File size: 1,666 Bytes
b836623 2638769 | 1 2 3 4 5 6 7 8 9 10 | # Performance optimization
The initial Caltech prototype implementation retained text-encoder autograd graphs for all 196 Cars classes and exhausted the 95 GiB GPU during smoke testing. The official `SPC/IsSynth/main_issynth.py` `get_mu_and_kappa` is explicitly decorated with `@torch.no_grad()` at line 47. Restoring that official behavior made the smoke test pass at 11.86 GiB allocated while preserving the source method's effective gradients. The original full-batch 64-real/64-synthetic sampling and 0.8/0.2 loss weights are unchanged.
The runner reuses Caltech's bfloat16 autocast, microbatch accumulation with effective batch 64, pinned memory, and two persistent loader workers. It avoids new synthetic generation and caches local images. Increasing effective batch size, changing losses, and caching trainable image embeddings were rejected because they change the scientific objective. No broad speed sweep was run. Training throughput will be recorded from completed epochs.
### Representative training observation
The first 7 logged Cars optimizer steps averaged a median 20.3 seconds per step (excluding the first model/warmup step). With 49 steps per epoch this implies about 16.6 minutes for the training portion of an epoch; validation and synchronization add time. This is a provisional estimate, not a measured full-epoch duration. A separate 24-class no-gradient prototype probe took 1.894 seconds with class-by-class calls and 0.847 seconds grouping four classes per call. The grouped result differed by up to 0.00159 in normalized embedding components and changes train-mode dropout draw order. It was rejected for the frozen canonical run.
|