# B200 BF16 execution and checkpoint validation Measurements from September 10, 2026 use disposable copies of the full step-488 checkpoint (63,963,136 tokens), including FP32 AdamW moments. The prepared dataset, tokenizer, model architecture, sequence length 2,048, and global batch of 131,072 tokens remain unchanged. No FP8 or FP16 training is used. Production resumed from step 488 with validation loss 4.79387 after the user requested continuation; see the production validation section below. The preserved input is `work/run/paused-step-488.pt`, SHA256 `69d37e902c41f541d55a07b3d10dcb7643721240cc6aae032455cd8904c64608`. ## Latest pause Training stopped gracefully at the user's request after step **680 / 89,128,960 tokens**, with training loss **4.60232** and validation loss **4.39469**. The full `work/run/latest.pt` checkpoint was published before the process exited. Final staging took **4.93 seconds**, followed by **17.74 seconds** of background writing. All 256 optimizer entries are at step 680; model/AdamW values are finite FP32, the schedule and data signature are unchanged, and the saved optimizer LR is **0.000102**. SHA256 and the complete inspection are recorded in `work/validation/b200/paused-checkpoint-step-680.json` and the current [handoff](B200_HANDOFF.md). The final 100 completed updates (581-680) averaged **16,501 wall-clock tokens/s**, including between-update overhead, with a median **72,418 tokens/s** and 18 slow updates. This window ends before the final pause evaluation/save. Peak reserved memory stayed **139.55 GiB**. The earlier measurements below remain historical comparisons. Production is now stopped; source settings, prepared data and earlier preserved checkpoint snapshots remain unchanged. ## Selected launcher and measured tradeoffs [train-b200.sh](train-b200.sh) now selects microbatch 16 / accumulation 4, compiled full loss, `max-autotune-no-cudagraphs`, and asynchronous full checkpoints. BF16 autocast, FP32 master weights/gradients/AdamW, the 40B token budget, original learning-rate schedule, seed, and save/evaluation cadence remain unchanged. Core activation checkpointing stays disabled and the allocator cap remains 150 GiB. Eight compiler workers are the default; initial autotuning can take several minutes. Every benchmark below uses two warmup and twenty measured updates. Sustained throughput divides measured tokens by measured update seconds, including provider scheduling waits. Median speed describes a typical update; it is not continuous throughput or a guarantee of exclusive GPU capacity. Loading, compilation, validation and checkpoint writes are outside these measurements. | Microbatch / accumulation | Loss | Core mode / BLAS preference | Sustained tokens/s | Median tokens/s | Peak reserved GiB | | --- | --- | --- | ---: | ---: | ---: | | 4 / 16 | Checkpointed | Default; stall diagnostics | 13,808 | 61,853 | 60.44 | | 4 / 16 | Checkpointed | Final baseline control | 15,344 | 61,839 | 60.44 | | 8 / 8 | Checkpointed | Default | 16,362 | 65,636 | 86.09 | | 16 / 4 | Checkpointed | Default | 16,467 | 67,421 | 137.72 | | 16 / 4 | Compiled full | Default | 16,698 | 71,370 | 141.25 | | 16 / 4 | Compiled full | Autotune, no graphs | 20,752 | 72,499 | 139.31 | | 16 / 4 | Compiled full | Default / cuBLASLt preference | 18,718 | 71,309 | 141.25 | | 16 / 4 | Compiled full | CUDA graphs, corrected timer | 17,432 | 70,660 | 141.31 | Autotuning gave the fastest typical update, about 17.2% above the initial baseline. Peak allocated memory was 138.18 GiB. Its first warmup took 206.16 seconds including compilation/tuning and scheduling waits, excluded above. The baseline diagnostic run encountered five slow measured updates, versus three in the autotune run. Scheduling phase contributes substantially to aggregate differences, so sustained gains cannot be attributed solely to the chosen execution settings. The final baseline control repeated a median of 61,839 tokens/s, within 0.03% of the original diagnostic baseline despite different scheduling waits; the typical-speed gain is repeatable. cuBLASLt preference produced exactly the same losses and gradient norms as the default compiled-full case and no meaningful median speed improvement. It is available for comparison but is not forced in the B200 launcher. The recorded preferences were `_BlasBackend.Cublas` and `_BlasBackend.Cublaslt`; a preference does not guarantee every operation uses that backend. Current `CUBLAS_WORKSPACE_CONFIG=:4096:8` provides 32 MiB workspace. See [PyTorch backend controls](https://docs.pytorch.org/docs/2.14/backends.html#torch.backends.cuda.preferred_blas_library) and [NVIDIA workspace guidance](https://docs.nvidia.com/cuda/archive/13.0.0/cublas/index.html#cublassetworkspace). ## CUDA graph implementation and evidence `--compile-mode reduce-overhead` now preallocates FP32 gradient buffers outside capture and zeros them in place between updates. Without this, the installed PyTorch rejected accumulation into graph-owned gradient outputs. Other modes retain `zero_grad(set_to_none=True)`. No per-core step markers were inserted; completed forward/backward microbatches use PyTorch's iteration tracking. The full-model graph run recorded **56 captures**, **zero compiler skips**, and **4,480 successful nonempty replays during measured updates**. Losses and gradient norms match the default compiled-full trajectory exactly. The optional [diagnostic helper](cudagraph_diagnostics.py) counts successful tree replay calls using private PyTorch 2.14 APIs, restores its wrapper on exit, and adds no CUDA operations or synchronization. Counts are not GPU timings. The first exploratory graph benchmark excluded gradient zeroing from the timer and logged capture details with `TORCH_LOGS=cudagraphs`. It is retained as evidence but not used in the table. The corrected run includes gradient zeroing, matching the production update timer. Future benchmark configurations all use this timing. First-update peak allocation was 138.15 GiB; steady allocator peaks of 34.69 GiB during replay do not represent the complete graph memory footprint. Reserved memory, including graph pools, remained 141.31 GiB. Graph mode is validated and available as an override, but was not selected as the default because it did not improve typical update speed. It captures cores; embedding, depth modulation, output projection, optimizer and the separately compiled full-loss reduction are outside those core graphs. See [PyTorch compile modes](https://docs.pytorch.org/docs/2.14/generated/torch.compile.html) and [CUDA graph constraints](https://docs.pytorch.org/docs/2.14/notes/cuda.html#constraints). ## Asynchronous full checkpoints `--async-checkpoint` is optional in the general runner and enabled in the B200 launcher. It retains the existing `warpstate-server-v1` single-file checkpoint, including model, complete AdamW state, RNG, token cursor, schedule/signature, configuration and provenance. `--no-async-checkpoint` restores synchronous saving. The asynchronous option currently requires a single training rank. The main thread first creates a fully independent CPU snapshot. GPU-to-CPU staging completes synchronously before the next optimizer update can mutate live state. One background writer then serializes, flushes and fsyncs the file, atomically replaces `latest.pt`, and fsyncs the parent directory. Only serialization and disk I/O overlap training. Staging needs roughly 24 GB of additional host memory; it does not allocate a full extra GPU checkpoint. A new save waits for the previous writer before allocating another snapshot, so only one snapshot/write can be outstanding. Background errors surface between updates and again on flush/close. Normal exit, graceful stop and training exceptions join any already-staged valid save before the pipeline lock is released. Exceptions do not snapshot partially updated live weights. Logs distinguish staging from completed publication and include snapshot step/tokens. A **Staged** message alone is not a durable checkpoint. This follows the stable-snapshot and bounded-outstanding-save principles in [PyTorch's asynchronous checkpoint recipe](https://docs.pytorch.org/tutorials/recipes/distributed_async_checkpoint_recipe.html) while preserving this repository's existing checkpoint format. ## Numerical and continuation checks All 22 updates in each completed benchmark had finite loss and gradient norm. Against the microbatch-4 trajectory, maximum absolute loss differences were 0.00006819 for microbatch 8, 0.00006199 for microbatch 16, 0.00031376 for compiled full loss at microbatch 16, and 0.00006914 for its autotuned version. Maximum relative gradient-norm differences were 0.2680%, 0.1688%, 1.1087%, and 0.1760%. Executions preserve the objective but need not be bitwise equal across modes. Checkpoint and dataset identities remained unchanged throughout the benchmarks. The final CPU runner/loss/checkpoint/launcher selection passed **91 tests** with nine GPU cases deselected. Default and autotuned compiled-full CUDA continuation cases passed earlier; the graph and asynchronous cases then passed **2/2** in 84.41 seconds. These regressions check exact model, full optimizer and RNG state, canonical keys, FP32 state, learning-rate schedule, finite validation, and the final partial update at token counts 32, 64 and 73. The asynchronous case compares asynchronous pause/resume against synchronous continuous training and verifies that the asynchronous path actually ran. CPU writer tests cover immutable snapshots during AdamW mutation, atomic publication and failure cleanup, bounded writes, persistent errors and both normal/exceptional exit flushing. Only one GPU workload ran at a time. ## Production continuation and first asynchronous save After the user requested continuation, production resumed under B200 Supervisor in tmux session `warpstate-b200`, window `training`, new pane `0.1`. Initial validation loss matched the pause snapshot at **4.79387**. The first update, step 489, took 8.11 seconds including cached compilation startup. Steps **490-500** processed 1,441,792 tokens at **19,223 sustained tokens/s**, with a median **72,509 tokens/s**, including provider waits in update timing. Peak allocated/reserved memory was **138.18 / 139.55 GiB**. This window excludes validation, CPU checkpoint staging and background disk-write overhead. At step **500 / 65,536,000 tokens**, training loss was **5.0160** and validation loss was **4.74471**. The different training batches explain why individual training losses can be above five while the fixed validation loss is lower. The complete snapshot took **35.15 seconds** to stage synchronously, followed by **44.44 seconds** of background serialization/fsync/publication. Steps **501-507** completed while that writer was active. These staging and write times are observed wall times on this host; the staging wait was not separately profiled and should not be attributed entirely to transfer bandwidth. The published checkpoint retains all **2,005,111,424 FP32 parameters**, **256 full AdamW entries at step 500**, finite FP32 moments, one rank's RNG state, the same data/origin, **131,072 tokens per update**, and scheduled optimizer LR **0.000075**. The full SHA256 and inspection are recorded in `work/validation/b200/optimized-checkpoint-step-500.json`. The checkpoint is preserved as `work/run/validated-step-500.pt`; later saves advanced `latest.pt`. The original `paused-step-488.pt` remained unchanged. Training continued after this first-save validation, until the later step-680 pause recorded above. Preparation, tokenizer, dataset, schedule and precision were preserved. ## Remaining bottleneck The recurring extra 30-second waits are strongly associated with the provider's HAIShare GPU scheduler, which waits for ownership beneath CUDA launch calls. Data transfer took about 5-6 ms/update; AdamW and CPU quota throttling did not account for sampled delays. See [B200_SCHEDULING.md](B200_SCHEDULING.md) for evidence and uncertainty. No provider scheduling or interposition settings were changed. Confirm exclusive compute with the provider; full visible VRAM alone does not establish exclusive allocation. ## Reproduction Use [b200ctl.sh](b200ctl.sh) to stop production gracefully and confirm the GPU is free before benchmarking. Preserve the pause snapshot. Benchmarks copy model and optimizer state into disposable CUDA storage and never save a training checkpoint. Their learning rate stays fixed at the loaded value and RNG is reseeded, so benchmark updates are timing experiments. From `/workspace/WarpState`, choose a new output directory for each run: ```bash source rtx6000_40b/ops/training-env.sh source rtx6000_40b/ops/workspace-tools.sh export TORCHINDUCTOR_COMPILE_THREADS=8 rtx6000_40b/ops/benchmark.sh \ --checkpoint /workspace/WarpState/rtx6000_40b/work/run/paused-step-488.pt \ --micro-batch 16 --seq-len 2048 --global-tokens-per-update 131072 \ --compile-cores --compile-mode max-autotune-no-cudagraphs \ --no-activation-checkpointing --autocast-cache --loss-mode compiled-full \ --loss-chunk-size 1024 --vram-limit-gib 150 \ --warmup-updates 2 --measured-updates 20 \ --output-dir /workspace/WarpState/rtx6000_40b/work/benchmark-b200-reproduction ``` For backend comparisons use `--compile-mode default` and select `--blas-backend default`, `--blas-backend cublas`, or `--blas-backend cublaslt` in otherwise identical cases. The completed cuBLASLt case used the equivalent `TORCH_BLAS_PREFER_CUBLASLT=1` environment preference, recorded in its metadata. For graph comparison use `--compile-mode reduce-overhead --cudagraph-diagnostics`. The same compile-mode override is accepted by `train-b200.sh` on a later production launch; the general runner's async flag is independent. Raw `summary.json` and rank records are under `work/benchmark-b200-*`; `work/validation/b200/optimization-grid.json` aggregates comparisons. Metadata records source hashes, input identities, versions, backend options, precision and each update's measurement. Shared logs are shown in tmux session `warpstate-b200`, window `optimization`; attach with `rtx6000_40b/ops/tmux-workspace.sh attach -t warpstate-b200`.