Download ops/B200_PERFORMANCE.md from zerdovzad/WarpStateAI: direct link, hf CLI and curl.
- Browser
- Download file 14.3 kB
-
https://huggingface.co/zerdovzad/WarpStateAI/resolve/main/ops/B200_PERFORMANCE.md
- Command line
-
hf download hf://zerdovzad/WarpStateAI/ops/B200_PERFORMANCE.md
-
curl -L -o B200_PERFORMANCE.md https://huggingface.co/zerdovzad/WarpStateAI/resolve/main/ops/B200_PERFORMANCE.md
B200 BF16 execution and checkpoint validation
Measurements from September 10, 2026 use disposable copies of the full
step-488 checkpoint (63,963,136 tokens), including FP32 AdamW moments.
The prepared dataset, tokenizer, model architecture, sequence length 2,048,
and global batch of 131,072 tokens remain unchanged. No FP8 or FP16 training
is used. Production resumed from step 488 with validation loss 4.79387 after
the user requested continuation; see the production validation section below.
The preserved input is work/run/paused-step-488.pt, SHA256
69d37e902c41f541d55a07b3d10dcb7643721240cc6aae032455cd8904c64608.
Latest pause
Training stopped gracefully at the user's request after step 680 /
89,128,960 tokens, with training loss 4.60232 and validation loss
4.39469. The full work/run/latest.pt checkpoint was published before the
process exited. Final staging took 4.93 seconds, followed by 17.74
seconds of background writing. All 256 optimizer entries are at step 680;
model/AdamW values are finite FP32, the schedule and data signature are
unchanged, and the saved optimizer LR is 0.000102. SHA256 and the complete
inspection are recorded in work/validation/b200/paused-checkpoint-step-680.json
and the current handoff.
The final 100 completed updates (581-680) averaged 16,501 wall-clock tokens/s, including between-update overhead, with a median 72,418 tokens/s and 18 slow updates. This window ends before the final pause evaluation/save. Peak reserved memory stayed 139.55 GiB. The earlier measurements below remain historical comparisons. Production is now stopped; source settings, prepared data and earlier preserved checkpoint snapshots remain unchanged.
Selected launcher and measured tradeoffs
train-b200.sh now selects microbatch 16 / accumulation 4,
compiled full loss, max-autotune-no-cudagraphs, and asynchronous full
checkpoints. BF16 autocast, FP32 master weights/gradients/AdamW, the 40B token
budget, original learning-rate schedule, seed, and save/evaluation cadence
remain unchanged. Core activation checkpointing stays disabled and the
allocator cap remains 150 GiB. Eight compiler workers are the default;
initial autotuning can take several minutes.
Every benchmark below uses two warmup and twenty measured updates. Sustained throughput divides measured tokens by measured update seconds, including provider scheduling waits. Median speed describes a typical update; it is not continuous throughput or a guarantee of exclusive GPU capacity. Loading, compilation, validation and checkpoint writes are outside these measurements.
| Microbatch / accumulation | Loss | Core mode / BLAS preference | Sustained tokens/s | Median tokens/s | Peak reserved GiB |
|---|---|---|---|---|---|
| 4 / 16 | Checkpointed | Default; stall diagnostics | 13,808 | 61,853 | 60.44 |
| 4 / 16 | Checkpointed | Final baseline control | 15,344 | 61,839 | 60.44 |
| 8 / 8 | Checkpointed | Default | 16,362 | 65,636 | 86.09 |
| 16 / 4 | Checkpointed | Default | 16,467 | 67,421 | 137.72 |
| 16 / 4 | Compiled full | Default | 16,698 | 71,370 | 141.25 |
| 16 / 4 | Compiled full | Autotune, no graphs | 20,752 | 72,499 | 139.31 |
| 16 / 4 | Compiled full | Default / cuBLASLt preference | 18,718 | 71,309 | 141.25 |
| 16 / 4 | Compiled full | CUDA graphs, corrected timer | 17,432 | 70,660 | 141.31 |
Autotuning gave the fastest typical update, about 17.2% above the initial baseline. Peak allocated memory was 138.18 GiB. Its first warmup took 206.16 seconds including compilation/tuning and scheduling waits, excluded above. The baseline diagnostic run encountered five slow measured updates, versus three in the autotune run. Scheduling phase contributes substantially to aggregate differences, so sustained gains cannot be attributed solely to the chosen execution settings. The final baseline control repeated a median of 61,839 tokens/s, within 0.03% of the original diagnostic baseline despite different scheduling waits; the typical-speed gain is repeatable.
cuBLASLt preference produced exactly the same losses and gradient norms as
the default compiled-full case and no meaningful median speed improvement.
It is available for comparison but is not forced in the B200 launcher. The
recorded preferences were _BlasBackend.Cublas and _BlasBackend.Cublaslt;
a preference does not guarantee every operation uses that backend. Current CUBLAS_WORKSPACE_CONFIG=:4096:8
provides 32 MiB workspace. See PyTorch backend controls
and NVIDIA workspace guidance.
CUDA graph implementation and evidence
--compile-mode reduce-overhead now preallocates FP32 gradient buffers outside
capture and zeros them in place between updates. Without this, the installed
PyTorch rejected accumulation into graph-owned gradient outputs. Other modes
retain zero_grad(set_to_none=True). No per-core step markers were inserted;
completed forward/backward microbatches use PyTorch's iteration tracking.
The full-model graph run recorded 56 captures, zero compiler skips, and 4,480 successful nonempty replays during measured updates. Losses and gradient norms match the default compiled-full trajectory exactly. The optional diagnostic helper counts successful tree replay calls using private PyTorch 2.14 APIs, restores its wrapper on exit, and adds no CUDA operations or synchronization. Counts are not GPU timings.
The first exploratory graph benchmark excluded gradient zeroing from the
timer and logged capture details with TORCH_LOGS=cudagraphs. It is retained
as evidence but not used in the table. The corrected run includes gradient
zeroing, matching the production update timer. Future benchmark configurations
all use this timing. First-update peak allocation was 138.15 GiB; steady
allocator peaks of 34.69 GiB during replay do not represent the complete graph
memory footprint. Reserved memory, including graph pools, remained 141.31 GiB.
Graph mode is validated and available as an override, but was not selected as the default because it did not improve typical update speed. It captures cores; embedding, depth modulation, output projection, optimizer and the separately compiled full-loss reduction are outside those core graphs. See PyTorch compile modes and CUDA graph constraints.
Asynchronous full checkpoints
--async-checkpoint is optional in the general runner and enabled in the B200
launcher. It retains the existing warpstate-server-v1 single-file checkpoint,
including model, complete AdamW state, RNG, token cursor, schedule/signature,
configuration and provenance. --no-async-checkpoint restores synchronous
saving. The asynchronous option currently requires a single training rank.
The main thread first creates a fully independent CPU snapshot. GPU-to-CPU
staging completes synchronously before the next optimizer update can mutate
live state. One background writer then serializes, flushes and fsyncs the file,
atomically replaces latest.pt, and fsyncs the parent directory. Only
serialization and disk I/O overlap training. Staging needs roughly 24 GB of
additional host memory; it does not allocate a full extra GPU checkpoint.
A new save waits for the previous writer before allocating another snapshot, so only one snapshot/write can be outstanding. Background errors surface between updates and again on flush/close. Normal exit, graceful stop and training exceptions join any already-staged valid save before the pipeline lock is released. Exceptions do not snapshot partially updated live weights. Logs distinguish staging from completed publication and include snapshot step/tokens. A Staged message alone is not a durable checkpoint. This follows the stable-snapshot and bounded-outstanding-save principles in PyTorch's asynchronous checkpoint recipe while preserving this repository's existing checkpoint format.
Numerical and continuation checks
All 22 updates in each completed benchmark had finite loss and gradient norm. Against the microbatch-4 trajectory, maximum absolute loss differences were 0.00006819 for microbatch 8, 0.00006199 for microbatch 16, 0.00031376 for compiled full loss at microbatch 16, and 0.00006914 for its autotuned version. Maximum relative gradient-norm differences were 0.2680%, 0.1688%, 1.1087%, and 0.1760%. Executions preserve the objective but need not be bitwise equal across modes. Checkpoint and dataset identities remained unchanged throughout the benchmarks.
The final CPU runner/loss/checkpoint/launcher selection passed 91 tests with nine GPU cases deselected. Default and autotuned compiled-full CUDA continuation cases passed earlier; the graph and asynchronous cases then passed 2/2 in 84.41 seconds. These regressions check exact model, full optimizer and RNG state, canonical keys, FP32 state, learning-rate schedule, finite validation, and the final partial update at token counts 32, 64 and 73. The asynchronous case compares asynchronous pause/resume against synchronous continuous training and verifies that the asynchronous path actually ran. CPU writer tests cover immutable snapshots during AdamW mutation, atomic publication and failure cleanup, bounded writes, persistent errors and both normal/exceptional exit flushing. Only one GPU workload ran at a time.
Production continuation and first asynchronous save
After the user requested continuation, production resumed under B200
Supervisor in tmux session warpstate-b200, window training, new pane 0.1.
Initial validation loss matched the pause snapshot at 4.79387. The first
update, step 489, took 8.11 seconds including cached compilation startup.
Steps 490-500 processed 1,441,792 tokens at 19,223 sustained tokens/s,
with a median 72,509 tokens/s, including provider waits in update timing.
Peak allocated/reserved memory was 138.18 / 139.55 GiB. This window excludes
validation, CPU checkpoint staging and background disk-write overhead.
At step 500 / 65,536,000 tokens, training loss was 5.0160 and validation loss was 4.74471. The different training batches explain why individual training losses can be above five while the fixed validation loss is lower. The complete snapshot took 35.15 seconds to stage synchronously, followed by 44.44 seconds of background serialization/fsync/publication. Steps 501-507 completed while that writer was active. These staging and write times are observed wall times on this host; the staging wait was not separately profiled and should not be attributed entirely to transfer bandwidth.
The published checkpoint retains all 2,005,111,424 FP32 parameters, 256
full AdamW entries at step 500, finite FP32 moments, one rank's RNG state,
the same data/origin, 131,072 tokens per update, and scheduled optimizer
LR 0.000075. The full SHA256 and inspection are recorded in
work/validation/b200/optimized-checkpoint-step-500.json. The checkpoint is
preserved as work/run/validated-step-500.pt; later saves advanced latest.pt.
The original paused-step-488.pt remained unchanged. Training continued after
this first-save validation, until the later step-680 pause recorded above.
Preparation, tokenizer, dataset, schedule and precision were preserved.
Remaining bottleneck
The recurring extra 30-second waits are strongly associated with the provider's HAIShare GPU scheduler, which waits for ownership beneath CUDA launch calls. Data transfer took about 5-6 ms/update; AdamW and CPU quota throttling did not account for sampled delays. See B200_SCHEDULING.md for evidence and uncertainty. No provider scheduling or interposition settings were changed. Confirm exclusive compute with the provider; full visible VRAM alone does not establish exclusive allocation.
Reproduction
Use b200ctl.sh to stop production gracefully and confirm the GPU is free before benchmarking. Preserve the pause snapshot. Benchmarks copy model and optimizer state into disposable CUDA storage and never save a training checkpoint. Their learning rate stays fixed at the loaded value and RNG is reseeded, so benchmark updates are timing experiments.
From /workspace/WarpState, choose a new output directory for each run:
source rtx6000_40b/ops/training-env.sh
source rtx6000_40b/ops/workspace-tools.sh
export TORCHINDUCTOR_COMPILE_THREADS=8
rtx6000_40b/ops/benchmark.sh \
--checkpoint /workspace/WarpState/rtx6000_40b/work/run/paused-step-488.pt \
--micro-batch 16 --seq-len 2048 --global-tokens-per-update 131072 \
--compile-cores --compile-mode max-autotune-no-cudagraphs \
--no-activation-checkpointing --autocast-cache --loss-mode compiled-full \
--loss-chunk-size 1024 --vram-limit-gib 150 \
--warmup-updates 2 --measured-updates 20 \
--output-dir /workspace/WarpState/rtx6000_40b/work/benchmark-b200-reproduction
For backend comparisons use --compile-mode default and select
--blas-backend default, --blas-backend cublas, or --blas-backend cublaslt
in otherwise identical cases. The completed cuBLASLt case used the equivalent
TORCH_BLAS_PREFER_CUBLASLT=1 environment preference, recorded in its metadata.
For graph comparison use --compile-mode reduce-overhead --cudagraph-diagnostics.
The same compile-mode override is accepted by train-b200.sh on a later
production launch; the general runner's async flag is independent.
Raw summary.json and rank records are under work/benchmark-b200-*;
work/validation/b200/optimization-grid.json aggregates comparisons.
Metadata records source hashes, input identities, versions, backend options,
precision and each update's measurement. Shared logs are shown in tmux session
warpstate-b200, window optimization; attach with
rtx6000_40b/ops/tmux-workspace.sh attach -t warpstate-b200.