WarpStateAI / ops /B200_PERFORMANCE.md
zerdovzad's picture
Upload 139 files
c8ba9c6 verified
|
Raw History Blame Contribute Delete
14.3 kB

B200 BF16 execution and checkpoint validation

Measurements from September 10, 2026 use disposable copies of the full step-488 checkpoint (63,963,136 tokens), including FP32 AdamW moments. The prepared dataset, tokenizer, model architecture, sequence length 2,048, and global batch of 131,072 tokens remain unchanged. No FP8 or FP16 training is used. Production resumed from step 488 with validation loss 4.79387 after the user requested continuation; see the production validation section below. The preserved input is work/run/paused-step-488.pt, SHA256 69d37e902c41f541d55a07b3d10dcb7643721240cc6aae032455cd8904c64608.

Latest pause

Training stopped gracefully at the user's request after step 680 / 89,128,960 tokens, with training loss 4.60232 and validation loss 4.39469. The full work/run/latest.pt checkpoint was published before the process exited. Final staging took 4.93 seconds, followed by 17.74 seconds of background writing. All 256 optimizer entries are at step 680; model/AdamW values are finite FP32, the schedule and data signature are unchanged, and the saved optimizer LR is 0.000102. SHA256 and the complete inspection are recorded in work/validation/b200/paused-checkpoint-step-680.json and the current handoff.

The final 100 completed updates (581-680) averaged 16,501 wall-clock tokens/s, including between-update overhead, with a median 72,418 tokens/s and 18 slow updates. This window ends before the final pause evaluation/save. Peak reserved memory stayed 139.55 GiB. The earlier measurements below remain historical comparisons. Production is now stopped; source settings, prepared data and earlier preserved checkpoint snapshots remain unchanged.

Selected launcher and measured tradeoffs

train-b200.sh now selects microbatch 16 / accumulation 4, compiled full loss, max-autotune-no-cudagraphs, and asynchronous full checkpoints. BF16 autocast, FP32 master weights/gradients/AdamW, the 40B token budget, original learning-rate schedule, seed, and save/evaluation cadence remain unchanged. Core activation checkpointing stays disabled and the allocator cap remains 150 GiB. Eight compiler workers are the default; initial autotuning can take several minutes.

Every benchmark below uses two warmup and twenty measured updates. Sustained throughput divides measured tokens by measured update seconds, including provider scheduling waits. Median speed describes a typical update; it is not continuous throughput or a guarantee of exclusive GPU capacity. Loading, compilation, validation and checkpoint writes are outside these measurements.

Microbatch / accumulation Loss Core mode / BLAS preference Sustained tokens/s Median tokens/s Peak reserved GiB
4 / 16 Checkpointed Default; stall diagnostics 13,808 61,853 60.44
4 / 16 Checkpointed Final baseline control 15,344 61,839 60.44
8 / 8 Checkpointed Default 16,362 65,636 86.09
16 / 4 Checkpointed Default 16,467 67,421 137.72
16 / 4 Compiled full Default 16,698 71,370 141.25
16 / 4 Compiled full Autotune, no graphs 20,752 72,499 139.31
16 / 4 Compiled full Default / cuBLASLt preference 18,718 71,309 141.25
16 / 4 Compiled full CUDA graphs, corrected timer 17,432 70,660 141.31

Autotuning gave the fastest typical update, about 17.2% above the initial baseline. Peak allocated memory was 138.18 GiB. Its first warmup took 206.16 seconds including compilation/tuning and scheduling waits, excluded above. The baseline diagnostic run encountered five slow measured updates, versus three in the autotune run. Scheduling phase contributes substantially to aggregate differences, so sustained gains cannot be attributed solely to the chosen execution settings. The final baseline control repeated a median of 61,839 tokens/s, within 0.03% of the original diagnostic baseline despite different scheduling waits; the typical-speed gain is repeatable.

cuBLASLt preference produced exactly the same losses and gradient norms as the default compiled-full case and no meaningful median speed improvement. It is available for comparison but is not forced in the B200 launcher. The recorded preferences were _BlasBackend.Cublas and _BlasBackend.Cublaslt; a preference does not guarantee every operation uses that backend. Current CUBLAS_WORKSPACE_CONFIG=:4096:8 provides 32 MiB workspace. See PyTorch backend controls and NVIDIA workspace guidance.

CUDA graph implementation and evidence

--compile-mode reduce-overhead now preallocates FP32 gradient buffers outside capture and zeros them in place between updates. Without this, the installed PyTorch rejected accumulation into graph-owned gradient outputs. Other modes retain zero_grad(set_to_none=True). No per-core step markers were inserted; completed forward/backward microbatches use PyTorch's iteration tracking.

The full-model graph run recorded 56 captures, zero compiler skips, and 4,480 successful nonempty replays during measured updates. Losses and gradient norms match the default compiled-full trajectory exactly. The optional diagnostic helper counts successful tree replay calls using private PyTorch 2.14 APIs, restores its wrapper on exit, and adds no CUDA operations or synchronization. Counts are not GPU timings.

The first exploratory graph benchmark excluded gradient zeroing from the timer and logged capture details with TORCH_LOGS=cudagraphs. It is retained as evidence but not used in the table. The corrected run includes gradient zeroing, matching the production update timer. Future benchmark configurations all use this timing. First-update peak allocation was 138.15 GiB; steady allocator peaks of 34.69 GiB during replay do not represent the complete graph memory footprint. Reserved memory, including graph pools, remained 141.31 GiB.

Graph mode is validated and available as an override, but was not selected as the default because it did not improve typical update speed. It captures cores; embedding, depth modulation, output projection, optimizer and the separately compiled full-loss reduction are outside those core graphs. See PyTorch compile modes and CUDA graph constraints.

Asynchronous full checkpoints

--async-checkpoint is optional in the general runner and enabled in the B200 launcher. It retains the existing warpstate-server-v1 single-file checkpoint, including model, complete AdamW state, RNG, token cursor, schedule/signature, configuration and provenance. --no-async-checkpoint restores synchronous saving. The asynchronous option currently requires a single training rank.

The main thread first creates a fully independent CPU snapshot. GPU-to-CPU staging completes synchronously before the next optimizer update can mutate live state. One background writer then serializes, flushes and fsyncs the file, atomically replaces latest.pt, and fsyncs the parent directory. Only serialization and disk I/O overlap training. Staging needs roughly 24 GB of additional host memory; it does not allocate a full extra GPU checkpoint.

A new save waits for the previous writer before allocating another snapshot, so only one snapshot/write can be outstanding. Background errors surface between updates and again on flush/close. Normal exit, graceful stop and training exceptions join any already-staged valid save before the pipeline lock is released. Exceptions do not snapshot partially updated live weights. Logs distinguish staging from completed publication and include snapshot step/tokens. A Staged message alone is not a durable checkpoint. This follows the stable-snapshot and bounded-outstanding-save principles in PyTorch's asynchronous checkpoint recipe while preserving this repository's existing checkpoint format.

Numerical and continuation checks

All 22 updates in each completed benchmark had finite loss and gradient norm. Against the microbatch-4 trajectory, maximum absolute loss differences were 0.00006819 for microbatch 8, 0.00006199 for microbatch 16, 0.00031376 for compiled full loss at microbatch 16, and 0.00006914 for its autotuned version. Maximum relative gradient-norm differences were 0.2680%, 0.1688%, 1.1087%, and 0.1760%. Executions preserve the objective but need not be bitwise equal across modes. Checkpoint and dataset identities remained unchanged throughout the benchmarks.

The final CPU runner/loss/checkpoint/launcher selection passed 91 tests with nine GPU cases deselected. Default and autotuned compiled-full CUDA continuation cases passed earlier; the graph and asynchronous cases then passed 2/2 in 84.41 seconds. These regressions check exact model, full optimizer and RNG state, canonical keys, FP32 state, learning-rate schedule, finite validation, and the final partial update at token counts 32, 64 and 73. The asynchronous case compares asynchronous pause/resume against synchronous continuous training and verifies that the asynchronous path actually ran. CPU writer tests cover immutable snapshots during AdamW mutation, atomic publication and failure cleanup, bounded writes, persistent errors and both normal/exceptional exit flushing. Only one GPU workload ran at a time.

Production continuation and first asynchronous save

After the user requested continuation, production resumed under B200 Supervisor in tmux session warpstate-b200, window training, new pane 0.1. Initial validation loss matched the pause snapshot at 4.79387. The first update, step 489, took 8.11 seconds including cached compilation startup. Steps 490-500 processed 1,441,792 tokens at 19,223 sustained tokens/s, with a median 72,509 tokens/s, including provider waits in update timing. Peak allocated/reserved memory was 138.18 / 139.55 GiB. This window excludes validation, CPU checkpoint staging and background disk-write overhead.

At step 500 / 65,536,000 tokens, training loss was 5.0160 and validation loss was 4.74471. The different training batches explain why individual training losses can be above five while the fixed validation loss is lower. The complete snapshot took 35.15 seconds to stage synchronously, followed by 44.44 seconds of background serialization/fsync/publication. Steps 501-507 completed while that writer was active. These staging and write times are observed wall times on this host; the staging wait was not separately profiled and should not be attributed entirely to transfer bandwidth.

The published checkpoint retains all 2,005,111,424 FP32 parameters, 256 full AdamW entries at step 500, finite FP32 moments, one rank's RNG state, the same data/origin, 131,072 tokens per update, and scheduled optimizer LR 0.000075. The full SHA256 and inspection are recorded in work/validation/b200/optimized-checkpoint-step-500.json. The checkpoint is preserved as work/run/validated-step-500.pt; later saves advanced latest.pt. The original paused-step-488.pt remained unchanged. Training continued after this first-save validation, until the later step-680 pause recorded above. Preparation, tokenizer, dataset, schedule and precision were preserved.

Remaining bottleneck

The recurring extra 30-second waits are strongly associated with the provider's HAIShare GPU scheduler, which waits for ownership beneath CUDA launch calls. Data transfer took about 5-6 ms/update; AdamW and CPU quota throttling did not account for sampled delays. See B200_SCHEDULING.md for evidence and uncertainty. No provider scheduling or interposition settings were changed. Confirm exclusive compute with the provider; full visible VRAM alone does not establish exclusive allocation.

Reproduction

Use b200ctl.sh to stop production gracefully and confirm the GPU is free before benchmarking. Preserve the pause snapshot. Benchmarks copy model and optimizer state into disposable CUDA storage and never save a training checkpoint. Their learning rate stays fixed at the loaded value and RNG is reseeded, so benchmark updates are timing experiments.

From /workspace/WarpState, choose a new output directory for each run:

source rtx6000_40b/ops/training-env.sh
source rtx6000_40b/ops/workspace-tools.sh
export TORCHINDUCTOR_COMPILE_THREADS=8
rtx6000_40b/ops/benchmark.sh \
  --checkpoint /workspace/WarpState/rtx6000_40b/work/run/paused-step-488.pt \
  --micro-batch 16 --seq-len 2048 --global-tokens-per-update 131072 \
  --compile-cores --compile-mode max-autotune-no-cudagraphs \
  --no-activation-checkpointing --autocast-cache --loss-mode compiled-full \
  --loss-chunk-size 1024 --vram-limit-gib 150 \
  --warmup-updates 2 --measured-updates 20 \
  --output-dir /workspace/WarpState/rtx6000_40b/work/benchmark-b200-reproduction

For backend comparisons use --compile-mode default and select --blas-backend default, --blas-backend cublas, or --blas-backend cublaslt in otherwise identical cases. The completed cuBLASLt case used the equivalent TORCH_BLAS_PREFER_CUBLASLT=1 environment preference, recorded in its metadata. For graph comparison use --compile-mode reduce-overhead --cudagraph-diagnostics. The same compile-mode override is accepted by train-b200.sh on a later production launch; the general runner's async flag is independent.

Raw summary.json and rank records are under work/benchmark-b200-*; work/validation/b200/optimization-grid.json aggregates comparisons. Metadata records source hashes, input identities, versions, backend options, precision and each update's measurement. Shared logs are shown in tmux session warpstate-b200, window optimization; attach with rtx6000_40b/ops/tmux-workspace.sh attach -t warpstate-b200.