WarpStateAI / ops /TRAINING.md
zerdovzad's picture
Upload 139 files
c8ba9c6 verified
|
Raw History Blame Contribute Delete
7.15 kB

Training environment and shared GPU configuration

This document records the original two-GPU workstation configuration. Its Supervisor and launchers are not the controls for the single-B200 host. Current B200 settings, checkpoint and pause/resume controls are in B200_HANDOFF.md. For workstation history, see OPTIMIZED_TRAINING.md; the shared GPU settings below preceded that workstation preset.

The completed corpus in work/data contains exactly 40,000,000,000 training tokens and 1,048,576 validation tokens. Its generated manifest records the SHA256 checksums. The model has 2,005,111,424 parameters; 40B is the token budget, not the parameter count.

No initial WarpState checkpoint was supplied. The shared-GPU launcher uses --from-scratch with seed 1337 for its first launch. Once work/run/latest.pt exists, subsequent launches restore its model, full AdamW state, token cursor, learning-rate schedule, and random states. --from-scratch never overwrites an existing run. To initialize another run from trusted weights, invoke run_40b.py --weights ... with a separate work directory instead.

Shared GPU settings

train-shared.sh selects physical GPU 1 by its UUID. VORTEX uses both GPUs, so concurrent training shares GPU compute and can slow both jobs. On another machine, set WARPSTATE_CUDA_DEVICE to the intended GPU UUID or index before launching.

Setting Value
Sequence length 2,048 tokens
Microbatch 1 sequence
Gradient accumulation 64 microbatches
Tokens per optimizer update 131,072
Parameters, gradients, AdamW moments FP32 on the GPU
Forward/backward matrix operations BF16 autocast
Activation checkpointing Enabled
Autocast weight cache Disabled to reduce retained weight copies
Loss projection chunks 256 tokens
PyTorch allocator limit 33.5 GiB
Required extra free memory at startup 3 GiB
Validation 8 batches initially and every 100 updates
Checkpoint Every 50 updates, on a graceful stop, and at completion

The September 9, 2026 live trial completed two full updates, saved a 22.41 GiB checkpoint, and resumed through step 5. Peak memory was 31.54 GiB allocated and 32.02 GiB reserved, leaving about 4.7 GiB physically free. Observed training throughput was approximately 5,500–6,000 tokens/second. At that unchanged rate, 40B tokens would take roughly 77–84 days before checkpoint/evaluation overhead; this is an early measurement, not an ETA that accounts for future GPU availability. VORTEX continued advancing, with overlapping updates around 68–69K tokens/second versus roughly 100K before the trial.

The fixed FP32 parameters, gradients, and two AdamW moments alone occupy about 29.88 GiB. Activations and allocator overhead must also fit the cap. The launcher refuses to start if the cap plus startup headroom exceeds available memory. This is a check, not a reservation for VORTEX.

PyTorch's allocator memory limit excludes CUDA context and library allocations. Compare both the allocated and reserved peaks in work/run/metrics.jsonl with physical memory in nvidia-smi. A different VORTEX workload may need different headroom.

Each full checkpoint is approximately 22.4 GiB. Keep at least 53 GiB free for checkpoint writing and atomic replacement. Checkpoints, metrics, data, and environments stay inside this project and are ignored by Git.

Environment

The CUDA training environment is separate from the CPU data-preparation environment and from VORTEX's environment. Recreate it with Python 3.12:

cd /workspace/WarpState
source rtx6000_40b/ops/training-env.sh
python3 -m venv rtx6000_40b/.server-venv
uv pip install --python rtx6000_40b/.server-venv/bin/python \
  -r rtx6000_40b/ops/training-requirements-resolved.txt \
  --extra-index-url https://download.pytorch.org/whl/cu130 \
  --index-strategy unsafe-best-match
uv pip check --python rtx6000_40b/.server-venv/bin/python

The lock pins PyTorch 2.14.0+cu130 and its installed dependencies. Always source training-env.sh when testing or provisioning so caches and temporary files remain under /workspace/WarpState. The launcher sets these paths itself and uses --no-bootstrap; the minimal GPU environment does not contain the separate data-preparation dependencies.

Operation

The existing project-private Supervisor manages both preparation and training. Its training command is now train-fast.sh, which needs both GPUs available. datactl.sh is its control wrapper, including for the training job:

cd /workspace/WarpState
rtx6000_40b/ops/datactl.sh status
rtx6000_40b/ops/datactl.sh start warpstate-train
tail -n 10 rtx6000_40b/work/logs/train.log
tail -n 3 rtx6000_40b/work/run/metrics.jsonl
nvidia-smi

To stop, use rtx6000_40b/ops/datactl.sh stop warpstate-train. The runner finishes its current optimizer update and saves before exiting. Supervisor allows up to 15 minutes for that. Starting the same job resumes its latest checkpoint. --reconfigure permits changes to execution settings while keeping the sequence length, global tokens per update, data, and optimizer schedule unchanged. Changes to accumulation or GPU count can change floating-point rounding; the model and full optimizer state are retained.

Training is started explicitly and does not automatically restart after a failure, avoiding repeated allocation attempts on a shared GPU. Inspect the log and available memory before restarting a failed job. A container restart also requires starting the project Supervisor and the training job again.

For a bounded trial of the current configuration, run ops/train-fast.sh --max-steps 2 from the package directory while the supervised training job is stopped. This saves after two additional updates while preserving the full 40B schedule. Only one runner can hold work/pipeline.lock at a time.

To return to the original shared GPU configuration, stop the supervised job and run ops/train-shared.sh --reconfigure with enough free memory on GPU 1. For persistent supervision, update the program command to that launcher plus --reconfigure and use stopsignal=TERM before datactl.sh reread and datactl.sh update warpstate-train. The shared launcher runs the worker directly; the optimized launcher relays stop requests to distributed workers. Do not run the benchmark or another launcher alongside the production job.

Checks

The tests exercise interrupted data preparation, exact final partial updates, the memory-headroom refusal, and exact CUDA checkpoint/resume behavior for both imported weights and fresh initialization. The CUDA tests use a tiny model and a 0.5 GiB allocator cap:

cd /workspace/WarpState
source rtx6000_40b/ops/training-env.sh
CUDA_VISIBLE_DEVICES=1 rtx6000_40b/.server-venv/bin/python \
  -m pytest rtx6000_40b/test_server.py -q

Use CUDA_VISIBLE_DEVICES='' to skip CUDA tests when there is insufficient free memory. Test artifacts use the project-local TMPDIR.