|
Download ops/TRAINING.md from zerdovzad/WarpStateAI: direct link, hf CLI and curl.
- Browser
- Download file 7.15 kB
-
https://huggingface.co/zerdovzad/WarpStateAI/resolve/main/ops/TRAINING.md
- Command line
-
hf download hf://zerdovzad/WarpStateAI/ops/TRAINING.md
-
curl -L -o TRAINING.md https://huggingface.co/zerdovzad/WarpStateAI/resolve/main/ops/TRAINING.md
7.15 kB
| # Training environment and shared GPU configuration | |
| This document records the original two-GPU workstation configuration. Its | |
| Supervisor and launchers are not the controls for the single-B200 host. | |
| Current B200 settings, checkpoint and pause/resume controls are in | |
| [B200_HANDOFF.md](B200_HANDOFF.md). For workstation history, see | |
| [OPTIMIZED_TRAINING.md](OPTIMIZED_TRAINING.md); the shared GPU settings below | |
| preceded that workstation preset. | |
| The completed corpus in `work/data` contains exactly 40,000,000,000 training | |
| tokens and 1,048,576 validation tokens. Its generated manifest records the | |
| SHA256 checksums. The model has 2,005,111,424 parameters; 40B is the token | |
| budget, not the parameter count. | |
| No initial WarpState checkpoint was supplied. The shared-GPU launcher uses | |
| `--from-scratch` with seed 1337 for its first launch. Once `work/run/latest.pt` | |
| exists, subsequent launches restore its model, full AdamW state, token cursor, | |
| learning-rate schedule, and random states. `--from-scratch` never overwrites | |
| an existing run. To initialize another run from trusted weights, invoke | |
| `run_40b.py --weights ...` with a separate work directory instead. | |
| ## Shared GPU settings | |
| `train-shared.sh` selects physical GPU 1 by its UUID. VORTEX uses both GPUs, | |
| so concurrent training shares GPU compute and can slow both jobs. | |
| On another machine, set `WARPSTATE_CUDA_DEVICE` to the intended GPU UUID or | |
| index before launching. | |
| | Setting | Value | | |
| | --- | --- | | |
| | Sequence length | 2,048 tokens | | |
| | Microbatch | 1 sequence | | |
| | Gradient accumulation | 64 microbatches | | |
| | Tokens per optimizer update | 131,072 | | |
| | Parameters, gradients, AdamW moments | FP32 on the GPU | | |
| | Forward/backward matrix operations | BF16 autocast | | |
| | Activation checkpointing | Enabled | | |
| | Autocast weight cache | Disabled to reduce retained weight copies | | |
| | Loss projection chunks | 256 tokens | | |
| | PyTorch allocator limit | 33.5 GiB | | |
| | Required extra free memory at startup | 3 GiB | | |
| | Validation | 8 batches initially and every 100 updates | | |
| | Checkpoint | Every 50 updates, on a graceful stop, and at completion | | |
| The September 9, 2026 live trial completed two full updates, saved a | |
| 22.41 GiB checkpoint, and resumed through step 5. Peak memory was 31.54 GiB | |
| allocated and 32.02 GiB reserved, leaving about 4.7 GiB physically free. | |
| Observed training throughput was approximately 5,500–6,000 tokens/second. | |
| At that unchanged rate, 40B tokens would take roughly 77–84 days before | |
| checkpoint/evaluation overhead; this is an early measurement, not an ETA | |
| that accounts for future GPU availability. VORTEX continued advancing, | |
| with overlapping updates around 68–69K tokens/second versus roughly 100K | |
| before the trial. | |
| The fixed FP32 parameters, gradients, and two AdamW moments alone occupy | |
| about 29.88 GiB. Activations and allocator overhead must also fit the cap. | |
| The launcher refuses to start if the cap plus startup headroom exceeds | |
| available memory. This is a check, not a reservation for VORTEX. | |
| PyTorch's [allocator memory limit](https://docs.pytorch.org/docs/2.14/generated/torch.cuda.memory.set_per_process_memory_fraction.html) | |
| excludes CUDA context and library allocations. Compare both the allocated | |
| and reserved peaks in `work/run/metrics.jsonl` with physical memory in | |
| `nvidia-smi`. A different VORTEX workload may need different headroom. | |
| Each full checkpoint is approximately 22.4 GiB. Keep at least 53 GiB free | |
| for checkpoint writing and atomic replacement. Checkpoints, metrics, data, | |
| and environments stay inside this project and are ignored by Git. | |
| ## Environment | |
| The CUDA training environment is separate from the CPU data-preparation | |
| environment and from VORTEX's environment. Recreate it with Python 3.12: | |
| ```bash | |
| cd /workspace/WarpState | |
| source rtx6000_40b/ops/training-env.sh | |
| python3 -m venv rtx6000_40b/.server-venv | |
| uv pip install --python rtx6000_40b/.server-venv/bin/python \ | |
| -r rtx6000_40b/ops/training-requirements-resolved.txt \ | |
| --extra-index-url https://download.pytorch.org/whl/cu130 \ | |
| --index-strategy unsafe-best-match | |
| uv pip check --python rtx6000_40b/.server-venv/bin/python | |
| ``` | |
| The lock pins PyTorch 2.14.0+cu130 and its installed dependencies. Always | |
| source `training-env.sh` when testing or provisioning so caches and temporary | |
| files remain under `/workspace/WarpState`. The launcher sets these paths | |
| itself and uses `--no-bootstrap`; the minimal GPU environment does not | |
| contain the separate data-preparation dependencies. | |
| ## Operation | |
| The existing project-private Supervisor manages both preparation and training. | |
| Its training command is now `train-fast.sh`, which needs both GPUs available. | |
| `datactl.sh` is its control wrapper, including for the training job: | |
| ```bash | |
| cd /workspace/WarpState | |
| rtx6000_40b/ops/datactl.sh status | |
| rtx6000_40b/ops/datactl.sh start warpstate-train | |
| tail -n 10 rtx6000_40b/work/logs/train.log | |
| tail -n 3 rtx6000_40b/work/run/metrics.jsonl | |
| nvidia-smi | |
| ``` | |
| To stop, use `rtx6000_40b/ops/datactl.sh stop warpstate-train`. The runner | |
| finishes its current optimizer update and saves before exiting. Supervisor | |
| allows up to 15 minutes for that. Starting the same job resumes its latest | |
| checkpoint. `--reconfigure` permits changes to execution settings while keeping | |
| the sequence length, global tokens per update, data, and optimizer schedule | |
| unchanged. Changes to accumulation or GPU count can change floating-point | |
| rounding; the model and full optimizer state are retained. | |
| Training is started explicitly and does not automatically restart after a | |
| failure, avoiding repeated allocation attempts on a shared GPU. Inspect the | |
| log and available memory before restarting a failed job. A container restart | |
| also requires starting the project Supervisor and the training job again. | |
| For a bounded trial of the current configuration, run | |
| `ops/train-fast.sh --max-steps 2` from the package directory while the | |
| supervised training job is stopped. This saves after two | |
| additional updates while preserving the full 40B schedule. Only one runner | |
| can hold `work/pipeline.lock` at a time. | |
| To return to the original shared GPU configuration, stop the supervised job | |
| and run `ops/train-shared.sh --reconfigure` with enough free memory on GPU 1. | |
| For persistent supervision, update the program command to that launcher plus | |
| `--reconfigure` and use `stopsignal=TERM` before `datactl.sh reread` and | |
| `datactl.sh update warpstate-train`. The shared launcher runs the worker | |
| directly; the optimized launcher relays stop requests to distributed workers. | |
| Do not run the benchmark or another launcher alongside the production job. | |
| ## Checks | |
| The tests exercise interrupted data preparation, exact final partial updates, | |
| the memory-headroom refusal, and exact CUDA checkpoint/resume behavior for | |
| both imported weights and fresh initialization. The CUDA tests use a tiny | |
| model and a 0.5 GiB allocator cap: | |
| ```bash | |
| cd /workspace/WarpState | |
| source rtx6000_40b/ops/training-env.sh | |
| CUDA_VISIBLE_DEVICES=1 rtx6000_40b/.server-venv/bin/python \ | |
| -m pytest rtx6000_40b/test_server.py -q | |
| ``` | |
| Use `CUDA_VISIBLE_DEVICES=''` to skip CUDA tests when there is insufficient | |
| free memory. Test artifacts use the project-local `TMPDIR`. | |