|
Download ops/POSTTRAIN.md from zerdovzad/WarpStateAI: direct link, hf CLI and curl.
- Browser
- Download file 9.14 kB
-
https://huggingface.co/zerdovzad/WarpStateAI/resolve/main/ops/POSTTRAIN.md
- Command line
-
hf download hf://zerdovzad/WarpStateAI/ops/POSTTRAIN.md
-
curl -L -o POSTTRAIN.md https://huggingface.co/zerdovzad/WarpStateAI/resolve/main/ops/POSTTRAIN.md
9.14 kB
| # Step 76,294 archive and supervised fine-tuning | |
| The SFT stage runs from | |
| [`../configs/posttrain_10b.json`](../configs/posttrain_10b.json). The user later | |
| authorized **archiving the full completed SFT state, then automatically running | |
| DPO**. A separate `warpstate-dpo` Supervisor job implements that continuation; | |
| see [DPO_HANDOFF.md](DPO_HANDOFF.md). The active SFT plan/source bytes stay fixed | |
| to preserve their checkpoint/resume signatures. Its original guardian still | |
| finishes and exits cleanly before the DPO guardian can take over. | |
| `prepare_posttrain_data.py` prepares conversations and preference pairs. The | |
| separate `posttrain.py` runner implements assistant-only supervised training. | |
| ## Automatic sequence | |
| 1. `warpstate-posttrain-data` prepares the pinned dataset recipe on CPU now. | |
| 2. `warpstate-posttrain` waits for `warpstate-train` to exit successfully. It | |
| never changes the base run or resumes an earlier stopped base checkpoint. | |
| 3. Under the base launch/pipeline locks, require the full checkpoint to contain | |
| step **76,294**, **10,000,007,168 tokens**, and final validation at that same | |
| boundary. A checkpoint before the boundary remains waiting; an overshoot, | |
| stale validation, or mismatched tokenizer fails without starting SFT. | |
| 4. Export the original FP32 model using the same `chat.capture_latest` method as | |
| the 8K, 16K and 40K archives. Compare every exported tensor exactly against the | |
| stopped checkpoint; copy the exact tokenizer; write provenance/checksums; | |
| seal files read-only; publish the complete archive atomically without | |
| replacing an existing directory. | |
| 5. Make a physical, independent copy of the archive as SFT initialization. The | |
| full pretraining checkpoint, corpus, historical archives and chat selection | |
| are retained. SFT starts with a fresh AdamW optimizer on its first launch. | |
| 6. Verify prepared SFT data, run a tiny two-GPU BF16 training/resume preflight, | |
| then start the full SFT model. Failed preflight or data checks stop the | |
| handoff and retain the raw archive for diagnosis. | |
| 7. Save final SFT state and evaluation results, verify the raw archive and | |
| initialization remain unchanged, then exit. Neither program automatically | |
| restarts training after completion or failure. The separately armed DPO job | |
| then archives this full SFT state and continues according to DPO_HANDOFF.md. | |
| The boundary triggers archival, checksum verification, compilation/preflight | |
| and initialization before the first SFT update; allow several minutes for this | |
| handoff. GPU preflight is deliberately deferred until base training releases | |
| the GPUs. Its result is not claimed by the CPU-only setup tests. | |
| ## Paths | |
| All paths below are relative to `/workspace/WarpState/rtx6000_40b`. | |
| | Purpose | Path | | |
| | --- | --- | | |
| | Untouched full base resume checkpoint | `work/run/latest.pt` | | |
| | Raw historical model archive | `work/archives/step-00076294/` | | |
| | Independent SFT initialization | `work/posttrain/step-00076294/initialization/` | | |
| | Prepared SFT/DPO data and final manifest | `work/posttrain/data/` | | |
| | SFT checkpoints, settings and metrics | `work/posttrain/step-00076294/run/` | | |
| | Final evaluation / completion | `run/evaluation.json`, `run/completion.json` under the SFT directory | | |
| | Handoff state and frozen execution provenance | `work/posttrain/step-00076294/control/` | | |
| | GPU preflight artifacts | `work/posttrain/step-00076294/preflight/` | | |
| | Handoff and training log | `work/logs/posttrain.log` | | |
| | CPU preparation log | `work/logs/posttrain-data.log` | | |
| Archives contain model weights only, without Adam or training RNG state. SFT's | |
| `latest.pt` is a separate full `warpstate-posttrain-v1` checkpoint with optimizer, | |
| per-rank RNG, deterministic sample cursor, source/data identities and base | |
| provenance. It is not interchangeable with the base runner's checkpoint. | |
| ## SFT recipe and validation | |
| This first run uses one epoch, two GPUs, sequence length 2,048, microbatch 4 per | |
| rank and four accumulation microbatches: **32 conversations per update**. There | |
| is no cross-conversation packing. All parameters train with BF16 compute, | |
| original FP32 parameters/gradients and fresh FP32 AdamW state; no FP8 is used. | |
| The learning rate peaks at `5e-6`, with 3% warmup and cosine decay to zero, | |
| zero weight decay and gradient clipping at 1. Evaluation/checkpoints occur | |
| every 100 SFT updates and at the end. Initial and final validation use the full | |
| SFT holdout; intermediate checks use the same fixed 128-example subset. | |
| Settings and effective epoch length are | |
| recorded in the SFT run's `settings.json` once data is loaded. | |
| The learning rate is a conservative experimental starting point, not a | |
| validated optimum for this 2B model. This is not a reproduction of Tülu 3's | |
| SFT schedule: its published learning rate was calibrated using summed token | |
| loss, and its models/context/batch sizes differ. See the | |
| [Tülu 3 paper, section 4.3.2](https://arxiv.org/html/2411.15124v3). | |
| Role headers use ordinary tokens from the existing SentencePiece vocabulary. | |
| Only assistant content and the actual EOS ID contribute to the loss. Prompt, | |
| role-header and padding labels are masked; DDP gradients are normalized by the | |
| global count of supervised tokens, including uneven final batches. The shared | |
| renderer is `ops/posttrain_data.py`; its hash is pinned in the data manifest. | |
| SFT metrics report actual nonpadding input tokens and supervised assistant | |
| tokens separately. SFT validation loss/perplexity uses the assistant-token | |
| objective on the prepared holdout; it is **not directly comparable** with the | |
| base model's all-token pretraining validation loss. Evaluation reports include | |
| the exact evaluated sample/token counts, dataset and base-weight identities. | |
| Behavioral instruction-following and tool-use evaluation is the next manual | |
| decision after the full authorized SFT/DPO sequence pauses; a lower SFT loss | |
| alone does not establish those capabilities. The final SFT archive preserves | |
| the exact model and its evaluation for comparison with DPO. | |
| ## Monitor, pause and resume | |
| From the repository root: | |
| ```bash | |
| rtx6000_40b/ops/datactl.sh status warpstate-train warpstate-posttrain-data warpstate-posttrain | |
| rtx6000_40b/ops/posttrain-handoff.sh --check | |
| tmux attach -t warpstate-posttrain | |
| tail -f rtx6000_40b/work/logs/posttrain.log | |
| ``` | |
| The tmux session displays the durable Supervisor log. Detaching tmux does not | |
| stop the handoff or training. `RUNNING` on the posttraining program can mean | |
| waiting for the base boundary; inspect `--check` or its phase log. | |
| To pause the handoff or gracefully checkpoint an active SFT run: | |
| ```bash | |
| rtx6000_40b/ops/datactl.sh stop warpstate-posttrain | |
| ``` | |
| To explicitly re-enable the unchanged handoff or resume an interrupted SFT run: | |
| ```bash | |
| rtx6000_40b/ops/datactl.sh start warpstate-posttrain | |
| ``` | |
| The saved plan/source identity must still match. A completed run stays paused | |
| even if the program is started again. Do not remove its completion state to | |
| repeat SFT. Use a separately reviewed plan and output directory for a new | |
| experiment. Do not clear lock files or launch the base trainer manually while | |
| SFT workers are alive: every SFT rank holds a shared base pipeline lock, and | |
| the base/handoff require its exclusive lock. The launcher also retains its | |
| exclusion locks across child launch and unexpected watcher death. | |
| SFT Supervisor includes are in `ops/posttrain-supervisord.conf`; both programs | |
| have `autostart=false` and `autorestart=false`. Setup explicitly starts the | |
| authorized jobs. A whole-machine/Supervisor restart requires an explicit | |
| restart of the appropriate jobs; there is no hidden reboot training launch. | |
| To cancel automatic continuation into DPO while allowing SFT to finish, stop | |
| the separate waiting job with `rtx6000_40b/ops/datactl.sh stop warpstate-dpo`. | |
| Stopping SFT itself leaves the DPO guardian waiting for verified completion. | |
| ## Rebuild the prepared inputs | |
| The recipe in `configs/posttrain_data.json` pins immutable dataset revisions, | |
| the tokenizer identity, sample counts/seed, split rule, renderer and required | |
| preparation packages. Preparation records downloaded file hashes, schemas, | |
| source/licensing metadata, row/token counts, rejections and output hashes. | |
| Prompt groups use the same deterministic partition across SFT and DPO; exact | |
| duplicate conversations are excluded. Overlength examples and DPO branches | |
| are rejected rather than truncated. Gated xLAM data is recorded as skipped | |
| without automatically accepting terms or requiring interactive input. | |
| To rebuild the isolated CPU environment, source `ops/training-env.sh`, create | |
| `work/posttrain-data-venv` with Python 3.12, then install | |
| `ops/requirements-posttrain-data.lock` into that environment. Do not change the | |
| active training environment. `ops/prepare-posttrain-data.sh` runs the recipe; | |
| do not regenerate/replace data while SFT is running. The completed manifest is | |
| published only after every selected output has been written and hashed. | |
| Source/configuration/tests/dependency metadata belong in Git. Downloaded data, | |
| token caches, generated manifests/reports, checkpoints, archives, environments | |
| and logs under `work/` remain ignored. | |