WarpStateAI / ops /POSTTRAIN.md
zerdovzad's picture
Upload 139 files
c8ba9c6 verified
|
Raw History Blame Contribute Delete
9.14 kB

Step 76,294 archive and supervised fine-tuning

The SFT stage runs from ../configs/posttrain_10b.json. The user later authorized archiving the full completed SFT state, then automatically running DPO. A separate warpstate-dpo Supervisor job implements that continuation; see DPO_HANDOFF.md. The active SFT plan/source bytes stay fixed to preserve their checkpoint/resume signatures. Its original guardian still finishes and exits cleanly before the DPO guardian can take over.

prepare_posttrain_data.py prepares conversations and preference pairs. The separate posttrain.py runner implements assistant-only supervised training.

Automatic sequence

  1. warpstate-posttrain-data prepares the pinned dataset recipe on CPU now.
  2. warpstate-posttrain waits for warpstate-train to exit successfully. It never changes the base run or resumes an earlier stopped base checkpoint.
  3. Under the base launch/pipeline locks, require the full checkpoint to contain step 76,294, 10,000,007,168 tokens, and final validation at that same boundary. A checkpoint before the boundary remains waiting; an overshoot, stale validation, or mismatched tokenizer fails without starting SFT.
  4. Export the original FP32 model using the same chat.capture_latest method as the 8K, 16K and 40K archives. Compare every exported tensor exactly against the stopped checkpoint; copy the exact tokenizer; write provenance/checksums; seal files read-only; publish the complete archive atomically without replacing an existing directory.
  5. Make a physical, independent copy of the archive as SFT initialization. The full pretraining checkpoint, corpus, historical archives and chat selection are retained. SFT starts with a fresh AdamW optimizer on its first launch.
  6. Verify prepared SFT data, run a tiny two-GPU BF16 training/resume preflight, then start the full SFT model. Failed preflight or data checks stop the handoff and retain the raw archive for diagnosis.
  7. Save final SFT state and evaluation results, verify the raw archive and initialization remain unchanged, then exit. Neither program automatically restarts training after completion or failure. The separately armed DPO job then archives this full SFT state and continues according to DPO_HANDOFF.md.

The boundary triggers archival, checksum verification, compilation/preflight and initialization before the first SFT update; allow several minutes for this handoff. GPU preflight is deliberately deferred until base training releases the GPUs. Its result is not claimed by the CPU-only setup tests.

Paths

All paths below are relative to /workspace/WarpState/rtx6000_40b.

Purpose Path
Untouched full base resume checkpoint work/run/latest.pt
Raw historical model archive work/archives/step-00076294/
Independent SFT initialization work/posttrain/step-00076294/initialization/
Prepared SFT/DPO data and final manifest work/posttrain/data/
SFT checkpoints, settings and metrics work/posttrain/step-00076294/run/
Final evaluation / completion run/evaluation.json, run/completion.json under the SFT directory
Handoff state and frozen execution provenance work/posttrain/step-00076294/control/
GPU preflight artifacts work/posttrain/step-00076294/preflight/
Handoff and training log work/logs/posttrain.log
CPU preparation log work/logs/posttrain-data.log

Archives contain model weights only, without Adam or training RNG state. SFT's latest.pt is a separate full warpstate-posttrain-v1 checkpoint with optimizer, per-rank RNG, deterministic sample cursor, source/data identities and base provenance. It is not interchangeable with the base runner's checkpoint.

SFT recipe and validation

This first run uses one epoch, two GPUs, sequence length 2,048, microbatch 4 per rank and four accumulation microbatches: 32 conversations per update. There is no cross-conversation packing. All parameters train with BF16 compute, original FP32 parameters/gradients and fresh FP32 AdamW state; no FP8 is used. The learning rate peaks at 5e-6, with 3% warmup and cosine decay to zero, zero weight decay and gradient clipping at 1. Evaluation/checkpoints occur every 100 SFT updates and at the end. Initial and final validation use the full SFT holdout; intermediate checks use the same fixed 128-example subset. Settings and effective epoch length are recorded in the SFT run's settings.json once data is loaded.

The learning rate is a conservative experimental starting point, not a validated optimum for this 2B model. This is not a reproduction of Tülu 3's SFT schedule: its published learning rate was calibrated using summed token loss, and its models/context/batch sizes differ. See the Tülu 3 paper, section 4.3.2.

Role headers use ordinary tokens from the existing SentencePiece vocabulary. Only assistant content and the actual EOS ID contribute to the loss. Prompt, role-header and padding labels are masked; DDP gradients are normalized by the global count of supervised tokens, including uneven final batches. The shared renderer is ops/posttrain_data.py; its hash is pinned in the data manifest.

SFT metrics report actual nonpadding input tokens and supervised assistant tokens separately. SFT validation loss/perplexity uses the assistant-token objective on the prepared holdout; it is not directly comparable with the base model's all-token pretraining validation loss. Evaluation reports include the exact evaluated sample/token counts, dataset and base-weight identities. Behavioral instruction-following and tool-use evaluation is the next manual decision after the full authorized SFT/DPO sequence pauses; a lower SFT loss alone does not establish those capabilities. The final SFT archive preserves the exact model and its evaluation for comparison with DPO.

Monitor, pause and resume

From the repository root:

rtx6000_40b/ops/datactl.sh status warpstate-train warpstate-posttrain-data warpstate-posttrain
rtx6000_40b/ops/posttrain-handoff.sh --check
tmux attach -t warpstate-posttrain
tail -f rtx6000_40b/work/logs/posttrain.log

The tmux session displays the durable Supervisor log. Detaching tmux does not stop the handoff or training. RUNNING on the posttraining program can mean waiting for the base boundary; inspect --check or its phase log.

To pause the handoff or gracefully checkpoint an active SFT run:

rtx6000_40b/ops/datactl.sh stop warpstate-posttrain

To explicitly re-enable the unchanged handoff or resume an interrupted SFT run:

rtx6000_40b/ops/datactl.sh start warpstate-posttrain

The saved plan/source identity must still match. A completed run stays paused even if the program is started again. Do not remove its completion state to repeat SFT. Use a separately reviewed plan and output directory for a new experiment. Do not clear lock files or launch the base trainer manually while SFT workers are alive: every SFT rank holds a shared base pipeline lock, and the base/handoff require its exclusive lock. The launcher also retains its exclusion locks across child launch and unexpected watcher death.

SFT Supervisor includes are in ops/posttrain-supervisord.conf; both programs have autostart=false and autorestart=false. Setup explicitly starts the authorized jobs. A whole-machine/Supervisor restart requires an explicit restart of the appropriate jobs; there is no hidden reboot training launch.

To cancel automatic continuation into DPO while allowing SFT to finish, stop the separate waiting job with rtx6000_40b/ops/datactl.sh stop warpstate-dpo. Stopping SFT itself leaves the DPO guardian waiting for verified completion.

Rebuild the prepared inputs

The recipe in configs/posttrain_data.json pins immutable dataset revisions, the tokenizer identity, sample counts/seed, split rule, renderer and required preparation packages. Preparation records downloaded file hashes, schemas, source/licensing metadata, row/token counts, rejections and output hashes. Prompt groups use the same deterministic partition across SFT and DPO; exact duplicate conversations are excluded. Overlength examples and DPO branches are rejected rather than truncated. Gated xLAM data is recorded as skipped without automatically accepting terms or requiring interactive input.

To rebuild the isolated CPU environment, source ops/training-env.sh, create work/posttrain-data-venv with Python 3.12, then install ops/requirements-posttrain-data.lock into that environment. Do not change the active training environment. ops/prepare-posttrain-data.sh runs the recipe; do not regenerate/replace data while SFT is running. The completed manifest is published only after every selected output has been written and hashed.

Source/configuration/tests/dependency metadata belong in Git. Downloaded data, token caches, generated manifests/reports, checkpoints, archives, environments and logs under work/ remain ignored.