|
Download ops/DPO_HANDOFF.md from zerdovzad/WarpStateAI: direct link, hf CLI and curl.
- Browser
- Download file 7.65 kB
-
https://huggingface.co/zerdovzad/WarpStateAI/resolve/main/ops/DPO_HANDOFF.md
- Command line
-
hf download hf://zerdovzad/WarpStateAI/ops/DPO_HANDOFF.md
-
curl -L -o DPO_HANDOFF.md https://huggingface.co/zerdovzad/WarpStateAI/resolve/main/ops/DPO_HANDOFF.md
7.65 kB
| # Archive completed SFT and continue into DPO | |
| The user authorized this sequence on 2026-09-14: | |
| **Finish SFT → archive its full state → independent initialization → DPO → pause.** | |
| The separate `warpstate-dpo` Supervisor job waits while the existing SFT job | |
| continues unchanged. The original SFT plan still says pause because it is part | |
| of that run's immutable resume signature; its guardian must complete and exit | |
| before this newly authorized continuation can start. The active DPO plan is | |
| [`../configs/dpo_10b.json`](../configs/dpo_10b.json). | |
| ## Completion and archival gate | |
| The DPO guardian requires all of the following before any GPU work: | |
| - SFT Supervisor state `EXITED`, exit status 0, and its guardian's durable state | |
| `complete`. | |
| - Exclusive SFT guardian, base launcher and pipeline locks, then recheck. | |
| - Exactly 10,312 SFT updates, 329,973 samples, 212,321,372 shifted input tokens, | |
| and 143,695,039 supervised assistant tokens. | |
| - A completed full SFT checkpoint with Adam and both ranks' RNG state, and its | |
| matching final evaluation of all 1,603 held-out conversations. | |
| - Unchanged parent SFT plan, tokenizer, prepared-data manifest and source pins. | |
| The archive is `rtx6000_40b/work/archives/step-00076294-sft-final/`. It contains: | |
| | File | Purpose | | |
| | --- | --- | | |
| | `checkpoint.pt` | Physical byte-for-byte copy of the full SFT checkpoint, including optimizer/RNG/cursor/signatures | | |
| | `model.pt`, `snapshot.json` | Original FP32 model-only export, with every tensor compared exactly to the full checkpoint | | |
| | `tokenizer.model` | Exact original SentencePiece tokenizer | | |
| | `evaluation.json`, `completion.json`, `settings.json`, `metrics.jsonl` | Original final evaluation, completion, execution settings and complete SFT history | | |
| | `sft-execution.json`, `sft-handoff.json`, `PROVENANCE.json` | Source/data identities and both raw-pretraining and SFT stage counters | | |
| | `ARCHIVE.md`, `SHA256SUMS` | Retention instructions and integrity checks | | |
| All files are sealed read-only before the complete directory is atomically | |
| published without replacement. Existing archives are validated and reused, or | |
| rejected without alteration. Interrupted hidden staging directories remain for | |
| diagnosis. The original SFT resume checkpoint and all raw historical archives | |
| remain in place. | |
| DPO reads a physically independent copy of `model.pt`, `snapshot.json` and the | |
| tokenizer under `work/posttrain/step-00076294-dpo/initialization/`. It does not | |
| inherit SFT Adam state or write into either SFT archive or SFT run directory. | |
| The full SFT archive is roughly 32 GB decimal; its counters describe SFT after | |
| the raw step-76,294 model, not an additional 10B-token pretraining run. | |
| ## Preference objective and execution | |
| This first DPO experiment uses: | |
| | Setting | Value | | |
| | --- | --- | | |
| | Data | 74,661 training pairs and 339 held-out pairs already prepared | | |
| | Objective | Length-normalized DPO, beta 5 | | |
| | Epochs | 1; 2,334 optimizer updates | | |
| | Global batch | 32 preference pairs; 2 pairs per rank × 2 ranks × 8 accumulation microbatches | | |
| | Context | 2,048 tokens per chosen/rejected branch; no truncation or cross-conversation packing | | |
| | Learning rate | Peak `5e-7`, 10% warmup, linear decay to zero | | |
| | Optimizer | Fresh FP32 AdamW, betas 0.9/0.95, zero weight decay, clipping at 1 | | |
| | Precision | BF16 compute, original FP32 parameters/gradients/Adam; no FP8 | | |
| | Evaluation/save | Every 100 DPO updates; full holdout at start and completion | | |
| For each branch, sum the log probabilities of **only the final assistant | |
| response and its actual EOS token**. System/user/tool messages, earlier | |
| assistant turns, role headers and padding are conditioning or masked. Divide | |
| each policy-minus-reference log-probability sum by that branch's response-token | |
| count, subtract rejected from chosen, multiply by beta, and apply negative | |
| log-sigmoid. Average over actual preference pairs across ranks; short final | |
| batches and empty rank slots do not change the normalization. | |
| This uses the length-normalized objective and conservative hyperparameters | |
| described in the [Tülu 3 paper, equation 6 and table 20](https://arxiv.org/html/2411.15124v3). | |
| It is a pilot for this 2B/10B base, not a reproduced or tuned Tülu result. | |
| Ordinary summed-log-probability DPO is a separate supported objective and its | |
| usual beta cannot be substituted for beta 5 here without changing the recipe. | |
| Before policy updates, score the immutable SFT model in BF16 on all selected | |
| chosen/rejected branches. Each GPU saves resumable reference-score shards; the | |
| final cache is bound to SFT weights, data, tokenizer, masks, execution settings, | |
| library version and source hashes. Reusing or rebuilding it must preserve that | |
| identity. The reference remains the completed SFT model after every policy | |
| checkpoint/resume. Once scores are cached, release the reference model before | |
| allocating policy optimizer state, avoiding a second resident full model. | |
| The output head uses checkpointed token chunks to avoid retaining full-sequence | |
| vocabulary logits for both responses. A tiny two-GPU BF16 preflight checks the | |
| actual DPO/reference/continuation path after SFT releases the GPUs. CPU setup | |
| tests do not claim that this deferred GPU gate or production-size startup has | |
| already run. Any failed gate stops automatic progression with the archive kept. | |
| ## State, monitoring and controls | |
| Paths below are relative to `rtx6000_40b/work/posttrain/step-00076294-dpo/`. | |
| | Path | Purpose | | |
| | --- | --- | | |
| | `control/handoff.json`, `control/execution.json` | Guardian phase and frozen execution provenance | | |
| | `initialization/` | Independent immutable SFT model copy | | |
| | `preflight/` | Tiny GPU validation and its acceptance report | | |
| | `reference-cache/` | Fixed SFT reference log probabilities and resumable scoring shards | | |
| | `run/latest.pt` | Independent full DPO resume checkpoint, format `warpstate-dpo-v1` | | |
| | `run/metrics.jsonl`, `run/evaluation.json`, `run/completion.json` | Training history, evaluation and durable completion | | |
| From `/workspace/WarpState`: | |
| ```bash | |
| rtx6000_40b/ops/datactl.sh status warpstate-posttrain warpstate-dpo | |
| rtx6000_40b/ops/dpo-handoff.sh --check | |
| tmux attach -t warpstate-dpo | |
| tail -f rtx6000_40b/work/logs/dpo.log | |
| ``` | |
| To cancel automatic DPO while SFT continues, or gracefully stop active DPO: | |
| ```bash | |
| rtx6000_40b/ops/datactl.sh stop warpstate-dpo | |
| ``` | |
| To explicitly resume the unchanged guardian/reference preparation/DPO state: | |
| ```bash | |
| rtx6000_40b/ops/datactl.sh start warpstate-dpo | |
| ``` | |
| No restart of SFT is needed. The new guardian preserves the SFT guardian and | |
| base-launch exclusion locks throughout DPO. Every worker holds a shared base | |
| pipeline lock, so surviving workers block accidental base/SFT/DPO relaunches | |
| even if their launcher dies. Stop files and lock files are never shared with | |
| the live SFT stop request. The program has `autostart=false` and | |
| `autorestart=false`; a whole-machine/Supervisor restart requires explicitly | |
| restarting the appropriate jobs. A completed DPO run stays paused when its | |
| guardian is started again. | |
| On normal completion, validate final counters, all 339 held-out pairs, fixed | |
| reference-cache identity and full archive/init hashes before recording success. | |
| No additional training stage starts automatically. Next compare the archived | |
| SFT model with DPO on instruction following and tool use. DPO loss, reward | |
| margin and preference accuracy describe this objective; exponentiating DPO | |
| loss does not produce language-model perplexity. | |
| All generated data, scores, archives, checkpoints, logs and environments remain | |
| ignored under `work/`. Only source/configuration/tests/documentation are tracked. | |