WarpStateAI / ops /DPO_HANDOFF.md
zerdovzad's picture
Upload 139 files
c8ba9c6 verified
|
Raw History Blame Contribute Delete
7.65 kB

Archive completed SFT and continue into DPO

The user authorized this sequence on 2026-09-14:

Finish SFT → archive its full state → independent initialization → DPO → pause.

The separate warpstate-dpo Supervisor job waits while the existing SFT job continues unchanged. The original SFT plan still says pause because it is part of that run's immutable resume signature; its guardian must complete and exit before this newly authorized continuation can start. The active DPO plan is ../configs/dpo_10b.json.

Completion and archival gate

The DPO guardian requires all of the following before any GPU work:

  • SFT Supervisor state EXITED, exit status 0, and its guardian's durable state complete.
  • Exclusive SFT guardian, base launcher and pipeline locks, then recheck.
  • Exactly 10,312 SFT updates, 329,973 samples, 212,321,372 shifted input tokens, and 143,695,039 supervised assistant tokens.
  • A completed full SFT checkpoint with Adam and both ranks' RNG state, and its matching final evaluation of all 1,603 held-out conversations.
  • Unchanged parent SFT plan, tokenizer, prepared-data manifest and source pins.

The archive is rtx6000_40b/work/archives/step-00076294-sft-final/. It contains:

File Purpose
checkpoint.pt Physical byte-for-byte copy of the full SFT checkpoint, including optimizer/RNG/cursor/signatures
model.pt, snapshot.json Original FP32 model-only export, with every tensor compared exactly to the full checkpoint
tokenizer.model Exact original SentencePiece tokenizer
evaluation.json, completion.json, settings.json, metrics.jsonl Original final evaluation, completion, execution settings and complete SFT history
sft-execution.json, sft-handoff.json, PROVENANCE.json Source/data identities and both raw-pretraining and SFT stage counters
ARCHIVE.md, SHA256SUMS Retention instructions and integrity checks

All files are sealed read-only before the complete directory is atomically published without replacement. Existing archives are validated and reused, or rejected without alteration. Interrupted hidden staging directories remain for diagnosis. The original SFT resume checkpoint and all raw historical archives remain in place.

DPO reads a physically independent copy of model.pt, snapshot.json and the tokenizer under work/posttrain/step-00076294-dpo/initialization/. It does not inherit SFT Adam state or write into either SFT archive or SFT run directory. The full SFT archive is roughly 32 GB decimal; its counters describe SFT after the raw step-76,294 model, not an additional 10B-token pretraining run.

Preference objective and execution

This first DPO experiment uses:

Setting Value
Data 74,661 training pairs and 339 held-out pairs already prepared
Objective Length-normalized DPO, beta 5
Epochs 1; 2,334 optimizer updates
Global batch 32 preference pairs; 2 pairs per rank × 2 ranks × 8 accumulation microbatches
Context 2,048 tokens per chosen/rejected branch; no truncation or cross-conversation packing
Learning rate Peak 5e-7, 10% warmup, linear decay to zero
Optimizer Fresh FP32 AdamW, betas 0.9/0.95, zero weight decay, clipping at 1
Precision BF16 compute, original FP32 parameters/gradients/Adam; no FP8
Evaluation/save Every 100 DPO updates; full holdout at start and completion

For each branch, sum the log probabilities of only the final assistant response and its actual EOS token. System/user/tool messages, earlier assistant turns, role headers and padding are conditioning or masked. Divide each policy-minus-reference log-probability sum by that branch's response-token count, subtract rejected from chosen, multiply by beta, and apply negative log-sigmoid. Average over actual preference pairs across ranks; short final batches and empty rank slots do not change the normalization.

This uses the length-normalized objective and conservative hyperparameters described in the Tülu 3 paper, equation 6 and table 20. It is a pilot for this 2B/10B base, not a reproduced or tuned Tülu result. Ordinary summed-log-probability DPO is a separate supported objective and its usual beta cannot be substituted for beta 5 here without changing the recipe.

Before policy updates, score the immutable SFT model in BF16 on all selected chosen/rejected branches. Each GPU saves resumable reference-score shards; the final cache is bound to SFT weights, data, tokenizer, masks, execution settings, library version and source hashes. Reusing or rebuilding it must preserve that identity. The reference remains the completed SFT model after every policy checkpoint/resume. Once scores are cached, release the reference model before allocating policy optimizer state, avoiding a second resident full model.

The output head uses checkpointed token chunks to avoid retaining full-sequence vocabulary logits for both responses. A tiny two-GPU BF16 preflight checks the actual DPO/reference/continuation path after SFT releases the GPUs. CPU setup tests do not claim that this deferred GPU gate or production-size startup has already run. Any failed gate stops automatic progression with the archive kept.

State, monitoring and controls

Paths below are relative to rtx6000_40b/work/posttrain/step-00076294-dpo/.

Path Purpose
control/handoff.json, control/execution.json Guardian phase and frozen execution provenance
initialization/ Independent immutable SFT model copy
preflight/ Tiny GPU validation and its acceptance report
reference-cache/ Fixed SFT reference log probabilities and resumable scoring shards
run/latest.pt Independent full DPO resume checkpoint, format warpstate-dpo-v1
run/metrics.jsonl, run/evaluation.json, run/completion.json Training history, evaluation and durable completion

From /workspace/WarpState:

rtx6000_40b/ops/datactl.sh status warpstate-posttrain warpstate-dpo
rtx6000_40b/ops/dpo-handoff.sh --check
tmux attach -t warpstate-dpo
tail -f rtx6000_40b/work/logs/dpo.log

To cancel automatic DPO while SFT continues, or gracefully stop active DPO:

rtx6000_40b/ops/datactl.sh stop warpstate-dpo

To explicitly resume the unchanged guardian/reference preparation/DPO state:

rtx6000_40b/ops/datactl.sh start warpstate-dpo

No restart of SFT is needed. The new guardian preserves the SFT guardian and base-launch exclusion locks throughout DPO. Every worker holds a shared base pipeline lock, so surviving workers block accidental base/SFT/DPO relaunches even if their launcher dies. Stop files and lock files are never shared with the live SFT stop request. The program has autostart=false and autorestart=false; a whole-machine/Supervisor restart requires explicitly restarting the appropriate jobs. A completed DPO run stays paused when its guardian is started again.

On normal completion, validate final counters, all 339 held-out pairs, fixed reference-cache identity and full archive/init hashes before recording success. No additional training stage starts automatically. Next compare the archived SFT model with DPO on instruction following and tool use. DPO loss, reward margin and preference accuracy describe this objective; exponentiating DPO loss does not produce language-model perplexity.

All generated data, scores, archives, checkpoints, logs and environments remain ignored under work/. Only source/configuration/tests/documentation are tracked.