WarpStateAI / ops /DPO_CHAT.md
zerdovzad's picture
Upload 139 files
c8ba9c6 verified
|
Raw History Blame Contribute Delete
2.72 kB

Final DPO archive and chat

DPO finished on 2026-09-15 at 01:05:06.882 UTC, with the guardian's final verification complete at 01:05:42 UTC. It completed 2,334 updates over 74,661 preference pairs. Training remains paused.

The historical archive is work/archives/step-00076294-dpo-final/ relative to rtx6000_40b. It preserves the full FP32 model/Adam/RNG checkpoint and a tensor-exact inference export, plus the tokenizer, evaluation, recipes, reference scores, and provenance. Files are sealed read-only and checksummed. Existing raw and SFT archives are preserved.

From /workspace/WarpState, create or verify the archive and prepare the independent chat copy:

source rtx6000_40b/ops/training-env.sh
rtx6000_40b/.server-venv/bin/python -B rtx6000_40b/ops/archive_dpo.py
rtx6000_40b/.server-venv/bin/python -B rtx6000_40b/ops/prepare_dpo_chat.py

The chat copy is rtx6000_40b/work/chat/dpo-final/: separate physical files, with hashes equal to the archive. The chat launcher reads this copy by default. Neither chat nor preparation modifies archived weights.

rtx6000_40b/ops/chat-dpo.sh
# Or attach to the already-open interface:
tmux attach -t warpstate-chat

Type a message at You:. Ctrl-C discards unfinished input or cancels the current reply and returns to You:, preserving completed conversation turns. /reset clears conversation history; /quit exits. Detach tmux with Ctrl-B followed by D. The interface uses GPU 0, BF16 compute, original FP32 master weights, a 2,048-token context, and greedy decoding. Responses print when complete. Alternate CUDA devices can be selected with --device cuda:1; --temperature 0.7 enables sampling.

Prompt serialization matches the segmented role headers, BOS and assistant EOS used during posttraining. Cached inference follows the model's parallel chunk memory; shape-dependent BF16 rounding can differ from full-sequence execution.

Run reproducible sample prompts, choosing a new output filename each time:

rtx6000_40b/ops/chat-dpo.sh --prompts-json rtx6000_40b/configs/posttrain_smoke_prompts.json --max-new-tokens 128 --output-json rtx6000_40b/work/validation/dpo-smoke.json

These few prompts are sanity checks, not a capability benchmark. DPO's final held-out preference loss was 0.5790; 245 of 339 pairs had positive relative reward margins. This 72.27% is not a generation win rate or answer accuracy. Exponentiating DPO loss does not produce language-model perplexity.

For a matched full SFT holdout evaluation, ops/evaluate_sft_snapshot.py accepts --snapshot, --device, and a new --output path. Reports and generated responses stay under ignored work/; source scripts and prompts are tracked.