gpu-sft — Qwen3 self-instill SFT + eval (TPU→GPU port)

Downstream of the SDG pipeline: train Qwen3 on the self-instill datasets (the fzzhang/qwen3_8b_*_instill_... sets our SDG runs produced) → eval on math/science/code → regenerate results.md.

Provenance

All scripts here were extracted verbatim from Tony's handoff doc fangzhao.md (Appendix A), via a fenced-code parser — not hand-typed. They have NOT been run on a GPU (written on macOS, no CUDA). Treat as a strong first draft to smoke-test on one node; the acceptance tests are the two parity gates in §10 of the handoff. Do not trust any experiment number until Gate 1 (training-loss parity) and Gate 2 (eval-baseline parity) pass.

Pipeline

HF dataset@rev → scripts/gpu_sft/prepare_sft_data.py  (CPU) → packed .npy + assistant mask
              → scripts/gpu_sft/train_sft_qwen3.py     (GPU FSDP) → hf/step-N/ (bf16)
              → scripts/gpu_eval/gpu_eval_driver.py    (vLLM+evalchemy) → eval JSON
              → claude/compile_results_local.py        → results.md

Environment (two venvs — mandatory; see handoff §5.1)

sft-train: torch 2.7 cu124 + transformers 4.57 + flash-attn (this dir's requirements-gpu.txt).
sft-eval: vLLM (its own torch) + datasets>=3.1,<4 + pinned evalchemy (via scripts/gpu_eval/bootstrap_gpu_eval.sh).
HF_HOME on the big local disk, NOT $HOME. Pin datasets<4.0 in both (v4 breaks LiveCodeBench).
Gated licences: the fzzhang HF account must accept cais/hle AND Idavidrein/gpqa before any eval.

Storage plan (this cluster: Azure ND96isr H100 v5, 4 nodes × 8×H100-80G, not preemptible)

No HDFS, no cross-node FS. 8B@32K fits on ONE node → run 4 independent single-node experiments, one per node.
Working dir + HF_HOME + active checkpoints → local NVMe (/, ~3.3 TB free; one 8B experiment ~2 TB fits).
Durable artifact = bf16 hf/step-N exports → private HF repo fzzhang/<exp>-stepN (existing workflow; durable + eval-anywhere).
Tradeoff: HF export drops the fp32 optimizer state, so no bit-exact training resume. Given not-preemptible + ~1.5-day runs, accept re-run risk. Revisit HDFS write-perm only if bit-exact auto-resume is needed.

Status

Phase 0 (zero-GPU): DONE — all 8 Python files compile; the results.md compiler is validated end-to-end via claude/make_fixture_eval_tree.py → claude/compile_results_local.py (correct §9 format).
Phase 1 (1 node): pending — two venvs + gated-licence preflight → prepare_sft_data.py on one dataset (with the §10 sanity asserts) → tiny smoke train → parity gate (coding_depth4v3_n8_vr5, 2000 steps) vs Tony's W&B (tonyhlee/instill_ot4).
Phase 2: GPU baselines (Qwen3-8B) into a fresh results root → experiments.

Zero-GPU compiler self-test (reproduce anywhere)

python claude/make_fixture_eval_tree.py /tmp/fixture
python claude/compile_results_local.py --root /tmp/fixture/evalout \
    --models /tmp/fixture/evals.md --output /tmp/fixture/results.md \
    --timestamp "2026-09-01 12:00" --no-cache
cat /tmp/fixture/results.md