Upload docs/ClaudeReport_session_2026-09-15_680m_tied_head_fix_and_phase1_extension.md with huggingface_hub

docs/ClaudeReport_session_2026-09-15_680m_tied_head_fix_and_phase1_extension.md

@@ -0,0 +1,82 @@
+# 2026-09-15 - 680M tied-head bf16 fix, then phase-1 context extension (4k -> 16k)
+
+Workspace: dist/Context_Parallelism
+Files: Scripts/Blutrain/bluscriptCP.cpp, p1_680m.sh, p1_shipper.sh, dl_p23.sh, mix_buckets.py
+
+## Why this session exists
+
+Two long-context extension runs on the 1.06B model collapsed. Both broke at roughly
+80 percent of a cumulative b5 (64k-131k) epoch, and the collapse tracked cumulative
+long-document exposure rather than data composition -- mixing shorter buckets in
+delayed it but did not prevent it. The degraded model showed 62x more NLL<0.5
+positions, the fingerprint of copying rather than retrieval. Long-position
+processing itself stayed intact.
+
+Working hypothesis: the 1.06B is undertrained at 0.99 tok/param (Chinchilla is 20),
+so copying is the cheaper solution than retrieval and the model falls into it. That
+predicts a better-trained base has a higher ceiling. varybatch is a 680M at
+10.5 tok/param -- the test.
+
+## The bug that had to be fixed first
+
+varybatch scored NLL 12.13, worse than uniform. Bisected with CP_DUMP_ACTS against
+a PyTorch reference implementation. Cause: in GPT::forward, the weight_tying path
+cast the TRANSPOSED embedding matrix to bf16. The cast ignores strides, so the tied
+head multiplied a garbled weight.
+
+  before: transpose -> to_compute(bf16)          [strided cast, silently wrong]
+  after:  to_compute(bf16) -> transpose -> contiguous()
+
+Never surfaced before because every model trained in this repo uses weight_tying=0,
+which takes the separate contiguous lm_head path. bluscript_zero_ckpt.cpp -- the
+trainer that actually produced varybatch -- documents the same hazard in a comment
+about contiguous() and trans_b.
+
+Confirmed by bf16 mask bisection: mask=15 gave 12.1295, mask=7 (head in fp32) gave
+3.0496. After the fix the 680M scores 3.0497 (PyTorch reference: 3.0793) and the
+1.06B is unchanged at 3.1719 (was 3.1717), so no regression.
+
+Result: the 680M beats the 1.06B on identical data, 3.05 vs 3.17.
+
+## Phase-1 run (launched, in progress)
+
+2x H100 80GB, cp=2. Parent varybatch step 3400.
+
+  window     16,384 via YaRN SCALE=4 x ORIG_MAXPOS=4096 (must equal CP_T exactly)
+  data       Data_Loader/Data/p1mix, 640,009,970 tokens, b3 50 / b2 30 / b1 20
+  steps      1,200 of 1,220 usable at global_batch 524,288 (grad_accum 32)
+  WSD        warmup 100 / stable 1,098 / tail 2
+  measured   50,117 MB of 81,559 MB per card, 7.83 s/step, ETA 2h34m
+
+Both YaRN inputs were verified against the trainer that produced the checkpoint
+rather than assumed: rope_theta 500000 (bluscript_zero_ckpt.cpp:121 and config.md)
+and context_length 4096. The trainer prints a "provenance UNVERIFIED" warning only
+because no sidecar file exists to compare against; the values are correct.
+
+Ladder (b5 deliberately ends at 36 percent of its epoch -- both prior collapses
+were at ~80 percent):
+
+  phase 1   4k -> 16k    b3 50, b2 30, b1 20
+  phase 2   16k -> 32k   b4 50, b3 30, b1 20          fits 2x H100, tight
+  phase 3   32k -> 64k   b5 50, b4 25, b3 15, b1 10   needs >= 4 GPUs (117 GB at cp=2)
+
+## Process notes worth keeping
+
+- The shipper reads $LOG/runnum, but the training script only writes it after
+  training COMPLETES. That left the mixcont shipper idle for 50 minutes. Wrote it
+  manually at launch this time.
+- The shipper's `STEP % FREQ` gate assumes checkpoints are named 200/400/...
+  If the trainer names them 199/399/... it silently ships nothing. The phase-1 copy
+  ships every checkpoint the glob actually finds instead of predicting the numbering.
+- Every hf download used an explicit --include. An unfiltered one earlier in this
+  project pulled 167 GB and broke the build.
+- Every upload is verified by byte size against what HF reports back, never by the
+  exit code of `hf upload` -- that returned success for a missing file on 2026-09-10
+  and 17 GB of checkpoints were lost.
+
+## Open
+
+- CP_EVAL_PPL is broken and deferred by decision. Not required: training, the NIAH
+  evals in run_longeval.sh, and the trainer's own "validation loss:" line are all
+  separate code paths.
+- Phase 3 cannot run on this 2-GPU box.