Upload scripts/HANDOFF_H200_32K.md with huggingface_hub

scripts/HANDOFF_H200_32K.md

@@ -0,0 +1,196 @@
+# Handoff: 16k -> 32k YaRN extension on the 2x H200
+
+Written 2026-09-09 from the 2x H100 box. Everything below is measured on this
+box unless marked as a projection.
+
+## What the 2x H200 CAN and CANNOT do
+
+`cp` cannot exceed the GPU count, so on 2 GPUs `tokens/rank = T/2`.
+
+| target | tok/rank | total/GPU | 2x H200 (143 GB) |
+|---|---|---|---|
+| 16k | 8,192 | 70.7 GB (measured) | comfortable |
+| **32k** | 16,384 | **130.8 GB (projected)** | **91.4% -- tight, probe first** |
+| 64k | 32,768 | 250.9 GB | 175% -- impossible |
+
+- **32k: run it.** `ext_32k.sh` starts with a 3-step probe and aborts on OOM.
+- **64k: needs 4+ GPUs.** 4x H200 at cp=4 = 127.6 GB/GPU fits. Nothing to tune
+  on 2 GPUs: `CP_B` is already 1, and there is no activation-recompute flag in
+  bluscriptCP.cpp (checked: no `CP_RECOMPUTE`, no gradient checkpointing).
+- 64k **evaluation** is fine on 2 GPUs -- generation stores no activations for
+  backward, so the training memory figure does not apply.
+
+Measured activation cost is 7.51 MB/token/rank. A first-principles estimate for
+this architecture is 1.3-2.5 MB, so actual is 3-6x higher than it needs to be
+(the config keeps fp32 norms/residual/logits across 36 layers). If that gap is
+real, a memory pass could make 32k comfortable and 64k possible on 2 GPUs -- but
+that is a C++ change, not a flag, and it is unprofiled.
+
+## Setup on the new box (nothing is pre-installed)
+
+```bash
+cd /root/Adhi/BluTrain/dist/Context_Parallelism
+
+# 1. build -- BOTH targets, libtensor first
+make libtensor SM_ARCH=90 -j$(nproc)
+make CP_FUSED_ROPE=1 CP_ATTN_SPLIT=1 SM_ARCH=90 bluscript-cp -j$(nproc)
+
+# 2. tiktoken -- longeval_gen.py needs it. WITHOUT IT training runs fine and the
+#    eval silently produces "no prompts" / "no scores yet". This has bitten twice.
+/root/Adhi/BluTrain/envv/bin/pip install tiktoken
+
+# 3. hf auth (the dataset is public now, but the model repo push needs a token)
+hf auth login
+
+# 4. parent checkpoint: l2, the best 16k arm. cp=2 needs it on BOTH ranks.
+python - <<'PY'
+from huggingface_hub import hf_hub_download
+import shutil, os
+p = hf_hub_download("unparallelled/BluScriptCP",
+                    "checkpoints/lenbucket_l2_blumodelcp_run2_step_400.ckpt")
+for r in (0, 1):
+    d = f"checkpoints_bluscriptcp/rank_{r}/lenbucket_l2_blumodelcp_run2_step_400.ckpt"
+    os.makedirs(os.path.dirname(d), exist_ok=True)
+    if not os.path.exists(d):
+        shutil.copyfile(p, d)
+PY
+
+# 5. b3 data (4 train shards + val ~ 930 MB) and the b2-val haystack
+hf download unparallelled/BluScriptCP-data --repo-type dataset \
+  --include "cp_data_by_length/b3_16384_32768/train_shard_0000[0-3].bin" \
+  --include "cp_data_by_length/b3_16384_32768/val_shard_00000.bin" \
+  --include "cp_data_by_length/b2_8192_16384/val_shard_00000.bin" \
+  --local-dir ./dl
+mkdir -p lendata/b3_16384_32768 lendata/haystack
+cp dl/cp_data_by_length/b3_16384_32768/*.bin lendata/b3_16384_32768/
+cp dl/cp_data_by_length/b2_8192_16384/val_shard_00000.bin lendata/haystack/b2val_haystack.bin
+
+# VERIFY -- `hf download` exits 0 and prints a path even when --include matches
+# NOTHING. Always ls, never trust the exit code.
+ls -la lendata/b3_16384_32768/ lendata/haystack/
+
+# 6. run it
+./ext_32k.sh
+```
+
+## The run
+
+```
+CP_T=32768   YARN_SCALE=2   YARN_ORIG_MAXPOS=16384
+parent: lenbucket_l2_blumodelcp_run2_step_400.ckpt
+data:   b3_16384_32768   (docs 16-32k -- these now FIT inside the window)
+750 steps x 1,048,576 = 786 M tokens, matching run7's original 4k->16k extension
+```
+
+**The YaRN invariant, which cost 27pp when it was wrong:** `YARN_SCALE *
+YARN_ORIG_MAXPOS` must equal `CP_T` exactly, and `ORIG_MAXPOS` must be the
+model's ORIGINAL PRETRAIN length -- for this family, 4096. The default at
+`YARNOps.cpp:57` is **1024** -- always set it explicitly. `ext_32k.sh` asserts
+the product.
+
+**CORRECTION 2026-09-15.** This rule previously read "`ORIG_MAXPOS` must be the
+length the weights were ACTUALLY trained at". That is right for a single
+extension but WRONG for a staged ladder, where after stage one the two stop
+being the same number. Hold `ORIG_MAXPOS` at 4096 and raise `SCALE` instead:
+4 -> 8 -> 16 for 16k -> 32k -> 64k.
+
+Why: `build_rope_cache` is rebuilt FROM BASE THETA every run (it takes
+`rope_theta`, not the previous stage's cache), so `SCALE` always applies to the
+ORIGINAL frequencies and never composes with the previous stage's. Re-basing
+therefore freezes interpolation instead of deepening it:
+
+    stage  as-run (re-based)      interp   last tok -> virtual   ramp
+    16k    4 x 4096               theta/4        4,096          [14,32]
+    32k    2 x 16384              theta/2       16,384          [21,39]
+    64k    2 x 32768              theta/2       32,768          [24,42]
+
+    stage  corrected (orig=4096)  interp   last tok -> virtual   ramp
+    16k    4 x 4096               theta/4        4,096          [14,32]
+    32k    8 x 4096               theta/8        4,096          [14,32]
+    64k   16 x 4096               theta/16       4,096          [14,32]
+
+Holding orig=4096 pins the far end of every window onto the 4096 range the base
+was actually pretrained on, and keeps the ramp fixed, because `find_dim()` takes
+(num_rot, L, head_dim, base) -- `scale` is NOT an argument, so the interpolated
+dimension set moves only when `ORIG_MAXPOS` moves. Re-basing instead puts the
+64k model at virtual position 32,768, i.e. 8x beyond anything it ever saw, with
+YaRN doing almost none of the work it exists to do.
+
+The 1.06B ladder (ext_32k.sh, b4_full_epoch.sh, ext64k_b5.sh, mix64k_h200.sh,
+mixcont_h100.sh) followed the OLD rule and is affected from the 32k stage on.
+This is NOT thought to be the cause of the retrieval collapse -- a wrong cache
+is a static defect, while the collapse was progressive and tracked cumulative
+b5 exposure across two lineages -- but it plausibly lowered the ceiling.
+The 680M ladder (p1/p2/p3_680m.sh) uses the corrected scheme.
+
+Projected step time ~25-29 s (2x the 16k tokens/rank plus attention's quadratic
+term), so **750 steps ~ 5-6.5 h**. Unmeasured; the probe prints the real number.
+
+## Why this run, in one paragraph
+
+The document-length ladder is finished and it ended in a tie. Holding everything
+else fixed at a 16k window:
+
+| arm | training docs | 4096 | 8192 | 16360 |
+|---|---|---|---|---|
+| armA_c | flux, median 826 | 73.0 | 53.0 | 33.0 |
+| l1 | b1 4-8k (fits) | 75.0 | 48.0 | 30.0 |
+| l2 | b2 8-16k (fits) | **80.0** | 62.0 | **35.0** |
+| l3 | b3 16-32k (TRUNCATED) | 74.0 | **64.0** | **35.0** |
+
+l3 - l2 = -6.0 / +2.0 / +0.0 pp, none resolvable (combined 95% bars +-11.5 to
++-13.1). The haystack (b2-val) is in-distribution for l2 and out-of-distribution
+for l3, so the bias runs TOWARD l2 and l3 still only drew level. Conclusion:
+**document length helps only while documents fit inside the window.** Hence widen
+the window, then feed b3.
+
+## Gate the result on SHORT context, not perplexity
+
+On 2026-09-02 a mismatched cache IMPROVED 16k (0 -> 14-17%) while COLLAPSING 4k
+from 74% to 16%. Perplexity at the new length looked fine. `ext_32k.sh` therefore
+evaluates 1024 / 4096 / 16360 / 32744 and **a regression at 1k or 4k is a failure
+even if 32k improves.** Compare against l2's row above.
+
+(32744 is not arbitrary: `PROMPT_LEN + MAX_NEW` must pad up to <= `CP_T` with
+PAD=256, and 32744 + 24 = 32768 exactly.)
+
+## Traps already paid for
+
+1. **Never edit a running bash script.** Bash reads incrementally from a byte
+   offset; an insert shifts it and the interpreter resumes mid-line. This killed
+   `ladder_456.sh` with `syntax error near unexpected token 'do'`.
+2. **`pgrep -f <script>.sh` matches the shell running it.** Killed my own shell
+   twice (exit 144). Exclude `$$`.
+3. **Verify all rank shards** before trusting a checkpoint. A 3-of-4 set loads
+   silently wrong, and a zero exit code is not proof -- bluscriptCP catches a
+   failed save and only prints `[WARN]`.
+4. **`say()` must write to stderr.** On stdout it poisons `$(...)` capture; that
+   fed a 3-line path into `CP_INIT_FROM` and aborted a run on rank 2.
+5. **Run-number collisions.** `run_number` is the first free
+   `CP_BluScreipt_Training_logs/CP_Training_log<N>.csv`. A collision with an
+   existing checkpoint prefix let `CP_CKPT_KEEP=1` prune a live shard.
+6. **`CP_STABLE` cannot equal `CP_MAX_STEPS`** -- `bluscriptCP.cpp:1006` aborts
+   with "leaves no cosine decay tail". Leave at least a 2-step tail; the guard
+   reads `CP_WARMUP` while `get_lr` uses `CP_REWARMUP`, so pin both.
+7. **`CP_REWARMUP_ANCHOR=0` on every resume**, or each resume re-anchors a fresh
+   warmup ramp and the LR sawtooths (`bluscriptCP.cpp:2105`).
+
+## Concurrently, on the 2x H100 box
+
+Long b3 arm at 16k: run 17, 4,000 steps, WSD with flat 3.0e-4 for steps
+100-3899 then a 100-step tail. Tests whether more long-context STEPS flatten the
+slope -- the open question, since all arms sat at -18 to -29 pp/doubling while
+Phi-3.5-mini (3.8B) manages -3.6. Checkpointed every 25 steps, `KEEP=2`. An
+eval at step 1000 fires automatically, then the run resumes to 4000.
+
+## Not to do
+
+- **b4/b5/b6 at a 16k window** -- l3 already showed truncated documents add
+  nothing. b4 only becomes useful after a 64k extension.
+- **b6 in a matched ladder** -- 57 M training tokens is 0.27x of one 400-step
+  arm; it would cycle 3.7x through all 790 documents.
+- **Chasing 128k.** No 4B-class model has effective 128k: applying RULER's own
+  threshold (Llama2-7B at 4k = 85.6), Phi-3.5-mini (3.8B, 128k claimed) has an
+  effective context of **32k**, and so does Llama-3.1-8B-instruct. 32k is the
+  right target, and it is also where this corpus runs out (2.35 B tokens above
+  32k, total).