Blame for scripts/HANDOFF_H200_32K.md

# Handoff: 16k -> 32k YaRN extension on the 2x H200

Written 2026-09-09 from the 2x H100 box. Everything below is measured on this
box unless marked as a projection.

## What the 2x H200 CAN and CANNOT do

`cp` cannot exceed the GPU count, so on 2 GPUs `tokens/rank = T/2`.

| target | tok/rank | total/GPU | 2x H200 (143 GB) |
|---|---|---|---|
| 16k | 8,192 | 70.7 GB (measured) | comfortable |
| **32k** | 16,384 | **130.8 GB (projected)** | **91.4% -- tight, probe first** |
| 64k | 32,768 | 250.9 GB | 175% -- impossible |

- **32k: run it.** `ext_32k.sh` starts with a 3-step probe and aborts on OOM.
- **64k: needs 4+ GPUs.** 4x H200 at cp=4 = 127.6 GB/GPU fits. Nothing to tune
  on 2 GPUs: `CP_B` is already 1, and there is no activation-recompute flag in
  bluscriptCP.cpp (checked: no `CP_RECOMPUTE`, no gradient checkpointing).
- 64k **evaluation** is fine on 2 GPUs -- generation stores no activations for
  backward, so the training memory figure does not apply.

Measured activation cost is 7.51 MB/token/rank. A first-principles estimate for
this architecture is 1.3-2.5 MB, so actual is 3-6x higher than it needs to be
(the config keeps fp32 norms/residual/logits across 36 layers). If that gap is
real, a memory pass could make 32k comfortable and 64k possible on 2 GPUs -- but
that is a C++ change, not a flag, and it is unprofiled.

## Setup on the new box (nothing is pre-installed)

```bash
cd /root/Adhi/BluTrain/dist/Context_Parallelism

# 1. build -- BOTH targets, libtensor first
make libtensor SM_ARCH=90 -j$(nproc)
make CP_FUSED_ROPE=1 CP_ATTN_SPLIT=1 SM_ARCH=90 bluscript-cp -j$(nproc)

# 2. tiktoken -- longeval_gen.py needs it. WITHOUT IT training runs fine and the
#    eval silently produces "no prompts" / "no scores yet". This has bitten twice.
/root/Adhi/BluTrain/envv/bin/pip install tiktoken

# 3. hf auth (the dataset is public now, but the model repo push needs a token)
hf auth login

# 4. parent checkpoint: l2, the best 16k arm. cp=2 needs it on BOTH ranks.
python - <<'PY'
from huggingface_hub import hf_hub_download
import shutil, os
p = hf_hub_download("unparallelled/BluScriptCP",
                    "checkpoints/lenbucket_l2_blumodelcp_run2_step_400.ckpt")
for r in (0, 1):
    d = f"checkpoints_bluscriptcp/rank_{r}/lenbucket_l2_blumodelcp_run2_step_400.ckpt"
    os.makedirs(os.path.dirname(d), exist_ok=True)
    if not os.path.exists(d):
        shutil.copyfile(p, d)
PY

# 5. b3 data (4 train shards + val ~ 930 MB) and the b2-val haystack
hf download unparallelled/BluScriptCP-data --repo-type dataset \
  --include "cp_data_by_length/b3_16384_32768/train_shard_0000[0-3].bin" \
  --include "cp_data_by_length/b3_16384_32768/val_shard_00000.bin" \
  --include "cp_data_by_length/b2_8192_16384/val_shard_00000.bin" \
  --local-dir ./dl
mkdir -p lendata/b3_16384_32768 lendata/haystack
cp dl/cp_data_by_length/b3_16384_32768/*.bin lendata/b3_16384_32768/
cp dl/cp_data_by_length/b2_8192_16384/val_shard_00000.bin lendata/haystack/b2val_haystack.bin

# VERIFY -- `hf download` exits 0 and prints a path even when --include matches
# NOTHING. Always ls, never trust the exit code.
ls -la lendata/b3_16384_32768/ lendata/haystack/

# 6. run it
./ext_32k.sh
```

## The run

```
CP_T=32768   YARN_SCALE=2   YARN_ORIG_MAXPOS=16384
parent: lenbucket_l2_blumodelcp_run2_step_400.ckpt
data:   b3_16384_32768   (docs 16-32k -- these now FIT inside the window)
750 steps x 1,048,576 = 786 M tokens, matching run7's original 4k->16k extension
```

**The YaRN invariant, which cost 27pp when it was wrong:** `YARN_SCALE *
YARN_ORIG_MAXPOS` must equal `CP_T` exactly, and `ORIG_MAXPOS` must be the
model's ORIGINAL PRETRAIN length -- for this family, 4096. The default at
`YARNOps.cpp:57` is **1024** -- always set it explicitly. `ext_32k.sh` asserts
the product.

**CORRECTION 2026-09-15.** This rule previously read "`ORIG_MAXPOS` must be the
length the weights were ACTUALLY trained at". That is right for a single
extension but WRONG for a staged ladder, where after stage one the two stop
being the same number. Hold `ORIG_MAXPOS` at 4096 and raise `SCALE` instead:
4 -> 8 -> 16 for 16k -> 32k -> 64k.

Why: `build_rope_cache` is rebuilt FROM BASE THETA every run (it takes
`rope_theta`, not the previous stage's cache), so `SCALE` always applies to the
ORIGINAL frequencies and never composes with the previous stage's. Re-basing
therefore freezes interpolation instead of deepening it:

    stage  as-run (re-based)      interp   last tok -> virtual   ramp
    16k    4 x 4096               theta/4        4,096          [14,32]
    32k    2 x 16384              theta/2       16,384          [21,39]
    64k    2 x 32768              theta/2       32,768          [24,42]

    stage  corrected (orig=4096)  interp   last tok -> virtual   ramp
    16k    4 x 4096               theta/4        4,096          [14,32]
    32k    8 x 4096               theta/8        4,096          [14,32]
    64k   16 x 4096               theta/16       4,096          [14,32]

Holding orig=4096 pins the far end of every window onto the 4096 range the base
was actually pretrained on, and keeps the ramp fixed, because `find_dim()` takes
(num_rot, L, head_dim, base) -- `scale` is NOT an argument, so the interpolated
dimension set moves only when `ORIG_MAXPOS` moves. Re-basing instead puts the
64k model at virtual position 32,768, i.e. 8x beyond anything it ever saw, with
YaRN doing almost none of the work it exists to do.

The 1.06B ladder (ext_32k.sh, b4_full_epoch.sh, ext64k_b5.sh, mix64k_h200.sh,
mixcont_h100.sh) followed the OLD rule and is affected from the 32k stage on.
This is NOT thought to be the cause of the retrieval collapse -- a wrong cache
is a static defect, while the collapse was progressive and tracked cumulative
b5 exposure across two lineages -- but it plausibly lowered the ceiling.
The 680M ladder (p1/p2/p3_680m.sh) uses the corrected scheme.

Projected step time ~25-29 s (2x the 16k tokens/rank plus attention's quadratic
term), so **750 steps ~ 5-6.5 h**. Unmeasured; the probe prints the real number.

## Why this run, in one paragraph

The document-length ladder is finished and it ended in a tie. Holding everything
else fixed at a 16k window:

| arm | training docs | 4096 | 8192 | 16360 |
|---|---|---|---|---|
| armA_c | flux, median 826 | 73.0 | 53.0 | 33.0 |
| l1 | b1 4-8k (fits) | 75.0 | 48.0 | 30.0 |
| l2 | b2 8-16k (fits) | **80.0** | 62.0 | **35.0** |
| l3 | b3 16-32k (TRUNCATED) | 74.0 | **64.0** | **35.0** |

l3 - l2 = -6.0 / +2.0 / +0.0 pp, none resolvable (combined 95% bars +-11.5 to
+-13.1). The haystack (b2-val) is in-distribution for l2 and out-of-distribution
for l3, so the bias runs TOWARD l2 and l3 still only drew level. Conclusion:
**document length helps only while documents fit inside the window.** Hence widen
the window, then feed b3.

## Gate the result on SHORT context, not perplexity

On 2026-09-02 a mismatched cache IMPROVED 16k (0 -> 14-17%) while COLLAPSING 4k
from 74% to 16%. Perplexity at the new length looked fine. `ext_32k.sh` therefore
evaluates 1024 / 4096 / 16360 / 32744 and **a regression at 1k or 4k is a failure
even if 32k improves.** Compare against l2's row above.

(32744 is not arbitrary: `PROMPT_LEN + MAX_NEW` must pad up to <= `CP_T` with
PAD=256, and 32744 + 24 = 32768 exactly.)

## Traps already paid for

1. **Never edit a running bash script.** Bash reads incrementally from a byte
   offset; an insert shifts it and the interpreter resumes mid-line. This killed
   `ladder_456.sh` with `syntax error near unexpected token 'do'`.
2. **`pgrep -f <script>.sh` matches the shell running it.** Killed my own shell
   twice (exit 144). Exclude `$$`.
3. **Verify all rank shards** before trusting a checkpoint. A 3-of-4 set loads
   silently wrong, and a zero exit code is not proof -- bluscriptCP catches a
   failed save and only prints `[WARN]`.
4. **`say()` must write to stderr.** On stdout it poisons `$(...)` capture; that
   fed a 3-line path into `CP_INIT_FROM` and aborted a run on rank 2.
5. **Run-number collisions.** `run_number` is the first free
   `CP_BluScreipt_Training_logs/CP_Training_log<N>.csv`. A collision with an
   existing checkpoint prefix let `CP_CKPT_KEEP=1` prune a live shard.
6. **`CP_STABLE` cannot equal `CP_MAX_STEPS`** -- `bluscriptCP.cpp:1006` aborts
   with "leaves no cosine decay tail". Leave at least a 2-step tail; the guard
   reads `CP_WARMUP` while `get_lr` uses `CP_REWARMUP`, so pin both.
7. **`CP_REWARMUP_ANCHOR=0` on every resume**, or each resume re-anchors a fresh
   warmup ramp and the LR sawtooths (`bluscriptCP.cpp:2105`).

## Concurrently, on the 2x H100 box

Long b3 arm at 16k: run 17, 4,000 steps, WSD with flat 3.0e-4 for steps
100-3899 then a 100-step tail. Tests whether more long-context STEPS flatten the
slope -- the open question, since all arms sat at -18 to -29 pp/doubling while
Phi-3.5-mini (3.8B) manages -3.6. Checkpointed every 25 steps, `KEEP=2`. An
eval at step 1000 fires automatically, then the run resumes to 4000.

## Not to do

- **b4/b5/b6 at a 16k window** -- l3 already showed truncated documents add
  nothing. b4 only becomes useful after a 64k extension.
- **b6 in a matched ladder** -- 57 M training tokens is 0.27x of one 400-step
  arm; it would cycle 3.7x through all 790 documents.
- **Chasing 128k.** No 4B-class model has effective 128k: applying RULER's own
  threshold (Llama2-7B at 4k = 85.6), Phi-3.5-mini (3.8B, 128k claimed) has an
  effective context of **32k**, and so does Llama-3.1-8B-instruct. 32k is the
  right target, and it is also where this corpus runs out (2.35 B tokens above
  32k, total).