Blame for docs/HANDOFF_H200_32K.md

# Handoff: 16k -> 32k YaRN extension on the 2x H200

Written 2026-09-09 from the 2x H100 box. Everything below is measured on this
box unless marked as a projection.

## What the 2x H200 CAN and CANNOT do

`cp` cannot exceed the GPU count, so on 2 GPUs `tokens/rank = T/2`.

| target | tok/rank | total/GPU | 2x H200 (143 GB) |
|---|---|---|---|
| 16k | 8,192 | 70.7 GB (measured) | comfortable |
| **32k** | 16,384 | **130.8 GB (projected)** | **91.4% -- tight, probe first** |
| 64k | 32,768 | 250.9 GB | 175% -- impossible |

- **32k: run it.** `ext_32k.sh` starts with a 3-step probe and aborts on OOM.
- **64k: needs 4+ GPUs.** 4x H200 at cp=4 = 127.6 GB/GPU fits. Nothing to tune
  on 2 GPUs: `CP_B` is already 1, and there is no activation-recompute flag in
  bluscriptCP.cpp (checked: no `CP_RECOMPUTE`, no gradient checkpointing).
- 64k **evaluation** is fine on 2 GPUs -- generation stores no activations for
  backward, so the training memory figure does not apply.

Measured activation cost is 7.51 MB/token/rank. A first-principles estimate for
this architecture is 1.3-2.5 MB, so actual is 3-6x higher than it needs to be
(the config keeps fp32 norms/residual/logits across 36 layers). If that gap is
real, a memory pass could make 32k comfortable and 64k possible on 2 GPUs -- but
that is a C++ change, not a flag, and it is unprofiled.

## Setup on the new box (nothing is pre-installed)

```bash
cd /root/Adhi/BluTrain/dist/Context_Parallelism

# 1. build -- BOTH targets, libtensor first
make libtensor SM_ARCH=90 -j$(nproc)
make CP_FUSED_ROPE=1 CP_ATTN_SPLIT=1 SM_ARCH=90 bluscript-cp -j$(nproc)

# 2. tiktoken -- longeval_gen.py needs it. WITHOUT IT training runs fine and the
#    eval silently produces "no prompts" / "no scores yet". This has bitten twice.
/root/Adhi/BluTrain/envv/bin/pip install tiktoken

# 3. hf auth (the dataset is public now, but the model repo push needs a token)
hf auth login

# 4. parent checkpoint: l2, the best 16k arm. cp=2 needs it on BOTH ranks.
python - <<'PY'
from huggingface_hub import hf_hub_download
import shutil, os
p = hf_hub_download("unparallelled/BluScriptCP",
                    "checkpoints/lenbucket_l2_blumodelcp_run2_step_400.ckpt")
for r in (0, 1):
    d = f"checkpoints_bluscriptcp/rank_{r}/lenbucket_l2_blumodelcp_run2_step_400.ckpt"
    os.makedirs(os.path.dirname(d), exist_ok=True)
    if not os.path.exists(d):
        shutil.copyfile(p, d)
PY

# 5. b3 data (4 train shards + val ~ 930 MB) and the b2-val haystack
hf download unparallelled/BluScriptCP-data --repo-type dataset \
  --include "cp_data_by_length/b3_16384_32768/train_shard_0000[0-3].bin" \
  --include "cp_data_by_length/b3_16384_32768/val_shard_00000.bin" \
  --include "cp_data_by_length/b2_8192_16384/val_shard_00000.bin" \
  --local-dir ./dl
mkdir -p lendata/b3_16384_32768 lendata/haystack
cp dl/cp_data_by_length/b3_16384_32768/*.bin lendata/b3_16384_32768/
cp dl/cp_data_by_length/b2_8192_16384/val_shard_00000.bin lendata/haystack/b2val_haystack.bin

# VERIFY -- `hf download` exits 0 and prints a path even when --include matches
# NOTHING. Always ls, never trust the exit code.
ls -la lendata/b3_16384_32768/ lendata/haystack/

# 6. run it
./ext_32k.sh
```

## The run

```
CP_T=32768   YARN_SCALE=2   YARN_ORIG_MAXPOS=16384
parent: lenbucket_l2_blumodelcp_run2_step_400.ckpt
data:   b3_16384_32768   (docs 16-32k -- these now FIT inside the window)
750 steps x 1,048,576 = 786 M tokens, matching run7's original 4k->16k extension
```

**The YaRN invariant, which cost 27pp when it was wrong:** `YARN_SCALE *
YARN_ORIG_MAXPOS` must equal `CP_T` exactly, and `ORIG_MAXPOS` must be the length
the weights were ACTUALLY trained at. Here 2 * 16384 = 32768. The default at
`YARNOps.cpp:57` is **1024** -- always set it explicitly. `ext_32k.sh` asserts it.

Projected step time ~25-29 s (2x the 16k tokens/rank plus attention's quadratic
term), so **750 steps ~ 5-6.5 h**. Unmeasured; the probe prints the real number.

## Why this run, in one paragraph

The document-length ladder is finished and it ended in a tie. Holding everything
else fixed at a 16k window:

| arm | training docs | 4096 | 8192 | 16360 |
|---|---|---|---|---|
| armA_c | flux, median 826 | 73.0 | 53.0 | 33.0 |
| l1 | b1 4-8k (fits) | 75.0 | 48.0 | 30.0 |
| l2 | b2 8-16k (fits) | **80.0** | 62.0 | **35.0** |
| l3 | b3 16-32k (TRUNCATED) | 74.0 | **64.0** | **35.0** |

l3 - l2 = -6.0 / +2.0 / +0.0 pp, none resolvable (combined 95% bars +-11.5 to
+-13.1). The haystack (b2-val) is in-distribution for l2 and out-of-distribution
for l3, so the bias runs TOWARD l2 and l3 still only drew level. Conclusion:
**document length helps only while documents fit inside the window.** Hence widen
the window, then feed b3.

## Gate the result on SHORT context, not perplexity

On 2026-09-02 a mismatched cache IMPROVED 16k (0 -> 14-17%) while COLLAPSING 4k
from 74% to 16%. Perplexity at the new length looked fine. `ext_32k.sh` therefore
evaluates 1024 / 4096 / 16360 / 32744 and **a regression at 1k or 4k is a failure
even if 32k improves.** Compare against l2's row above.

(32744 is not arbitrary: `PROMPT_LEN + MAX_NEW` must pad up to <= `CP_T` with
PAD=256, and 32744 + 24 = 32768 exactly.)

## Traps already paid for

1. **Never edit a running bash script.** Bash reads incrementally from a byte
   offset; an insert shifts it and the interpreter resumes mid-line. This killed
   `ladder_456.sh` with `syntax error near unexpected token 'do'`.
2. **`pgrep -f <script>.sh` matches the shell running it.** Killed my own shell
   twice (exit 144). Exclude `$$`.
3. **Verify all rank shards** before trusting a checkpoint. A 3-of-4 set loads
   silently wrong, and a zero exit code is not proof -- bluscriptCP catches a
   failed save and only prints `[WARN]`.
4. **`say()` must write to stderr.** On stdout it poisons `$(...)` capture; that
   fed a 3-line path into `CP_INIT_FROM` and aborted a run on rank 2.
5. **Run-number collisions.** `run_number` is the first free
   `CP_BluScreipt_Training_logs/CP_Training_log<N>.csv`. A collision with an
   existing checkpoint prefix let `CP_CKPT_KEEP=1` prune a live shard.
6. **`CP_STABLE` cannot equal `CP_MAX_STEPS`** -- `bluscriptCP.cpp:1006` aborts
   with "leaves no cosine decay tail". Leave at least a 2-step tail; the guard
   reads `CP_WARMUP` while `get_lr` uses `CP_REWARMUP`, so pin both.
7. **`CP_REWARMUP_ANCHOR=0` on every resume**, or each resume re-anchors a fresh
   warmup ramp and the LR sawtooths (`bluscriptCP.cpp:2105`).

## Concurrently, on the 2x H100 box

Long b3 arm at 16k: run 17, 4,000 steps, WSD with flat 3.0e-4 for steps
100-3899 then a 100-step tail. Tests whether more long-context STEPS flatten the
slope -- the open question, since all arms sat at -18 to -29 pp/doubling while
Phi-3.5-mini (3.8B) manages -3.6. Checkpointed every 25 steps, `KEEP=2`. An
eval at step 1000 fires automatically, then the run resumes to 4000.

## Not to do

- **b4/b5/b6 at a 16k window** -- l3 already showed truncated documents add
  nothing. b4 only becomes useful after a 64k extension.
- **b6 in a matched ladder** -- 57 M training tokens is 0.27x of one 400-step
  arm; it would cycle 3.7x through all 790 documents.
- **Chasing 128k.** No 4B-class model has effective 128k: applying RULER's own
  threshold (Llama2-7B at 4k = 85.6), Phi-3.5-mini (3.8B, 128k claimed) has an
  effective context of **32k**, and so does Llama-3.1-8B-instruct. 32k is the
  right target, and it is also where this corpus runs out (2.35 B tokens above
  32k, total).