Handoff: 16k -> 32k YaRN extension on the 2x H200

Written 2026-09-09 from the 2x H100 box. Everything below is measured on this box unless marked as a projection.

What the 2x H200 CAN and CANNOT do

cp cannot exceed the GPU count, so on 2 GPUs tokens/rank = T/2.

target	tok/rank	total/GPU	2x H200 (143 GB)
16k	8,192	70.7 GB (measured)	comfortable
32k	16,384	130.8 GB (projected)	91.4% -- tight, probe first
64k	32,768	250.9 GB	175% -- impossible

32k: run it. ext_32k.sh starts with a 3-step probe and aborts on OOM.
64k: needs 4+ GPUs. 4x H200 at cp=4 = 127.6 GB/GPU fits. Nothing to tune on 2 GPUs: CP_B is already 1, and there is no activation-recompute flag in bluscriptCP.cpp (checked: no CP_RECOMPUTE, no gradient checkpointing).
64k evaluation is fine on 2 GPUs -- generation stores no activations for backward, so the training memory figure does not apply.

Measured activation cost is 7.51 MB/token/rank. A first-principles estimate for this architecture is 1.3-2.5 MB, so actual is 3-6x higher than it needs to be (the config keeps fp32 norms/residual/logits across 36 layers). If that gap is real, a memory pass could make 32k comfortable and 64k possible on 2 GPUs -- but that is a C++ change, not a flag, and it is unprofiled.

Setup on the new box (nothing is pre-installed)

cd /root/Adhi/BluTrain/dist/Context_Parallelism

# 1. build -- BOTH targets, libtensor first
make libtensor SM_ARCH=90 -j$(nproc)
make CP_FUSED_ROPE=1 CP_ATTN_SPLIT=1 SM_ARCH=90 bluscript-cp -j$(nproc)

# 2. tiktoken -- longeval_gen.py needs it. WITHOUT IT training runs fine and the
#    eval silently produces "no prompts" / "no scores yet". This has bitten twice.
/root/Adhi/BluTrain/envv/bin/pip install tiktoken

# 3. hf auth (the dataset is public now, but the model repo push needs a token)
hf auth login

# 4. parent checkpoint: l2, the best 16k arm. cp=2 needs it on BOTH ranks.
python - <<'PY'
from huggingface_hub import hf_hub_download
import shutil, os
p = hf_hub_download("unparallelled/BluScriptCP",
                    "checkpoints/lenbucket_l2_blumodelcp_run2_step_400.ckpt")
for r in (0, 1):
    d = f"checkpoints_bluscriptcp/rank_{r}/lenbucket_l2_blumodelcp_run2_step_400.ckpt"
    os.makedirs(os.path.dirname(d), exist_ok=True)
    if not os.path.exists(d):
        shutil.copyfile(p, d)
PY

# 5. b3 data (4 train shards + val ~ 930 MB) and the b2-val haystack
hf download unparallelled/BluScriptCP-data --repo-type dataset \
  --include "cp_data_by_length/b3_16384_32768/train_shard_0000[0-3].bin" \
  --include "cp_data_by_length/b3_16384_32768/val_shard_00000.bin" \
  --include "cp_data_by_length/b2_8192_16384/val_shard_00000.bin" \
  --local-dir ./dl
mkdir -p lendata/b3_16384_32768 lendata/haystack
cp dl/cp_data_by_length/b3_16384_32768/*.bin lendata/b3_16384_32768/
cp dl/cp_data_by_length/b2_8192_16384/val_shard_00000.bin lendata/haystack/b2val_haystack.bin

# VERIFY -- `hf download` exits 0 and prints a path even when --include matches
# NOTHING. Always ls, never trust the exit code.
ls -la lendata/b3_16384_32768/ lendata/haystack/

# 6. run it
./ext_32k.sh

The run

CP_T=32768   YARN_SCALE=2   YARN_ORIG_MAXPOS=16384
parent: lenbucket_l2_blumodelcp_run2_step_400.ckpt
data:   b3_16384_32768   (docs 16-32k -- these now FIT inside the window)
750 steps x 1,048,576 = 786 M tokens, matching run7's original 4k->16k extension

The YaRN invariant, which cost 27pp when it was wrong: YARN_SCALE * YARN_ORIG_MAXPOS must equal CP_T exactly, and ORIG_MAXPOS must be the model's ORIGINAL PRETRAIN length -- for this family, 4096. The default at YARNOps.cpp:57 is 1024 -- always set it explicitly. ext_32k.sh asserts the product.

CORRECTION 2026-09-15. This rule previously read "ORIG_MAXPOS must be the length the weights were ACTUALLY trained at". That is right for a single extension but WRONG for a staged ladder, where after stage one the two stop being the same number. Hold ORIG_MAXPOS at 4096 and raise SCALE instead: 4 -> 8 -> 16 for 16k -> 32k -> 64k.

Why: build_rope_cache is rebuilt FROM BASE THETA every run (it takes rope_theta, not the previous stage's cache), so SCALE always applies to the ORIGINAL frequencies and never composes with the previous stage's. Re-basing therefore freezes interpolation instead of deepening it:

stage  as-run (re-based)      interp   last tok -> virtual   ramp
16k    4 x 4096               theta/4        4,096          [14,32]
32k    2 x 16384              theta/2       16,384          [21,39]
64k    2 x 32768              theta/2       32,768          [24,42]

stage  corrected (orig=4096)  interp   last tok -> virtual   ramp
16k    4 x 4096               theta/4        4,096          [14,32]
32k    8 x 4096               theta/8        4,096          [14,32]
64k   16 x 4096               theta/16       4,096          [14,32]

Holding orig=4096 pins the far end of every window onto the 4096 range the base was actually pretrained on, and keeps the ramp fixed, because find_dim() takes (num_rot, L, head_dim, base) -- scale is NOT an argument, so the interpolated dimension set moves only when ORIG_MAXPOS moves. Re-basing instead puts the 64k model at virtual position 32,768, i.e. 8x beyond anything it ever saw, with YaRN doing almost none of the work it exists to do.

The 1.06B ladder (ext_32k.sh, b4_full_epoch.sh, ext64k_b5.sh, mix64k_h200.sh, mixcont_h100.sh) followed the OLD rule and is affected from the 32k stage on. This is NOT thought to be the cause of the retrieval collapse -- a wrong cache is a static defect, while the collapse was progressive and tracked cumulative b5 exposure across two lineages -- but it plausibly lowered the ceiling. The 680M ladder (p1/p2/p3_680m.sh) uses the corrected scheme.

Projected step time ~25-29 s (2x the 16k tokens/rank plus attention's quadratic term), so 750 steps ~ 5-6.5 h. Unmeasured; the probe prints the real number.

Why this run, in one paragraph

The document-length ladder is finished and it ended in a tie. Holding everything else fixed at a 16k window:

arm	training docs	4096	8192	16360
armA_c	flux, median 826	73.0	53.0	33.0
l1	b1 4-8k (fits)	75.0	48.0	30.0
l2	b2 8-16k (fits)	80.0	62.0	35.0
l3	b3 16-32k (TRUNCATED)	74.0	64.0	35.0

l3 - l2 = -6.0 / +2.0 / +0.0 pp, none resolvable (combined 95% bars +-11.5 to +-13.1). The haystack (b2-val) is in-distribution for l2 and out-of-distribution for l3, so the bias runs TOWARD l2 and l3 still only drew level. Conclusion: document length helps only while documents fit inside the window. Hence widen the window, then feed b3.

Gate the result on SHORT context, not perplexity

On 2026-09-02 a mismatched cache IMPROVED 16k (0 -> 14-17%) while COLLAPSING 4k from 74% to 16%. Perplexity at the new length looked fine. ext_32k.sh therefore evaluates 1024 / 4096 / 16360 / 32744 and a regression at 1k or 4k is a failure even if 32k improves. Compare against l2's row above.

(32744 is not arbitrary: PROMPT_LEN + MAX_NEW must pad up to <= CP_T with PAD=256, and 32744 + 24 = 32768 exactly.)

Traps already paid for

Never edit a running bash script. Bash reads incrementally from a byte offset; an insert shifts it and the interpreter resumes mid-line. This killed ladder_456.sh with syntax error near unexpected token 'do'.
pgrep -f <script>.sh matches the shell running it. Killed my own shell twice (exit 144). Exclude $$.
Verify all rank shards before trusting a checkpoint. A 3-of-4 set loads silently wrong, and a zero exit code is not proof -- bluscriptCP catches a failed save and only prints [WARN].
say() must write to stderr. On stdout it poisons $(...) capture; that fed a 3-line path into CP_INIT_FROM and aborted a run on rank 2.
Run-number collisions. run_number is the first free CP_BluScreipt_Training_logs/CP_Training_log<N>.csv. A collision with an existing checkpoint prefix let CP_CKPT_KEEP=1 prune a live shard.
CP_STABLE cannot equal CP_MAX_STEPS -- bluscriptCP.cpp:1006 aborts with "leaves no cosine decay tail". Leave at least a 2-step tail; the guard reads CP_WARMUP while get_lr uses CP_REWARMUP, so pin both.
CP_REWARMUP_ANCHOR=0 on every resume, or each resume re-anchors a fresh warmup ramp and the LR sawtooths (bluscriptCP.cpp:2105).

Concurrently, on the 2x H100 box

Long b3 arm at 16k: run 17, 4,000 steps, WSD with flat 3.0e-4 for steps 100-3899 then a 100-step tail. Tests whether more long-context STEPS flatten the slope -- the open question, since all arms sat at -18 to -29 pp/doubling while Phi-3.5-mini (3.8B) manages -3.6. Checkpointed every 25 steps, KEEP=2. An eval at step 1000 fires automatically, then the run resumes to 4000.

Not to do

b4/b5/b6 at a 16k window -- l3 already showed truncated documents add nothing. b4 only becomes useful after a 64k extension.
b6 in a matched ladder -- 57 M training tokens is 0.27x of one 400-step arm; it would cycle 3.7x through all 790 documents.
Chasing 128k. No 4B-class model has effective 128k: applying RULER's own threshold (Llama2-7B at 4k = 85.6), Phi-3.5-mini (3.8B, 128k claimed) has an effective context of 32k, and so does Llama-3.1-8B-instruct. 32k is the right target, and it is also where this corpus runs out (2.35 B tokens above 32k, total).