Handoff: 16k -> 32k YaRN extension on the 2x H200

Written 2026-09-09 from the 2x H100 box. Everything below is measured on this box unless marked as a projection.

What the 2x H200 CAN and CANNOT do

cp cannot exceed the GPU count, so on 2 GPUs tokens/rank = T/2.

target	tok/rank	total/GPU	2x H200 (143 GB)
16k	8,192	70.7 GB (measured)	comfortable
32k	16,384	130.8 GB (projected)	91.4% -- tight, probe first
64k	32,768	250.9 GB	175% -- impossible

32k: run it. ext_32k.sh starts with a 3-step probe and aborts on OOM.
64k: needs 4+ GPUs. 4x H200 at cp=4 = 127.6 GB/GPU fits. Nothing to tune on 2 GPUs: CP_B is already 1, and there is no activation-recompute flag in bluscriptCP.cpp (checked: no CP_RECOMPUTE, no gradient checkpointing).
64k evaluation is fine on 2 GPUs -- generation stores no activations for backward, so the training memory figure does not apply.

Measured activation cost is 7.51 MB/token/rank. A first-principles estimate for this architecture is 1.3-2.5 MB, so actual is 3-6x higher than it needs to be (the config keeps fp32 norms/residual/logits across 36 layers). If that gap is real, a memory pass could make 32k comfortable and 64k possible on 2 GPUs -- but that is a C++ change, not a flag, and it is unprofiled.

Setup on the new box (nothing is pre-installed)

cd /root/Adhi/BluTrain/dist/Context_Parallelism

# 1. build -- BOTH targets, libtensor first
make libtensor SM_ARCH=90 -j$(nproc)
make CP_FUSED_ROPE=1 CP_ATTN_SPLIT=1 SM_ARCH=90 bluscript-cp -j$(nproc)

# 2. tiktoken -- longeval_gen.py needs it. WITHOUT IT training runs fine and the
#    eval silently produces "no prompts" / "no scores yet". This has bitten twice.
/root/Adhi/BluTrain/envv/bin/pip install tiktoken

# 3. hf auth (the dataset is public now, but the model repo push needs a token)
hf auth login

# 4. parent checkpoint: l2, the best 16k arm. cp=2 needs it on BOTH ranks.
python - <<'PY'
from huggingface_hub import hf_hub_download
import shutil, os
p = hf_hub_download("unparallelled/BluScriptCP",
                    "checkpoints/lenbucket_l2_blumodelcp_run2_step_400.ckpt")
for r in (0, 1):
    d = f"checkpoints_bluscriptcp/rank_{r}/lenbucket_l2_blumodelcp_run2_step_400.ckpt"
    os.makedirs(os.path.dirname(d), exist_ok=True)
    if not os.path.exists(d):
        shutil.copyfile(p, d)
PY

# 5. b3 data (4 train shards + val ~ 930 MB) and the b2-val haystack
hf download unparallelled/BluScriptCP-data --repo-type dataset \
  --include "cp_data_by_length/b3_16384_32768/train_shard_0000[0-3].bin" \
  --include "cp_data_by_length/b3_16384_32768/val_shard_00000.bin" \
  --include "cp_data_by_length/b2_8192_16384/val_shard_00000.bin" \
  --local-dir ./dl
mkdir -p lendata/b3_16384_32768 lendata/haystack
cp dl/cp_data_by_length/b3_16384_32768/*.bin lendata/b3_16384_32768/
cp dl/cp_data_by_length/b2_8192_16384/val_shard_00000.bin lendata/haystack/b2val_haystack.bin

# VERIFY -- `hf download` exits 0 and prints a path even when --include matches
# NOTHING. Always ls, never trust the exit code.
ls -la lendata/b3_16384_32768/ lendata/haystack/

# 6. run it
./ext_32k.sh

The run

CP_T=32768   YARN_SCALE=2   YARN_ORIG_MAXPOS=16384
parent: lenbucket_l2_blumodelcp_run2_step_400.ckpt
data:   b3_16384_32768   (docs 16-32k -- these now FIT inside the window)
750 steps x 1,048,576 = 786 M tokens, matching run7's original 4k->16k extension

The YaRN invariant, which cost 27pp when it was wrong: YARN_SCALE * YARN_ORIG_MAXPOS must equal CP_T exactly, and ORIG_MAXPOS must be the length the weights were ACTUALLY trained at. Here 2 * 16384 = 32768. The default at YARNOps.cpp:57 is 1024 -- always set it explicitly. ext_32k.sh asserts it.

Projected step time ~25-29 s (2x the 16k tokens/rank plus attention's quadratic term), so 750 steps ~ 5-6.5 h. Unmeasured; the probe prints the real number.

Why this run, in one paragraph

The document-length ladder is finished and it ended in a tie. Holding everything else fixed at a 16k window:

arm	training docs	4096	8192	16360
armA_c	flux, median 826	73.0	53.0	33.0
l1	b1 4-8k (fits)	75.0	48.0	30.0
l2	b2 8-16k (fits)	80.0	62.0	35.0
l3	b3 16-32k (TRUNCATED)	74.0	64.0	35.0

l3 - l2 = -6.0 / +2.0 / +0.0 pp, none resolvable (combined 95% bars +-11.5 to +-13.1). The haystack (b2-val) is in-distribution for l2 and out-of-distribution for l3, so the bias runs TOWARD l2 and l3 still only drew level. Conclusion: document length helps only while documents fit inside the window. Hence widen the window, then feed b3.

Gate the result on SHORT context, not perplexity

On 2026-09-02 a mismatched cache IMPROVED 16k (0 -> 14-17%) while COLLAPSING 4k from 74% to 16%. Perplexity at the new length looked fine. ext_32k.sh therefore evaluates 1024 / 4096 / 16360 / 32744 and a regression at 1k or 4k is a failure even if 32k improves. Compare against l2's row above.

(32744 is not arbitrary: PROMPT_LEN + MAX_NEW must pad up to <= CP_T with PAD=256, and 32744 + 24 = 32768 exactly.)

Traps already paid for

Never edit a running bash script. Bash reads incrementally from a byte offset; an insert shifts it and the interpreter resumes mid-line. This killed ladder_456.sh with syntax error near unexpected token 'do'.
pgrep -f <script>.sh matches the shell running it. Killed my own shell twice (exit 144). Exclude $$.
Verify all rank shards before trusting a checkpoint. A 3-of-4 set loads silently wrong, and a zero exit code is not proof -- bluscriptCP catches a failed save and only prints [WARN].
say() must write to stderr. On stdout it poisons $(...) capture; that fed a 3-line path into CP_INIT_FROM and aborted a run on rank 2.
Run-number collisions. run_number is the first free CP_BluScreipt_Training_logs/CP_Training_log<N>.csv. A collision with an existing checkpoint prefix let CP_CKPT_KEEP=1 prune a live shard.
CP_STABLE cannot equal CP_MAX_STEPS -- bluscriptCP.cpp:1006 aborts with "leaves no cosine decay tail". Leave at least a 2-step tail; the guard reads CP_WARMUP while get_lr uses CP_REWARMUP, so pin both.
CP_REWARMUP_ANCHOR=0 on every resume, or each resume re-anchors a fresh warmup ramp and the LR sawtooths (bluscriptCP.cpp:2105).

Concurrently, on the 2x H100 box

Long b3 arm at 16k: run 17, 4,000 steps, WSD with flat 3.0e-4 for steps 100-3899 then a 100-step tail. Tests whether more long-context STEPS flatten the slope -- the open question, since all arms sat at -18 to -29 pp/doubling while Phi-3.5-mini (3.8B) manages -3.6. Checkpointed every 25 steps, KEEP=2. An eval at step 1000 fires automatically, then the run resumes to 4000.

Not to do

b4/b5/b6 at a 16k window -- l3 already showed truncated documents add nothing. b4 only becomes useful after a 64k extension.
b6 in a matched ladder -- 57 M training tokens is 0.27x of one 400-step arm; it would cycle 3.7x through all 790 documents.
Chasing 128k. No 4B-class model has effective 128k: applying RULER's own threshold (Llama2-7B at 4k = 85.6), Phi-3.5-mini (3.8B, 128k claimed) has an effective context of 32k, and so does Llama-3.1-8B-instruct. 32k is the right target, and it is also where this corpus runs out (2.35 B tokens above 32k, total).