# Handoff: 16k -> 32k YaRN extension on the 2x H200 Written 2026-09-09 from the 2x H100 box. Everything below is measured on this box unless marked as a projection. ## What the 2x H200 CAN and CANNOT do `cp` cannot exceed the GPU count, so on 2 GPUs `tokens/rank = T/2`. | target | tok/rank | total/GPU | 2x H200 (143 GB) | |---|---|---|---| | 16k | 8,192 | 70.7 GB (measured) | comfortable | | **32k** | 16,384 | **130.8 GB (projected)** | **91.4% -- tight, probe first** | | 64k | 32,768 | 250.9 GB | 175% -- impossible | - **32k: run it.** `ext_32k.sh` starts with a 3-step probe and aborts on OOM. - **64k: needs 4+ GPUs.** 4x H200 at cp=4 = 127.6 GB/GPU fits. Nothing to tune on 2 GPUs: `CP_B` is already 1, and there is no activation-recompute flag in bluscriptCP.cpp (checked: no `CP_RECOMPUTE`, no gradient checkpointing). - 64k **evaluation** is fine on 2 GPUs -- generation stores no activations for backward, so the training memory figure does not apply. Measured activation cost is 7.51 MB/token/rank. A first-principles estimate for this architecture is 1.3-2.5 MB, so actual is 3-6x higher than it needs to be (the config keeps fp32 norms/residual/logits across 36 layers). If that gap is real, a memory pass could make 32k comfortable and 64k possible on 2 GPUs -- but that is a C++ change, not a flag, and it is unprofiled. ## Setup on the new box (nothing is pre-installed) ```bash cd /root/Adhi/BluTrain/dist/Context_Parallelism # 1. build -- BOTH targets, libtensor first make libtensor SM_ARCH=90 -j$(nproc) make CP_FUSED_ROPE=1 CP_ATTN_SPLIT=1 SM_ARCH=90 bluscript-cp -j$(nproc) # 2. tiktoken -- longeval_gen.py needs it. WITHOUT IT training runs fine and the # eval silently produces "no prompts" / "no scores yet". This has bitten twice. /root/Adhi/BluTrain/envv/bin/pip install tiktoken # 3. hf auth (the dataset is public now, but the model repo push needs a token) hf auth login # 4. parent checkpoint: l2, the best 16k arm. cp=2 needs it on BOTH ranks. python - <<'PY' from huggingface_hub import hf_hub_download import shutil, os p = hf_hub_download("unparallelled/BluScriptCP", "checkpoints/lenbucket_l2_blumodelcp_run2_step_400.ckpt") for r in (0, 1): d = f"checkpoints_bluscriptcp/rank_{r}/lenbucket_l2_blumodelcp_run2_step_400.ckpt" os.makedirs(os.path.dirname(d), exist_ok=True) if not os.path.exists(d): shutil.copyfile(p, d) PY # 5. b3 data (4 train shards + val ~ 930 MB) and the b2-val haystack hf download unparallelled/BluScriptCP-data --repo-type dataset \ --include "cp_data_by_length/b3_16384_32768/train_shard_0000[0-3].bin" \ --include "cp_data_by_length/b3_16384_32768/val_shard_00000.bin" \ --include "cp_data_by_length/b2_8192_16384/val_shard_00000.bin" \ --local-dir ./dl mkdir -p lendata/b3_16384_32768 lendata/haystack cp dl/cp_data_by_length/b3_16384_32768/*.bin lendata/b3_16384_32768/ cp dl/cp_data_by_length/b2_8192_16384/val_shard_00000.bin lendata/haystack/b2val_haystack.bin # VERIFY -- `hf download` exits 0 and prints a path even when --include matches # NOTHING. Always ls, never trust the exit code. ls -la lendata/b3_16384_32768/ lendata/haystack/ # 6. run it ./ext_32k.sh ``` ## The run ``` CP_T=32768 YARN_SCALE=2 YARN_ORIG_MAXPOS=16384 parent: lenbucket_l2_blumodelcp_run2_step_400.ckpt data: b3_16384_32768 (docs 16-32k -- these now FIT inside the window) 750 steps x 1,048,576 = 786 M tokens, matching run7's original 4k->16k extension ``` **The YaRN invariant, which cost 27pp when it was wrong:** `YARN_SCALE * YARN_ORIG_MAXPOS` must equal `CP_T` exactly, and `ORIG_MAXPOS` must be the model's ORIGINAL PRETRAIN length -- for this family, 4096. The default at `YARNOps.cpp:57` is **1024** -- always set it explicitly. `ext_32k.sh` asserts the product. **CORRECTION 2026-09-15.** This rule previously read "`ORIG_MAXPOS` must be the length the weights were ACTUALLY trained at". That is right for a single extension but WRONG for a staged ladder, where after stage one the two stop being the same number. Hold `ORIG_MAXPOS` at 4096 and raise `SCALE` instead: 4 -> 8 -> 16 for 16k -> 32k -> 64k. Why: `build_rope_cache` is rebuilt FROM BASE THETA every run (it takes `rope_theta`, not the previous stage's cache), so `SCALE` always applies to the ORIGINAL frequencies and never composes with the previous stage's. Re-basing therefore freezes interpolation instead of deepening it: stage as-run (re-based) interp last tok -> virtual ramp 16k 4 x 4096 theta/4 4,096 [14,32] 32k 2 x 16384 theta/2 16,384 [21,39] 64k 2 x 32768 theta/2 32,768 [24,42] stage corrected (orig=4096) interp last tok -> virtual ramp 16k 4 x 4096 theta/4 4,096 [14,32] 32k 8 x 4096 theta/8 4,096 [14,32] 64k 16 x 4096 theta/16 4,096 [14,32] Holding orig=4096 pins the far end of every window onto the 4096 range the base was actually pretrained on, and keeps the ramp fixed, because `find_dim()` takes (num_rot, L, head_dim, base) -- `scale` is NOT an argument, so the interpolated dimension set moves only when `ORIG_MAXPOS` moves. Re-basing instead puts the 64k model at virtual position 32,768, i.e. 8x beyond anything it ever saw, with YaRN doing almost none of the work it exists to do. The 1.06B ladder (ext_32k.sh, b4_full_epoch.sh, ext64k_b5.sh, mix64k_h200.sh, mixcont_h100.sh) followed the OLD rule and is affected from the 32k stage on. This is NOT thought to be the cause of the retrieval collapse -- a wrong cache is a static defect, while the collapse was progressive and tracked cumulative b5 exposure across two lineages -- but it plausibly lowered the ceiling. The 680M ladder (p1/p2/p3_680m.sh) uses the corrected scheme. Projected step time ~25-29 s (2x the 16k tokens/rank plus attention's quadratic term), so **750 steps ~ 5-6.5 h**. Unmeasured; the probe prints the real number. ## Why this run, in one paragraph The document-length ladder is finished and it ended in a tie. Holding everything else fixed at a 16k window: | arm | training docs | 4096 | 8192 | 16360 | |---|---|---|---|---| | armA_c | flux, median 826 | 73.0 | 53.0 | 33.0 | | l1 | b1 4-8k (fits) | 75.0 | 48.0 | 30.0 | | l2 | b2 8-16k (fits) | **80.0** | 62.0 | **35.0** | | l3 | b3 16-32k (TRUNCATED) | 74.0 | **64.0** | **35.0** | l3 - l2 = -6.0 / +2.0 / +0.0 pp, none resolvable (combined 95% bars +-11.5 to +-13.1). The haystack (b2-val) is in-distribution for l2 and out-of-distribution for l3, so the bias runs TOWARD l2 and l3 still only drew level. Conclusion: **document length helps only while documents fit inside the window.** Hence widen the window, then feed b3. ## Gate the result on SHORT context, not perplexity On 2026-09-02 a mismatched cache IMPROVED 16k (0 -> 14-17%) while COLLAPSING 4k from 74% to 16%. Perplexity at the new length looked fine. `ext_32k.sh` therefore evaluates 1024 / 4096 / 16360 / 32744 and **a regression at 1k or 4k is a failure even if 32k improves.** Compare against l2's row above. (32744 is not arbitrary: `PROMPT_LEN + MAX_NEW` must pad up to <= `CP_T` with PAD=256, and 32744 + 24 = 32768 exactly.) ## Traps already paid for 1. **Never edit a running bash script.** Bash reads incrementally from a byte offset; an insert shifts it and the interpreter resumes mid-line. This killed `ladder_456.sh` with `syntax error near unexpected token 'do'`. 2. **`pgrep -f