16k -> 32k YaRN extension on b4 -- 2026-09-09
Parent: l2 (lenbucket_l2_blumodelcp_run2_step_400.ckpt), the best 16k arm. Data: b4_32768_65536 -- min doc 32,768 == the window, so EVERY training sample is one contiguous document and a 32k-spanning dependency is possible. b3 (min 16,384) would have straddled an EOT in every window. YaRN: SCALE=2 ORIG_MAXPOS=16384 -> 32768 exactly. Train: 750 steps, global batch 524,288 (B=1 x accum 16), WSD stable 550 + 100-step decay, 2x H200 cp=2, peak 108.8 GB of 143 GB. Final loss 2.570224. 393 M tokens of 586 M available (no repetition).
niah_single, N=100, ONE haystack (b2 val)
| context | l2 (16k model, 16k cache) | ext32k (32k model, 32k cache) | delta |
|---|---|---|---|
| 4,096 | 80.0 | 75.0 | -5.0 |
| 8,192 | 62.0 | 60.0 | -2.0 |
| 16,360 | 35.0 | 45.0 | +10.0 |
| 32,744 | cannot run | 27.0 | new capability |
95% Wilson at N=100 is +-9 to 10 pp, so the combined bar is +-11 to 13 pp: NONE of the three deltas is individually resolvable. The 32,744 row is not a delta -- the parent's RoPE cache is sized 16,384 and cannot process 32k at all.
KNOWN BUG in this run's raw summary.csv files
ext32k_b4.sh line 128 passed YARN_ORIG=16384 YARN=2 to BOTH models, so the l2 column in ext32k_len_*/summary.csv is a MISMATCHED readout -- l2 was trained at scale 4 / orig 4096. Measured damage: 4k -4.0, 8k -26.0, 16k -15.0. The l2 numbers in the table above come from the matched H100 decision eval. DO NOT use the l2 column in the raw CSVs.
PROMPT_LEN=4096 (raw -- l2 column mismatched, see above)
task,ext32k,l2
niah_single,75.0,76.0
PROMPT_LEN=8192 (raw -- l2 column mismatched, see above)
task,ext32k,l2
niah_single,60.0,36.0
PROMPT_LEN=16360 (raw -- l2 column mismatched, see above)
task,ext32k,l2
niah_single,45.0,20.0
PROMPT_LEN=32744 (raw -- l2 column mismatched, see above)
task,ext32k,l2
niah_single,27.0,11.0