| # 16k -> 32k YaRN extension on b4 -- 2026-09-09 |
|
|
| Parent: l2 (lenbucket_l2_blumodelcp_run2_step_400.ckpt), the best 16k arm. |
| Data: b4_32768_65536 -- min doc 32,768 == the window, so EVERY training |
| sample is one contiguous document and a 32k-spanning dependency is |
| possible. b3 (min 16,384) would have straddled an EOT in every window. |
| YaRN: SCALE=2 ORIG_MAXPOS=16384 -> 32768 exactly. |
| Train: 750 steps, global batch 524,288 (B=1 x accum 16), WSD stable 550 |
| + 100-step decay, 2x H200 cp=2, peak 108.8 GB of 143 GB. |
| Final loss 2.570224. 393 M tokens of 586 M available (no repetition). |
| |
| ## niah_single, N=100, ONE haystack (b2 val) |
| |
| | context | l2 (16k model, 16k cache) | ext32k (32k model, 32k cache) | delta | |
| |---|---|---|---| |
| | 4,096 | 80.0 | 75.0 | -5.0 | |
| | 8,192 | 62.0 | 60.0 | -2.0 | |
| | 16,360 | 35.0 | 45.0 | +10.0 | |
| | 32,744 | cannot run | **27.0** | new capability | |
| |
| 95% Wilson at N=100 is +-9 to 10 pp, so the combined bar is +-11 to 13 pp: |
| NONE of the three deltas is individually resolvable. The 32,744 row is not a |
| delta -- the parent's RoPE cache is sized 16,384 and cannot process 32k at all. |
| |
| ## KNOWN BUG in this run's raw summary.csv files |
| |
| ext32k_b4.sh line 128 passed YARN_ORIG=16384 YARN=2 to BOTH models, so the |
| l2 column in ext32k_len_*/summary.csv is a MISMATCHED readout -- l2 was |
| trained at scale 4 / orig 4096. Measured damage: 4k -4.0, 8k -26.0, 16k -15.0. |
| The l2 numbers in the table above come from the matched H100 decision eval. |
| DO NOT use the l2 column in the raw CSVs. |
| |
| ### PROMPT_LEN=4096 (raw -- l2 column mismatched, see above) |
| ``` |
| task,ext32k,l2
|
| niah_single,75.0,76.0
|
| ``` |
|
|
| ### PROMPT_LEN=8192 (raw -- l2 column mismatched, see above) |
| ``` |
| task,ext32k,l2
|
| niah_single,60.0,36.0
|
| ``` |
| |
| ### PROMPT_LEN=16360 (raw -- l2 column mismatched, see above) |
| ``` |
| task,ext32k,l2
|
| niah_single,45.0,20.0
|
| ``` |
| |
| ### PROMPT_LEN=32744 (raw -- l2 column mismatched, see above) |
| ``` |
| task,ext32k,l2
|
| niah_single,27.0,11.0
|
| ``` |
|
|
|
|
