# 16k -> 32k YaRN extension on b4 -- 2026-09-09

Parent: l2 (lenbucket_l2_blumodelcp_run2_step_400.ckpt), the best 16k arm.
Data:   b4_32768_65536 -- min doc 32,768 == the window, so EVERY training
        sample is one contiguous document and a 32k-spanning dependency is
        possible. b3 (min 16,384) would have straddled an EOT in every window.
YaRN:   SCALE=2 ORIG_MAXPOS=16384 -> 32768 exactly.
Train:  750 steps, global batch 524,288 (B=1 x accum 16), WSD stable 550
        + 100-step decay, 2x H200 cp=2, peak 108.8 GB of 143 GB.
        Final loss 2.570224. 393 M tokens of 586 M available (no repetition).

## niah_single, N=100, ONE haystack (b2 val)

| context | l2 (16k model, 16k cache) | ext32k (32k model, 32k cache) | delta |
|---|---|---|---|
| 4,096 | 80.0 | 75.0 | -5.0 |
| 8,192 | 62.0 | 60.0 | -2.0 |
| 16,360 | 35.0 | 45.0 | +10.0 |
| 32,744 | cannot run | **27.0** | new capability |

95% Wilson at N=100 is +-9 to 10 pp, so the combined bar is +-11 to 13 pp:
NONE of the three deltas is individually resolvable. The 32,744 row is not a
delta -- the parent's RoPE cache is sized 16,384 and cannot process 32k at all.

## KNOWN BUG in this run's raw summary.csv files

ext32k_b4.sh line 128 passed YARN_ORIG=16384 YARN=2 to BOTH models, so the
l2 column in ext32k_len_*/summary.csv is a MISMATCHED readout -- l2 was
trained at scale 4 / orig 4096. Measured damage: 4k -4.0, 8k -26.0, 16k -15.0.
The l2 numbers in the table above come from the matched H100 decision eval.
DO NOT use the l2 column in the raw CSVs.

### PROMPT_LEN=4096 (raw -- l2 column mismatched, see above)
```
task,ext32k,l2
niah_single,75.0,76.0
```

### PROMPT_LEN=8192 (raw -- l2 column mismatched, see above)
```
task,ext32k,l2
niah_single,60.0,36.0
```

### PROMPT_LEN=16360 (raw -- l2 column mismatched, see above)
```
task,ext32k,l2
niah_single,45.0,20.0
```

### PROMPT_LEN=32744 (raw -- l2 column mismatched, see above)
```
task,ext32k,l2
niah_single,27.0,11.0
```