Upload docs/EXT32K_RESULTS_2026-09-09.md with huggingface_hub

docs/EXT32K_RESULTS_2026-09-09.md

@@ -0,0 +1,56 @@
+# 16k -> 32k YaRN extension on b4 -- 2026-09-09
+
+Parent: l2 (lenbucket_l2_blumodelcp_run2_step_400.ckpt), the best 16k arm.
+Data:   b4_32768_65536 -- min doc 32,768 == the window, so EVERY training
+        sample is one contiguous document and a 32k-spanning dependency is
+        possible. b3 (min 16,384) would have straddled an EOT in every window.
+YaRN:   SCALE=2 ORIG_MAXPOS=16384 -> 32768 exactly.
+Train:  750 steps, global batch 524,288 (B=1 x accum 16), WSD stable 550
+        + 100-step decay, 2x H200 cp=2, peak 108.8 GB of 143 GB.
+        Final loss 2.570224. 393 M tokens of 586 M available (no repetition).
+
+## niah_single, N=100, ONE haystack (b2 val)
+
+| context | l2 (16k model, 16k cache) | ext32k (32k model, 32k cache) | delta |
+|---|---|---|---|
+| 4,096 | 80.0 | 75.0 | -5.0 |
+| 8,192 | 62.0 | 60.0 | -2.0 |
+| 16,360 | 35.0 | 45.0 | +10.0 |
+| 32,744 | cannot run | **27.0** | new capability |
+
+95% Wilson at N=100 is +-9 to 10 pp, so the combined bar is +-11 to 13 pp:
+NONE of the three deltas is individually resolvable. The 32,744 row is not a
+delta -- the parent's RoPE cache is sized 16,384 and cannot process 32k at all.
+
+## KNOWN BUG in this run's raw summary.csv files
+
+ext32k_b4.sh line 128 passed YARN_ORIG=16384 YARN=2 to BOTH models, so the
+l2 column in ext32k_len_*/summary.csv is a MISMATCHED readout -- l2 was
+trained at scale 4 / orig 4096. Measured damage: 4k -4.0, 8k -26.0, 16k -15.0.
+The l2 numbers in the table above come from the matched H100 decision eval.
+DO NOT use the l2 column in the raw CSVs.
+
+### PROMPT_LEN=4096 (raw -- l2 column mismatched, see above)
+```
+task,ext32k,l2
+niah_single,75.0,76.0
+```
+
+### PROMPT_LEN=8192 (raw -- l2 column mismatched, see above)
+```
+task,ext32k,l2
+niah_single,60.0,36.0
+```
+
+### PROMPT_LEN=16360 (raw -- l2 column mismatched, see above)
+```
+task,ext32k,l2
+niah_single,45.0,20.0
+```
+
+### PROMPT_LEN=32744 (raw -- l2 column mismatched, see above)
+```
+task,ext32k,l2
+niah_single,27.0,11.0
+```
+