# Training document length vs 16k retrieval -- 2026-09-08 All arms: 400 steps, global batch 524,288, from run7' (run 6 step 750), YARN_ORIG_MAXPOS=4096 matched, Muon+ZeRO-1 bf16, 4xH100. Only the training data differs. NIAH scored generatively (substring match), greedy, N=100 per cell, one shared haystack held out from every arm. | arm | training docs | |---|---| | armA_c | flux plain text, median 826 / p90 2,721 tok | | l1 | b1_4096_8192, docs in [4096, 8192) | | l2 | b2_8192_16384, docs in [8192, 16384) | ## PROMPT_LEN=1024 ``` task,armA_c,l1,l2 niah_single,94.0,95.0,96.0 val_backward,1.0,1.0,1.0 ``` ## PROMPT_LEN=2048 ``` task,armA_c,l1,l2 niah_single,94.0,96.0,94.0 val_backward,2.0,2.0,1.0 ``` ## PROMPT_LEN=4096 ``` task,armA_c,l1,l2 niah_single,84.0,88.0,92.0 val_backward,0.0,0.0,0.0 ``` ## PROMPT_LEN=8192 ``` task,armA_c,l1,l2 niah_single,62.0,67.0,78.0 val_backward,2.0,1.0,2.0 ``` ## PROMPT_LEN=16360 ``` task,armA_c,l1,l2 niah_single,33.0,39.0,48.0 val_backward,1.0,1.0,0.0 ``` ## Training losses ``` l1: loss: 2.458802 l2: loss: 2.794149 ``` NOTE: losses are NOT comparable across arms -- each is measured on its own data distribution. Longer documents are a different prediction problem, so a higher loss on b2 than on flux says nothing about model quality. Only the retrieval numbers compare. ┌─────────┬────────────────────────┬────────────────┬─────────────────┐ │ context │ armA_c (flux, med 826) │ l1 (docs 4-8k) │ l2 (docs 8-16k) │ ├─────────┼────────────────────────┼────────────────┼─────────────────┤ │ 1k │ 94.0 ±4.8 │ 95.0 ±4.5 │ 96.0 ±4.1 │ ├─────────┼────────────────────────┼────────────────┼─────────────────┤ │ 2k │ 94.0 ±4.8 │ 96.0 ±4.1 │ 94.0 ±4.8 │ ├─────────┼────────────────────────┼────────────────┼─────────────────┤ │ 4k │ 84.0 ±7.2 │ 88.0 ±6.4 │ 92.0 ±5.4 │ ├─────────┼────────────────────────┼────────────────┼─────────────────┤ │ 8k │ 62.0 ±9.3 │ 67.0 ±9.1 │ 78.0 ±8.0 │ ├─────────┼────────────────────────┼────────────────┼─────────────────┤ │ 16360 │ 33.0 ±9.1 │ 39.0 ±9.4 │ 48.0 ±9.6 │ └─────────┴────────────────────────┴────────────────┴─────────────────┘ ┌─────────┬───────────┬───────────┬─────────┬─────────────────┐ │ context │ l1 − armA │ l2 − armA │ l2 − l1 │ resolvable │ ├─────────┼───────────┼───────────┼─────────┼─────────────────┤ │ 1k │ +1.0 │ +2.0 │ +1.0 │ no │ ├─────────┼───────────┼───────────┼─────────┼─────────────────┤ │ 2k │ +2.0 │ +0.0 │ −2.0 │ no │ ├─────────┼───────────┼───────────┼─────────┼─────────────────┤ │ 4k │ +4.0 │ +8.0 │ +4.0 │ no (bar ±9.0) │ ├─────────┼───────────┼───────────┼─────────┼─────────────────┤ │ 8k │ +5.0 │ +16.0 │ +11.0 │ YES (bar ±12.3) │ ├─────────┼───────────┼───────────┼─────────┼─────────────────┤ │ 16360 │ +6.0 │ +15.0 │ +9.0 │ YES (bar ±13.2) │ └─────────┴───────────┴───────────┴─────────┴─────────────────┘ The effect is monotone in training document length and grows with eval context. Nothing distinguishable at 1-2k, +8pp at 4k, then +16pp at 8k and +15pp at 16k. Ordering is l2 > l1 > armA_c at every length where there's signal — three independent lengths agreeing on the same ordering, with the two long ones individually resolvable. Your hypothesis is supported. The flux corpus's short documents (median 826, p90 2,721) were a genuine ceiling on long-context retrieval, not just the YaRN cache. Training on documents that actually span the window recovers a large chunk: at the full 16,384 window, 33% → 48%, a +45% relative improvement from changing nothing but document length. Backward retrieval is untouched — val_backward sits at 0-2% everywhere against ~2.9% chance, identical across all three arms. Document length does nothing for value→key retrieval. That failure mode is independent of both the cache fix and the data fix. Two caveats that matter for how you read this The control's absolute numbers moved. armA_c scored 25% at 16k on the flux haystack; here it's 33%. Same checkpoint — the haystack changed (b3's 16-32k documents, held out from all arms). That's exactly why I re-scored the control rather than reusing its published numbers; all three columns above share one haystack, so they compare, but they don't compare to the 2026-09-03 table. Training losses do not compare across arms (l1 2.459, l2 2.794). Each is measured on its own distribution, and predicting inside 8-16k documents is a harder problem than inside 4-8k ones. l2 has the worst loss and the best retrieval — the same divergence as the cache experiment. Only the retrieval numbers compare. Results are in lenbucket_logs/RESULTS.md.