Upload docs/LENBUCKET_RESULTS_2026-09-08.md with huggingface_hub

docs/LENBUCKET_RESULTS_2026-09-08.md

@@ -0,0 +1,102 @@
+# Training document length vs 16k retrieval -- 2026-09-08
+
+All arms: 400 steps, global batch 524,288, from run7' (run 6 step 750),
+YARN_ORIG_MAXPOS=4096 matched, Muon+ZeRO-1 bf16, 4xH100.
+Only the training data differs. NIAH scored generatively (substring match),
+greedy, N=100 per cell, one shared haystack held out from every arm.
+
+| arm | training docs |
+|---|---|
+| armA_c | flux plain text, median 826 / p90 2,721 tok |
+| l1 | b1_4096_8192, docs in [4096, 8192) |
+| l2 | b2_8192_16384, docs in [8192, 16384) |
+
+## PROMPT_LEN=1024
+```
+task,armA_c,l1,l2
+niah_single,94.0,95.0,96.0
+val_backward,1.0,1.0,1.0
+```
+
+## PROMPT_LEN=2048
+```
+task,armA_c,l1,l2
+niah_single,94.0,96.0,94.0
+val_backward,2.0,2.0,1.0
+```
+
+## PROMPT_LEN=4096
+```
+task,armA_c,l1,l2
+niah_single,84.0,88.0,92.0
+val_backward,0.0,0.0,0.0
+```
+
+## PROMPT_LEN=8192
+```
+task,armA_c,l1,l2
+niah_single,62.0,67.0,78.0
+val_backward,2.0,1.0,2.0
+```
+
+## PROMPT_LEN=16360
+```
+task,armA_c,l1,l2
+niah_single,33.0,39.0,48.0
+val_backward,1.0,1.0,0.0
+```
+
+## Training losses
+```
+l1: loss: 2.458802
+l2: loss: 2.794149
+```
+
+NOTE: losses are NOT comparable across arms -- each is measured on its own
+data distribution. Longer documents are a different prediction problem, so
+a higher loss on b2 than on flux says nothing about model quality. Only the
+retrieval numbers compare.
+
+
+┌─────────┬────────────────────────┬────────────────┬─────────────────┐
+│ context │ armA_c (flux, med 826) │ l1 (docs 4-8k) │ l2 (docs 8-16k) │
+├─────────┼────────────────────────┼────────────────┼─────────────────┤
+│ 1k      │ 94.0 ±4.8              │ 95.0 ±4.5      │ 96.0 ±4.1       │
+├─────────┼────────────────────────┼────────────────┼─────────────────┤
+│ 2k      │ 94.0 ±4.8              │ 96.0 ±4.1      │ 94.0 ±4.8       │
+├─────────┼────────────────────────┼────────────────┼─────────────────┤
+│ 4k      │ 84.0 ±7.2              │ 88.0 ±6.4      │ 92.0 ±5.4       │
+├─────────┼────────────────────────┼────────────────┼─────────────────┤
+│ 8k      │ 62.0 ±9.3              │ 67.0 ±9.1      │ 78.0 ±8.0       │
+├─────────┼────────────────────────┼────────────────┼─────────────────┤
+│ 16360   │ 33.0 ±9.1              │ 39.0 ±9.4      │ 48.0 ±9.6       │
+└─────────┴────────────────────────┴────────────────┴─────────────────┘
+
+┌─────────┬───────────┬───────────┬─────────┬─────────────────┐
+│ context │ l1 − armA │ l2 − armA │ l2 − l1 │   resolvable    │
+├─────────┼───────────┼───────────┼─────────┼─────────────────┤
+│ 1k      │ +1.0      │ +2.0      │ +1.0    │ no              │
+├─────────┼───────────┼───────────┼─────────┼─────────────────┤
+│ 2k      │ +2.0      │ +0.0      │ −2.0    │ no              │
+├─────────┼───────────┼───────────┼─────────┼─────────────────┤
+│ 4k      │ +4.0      │ +8.0      │ +4.0    │ no (bar ±9.0)   │
+├─────────┼───────────┼───────────┼─────────┼─────────────────┤
+│ 8k      │ +5.0      │ +16.0     │ +11.0   │ YES (bar ±12.3) │
+├─────────┼───────────┼───────────┼─────────┼─────────────────┤
+│ 16360   │ +6.0      │ +15.0     │ +9.0    │ YES (bar ±13.2) │
+└─────────┴───────────┴─��─────────┴─────────┴─────────────────┘
+
+
+The effect is monotone in training document length and grows with eval context. Nothing distinguishable at 1-2k, +8pp at 4k, then +16pp at 8k and +15pp at 16k. Ordering is l2 > l1 > armA_c at every length where there's signal — three independent lengths agreeing on the same ordering, with the two long ones individually resolvable.
+
+Your hypothesis is supported. The flux corpus's short documents (median 826, p90 2,721) were a genuine ceiling on long-context retrieval, not just the YaRN cache. Training on documents that actually span the window recovers a large chunk: at the full 16,384 window, 33% → 48%, a +45% relative improvement from changing nothing but document length.
+
+Backward retrieval is untouched — val_backward sits at 0-2% everywhere against ~2.9% chance, identical across all three arms. Document length does nothing for value→key retrieval. That failure mode is independent of both the cache fix and the data fix.
+
+Two caveats that matter for how you read this
+
+The control's absolute numbers moved. armA_c scored 25% at 16k on the flux haystack; here it's 33%. Same checkpoint — the haystack changed (b3's 16-32k documents, held out from all arms). That's exactly why I re-scored the control rather than reusing its published numbers; all three columns above share one haystack, so they compare, but they don't compare to the 2026-09-03 table.
+
+Training losses do not compare across arms (l1 2.459, l2 2.794). Each is measured on its own distribution, and predicting inside 8-16k documents is a harder problem than inside 4-8k ones. l2 has the worst loss and the best retrieval — the same divergence as the cache experiment. Only the retrieval numbers compare.
+
+Results are in lenbucket_logs/RESULTS.md.