BluScriptCP / logs /lenbucket /RESULTS.md
unparallelled's picture
Upload folder using huggingface_hub
b50ee6a verified
|
Raw History Blame Contribute Delete
6.87 kB

Training document length vs 16k retrieval -- 2026-09-08

All arms: 400 steps, global batch 524,288, from run7' (run 6 step 750), YARN_ORIG_MAXPOS=4096 matched, Muon+ZeRO-1 bf16, 4xH100. Only the training data differs. NIAH scored generatively (substring match), greedy, N=100 per cell, one shared haystack held out from every arm.

arm training docs
armA_c flux plain text, median 826 / p90 2,721 tok
l1 b1_4096_8192, docs in [4096, 8192)
l2 b2_8192_16384, docs in [8192, 16384)

PROMPT_LEN=1024

task,armA_c,l1,l2
niah_single,94.0,95.0,96.0
val_backward,1.0,1.0,1.0

PROMPT_LEN=2048

task,armA_c,l1,l2
niah_single,94.0,96.0,94.0
val_backward,2.0,2.0,1.0

PROMPT_LEN=4096

task,armA_c,l1,l2
niah_single,84.0,88.0,92.0
val_backward,0.0,0.0,0.0

PROMPT_LEN=8192

task,armA_c,l1,l2
niah_single,62.0,67.0,78.0
val_backward,2.0,1.0,2.0

PROMPT_LEN=16360

task,armA_c,l1,l2
niah_single,33.0,39.0,48.0
val_backward,1.0,1.0,0.0

Training losses

l1: loss: 2.458802
l2: loss: 2.794149

NOTE: losses are NOT comparable across arms -- each is measured on its own data distribution. Longer documents are a different prediction problem, so a higher loss on b2 than on flux says nothing about model quality. Only the retrieval numbers compare.

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ context β”‚ armA_c (flux, med 826) β”‚ l1 (docs 4-8k) β”‚ l2 (docs 8-16k) β”‚ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ β”‚ 1k β”‚ 94.0 Β±4.8 β”‚ 95.0 Β±4.5 β”‚ 96.0 Β±4.1 β”‚ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ β”‚ 2k β”‚ 94.0 Β±4.8 β”‚ 96.0 Β±4.1 β”‚ 94.0 Β±4.8 β”‚ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ β”‚ 4k β”‚ 84.0 Β±7.2 β”‚ 88.0 Β±6.4 β”‚ 92.0 Β±5.4 β”‚ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ β”‚ 8k β”‚ 62.0 Β±9.3 β”‚ 67.0 Β±9.1 β”‚ 78.0 Β±8.0 β”‚ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ β”‚ 16360 β”‚ 33.0 Β±9.1 β”‚ 39.0 Β±9.4 β”‚ 48.0 Β±9.6 β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ context β”‚ l1 βˆ’ armA β”‚ l2 βˆ’ armA β”‚ l2 βˆ’ l1 β”‚ resolvable β”‚ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ β”‚ 1k β”‚ +1.0 β”‚ +2.0 β”‚ +1.0 β”‚ no β”‚ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ β”‚ 2k β”‚ +2.0 β”‚ +0.0 β”‚ βˆ’2.0 β”‚ no β”‚ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ β”‚ 4k β”‚ +4.0 β”‚ +8.0 β”‚ +4.0 β”‚ no (bar Β±9.0) β”‚ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ β”‚ 8k β”‚ +5.0 β”‚ +16.0 β”‚ +11.0 β”‚ YES (bar Β±12.3) β”‚ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ β”‚ 16360 β”‚ +6.0 β”‚ +15.0 β”‚ +9.0 β”‚ YES (bar Β±13.2) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

The effect is monotone in training document length and grows with eval context. Nothing distinguishable at 1-2k, +8pp at 4k, then +16pp at 8k and +15pp at 16k. Ordering is l2 > l1 > armA_c at every length where there's signal β€” three independent lengths agreeing on the same ordering, with the two long ones individually resolvable.

Your hypothesis is supported. The flux corpus's short documents (median 826, p90 2,721) were a genuine ceiling on long-context retrieval, not just the YaRN cache. Training on documents that actually span the window recovers a large chunk: at the full 16,384 window, 33% β†’ 48%, a +45% relative improvement from changing nothing but document length.

Backward retrieval is untouched — val_backward sits at 0-2% everywhere against ~2.9% chance, identical across all three arms. Document length does nothing for value→key retrieval. That failure mode is independent of both the cache fix and the data fix.

Two caveats that matter for how you read this

The control's absolute numbers moved. armA_c scored 25% at 16k on the flux haystack; here it's 33%. Same checkpoint β€” the haystack changed (b3's 16-32k documents, held out from all arms). That's exactly why I re-scored the control rather than reusing its published numbers; all three columns above share one haystack, so they compare, but they don't compare to the 2026-09-03 table.

Training losses do not compare across arms (l1 2.459, l2 2.794). Each is measured on its own distribution, and predicting inside 8-16k documents is a harder problem than inside 4-8k ones. l2 has the worst loss and the best retrieval β€” the same divergence as the cache experiment. Only the retrieval numbers compare.

Results are in lenbucket_logs/RESULTS.md.