|
Download logs/lenbucket/RESULTS.md from unparallelled/BluScriptCP: direct link, hf CLI and curl.
- Browser
- Download file 6.87 kB
-
https://huggingface.co/unparallelled/BluScriptCP/resolve/main/logs/lenbucket/RESULTS.md
- Command line
-
hf download hf://unparallelled/BluScriptCP/logs/lenbucket/RESULTS.md
-
curl -L -o RESULTS.md https://huggingface.co/unparallelled/BluScriptCP/resolve/main/logs/lenbucket/RESULTS.md
6.87 kB
| # Training document length vs 16k retrieval -- 2026-09-08 | |
| All arms: 400 steps, global batch 524,288, from run7' (run 6 step 750), | |
| YARN_ORIG_MAXPOS=4096 matched, Muon+ZeRO-1 bf16, 4xH100. | |
| Only the training data differs. NIAH scored generatively (substring match), | |
| greedy, N=100 per cell, one shared haystack held out from every arm. | |
| | arm | training docs | | |
| |---|---| | |
| | armA_c | flux plain text, median 826 / p90 2,721 tok | | |
| | l1 | b1_4096_8192, docs in [4096, 8192) | | |
| | l2 | b2_8192_16384, docs in [8192, 16384) | | |
| ## PROMPT_LEN=1024 | |
| ``` | |
| task,armA_c,l1,l2 | |
| niah_single,94.0,95.0,96.0 | |
| val_backward,1.0,1.0,1.0 | |
| ``` | |
| ## PROMPT_LEN=2048 | |
| ``` | |
| task,armA_c,l1,l2 | |
| niah_single,94.0,96.0,94.0 | |
| val_backward,2.0,2.0,1.0 | |
| ``` | |
| ## PROMPT_LEN=4096 | |
| ``` | |
| task,armA_c,l1,l2 | |
| niah_single,84.0,88.0,92.0 | |
| val_backward,0.0,0.0,0.0 | |
| ``` | |
| ## PROMPT_LEN=8192 | |
| ``` | |
| task,armA_c,l1,l2 | |
| niah_single,62.0,67.0,78.0 | |
| val_backward,2.0,1.0,2.0 | |
| ``` | |
| ## PROMPT_LEN=16360 | |
| ``` | |
| task,armA_c,l1,l2 | |
| niah_single,33.0,39.0,48.0 | |
| val_backward,1.0,1.0,0.0 | |
| ``` | |
| ## Training losses | |
| ``` | |
| l1: loss: 2.458802 | |
| l2: loss: 2.794149 | |
| ``` | |
| NOTE: losses are NOT comparable across arms -- each is measured on its own | |
| data distribution. Longer documents are a different prediction problem, so | |
| a higher loss on b2 than on flux says nothing about model quality. Only the | |
| retrieval numbers compare. | |
| βββββββββββ¬βββββββββββββββββββββββββ¬βββββββββββββββββ¬ββββββββββββββββββ | |
| β context β armA_c (flux, med 826) β l1 (docs 4-8k) β l2 (docs 8-16k) β | |
| βββββββββββΌβββββββββββββββββββββββββΌβββββββββββββββββΌββββββββββββββββββ€ | |
| β 1k β 94.0 Β±4.8 β 95.0 Β±4.5 β 96.0 Β±4.1 β | |
| βββββββββββΌβββββββββββββββββββββββββΌβββββββββββββββββΌββββββββββββββββββ€ | |
| β 2k β 94.0 Β±4.8 β 96.0 Β±4.1 β 94.0 Β±4.8 β | |
| βββββββββββΌβββββββββββββββββββββββββΌβββββββββββββββββΌββββββββββββββββββ€ | |
| β 4k β 84.0 Β±7.2 β 88.0 Β±6.4 β 92.0 Β±5.4 β | |
| βββββββββββΌβββββββββββββββββββββββββΌβββββββββββββββββΌββββββββββββββββββ€ | |
| β 8k β 62.0 Β±9.3 β 67.0 Β±9.1 β 78.0 Β±8.0 β | |
| βββββββββββΌβββββββββββββββββββββββββΌβββββββββββββββββΌββββββββββββββββββ€ | |
| β 16360 β 33.0 Β±9.1 β 39.0 Β±9.4 β 48.0 Β±9.6 β | |
| βββββββββββ΄βββββββββββββββββββββββββ΄βββββββββββββββββ΄ββββββββββββββββββ | |
| βββββββββββ¬ββββββββββββ¬ββββββββββββ¬ββββββββββ¬ββββββββββββββββββ | |
| β context β l1 β armA β l2 β armA β l2 β l1 β resolvable β | |
| βββββββββββΌββββββββββββΌββββββββββββΌββββββββββΌββββββββββββββββββ€ | |
| β 1k β +1.0 β +2.0 β +1.0 β no β | |
| βββββββββββΌββββββββββββΌββββββββββββΌββββββββββΌββββββββββββββββββ€ | |
| β 2k β +2.0 β +0.0 β β2.0 β no β | |
| βββββββββββΌββββββββββββΌββββββββββββΌββββββββββΌββββββββββββββββββ€ | |
| β 4k β +4.0 β +8.0 β +4.0 β no (bar Β±9.0) β | |
| βββββββββββΌββββββββββββΌββββββββββββΌββββββββββΌββββββββββββββββββ€ | |
| β 8k β +5.0 β +16.0 β +11.0 β YES (bar Β±12.3) β | |
| βββββββββββΌββββββββββββΌββββββββββββΌββββββββββΌββββββββββββββββββ€ | |
| β 16360 β +6.0 β +15.0 β +9.0 β YES (bar Β±13.2) β | |
| βββββββββββ΄ββββββββββββ΄ββββββββββββ΄ββββββββββ΄ββββββββββββββββββ | |
| The effect is monotone in training document length and grows with eval context. Nothing distinguishable at 1-2k, +8pp at 4k, then +16pp at 8k and +15pp at 16k. Ordering is l2 > l1 > armA_c at every length where there's signal β three independent lengths agreeing on the same ordering, with the two long ones individually resolvable. | |
| Your hypothesis is supported. The flux corpus's short documents (median 826, p90 2,721) were a genuine ceiling on long-context retrieval, not just the YaRN cache. Training on documents that actually span the window recovers a large chunk: at the full 16,384 window, 33% β 48%, a +45% relative improvement from changing nothing but document length. | |
| Backward retrieval is untouched β val_backward sits at 0-2% everywhere against ~2.9% chance, identical across all three arms. Document length does nothing for valueβkey retrieval. That failure mode is independent of both the cache fix and the data fix. | |
| Two caveats that matter for how you read this | |
| The control's absolute numbers moved. armA_c scored 25% at 16k on the flux haystack; here it's 33%. Same checkpoint β the haystack changed (b3's 16-32k documents, held out from all arms). That's exactly why I re-scored the control rather than reusing its published numbers; all three columns above share one haystack, so they compare, but they don't compare to the 2026-09-03 table. | |
| Training losses do not compare across arms (l1 2.459, l2 2.794). Each is measured on its own distribution, and predicting inside 8-16k documents is a harder problem than inside 4-8k ones. l2 has the worst loss and the best retrieval β the same divergence as the cache experiment. Only the retrieval numbers compare. | |
| Results are in lenbucket_logs/RESULTS.md. |