Upload docs/RESULTS_l3long_2026-09-09.md with huggingface_hub
Browse files
docs/RESULTS_l3long_2026-09-09.md
CHANGED
|
@@ -20,7 +20,10 @@ window. Step count was the remaining untested lever.
|
|
| 20 |
rate 11.56 s/step measured
|
| 21 |
|
| 22 |
IMPORTANT: this is a 16k run, NOT a 32k run. The "32768" in the data path is the
|
| 23 |
-
document-length bucket, not the context window.
|
|
|
|
|
|
|
|
|
|
| 24 |
|
| 25 |
## NIAH single-needle accuracy (%)
|
| 26 |
|
|
@@ -30,6 +33,20 @@ document-length bucket, not the context window. No 32k extension has been run.
|
|
| 30 |
| 8192 | 53.0 | 48.0 | 62.0 | 64.0 | 55.0 | 54.0 |
|
| 31 |
| 16360 | 33.0 | 30.0 | 35.0 | 35.0 | 36.0 | 43.0 |
|
| 32 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
## Reading
|
| 34 |
|
| 35 |
Step count IS a live lever at 16k, unlike document length. At 16360 l3long went
|
|
|
|
| 20 |
rate 11.56 s/step measured
|
| 21 |
|
| 22 |
IMPORTANT: this is a 16k run, NOT a 32k run. The "32768" in the data path is the
|
| 23 |
+
document-length bucket, not the context window. A SEPARATE 32k extension
|
| 24 |
+
(ext32k_b4, parent l2, YaRN scale 2 x orig 16384) was run and evaluated earlier
|
| 25 |
+
on 2026-09-09 -- see docs/EXT32K_RESULTS_2026-09-09.md. The local ext_32k.sh
|
| 26 |
+
(parent l2, b3 data) is a DIFFERENT, still-unrun variant.
|
| 27 |
|
| 28 |
## NIAH single-needle accuracy (%)
|
| 29 |
|
|
|
|
| 33 |
| 8192 | 53.0 | 48.0 | 62.0 | 64.0 | 55.0 | 54.0 |
|
| 34 |
| 16360 | 33.0 | 30.0 | 35.0 | 35.0 | 36.0 | 43.0 |
|
| 35 |
|
| 36 |
+
## Cross-comparison with the 32k extension at 16,360
|
| 37 |
+
|
| 38 |
+
| arm | what it changed | 4,096 | 8,192 | 16,360 | 32,744 |
|
| 39 |
+
|---|---|---|---|---|---|
|
| 40 |
+
| l2 (baseline) | -- | 80.0 | 62.0 | 35.0 | cannot run |
|
| 41 |
+
| l3long@2000 | more steps at 16k | 68.0 | 54.0 | 43.0 | cannot run |
|
| 42 |
+
| ext32k_b4 | widened window to 32k | 75.0 | 60.0 | 45.0 | 27.0 |
|
| 43 |
+
|
| 44 |
+
Widening the window and adding step count buy a SIMILAR gain at 16,360 (+10.0
|
| 45 |
+
and +8.0 over l2 respectively, neither resolvable at N=100), but only the
|
| 46 |
+
widening unlocks 32,744 at all, and it costs less short-context accuracy
|
| 47 |
+
(4,096: 75.0 vs 68.0). If only one lever survives, the window widening looks
|
| 48 |
+
like the better buy -- but note ext32k is decayed and l3long@2000 is not.
|
| 49 |
+
|
| 50 |
## Reading
|
| 51 |
|
| 52 |
Step count IS a live lever at 16k, unlike document length. At 16360 l3long went
|