Upload docs/RESULTS_l3long_2026-09-09.md with huggingface_hub

docs/RESULTS_l3long_2026-09-09.md

@@ -0,0 +1,80 @@
+# l3long (run 17) -- does MORE step count at 16k flatten the NIAH slope?
+
+Stopped early at step 2450 of 4000 on 2026-09-09 18:2x UTC for instance teardown.
+
+## Question
+
+l1/l2/l3 all landed on an 8k->16k slope of -18 to -29 pp/doubling and tied at 16k
+(l2 35.0 vs l3 35.0). Document length was exhausted as a lever inside a 16k
+window. Step count was the remaining untested lever.
+
+## Setup
+
+    parent          blumodelcp_run6_step_750.ckpt (run7', same parent as l3@400)
+    context_length  16384   (YaRN scale 4.000 x orig_maxpos 4096)
+    data            lendata/b3_16384_32768  (16k-32k docs, truncated to the window)
+    global_batch    524288 tokens (grad_accum 32)
+    max_lr/min_lr   6e-4 / 6e-5
+    schedule        WSD, warmup 100, stable 3800, decay tail 100 (decay from 3900)
+    rewarmup        CP_REWARMUP_PEAK 0.3 -> flat 1.8e-4
+    rate            11.56 s/step measured
+
+IMPORTANT: this is a 16k run, NOT a 32k run. The "32768" in the data path is the
+document-length bucket, not the context window. No 32k extension has been run.
+
+## NIAH single-needle accuracy (%)
+
+| prompt_len | armA_c | l1 | l2 | l3 | l3long@1000 | l3long@2000 |
+|---|---|---|---|---|---|---|
+| 4096  | 73.0 | 75.0 | 80.0 | 74.0 | 65.0 | 68.0 |
+| 8192  | 53.0 | 48.0 | 62.0 | 64.0 | 55.0 | 54.0 |
+| 16360 | 33.0 | 30.0 | 35.0 | 35.0 | 36.0 | 43.0 |
+
+## Reading
+
+Step count IS a live lever at 16k, unlike document length. At 16360 l3long went
+36.0 -> 43.0 between step 1000 and 2000: +7 pp from step count alone, and +8 pp
+over the previous ladder best (l2/l3 tied at 35.0). That is the largest margin
+anything has opened at 16k in this series.
+
+Short context is recovering rather than degrading: 4096 went 65.0 -> 68.0 and
+8192 held flat at 54-55.
+
+Two caveats that keep this suggestive rather than settled:
+  1. l3long is still 12 pp behind l2 at 4096 (68 vs 80). The short-context cost is
+     real, only shrinking.
+  2. 100 samples per cell, so +7 pp is roughly 1.5 sigma.
+  3. Every l3long number is UNDECAYED (decay would not have begun until step
+     3900). l3/l2 at step 400 are decayed. The comparison is biased AGAINST
+     l3long, so its wins are conservative and its 4096 deficit is overstated by
+     an unknown amount.
+
+## Operational finding: silent checkpoint loss
+
+CP_CKPT_KEEP prunes .ckpt files but NOT their .zero_g1 optimizer sidecars. At
+CP_CKPT_FREQ=25 that leaked ~49 GB of orphans and filled a 193 GB disk. Every
+save from step 1275 to 1625 then failed while training carried on normally,
+because a failed save only emits
+
+    [WARN] checkpoint save at step NNNN failed: Failed to flush checkpoint to: ...
+
+No crash, no backpressure, no exit code. 75 minutes of checkpointing was lost and
+the run would have rolled back to step 1250 had it died.
+
+Mitigation used here: ckpt_janitor.sh, a detached 5-minute loop that deletes
+sidecars with no surviving .ckpt, removes .tmp older than 10 minutes, and warns
+below 25 GB free. It reclaimed 928 MB per cycle, exactly offsetting the leak.
+
+The real fix belongs in CheckpointManager: prune the sidecar with its .ckpt, and
+treat a failed flush as fatal rather than a warning.
+
+## Next
+
+1. Decay check: STEPS=1100 STABLE=900 from the step-1000 ckpt, to size how much
+   of the 4096 deficit is just the missing decay tail.
+2. ext_32k.sh -- written, never run. YaRN-extend l2 to 32768 (2 x 16384), 750
+   steps. Needs both H200s at a projected 130.8 GB/GPU (91.4%); it self-gates
+   with a 3-step OOM probe. PARENT IS lenbucket_l2_blumodelcp_run2_step_400.ckpt
+   -- do not delete that checkpoint.
+3. Resume l3long from step 2450 to 4000 if the trend is worth confirming; the
+   sidecars are uploaded so the optimizer state survives.