Upload docs/EVAL_COMPARISON_2026-09-02.md with huggingface_hub

docs/EVAL_COMPARISON_2026-09-02.md

@@ -134,3 +134,73 @@ Caveat recorded: the flux haystack contains ~6 `<|endoftext|>` tokens per
 16,360-token prompt (document separators a base model may treat as resets). Not
 a factor for the depth-1.0 failure, and the NLL needles used the same filler, so
 the two remain comparable on that axis.
+
+---
+
+# ADDENDUM (same day, later) — a YaRN config bug, and why the numbers above are provisional
+
+## The bug
+
+`YARNOps.cpp:57` defaults `YARN_ORIG_MAXPOS` to **1024**, but the base model (run8)
+trained at `context_length 4096`. So every run in this project — training, the NLL
+probes, and all the generative numbers above — emitted:
+
+```
+[YaRN][warn] seq_len(16384) != scale(4.000)*orig_maxpos(1024.0)=4096.0
+             -- cache scaled for a different context than it is sized for
+```
+
+The YaRN interpolation ramp targeted 4096 positions while the cache was *sized* 16384.
+Setting `YARN_ORIG_MAXPOS=4096` makes `scale x orig == seq_len` exactly and moves the
+ramp from `low=7 high=25` to `low=14 high=32`.
+
+## What the fix does — niah_single, buggy -> corrected
+
+    context      base       arm_a       arm_b       arm_c
+      1,024   84 -> 90    78 -> 86    64 -> 70    76 -> 82
+      2,048   82 -> 94    66 -> 78    56 -> 64    70 -> 80
+      4,096   74 -> 16 !  62 -> 54    48 -> 42    60 -> 48
+      8,192   40 -> 36    36 -> 42    36 -> 32    40 -> 44
+     16,360    0 -> 17     0 -> 17     0 ->  4     0 -> 18
+
+At 16,360 with N=100/cell the depth curve goes from flat zero to:
+d0.50 0-5%, d0.75 20-30%, d1.00 55-65%. That is 0/100 -> 17/100, p<0.001.
+
+So the cache bug was suppressing real ability **at every depth and every length**, not
+only at 16k. Depth 0.0 (needle at the window start) improves too: base 75 -> 85%,
+arm_a 35 -> 55% when pooled over 1k+2k.
+
+## The unexplained 4,096 collapse
+
+Every model drops at 4,096 under the corrected cache, and `base` falls **74% -> 16%**.
+A 58-point collapse at one length for one model, while its neighbours at 2k and 8k both
+improve, is not a smooth capability change. The suspicious coincidence: 4,096 is
+exactly `orig_maxpos`, so the ramp hits a boundary case there.
+
+**This has not been diagnosed, and it invalidates the corrected-cache reach column** —
+`base`'s "best reach" appears to fall 3,482 -> 2,048 purely because that one row went
+to zero. Isolation needed: re-run base at 4,096 with N=100 to confirm it reproduces
+rather than being a 50-sample fluke, and bracket at 3,840 / 4,352 to see whether the
+failure is pinned to exactly `orig_maxpos`.
+
+## What this means for the main report above
+
+  - STILL HOLDS: no U-shape in either metric (pure recency, no primacy bump at depth 0)
+    under both cache parameterisations; arm_b worst in every measurement; forward
+    retrieval works and backward retrieval does not; the NLL/generative ranking
+    disagreement (NLL puts arm_c first, generation puts the untrained base first).
+  - NOW PROVISIONAL: every reach number, including the "saturates at 2,700-3,500
+    tokens" headline and the "NLL overstates 2-4.7x" factor. All of those were measured
+    under the buggy cache, and the corrected-cache column cannot replace them until the
+    4,096 anomaly is understood.
+  - PREMATURE: retraining with the corrected value. Training on a parameterisation that
+    has a pathological point would bake the pathology in.
+
+## Correction to a claim in section 6 above
+
+I wrote that reach and the p90 training-document length (2,721) are "the same number."
+That is arithmetically true but I selected the percentile that matched — the mean
+(1,340) and median (826) are both well below the measured reach. The defensible claim
+is that reach lands in the **upper range** of the document-length distribution, not that
+it equals p90. The mechanism (99.7% of documents end before 16k, so no gradient signal
+for attending further) is unaffected.