Upload docs/EVAL_COMPARISON_2026-09-02.md with huggingface_hub

docs/EVAL_COMPARISON_2026-09-02.md

@@ -134,73 +134,3 @@ Caveat recorded: the flux haystack contains ~6 `<|endoftext|>` tokens per
 16,360-token prompt (document separators a base model may treat as resets). Not
 a factor for the depth-1.0 failure, and the NLL needles used the same filler, so
 the two remain comparable on that axis.
-
----
-
-# ADDENDUM (same day, later) — a YaRN config bug, and why the numbers above are provisional
-
-## The bug
-
-`YARNOps.cpp:57` defaults `YARN_ORIG_MAXPOS` to **1024**, but the base model (run8)
-trained at `context_length 4096`. So every run in this project — training, the NLL
-probes, and all the generative numbers above — emitted:
-
-```
-[YaRN][warn] seq_len(16384) != scale(4.000)*orig_maxpos(1024.0)=4096.0
-             -- cache scaled for a different context than it is sized for
-```
-
-The YaRN interpolation ramp targeted 4096 positions while the cache was *sized* 16384.
-Setting `YARN_ORIG_MAXPOS=4096` makes `scale x orig == seq_len` exactly and moves the
-ramp from `low=7 high=25` to `low=14 high=32`.
-
-## What the fix does — niah_single, buggy -> corrected
-
-    context      base       arm_a       arm_b       arm_c
-      1,024   84 -> 90    78 -> 86    64 -> 70    76 -> 82
-      2,048   82 -> 94    66 -> 78    56 -> 64    70 -> 80
-      4,096   74 -> 16 !  62 -> 54    48 -> 42    60 -> 48
-      8,192   40 -> 36    36 -> 42    36 -> 32    40 -> 44
-     16,360    0 -> 17     0 -> 17     0 ->  4     0 -> 18
-
-At 16,360 with N=100/cell the depth curve goes from flat zero to:
-d0.50 0-5%, d0.75 20-30%, d1.00 55-65%. That is 0/100 -> 17/100, p<0.001.
-
-So the cache bug was suppressing real ability **at every depth and every length**, not
-only at 16k. Depth 0.0 (needle at the window start) improves too: base 75 -> 85%,
-arm_a 35 -> 55% when pooled over 1k+2k.
-
-## The unexplained 4,096 collapse
-
-Every model drops at 4,096 under the corrected cache, and `base` falls **74% -> 16%**.
-A 58-point collapse at one length for one model, while its neighbours at 2k and 8k both
-improve, is not a smooth capability change. The suspicious coincidence: 4,096 is
-exactly `orig_maxpos`, so the ramp hits a boundary case there.
-
-**This has not been diagnosed, and it invalidates the corrected-cache reach column** —
-`base`'s "best reach" appears to fall 3,482 -> 2,048 purely because that one row went
-to zero. Isolation needed: re-run base at 4,096 with N=100 to confirm it reproduces
-rather than being a 50-sample fluke, and bracket at 3,840 / 4,352 to see whether the
-failure is pinned to exactly `orig_maxpos`.
-
-## What this means for the main report above
-
-  - STILL HOLDS: no U-shape in either metric (pure recency, no primacy bump at depth 0)
-    under both cache parameterisations; arm_b worst in every measurement; forward
-    retrieval works and backward retrieval does not; the NLL/generative ranking
-    disagreement (NLL puts arm_c first, generation puts the untrained base first).
-  - NOW PROVISIONAL: every reach number, including the "saturates at 2,700-3,500
-    tokens" headline and the "NLL overstates 2-4.7x" factor. All of those were measured
-    under the buggy cache, and the corrected-cache column cannot replace them until the
-    4,096 anomaly is understood.
-  - PREMATURE: retraining with the corrected value. Training on a parameterisation that
-    has a pathological point would bake the pathology in.
-
-## Correction to a claim in section 6 above
-
-I wrote that reach and the p90 training-document length (2,721) are "the same number."
-That is arithmetically true but I selected the percentile that matched — the mean
-(1,340) and median (826) are both well below the measured reach. The defensible claim
-is that reach lands in the **upper range** of the document-length distribution, not that
-it equals p90. The mechanism (99.7% of documents end before 16k, so no gradient signal
-for attending further) is unaffected.