Upload docs/EVAL_COMPARISON_2026-09-02.md with huggingface_hub

docs/EVAL_COMPARISON_2026-09-02.md

@@ -0,0 +1,136 @@
+# Generative RULER / VAL / NIAH vs the NLL proxy — 2026-09-02
+
+Model: 1.06B, YaRN-4, T=16384. 4 checkpoints x 5 tasks x N=100 at 16,360 tokens,
+plus a context-length sweep at 1024/2048/4096/8192 (N=50, 2 tasks). Greedy
+decoding, bf16, substring-match scoring. `T` held at 16384 throughout so the
+YaRN RoPE cache is identical at every context length.
+
+## 1. Headline: retrieval collapses with CONTEXT LENGTH, not with distance
+
+`niah_single` (recall a random 7-digit number given its key; answer space 10^7):
+
+| context | base | arm_a | arm_b | arm_c |
+|---|---|---|---|---|
+| 1,024 | **84.0** | 78.0 | 64.0 | 76.0 |
+| 2,048 | **82.0** | 66.0 | 56.0 | 70.0 |
+| 4,096 | **74.0** | 62.0 | 48.0 | 60.0 |
+| 8,192 | 40.0 | 36.0 | 36.0 | 40.0 |
+| 16,360 | **0.0** | 0.0 | 0.0 | 0.0 |
+
+Graceful decline to 8k, then a cliff to zero. At 16,360 the model cannot copy a
+number sitting ~30 tokens before the question (verified: the needle IS in the
+prompt at 99.8% depth, no `<|endoftext|>` between needle and question). The same
+harness, format and scorer HIT at 1024, so this is a capability limit, not a
+harness artifact.
+
+At 16,360: **2 hits in 2000** (base 0.40%, every arm 0.00%).
+
+## 2. Forward retrieval works; backward retrieval does not
+
+`val_backward` (name the key given its number; answer space 35 words):
+
+| context | base | arm_a | arm_b | arm_c |
+|---|---|---|---|---|
+| 1,024 | 8.0 | 2.0 | 4.0 | 4.0 |
+| 2,048–16,360 | 0–4 | 0–4 | 0–2 | 0–4 |
+
+Near chance everywhere despite a 35-word answer space, while forward retrieval
+of a 7-digit number hits 84%. This is the one VAL Probing axis implemented, and
+it fails outright — the model resolves key->value but not value->key.
+
+## 3. Model ranking: generative DISAGREES with NLL
+
+`niah_single` pooled over 1024/2048/4096 (n=150/model, +-95% Wilson):
+
+| model | generative | NLL reach @16k |
+|---|---|---|
+| base (run7, YaRN only) | **80.0% ±6.4** | 6,047 |
+| arm_a (plain text) | 68.7% ±7.3 | 7,435 |
+| arm_c (IN2, far needles) | 68.7% ±7.3 | **12,976** |
+| arm_b (close needles) | 56.0% ±7.8 | 5,924 |
+
+- generative best->worst: **base, arm_a = arm_c, arm_b**
+- NLL reach best->worst: **arm_c, arm_a, base, arm_b**
+
+They agree on one thing: **arm_b is worst in both.** They disagree at the top —
+NLL puts arm_c first by 2.19x; generation puts the untrained base first, with
+arm_c tied for second.
+
+**Every form of arm training reduced generative retrieval relative to the base.**
+Among the arms, IN2-style far needles (arm_c) and plain text (arm_a) are equal,
+and close needles (arm_b) are clearly worst.
+
+## 4. Depth curves: no U-shape, in either metric
+
+Generative, `niah_single` pooled 1024/2048/4096 (n=30/cell):
+
+| depth | base | arm_a | arm_b | arm_c |
+|---|---|---|---|---|
+| 0.00 | 56.7 | 30.0 | 26.7 | 30.0 |
+| 0.25 | 73.3 | 66.7 | 43.3 | 50.0 |
+| 0.50 | 86.7 | 80.0 | 50.0 | 90.0 |
+| 0.75 | 90.0 | 86.7 | 80.0 | 80.0 |
+| 1.00 | 93.3 | 80.0 | 80.0 | 93.3 |
+
+Monotonically rising toward the question, no primacy bump at depth 0 — the same
+qualitative shape the NLL margins showed. **Both metrics agree: pure recency, not
+lost-in-the-middle.** This is the clearest point of agreement between them.
+
+## 5. Pre-registered predictions
+
+| prediction | threshold | result | verdict |
+|---|---|---|---|
+| 1. depth 0.0 ≈ 0% | ≤5% confirm / ≥15% refute | 0% at 16,360; **27-57% at 1-4k** | confirmed at 16k, refuted at 1-4k |
+| 2. arm_c >> arm_b @0.25 | ≥20pp confirm / ≤5pp refute | **+12.7pp, 2.3σ** | INCONCLUSIVE |
+| 3. arm_c depth 0.25 > 0.5 | ≥20pp confirm / ≤5pp refute | **-40.0pp, -3.8σ (reversed)** | refuted, effect reversed |
+
+### Two flaws in my own thresholds, stated plainly
+
+- **No power precondition.** At 16,360 every arm is at the floor, so
+  `|diff| <= 5pp` is satisfied automatically and the rule mechanically returns
+  "refuted" when the honest answer is "untestable". I should have required at
+  least one arm above ~10% before evaluating predictions 2 and 3.
+- **No sign handling.** Prediction 3 came out -40pp at -3.8σ — a significant
+  reversal — which my "else -> inconclusive" branch mislabels. A significant
+  effect in the opposite direction is a refutation, not an absence of evidence.
+
+## 6. Verdict on the NLL proxy
+
+**Valid as a weak directional signal; invalid as a magnitude or a ranking; and
+over-sensitive at long context.**
+
+- *Agrees:* depth ordering (pure recency, no U-shape) and arm_b being worst.
+- *Overstates magnitude:* NLL gave arm_c a 2.19x reach advantage over arm_b at
+  ~9.5σ. Generation gives +12.7pp at 2.3σ — real in direction, far smaller, and
+  short of the pre-registered 20pp bar.
+- *Wrong at the top:* NLL ranked base 3rd of 4; generation ranks it 1st.
+- *Over-sensitive:* at 16,360 tokens NLL reported reach of 5,924–12,976 tokens
+  while generative accuracy is 0.0-0.4%. The metric has signal in a regime where
+  the model cannot do the task at all.
+- *The 6.87σ anomaly does not reproduce.* arm_c's "better at depth 0.25 than
+  0.5" (1.0865 nats, 6.87σ on 8 blocks) comes out **reversed** generatively
+  (-40pp, -3.8σ).
+
+## 7. The limitation that bounds all of the above
+
+**The two metrics never overlap at a context length where both have signal.**
+NLL was measured only at T=16,384, where generation is at the floor. The
+generative depth curve comes from 1-4k, where NLL was never measured — and
+measuring it there would require a different YaRN cache, so it would not be the
+same model configuration. Every cross-metric statement above therefore compares
+16k NLL against 1-4k generation and should be read accordingly.
+
+Closing that gap is the obvious next experiment: run `needle_score.py` at
+T=1024/2048/4096 (accepting the RoPE-cache change) or extend generation with a
+KV cache so 16k becomes tractable enough to show non-zero signal.
+
+## 8. Not covered
+
+RULER's CWE / FWE / QA / multiquery; VAL's code and structured-data contexts and
+bi-directional retrieval; `niah_multikey` / `niah_multivalue` / `vt` at 16k (all
+at the floor there, and not run in the length sweep).
+
+Caveat recorded: the flux haystack contains ~6 `<|endoftext|>` tokens per
+16,360-token prompt (document separators a base model may treat as resets). Not
+a factor for the depth-1.0 failure, and the NLL needles used the same filler, so
+the two remain comparable on that axis.