Upload evals/2026-09-17/RULER_t64_table.txt with huggingface_hub

evals/2026-09-17/RULER_t64_table.txt

@@ -1,26 +1,39 @@
-  RULER coverage -- t64 (680M, 64k @1200 steps), N=100, virtual ~1,000 at every length
-  task             cat             4096    8192   16360
-  ---------------------------------------------------------
-  niah_single      retrieval       87.0    91.0    92.0
-  niah_single_1    retrieval      100.0    99.0    98.0
-  niah_single_3    retrieval       28.0    30.0    23.0
-  niah_multikey    retrieval       43.0    53.0    44.0
-  niah_multikey_2  retrieval       27.0    25.0    35.0
-  niah_multikey_3  retrieval       10.0     4.0    13.0
-  niah_multivalue  retrieval        0.0     0.0     0.0
-  niah_multiquery  retrieval        0.0     0.0     0.0
-  vt               multi-hop        3.0     3.0     2.0
-  cwe              aggregation      0.0     0.0     0.0
-  fwe              aggregation      0.0     0.0     0.0
-  qa_1             QA               3.0     7.0     8.0
-  qa_2             QA              16.0    12.0    17.0
-  ---------------------------------------------------------
-  mean over 13 tasks @ 4096: 24.4
-  mean over 13 tasks @ 8192: 24.9
-  mean over 13 tasks @ 16360: 25.5
+RULER coverage -- t64 (680M, 64k @1200 steps, invariant intact)
+N=100/point, virtual ~1,000 at every length, haystack identical to all prior evals
 
-  by category @ 16,360:
-    retrieval     38.1  (n=8)
-    multi-hop      2.0  (n=1)
-    aggregation    0.0  (n=2)
-    QA            12.5  (n=2)
+task             category         4096    8192   16360   32744
+------------------------------------------------------------------
+niah_single      retrieval        87.0    91.0    92.0    93.0
+niah_single_1    retrieval       100.0    99.0    98.0    99.0
+niah_single_3    retrieval        28.0    30.0    23.0    28.0
+niah_multikey    retrieval        43.0    53.0    44.0    38.0
+niah_multikey_2  retrieval        27.0    25.0    35.0    28.0
+niah_multikey_3  retrieval        10.0     4.0    13.0     5.0
+niah_multivalue  retrieval         0.0     0.0     0.0     0.0
+niah_multiquery  retrieval         0.0     0.0     0.0     0.0
+vt               multi-hop         3.0     3.0     2.0     6.0
+cwe              aggregation       0.0     0.0     0.0     0.0
+fwe              aggregation       0.0     0.0     0.0     0.0
+qa_1             QA                3.0     7.0     8.0    15.0
+qa_2             QA               16.0    12.0    17.0    15.0
+------------------------------------------------------------------
+MEAN (13 tasks)                   24.4    24.9    25.5    25.2
+
+by category:
+  retrieval        36.9    37.8    38.1    36.4
+  multi-hop         3.0     3.0     2.0     6.0
+  aggregation       0.0     0.0     0.0     0.0
+  QA                9.5     9.5    12.5    15.0
+
+reference (published, official RULER generators, INSTRUCT models):
+  Llama-3.1-8B-Instruct   95.0 @4k   94.6 @16k   87-90 @32k
+  Qwen3-8B                94.8 @4k   91.3 @16k   91.0 @32k
+  Phi-3.5-mini-instruct   84.1 (4B)
+  no published RULER exists for ANY sub-1B base model
+
+CAVEATS: our task implementations, not the official RULER generators --
+internally comparable across our models, NOT against published scores.
+65,512 block incomplete (8/13 generations, no summary).
+niah_multivalue / niah_multiquery / cwe / fwe are EXACTLY 0.0 at every
+length -- unverified whether capability floor or harness bug; the
+Qwen3-0.6B-Base control that would settle it did not get to run.