DJLougen commited on
Commit
6b554ee
·
verified ·
1 Parent(s): 9b8659b

Add AIME 2025 & AIME 2026 official evaluations + 7B-14B peer benchmark comparison

Browse files
Files changed (1) hide show
  1. README.md +75 -11
README.md CHANGED
@@ -101,10 +101,41 @@ model-index:
101
  name: HuggingFaceH4/aime_2024
102
  type: HuggingFaceH4/aime_2024
103
  split: train
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
104
  metrics:
105
  - type: accuracy
106
  value: 10.00
107
  name: ChatML Zero-Shot Accuracy (%)
 
 
 
108
  ---
109
 
110
  <p align="center">
@@ -281,22 +312,26 @@ Across the 60 unseen hard-math and function-graphing problems, `Qwen3.5-9B-Math-
281
  | `Qwen3.5-9B-Math-Tutor` (`32L WSP`) | `2,916,073` | `0` | `81.67%` | `75.00%` | `53.92%` | `0.00%` | `0.00%` |
282
  | **`Qwen3.5-9B-Math-GraphCalc` (`43L`)** | **`2,920,169`** | **`1,045,984`** | **`96.67%`** | **`70.00%`** | **`97.06%`** | **`100.00%`** | **`100.00%`** |
283
 
284
- ### 2. Official Public HuggingFace Math Benchmarks (`7,546` Total Zero-Shot Greedy Evaluations)
285
 
286
  <p align="center">
287
- <img src="assets/hf_public_benchmarks.png" alt="Official Public HuggingFace Math Benchmarks — GSM8K, MATH-500, AIME 2024, and MATH-500 Computational-50" width="100%">
288
  </p>
289
 
290
- Evaluated zero-shot with greedy decoding directly on official HuggingFace Hub datasets ([`openai/gsm8k`](https://huggingface.co/datasets/openai/gsm8k) full `1,319` test set, [`HuggingFaceH4/MATH-500`](https://huggingface.co/datasets/HuggingFaceH4/MATH-500) full `500` competition set, and [`HuggingFaceH4/aime_2024`](https://huggingface.co/datasets/HuggingFaceH4/aime_2024) full `30` Olympiad set across both raw `Plain Q&A` and `ChatML` formats):
291
 
292
  | Public HF Dataset | Arm | Mean Math Acc | Plain Q&A Acc | ChatML Acc | Plain Verified Teach | ChatML Verified Teach |
293
  |---|---|---:|---:|---:|---:|---:|
294
- | **`openai/gsm8k`** (`N = 1,319 × 2`) | `Qwen3.5-9B-Base` | `78.54%` | `82.79%` | `74.30%` | `51.40%` | `5.31%` |
295
- | **`openai/gsm8k`** (`N = 1,319 × 2`) | **`9B WSP Tutor / GraphCalc`** | **`75.78%`** | **`72.71%`** | **`78.85%`** | **`59.89%`** | **`53.07%`** |
296
- | **`HuggingFaceH4/MATH-500`** (`N = 500 × 2`) | `Qwen3.5-9B-Base` | `48.40%` | `40.60%` | `56.20%` | `17.60%` | `2.80%` |
297
- | **`HuggingFaceH4/MATH-500`** (`N = 500 × 2`) | **`9B WSP Tutor / GraphCalc`** | **`51.80%`** | **`45.40%`** | **`58.20%`** | **`28.00%`** | **`42.40%`** |
298
- | **`HuggingFaceH4/aime_2024`** (`N = 30 × 2`) | `Qwen3.5-9B-Base` | `6.67%` | `3.33%` | `10.00%` | `0.00%` | `0.00%` |
299
- | **`HuggingFaceH4/aime_2024`** (`N = 30 × 2`) | **`9B WSP Tutor / GraphCalc`** | **`8.34%`** | **`6.67%`** | **`10.00%`** | **`6.67%`** | **`0.00%`** |
 
 
 
 
300
 
301
  #### `MATH-500` Accuracy by Competition Difficulty Level (Zero-Shot ChatML)
302
 
@@ -308,9 +343,38 @@ Evaluated zero-shot with greedy decoding directly on official HuggingFace Hub da
308
  | Level 4 | `48.44%` | **`52.34%`** |
309
  | Level 5 | `29.10%` | **`29.85%`** |
310
 
311
- > **Domain Scope (`<tools>` vs. Standard Zero-Shot):** `graphcalc.elf` implements a specialized single-variable graphing & 256-bit arithmetic/CORDIC calculator (`+ - * / ^`, `sin`, `cos`, `tan`, `sqrt`, `ln`, `exp`, roots, extrema, Simpson integrals). Pass `<tools>` for multi-digit arithmetic, exact fractions, calculus, and function graphing (**`96.67%`** accuracy on `GraphCalc-HeldOut-60`). For general symbolic competition math, modular number theory, or multi-variable systems (where forcing a single-variable `<tool_call>` on a 50-item `MATH-500` scalar slice scored `58.00%` vs `72.00%` without `<tools>` across `2,927,967` RV32 steps), omit `<tools>` — the untied `E6` gate triggers **`0 / 360`** false tool calls and runs at full 32-layer speed.
 
 
 
 
 
 
312
 
313
- ### 3. Disjoint Zero-Shot Math Tutoring & Language Retention (`N = 180 × 2 = 360` Generations, No `<tools>`)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
314
 
315
  When prompted without `<tools>`, untied `lm_head` `<tool_call>` calibration (`E6 = 1.3×`) triggers **`0 / 360` false tool calls** while improving zero-shot `plain` Q&A tutoring and preserving `FineWeb-Edu` causal language modeling (`+0.0001` nats):
316
 
 
101
  name: HuggingFaceH4/aime_2024
102
  type: HuggingFaceH4/aime_2024
103
  split: train
104
+ metrics:
105
+ - type: accuracy
106
+ value: 20.00
107
+ name: ChatML Zero-Shot Accuracy (%)
108
+ - type: accuracy
109
+ value: 16.67
110
+ name: Mean Accuracy Plain+ChatML (%)
111
+ - task:
112
+ type: text-generation
113
+ name: AIME 2025 Olympiad Benchmark (Zero-Shot Greedy)
114
+ dataset:
115
+ name: MathArena/aime_2025
116
+ type: MathArena/aime_2025
117
+ split: train
118
+ metrics:
119
+ - type: accuracy
120
+ value: 20.00
121
+ name: ChatML Zero-Shot Accuracy (%)
122
+ - type: accuracy
123
+ value: 13.33
124
+ name: Mean Accuracy Plain+ChatML (%)
125
+ - task:
126
+ type: text-generation
127
+ name: AIME 2026 Olympiad Benchmark (Zero-Shot Greedy)
128
+ dataset:
129
+ name: MathArena/aime_2026
130
+ type: MathArena/aime_2026
131
+ split: train
132
  metrics:
133
  - type: accuracy
134
  value: 10.00
135
  name: ChatML Zero-Shot Accuracy (%)
136
+ - type: accuracy
137
+ value: 11.67
138
+ name: Mean Accuracy Plain+ChatML (%)
139
  ---
140
 
141
  <p align="center">
 
312
  | `Qwen3.5-9B-Math-Tutor` (`32L WSP`) | `2,916,073` | `0` | `81.67%` | `75.00%` | `53.92%` | `0.00%` | `0.00%` |
313
  | **`Qwen3.5-9B-Math-GraphCalc` (`43L`)** | **`2,920,169`** | **`1,045,984`** | **`96.67%`** | **`70.00%`** | **`97.06%`** | **`100.00%`** | **`100.00%`** |
314
 
315
+ ### 2. Official Public HuggingFace Math Benchmarks (`7,786` Total Zero-Shot Greedy Evaluations)
316
 
317
  <p align="center">
318
+ <img src="assets/hf_public_benchmarks.png" alt="Official Public HuggingFace Math Benchmarks — GSM8K, MATH-500, AIME 2024, AIME 2025, and AIME 2026" width="100%">
319
  </p>
320
 
321
+ Evaluated zero-shot with greedy decoding directly on official HuggingFace Hub datasets ([`openai/gsm8k`](https://huggingface.co/datasets/openai/gsm8k) full `1,319` test set, [`HuggingFaceH4/MATH-500`](https://huggingface.co/datasets/HuggingFaceH4/MATH-500) full `500` competition set, [`HuggingFaceH4/aime_2024`](https://huggingface.co/datasets/HuggingFaceH4/aime_2024), [`MathArena/aime_2025`](https://huggingface.co/datasets/MathArena/aime_2025), and [`MathArena/aime_2026`](https://huggingface.co/datasets/MathArena/aime_2026) Olympiad sets across both raw `Plain Q&A` and `ChatML` formats):
322
 
323
  | Public HF Dataset | Arm | Mean Math Acc | Plain Q&A Acc | ChatML Acc | Plain Verified Teach | ChatML Verified Teach |
324
  |---|---|---:|---:|---:|---:|---:|
325
+ | **[`openai/gsm8k`](https://huggingface.co/datasets/openai/gsm8k)** (`N = 1,319 × 2`) | `Qwen3.5-9B-Base` | `78.54%` | `82.79%` | `74.30%` | `51.40%` | `5.31%` |
326
+ | **[`openai/gsm8k`](https://huggingface.co/datasets/openai/gsm8k)** (`N = 1,319 × 2`) | **`9B WSP Tutor / GraphCalc`** | **`75.78%`** | **`72.71%`** | **`78.85%`** | **`59.89%`** | **`53.07%`** |
327
+ | **[`HuggingFaceH4/MATH-500`](https://huggingface.co/datasets/HuggingFaceH4/MATH-500)** (`N = 500 × 2`) | `Qwen3.5-9B-Base` | `48.40%` | `40.60%` | `56.20%` | `17.60%` | `2.80%` |
328
+ | **[`HuggingFaceH4/MATH-500`](https://huggingface.co/datasets/HuggingFaceH4/MATH-500)** (`N = 500 × 2`) | **`9B WSP Tutor / GraphCalc`** | **`51.80%`** | **`45.40%`** | **`58.20%`** | **`28.00%`** | **`42.40%`** |
329
+ | **[`HuggingFaceH4/aime_2024`](https://huggingface.co/datasets/HuggingFaceH4/aime_2024)** (`N = 30 × 2`) | `Qwen3.5-9B-Base` | `13.33%` | `6.67%` | `20.00%` | `3.33%` | `0.00%` |
330
+ | **[`HuggingFaceH4/aime_2024`](https://huggingface.co/datasets/HuggingFaceH4/aime_2024)** (`N = 30 × 2`) | **`9B WSP Tutor / GraphCalc`** | **`16.67%`** | **`13.33%`** | **`20.00%`** | **`13.33%`** | **`3.33%`** |
331
+ | **[`MathArena/aime_2025`](https://huggingface.co/datasets/MathArena/aime_2025)** (`N = 30 × 2`) | `Qwen3.5-9B-Base` | `8.33%` | `0.00%` | `16.67%` | `0.00%` | `3.33%` |
332
+ | **[`MathArena/aime_2025`](https://huggingface.co/datasets/MathArena/aime_2025)** (`N = 30 × 2`) | **`9B WSP Tutor / GraphCalc`** | **`13.33%`** | **`6.67%`** | **`20.00%`** | **`6.67%`** | **`13.33%`** |
333
+ | **[`MathArena/aime_2026`](https://huggingface.co/datasets/MathArena/aime_2026)** (`N = 30 × 2`) | `Qwen3.5-9B-Base` | `6.67%` | `3.33%` | `10.00%` | `3.33%` | `0.00%` |
334
+ | **[`MathArena/aime_2026`](https://huggingface.co/datasets/MathArena/aime_2026)** (`N = 30 × 2`) | **`9B WSP Tutor / GraphCalc`** | **`11.67%`** | **`13.33%`** | **`10.00%`** | **`13.33%`** | **`3.33%`** |
335
 
336
  #### `MATH-500` Accuracy by Competition Difficulty Level (Zero-Shot ChatML)
337
 
 
343
  | Level 4 | `48.44%` | **`52.34%`** |
344
  | Level 5 | `29.10%` | **`29.85%`** |
345
 
346
+ > **Domain Scope (`<tools>` vs. Standard Zero-Shot):** `graphcalc.elf` implements a specialized single-variable graphing & 256-bit arithmetic/CORDIC calculator (`+ - * / ^`, `sin`, `cos`, `tan`, `sqrt`, `ln`, `exp`, roots, extrema, Simpson integrals). Pass `<tools>` for multi-digit arithmetic, exact fractions, calculus, and function graphing (**`96.67%`** accuracy on `GraphCalc-HeldOut-60`). For general symbolic competition math, modular number theory, or multi-variable systems (where forcing a single-variable `<tool_call>` on a 50-item `MATH-500` scalar slice scored `58.00%` vs `72.00%` without `<tools>` across `2,927,967` RV32 steps), omit `<tools>` — the untied `E6` gate triggers **`0`** false tool calls across all zero-shot evaluations and runs at full 32-layer speed.
347
+
348
+ ### 3. Peer Model Comparison on Public HF & Technical-Report Benchmarks
349
+
350
+ <p align="center">
351
+ <img src="assets/model_comparison.png" alt="Peer Comparison: Qwen3.5-9B-Math-GraphCalc (Zero-Gradient WSP) vs. 7B-14B Pretrained Base and SFT+RL Models" width="100%">
352
+ </p>
353
 
354
+ Comparing **`Qwen3.5-9B-Math-GraphCalc`** (`0` gradient steps, `0` SFT, `0` RL, evaluated **0-shot greedy**) against published `7B–14B` pretrained base peers (`0` post-training gradients) and gradient post-trained (`SFT + RL`) reference models from official technical reports ([`arXiv:2505.09388`](https://arxiv.org/abs/2505.09388), [`arXiv:2412.15115`](https://arxiv.org/abs/2412.15115), [`arXiv:2409.12122`](https://arxiv.org/abs/2409.12122), [`arXiv:2503.19786`](https://arxiv.org/abs/2503.19786)) and [`MathArena`](https://huggingface.co/MathArena):
355
+
356
+ | Model | Post-Training | Eval Protocol | GSM8K | MATH / MATH-500 | AIME 2024 | AIME 2025 | AIME 2026 |
357
+ |---|---|---|---:|---:|---:|---:|---:|
358
+ | `Mistral-7B-v0.1` | **0-Grad Base** | 8s / 4s | `36.2%` | `10.2%` | — | — | — |
359
+ | `Meta-Llama-3.1-8B` | **0-Grad Base** | 8s / 4s | `57.2%` | `20.3%` | — | — | — |
360
+ | `DeepSeekMath-Base-7B` | **0-Grad Base** | 8s / 4s | `64.2%` | `36.2%` | — | — | — |
361
+ | `Gemma-2-9B` | **0-Grad Base** | 5s / 4s | `68.6%` | `36.6%` | — | — | — |
362
+ | `Gemma-3-12B-PT` (`12B`) | **0-Grad Base** | 8s / 4s | `71.0%` | `43.3%` | — | — | — |
363
+ | `Qwen2.5-7B` | **0-Grad Base** | 8s / 4s | `85.36%` | `49.80%` | — | — | — |
364
+ | `Qwen2.5-Math-7B` | **0-Grad Base** | 8s / 4s | `91.6%` | `55.4%` | — | — | — |
365
+ | `Qwen2.5-14B` (`14B`) | **0-Grad Base** | 8s / 4s | `90.22%` | `55.64%` | — | — | — |
366
+ | `Qwen3-8B-Base` | **0-Grad Base** | 8s / 4s | `89.84%` | `60.80%` | — | — | — |
367
+ | `Qwen/Qwen3.5-9B-Base` | **0-Grad Base** | **0-shot ChatML** | `74.30%` | `56.20%` | `20.00%` | `16.67%` | `10.00%` |
368
+ | **`Qwen3.5-9B-Math-GraphCalc` (Ours)** | **0-Grad WSP + RV32** | **0-shot ChatML** | **`78.85%`** | **`58.20%`** | **`20.00%`** | **`20.00%`** | **`10.00%`** |
369
+ | `Llama-3.1-8B-Instruct` | SFT + RL | Non-Think (`≤2K`) | `84.5%` | `54.8%` | `6.3%` | `2.7%` | — |
370
+ | `Qwen2.5-7B-Instruct` | SFT + RL | Non-Think (`≤2K`) | `91.6%` | `77.6%` | `9.1%` | `12.1%` | — |
371
+ | `Qwen2.5-Math-7B-Instruct` | SFT + GRPO RL | CoT (`≤2K`) | `95.2%` | `83.6%` | `16.7%` | — | — |
372
+ | `Gemma-3-12B-IT` (`12B`) | SFT + RL Distill | Non-Think (`≤2K`) | — | `85.6%` | `22.4%` | `18.8%` | — |
373
+ | `Qwen3-8B` (Non-Thinking) | SFT + 4-Stage RL | Non-Think (`≤2K`) | — | `87.4%` | `29.1%` | `20.9%` | — |
374
+ | `Qwen3-8B` (Thinking) | SFT + 4-Stage RL | `32K` `<think>` | — | `97.4%` | `76.0%` | `67.3%` | — |
375
+ | `Qwen3.5-9B` (Thinking) | SFT + Scaled RL | `31K` avg `<think>` | — | — | — | — | `92.50%` |
376
+
377
+ ### 4. Disjoint Zero-Shot Math Tutoring & Language Retention (`N = 180 × 2 = 360` Generations, No `<tools>`)
378
 
379
  When prompted without `<tools>`, untied `lm_head` `<tool_call>` calibration (`E6 = 1.3×`) triggers **`0 / 360` false tool calls** while improving zero-shot `plain` Q&A tutoring and preserving `FineWeb-Edu` causal language modeling (`+0.0001` nats):
380