DJLougen commited on
Commit
ff3cfdc
·
verified ·
1 Parent(s): e44ce77

Add AIME 2025 & AIME 2026 official evaluations + 7B-14B peer benchmark comparison

Browse files
Files changed (1) hide show
  1. README.md +32 -24
README.md CHANGED
@@ -312,7 +312,7 @@ Across the 60 unseen hard-math and function-graphing problems, `Qwen3.5-9B-Math-
312
  | `Qwen3.5-9B-Math-Tutor` (`32L WSP`) | `2,916,073` | `0` | `81.67%` | `75.00%` | `53.92%` | `0.00%` | `0.00%` |
313
  | **`Qwen3.5-9B-Math-GraphCalc` (`43L`)** | **`2,920,169`** | **`1,045,984`** | **`96.67%`** | **`70.00%`** | **`97.06%`** | **`100.00%`** | **`100.00%`** |
314
 
315
- ### 2. Official Public HuggingFace Math Benchmarks (`7,786` Total Zero-Shot Greedy Evaluations)
316
 
317
  <p align="center">
318
  <img src="assets/hf_public_benchmarks.png" alt="Official Public HuggingFace Math Benchmarks — GSM8K, MATH-500, AIME 2024, AIME 2025, and AIME 2026" width="100%">
@@ -343,7 +343,7 @@ Evaluated zero-shot with greedy decoding directly on official HuggingFace Hub da
343
  | Level 4 | `48.44%` | **`52.34%`** |
344
  | Level 5 | `29.10%` | **`29.85%`** |
345
 
346
- > **Domain Scope (`<tools>` vs. Standard Zero-Shot):** `graphcalc.elf` implements a specialized single-variable graphing & 256-bit arithmetic/CORDIC calculator (`+ - * / ^`, `sin`, `cos`, `tan`, `sqrt`, `ln`, `exp`, roots, extrema, Simpson integrals). Pass `<tools>` for multi-digit arithmetic, exact fractions, calculus, and function graphing (**`96.67%`** accuracy on `GraphCalc-HeldOut-60`). For general symbolic competition math, modular number theory, or multi-variable systems (where forcing a single-variable `<tool_call>` on a 50-item `MATH-500` scalar slice scored `58.00%` vs `72.00%` without `<tools>` across `2,927,967` RV32 steps), omit `<tools>` — the untied `E6` gate triggers **`0`** false tool calls across all zero-shot evaluations and runs at full 32-layer speed.
347
 
348
  ### 3. Peer Model Comparison on Public HF & Technical-Report Benchmarks
349
 
@@ -351,28 +351,36 @@ Evaluated zero-shot with greedy decoding directly on official HuggingFace Hub da
351
  <img src="assets/model_comparison.png" alt="Peer Comparison: Qwen3.5-9B-Math-GraphCalc (Zero-Gradient WSP) vs. 7B-14B Pretrained Base and SFT+RL Models" width="100%">
352
  </p>
353
 
354
- Comparing **`Qwen3.5-9B-Math-GraphCalc`** (`0` gradient steps, `0` SFT, `0` RL, evaluated **0-shot greedy**) against published `7B–14B` pretrained base peers (`0` post-training gradients) and gradient post-trained (`SFT + RL`) reference models from official technical reports ([`arXiv:2505.09388`](https://arxiv.org/abs/2505.09388), [`arXiv:2412.15115`](https://arxiv.org/abs/2412.15115), [`arXiv:2409.12122`](https://arxiv.org/abs/2409.12122), [`arXiv:2503.19786`](https://arxiv.org/abs/2503.19786)) and [`MathArena`](https://huggingface.co/MathArena):
355
-
356
- | Model | Post-Training | Eval Protocol | GSM8K | MATH / MATH-500 | AIME 2024 | AIME 2025 | AIME 2026 |
357
- |---|---|---|---:|---:|---:|---:|---:|
358
- | `Mistral-7B-v0.1` | **0-Grad Base** | 8s / 4s | `36.2%` | `10.2%` | — | — | — |
359
- | `Meta-Llama-3.1-8B` | **0-Grad Base** | 8s / 4s | `57.2%` | `20.3%` | — | — | — |
360
- | `DeepSeekMath-Base-7B` | **0-Grad Base** | 8s / 4s | `64.2%` | `36.2%` | — | — | — |
361
- | `Gemma-2-9B` | **0-Grad Base** | 5s / 4s | `68.6%` | `36.6%` | — | — | — |
362
- | `Gemma-3-12B-PT` (`12B`) | **0-Grad Base** | 8s / 4s | `71.0%` | `43.3%` | — | — | — |
363
- | `Qwen2.5-7B` | **0-Grad Base** | 8s / 4s | `85.36%` | `49.80%` | — | — | — |
364
- | `Qwen2.5-Math-7B` | **0-Grad Base** | 8s / 4s | `91.6%` | `55.4%` | — | — | — |
365
- | `Qwen2.5-14B` (`14B`) | **0-Grad Base** | 8s / 4s | `90.22%` | `55.64%` | — | — | — |
366
- | `Qwen3-8B-Base` | **0-Grad Base** | 8s / 4s | `89.84%` | `60.80%` | — | — | — |
367
- | `Qwen/Qwen3.5-9B-Base` | **0-Grad Base** | **0-shot ChatML** | `74.30%` | `56.20%` | `20.00%` | `16.67%` | `10.00%` |
368
- | **`Qwen3.5-9B-Math-GraphCalc` (Ours)** | **0-Grad WSP + RV32** | **0-shot ChatML** | **`78.85%`** | **`58.20%`** | **`20.00%`** | **`20.00%`** | **`10.00%`** |
369
- | `Llama-3.1-8B-Instruct` | SFT + RL | Non-Think (`≤2K`) | `84.5%` | `54.8%` | `6.3%` | `2.7%` | — |
370
- | `Qwen2.5-7B-Instruct` | SFT + RL | Non-Think (`≤2K`) | `91.6%` | `77.6%` | `9.1%` | `12.1%` | — |
371
- | `Qwen2.5-Math-7B-Instruct` | SFT + GRPO RL | CoT (`≤2K`) | `95.2%` | `83.6%` | `16.7%` | — | — |
372
- | `Gemma-3-12B-IT` (`12B`) | SFT + RL Distill | Non-Think (`≤2K`) | — | `85.6%` | `22.4%` | `18.8%` | — |
373
- | `Qwen3-8B` (Non-Thinking) | SFT + 4-Stage RL | Non-Think (`≤2K`) | — | `87.4%` | `29.1%` | `20.9%` | — |
374
- | `Qwen3-8B` (Thinking) | SFT + 4-Stage RL | `32K` `<think>` | — | `97.4%` | `76.0%` | `67.3%` | — |
375
- | `Qwen3.5-9B` (Thinking) | SFT + Scaled RL | `31K` avg `<think>` | — | — | — | — | `92.50%` |
 
 
 
 
 
 
 
 
376
 
377
  ### 4. Disjoint Zero-Shot Math Tutoring & Language Retention (`N = 180 × 2 = 360` Generations, No `<tools>`)
378
 
 
312
  | `Qwen3.5-9B-Math-Tutor` (`32L WSP`) | `2,916,073` | `0` | `81.67%` | `75.00%` | `53.92%` | `0.00%` | `0.00%` |
313
  | **`Qwen3.5-9B-Math-GraphCalc` (`43L`)** | **`2,920,169`** | **`1,045,984`** | **`96.67%`** | **`70.00%`** | **`97.06%`** | **`100.00%`** | **`100.00%`** |
314
 
315
+ ### 2. Official Public HuggingFace Math Benchmarks (`7,636` Total Zero-Shot Greedy Evaluations)
316
 
317
  <p align="center">
318
  <img src="assets/hf_public_benchmarks.png" alt="Official Public HuggingFace Math Benchmarks — GSM8K, MATH-500, AIME 2024, AIME 2025, and AIME 2026" width="100%">
 
343
  | Level 4 | `48.44%` | **`52.34%`** |
344
  | Level 5 | `29.10%` | **`29.85%`** |
345
 
346
+ > **Domain Scope (`<tools>` vs. Standard Zero-Shot):** `graphcalc.elf` implements a specialized single-variable graphing & 256-bit arithmetic/CORDIC calculator (`+ - * / ^`, `sin`, `cos`, `tan`, `sqrt`, `ln`, `exp`, roots, extrema, Simpson integrals). Pass `<tools>` for multi-digit arithmetic, exact fractions, calculus, and function graphing (**`96.67%`** accuracy on `GraphCalc-HeldOut-60`). For general symbolic competition math, modular number theory, or multi-variable systems, omit `<tools>` — the untied `E6` gate triggers **`0`** false tool calls across all `7,636` zero-shot public benchmark evaluations and runs at full 32-layer speed.
347
 
348
  ### 3. Peer Model Comparison on Public HF & Technical-Report Benchmarks
349
 
 
351
  <img src="assets/model_comparison.png" alt="Peer Comparison: Qwen3.5-9B-Math-GraphCalc (Zero-Gradient WSP) vs. 7B-14B Pretrained Base and SFT+RL Models" width="100%">
352
  </p>
353
 
354
+ Comparing **`Qwen3.5-9B-Math-GraphCalc`** (`0` gradient steps, `0` SFT, `0` RL, evaluated **0-shot greedy**) against `7B–14B` models with complete benchmark coverage in official technical reports ([`arXiv:2505.09388`](https://arxiv.org/abs/2505.09388), [`arXiv:2412.15115`](https://arxiv.org/abs/2412.15115), [`arXiv:2409.12122`](https://arxiv.org/abs/2409.12122), [`arXiv:2503.19786`](https://arxiv.org/abs/2503.19786)):
355
+
356
+ #### 3A. Pretrained Base Peers (`0` Post-Training Gradients — `GSM8K` & `MATH / MATH-500`)
357
+
358
+ | Model | Post-Training | Eval Protocol | GSM8K | MATH / MATH-500 |
359
+ |---|---|---|---:|---:|
360
+ | `Mistral-7B-v0.1` | **0-Grad Base** | 8s / 4s | `36.20%` | `10.20%` |
361
+ | `Meta-Llama-3.1-8B` | **0-Grad Base** | 8s / 4s | `57.20%` | `20.30%` |
362
+ | `DeepSeekMath-Base-7B` | **0-Grad Base** | 8s / 4s | `64.20%` | `36.20%` |
363
+ | `Gemma-2-9B` | **0-Grad Base** | 5s / 4s | `68.60%` | `36.60%` |
364
+ | `Gemma-3-12B-PT` (`12B`) | **0-Grad Base** | 8s / 4s | `71.00%` | `43.30%` |
365
+ | `Qwen2.5-7B` | **0-Grad Base** | 8s / 4s | `85.36%` | `49.80%` |
366
+ | `Qwen2.5-Math-7B` | **0-Grad Base** | 8s / 4s | `91.60%` | `55.40%` |
367
+ | `Qwen2.5-14B` (`14B`) | **0-Grad Base** | 8s / 4s | `90.22%` | `55.64%` |
368
+ | `Qwen/Qwen3.5-9B-Base` | **0-Grad Base** | **0-shot ChatML** | `74.30%` | `56.20%` |
369
+ | **`Qwen3.5-9B-Math-GraphCalc` (Ours)** | **0-Grad WSP + RV32** | **0-shot ChatML** | **`78.85%`** | **`58.20%`** |
370
+ | `Qwen3-8B-Base` | **0-Grad Base** | 8s / 4s | `89.84%` | `60.80%` |
371
+
372
+ #### 3B. Post-Trained (`SFT + RL`) Reference Peers (`MATH-500`, `AIME 2024` & `AIME 2025`)
373
+
374
+ | Model | Post-Training | Eval Protocol | MATH-500 | AIME 2024 | AIME 2025 |
375
+ |---|---|---|---:|---:|---:|
376
+ | `Llama-3.1-8B-Instruct` | SFT + RL | Non-Think (`≤2K`) | `54.80%` | `6.30%` | `2.70%` |
377
+ | `Qwen2.5-7B-Instruct` | SFT + RL | Non-Think (`≤2K`) | `77.60%` | `9.10%` | `12.10%` |
378
+ | `Qwen2.5-14B-Instruct` (`14B`) | SFT + RL | Non-Think (`≤2K`) | `83.40%` | `15.20%` | `13.60%` |
379
+ | `Qwen/Qwen3.5-9B-Base` | **0-Grad Base** | **0-shot ChatML (`2K`)** | `56.20%` | `20.00%` | `16.67%` |
380
+ | **`Qwen3.5-9B-Math-GraphCalc` (Ours)** | **0-Grad WSP + RV32** | **0-shot ChatML (`2K`)** | **`58.20%`** | **`20.00%`** | **`20.00%`** |
381
+ | `Gemma-3-12B-IT` (`12B`) | SFT + RL Distill | Non-Think (`≤2K`) | `85.60%` | `22.40%` | `18.80%` |
382
+ | `Qwen3-8B` (Non-Thinking) | SFT + 4-Stage RL | Non-Think (`≤2K`) | `87.40%` | `29.10%` | `20.90%` |
383
+ | `Qwen3-8B` (Thinking) | SFT + 4-Stage RL | `32K` `<think>` | `97.40%` | `76.00%` | `67.30%` |
384
 
385
  ### 4. Disjoint Zero-Shot Math Tutoring & Language Retention (`N = 180 × 2 = 360` Generations, No `<tools>`)
386