| ---
|
| license: apache-2.0
|
| license_link: https://huggingface.co/Qwen/Qwen3.5-9B-Base/blob/main/LICENSE
|
| base_model: Qwen/Qwen3.5-9B-Base
|
| library_name: transformers
|
| pipeline_tag: text-generation
|
| language:
|
| - en
|
| tags:
|
| - weight-space-modeling
|
| - zero-gradient
|
| - risc-v
|
| - in-weights-computer
|
| - gated-deltanet
|
| - math
|
| - graphing-calculator
|
| - pedagogy
|
| - qwen3_5
|
| model-index:
|
| - name: Qwen3.5-9B-Math-GraphCalc
|
| results:
|
| - task:
|
| type: text-generation
|
| name: Hard Math & Function Graphing (Disjoint Held-Out)
|
| dataset:
|
| name: Carbon-GraphCalc-HeldOut-60
|
| type: custom
|
| split: test
|
| metrics:
|
| - type: accuracy
|
| value: 96.67
|
| name: Math Accuracy (%)
|
| - type: accuracy
|
| value: 70.00
|
| name: Verified Pedagogical Teaching (%)
|
| - type: accuracy
|
| value: 97.06
|
| name: Graph Feature Accuracy (%)
|
| - type: accuracy
|
| value: 100.00
|
| name: Function Plotting Rate (%)
|
| - type: accuracy
|
| value: 100.00
|
| name: In-Weights RV32 Execution Lockstep (%)
|
| - task:
|
| type: text-generation
|
| name: Zero-Shot Math Tutoring (Plain + ChatML Held-Out)
|
| dataset:
|
| name: Carbon-MathTutor-HeldOut-180
|
| type: custom
|
| split: test
|
| metrics:
|
| - type: accuracy
|
| value: 86.67
|
| name: Overall Math Accuracy (%)
|
| - type: accuracy
|
| value: 80.56
|
| name: Plain Q&A Math Accuracy (%)
|
| - type: accuracy
|
| value: 73.06
|
| name: Verified Pedagogical Teaching (%)
|
| - task:
|
| type: text-generation
|
| name: GSM8K Official Test (Zero-Shot Greedy)
|
| dataset:
|
| name: openai/gsm8k
|
| type: openai/gsm8k
|
| config: main
|
| split: test
|
| metrics:
|
| - type: accuracy
|
| value: 78.85
|
| name: ChatML Zero-Shot Accuracy (%)
|
| - type: accuracy
|
| value: 75.78
|
| name: Mean Accuracy Plain+ChatML (%)
|
| - type: accuracy
|
| value: 53.07
|
| name: Verified Pedagogical Teaching (%)
|
| - task:
|
| type: text-generation
|
| name: MATH-500 Official Competition Benchmark (Zero-Shot Greedy)
|
| dataset:
|
| name: HuggingFaceH4/MATH-500
|
| type: HuggingFaceH4/MATH-500
|
| split: test
|
| metrics:
|
| - type: accuracy
|
| value: 58.20
|
| name: ChatML Zero-Shot Accuracy (%)
|
| - type: accuracy
|
| value: 51.80
|
| name: Mean Accuracy Plain+ChatML (%)
|
| - type: accuracy
|
| value: 42.40
|
| name: Verified Pedagogical Teaching (%)
|
| - task:
|
| type: text-generation
|
| name: AIME 2024 Olympiad Benchmark (Zero-Shot Greedy)
|
| dataset:
|
| name: HuggingFaceH4/aime_2024
|
| type: HuggingFaceH4/aime_2024
|
| split: train
|
| metrics:
|
| - type: accuracy
|
| value: 20.00
|
| name: ChatML Zero-Shot Accuracy (%)
|
| - type: accuracy
|
| value: 16.67
|
| name: Mean Accuracy Plain+ChatML (%)
|
| - task:
|
| type: text-generation
|
| name: AIME 2025 Olympiad Benchmark (Zero-Shot Greedy)
|
| dataset:
|
| name: MathArena/aime_2025
|
| type: MathArena/aime_2025
|
| split: train
|
| metrics:
|
| - type: accuracy
|
| value: 20.00
|
| name: ChatML Zero-Shot Accuracy (%)
|
| - type: accuracy
|
| value: 13.33
|
| name: Mean Accuracy Plain+ChatML (%)
|
| - task:
|
| type: text-generation
|
| name: AIME 2026 Olympiad Benchmark (Zero-Shot Greedy)
|
| dataset:
|
| name: MathArena/aime_2026
|
| type: MathArena/aime_2026
|
| split: train
|
| metrics:
|
| - type: accuracy
|
| value: 10.00
|
| name: ChatML Zero-Shot Accuracy (%)
|
| - type: accuracy
|
| value: 11.67
|
| name: Mean Accuracy Plain+ChatML (%)
|
| ---
|
|
|
| <p align="center">
|
| <img src="assets/hero_banner.png" alt="Qwen3.5-9B-Math-GraphCalc — 9.0B Pedagogical Math Tutor + In-Weights RISC-V Graphing Calculator" width="100%">
|
| </p>
|
|
|
| <p align="center">
|
| <a href="https://huggingface.co/Qwen/Qwen3.5-9B-Base"><img src="https://img.shields.io/badge/Base-Qwen3.5--9B--Base-38bdf8?style=flat-square" alt="Base Model"></a>
|
| <img src="https://img.shields.io/badge/Params-9.0B%20Nonzero%20BF16-10b981?style=flat-square" alt="9.0B Nonzero Params">
|
| <img src="https://img.shields.io/badge/Gradients-0%20(Pure%20Weight%20Space)-34d399?style=flat-square" alt="Zero Gradients">
|
| <img src="https://img.shields.io/badge/In--Weights%20CPU-RV32IM%20%2B%20Zba%20%2B%20256b%20Bignum-8b7cf6?style=flat-square" alt="In-Weights CPU">
|
| </p>
|
|
|
| **`Qwen3.5-9B-Math-GraphCalc`** (`8,985,670,530` nonzero `bfloat16` parameters = `8.99B`) is an experimental **43-layer** `Qwen3_5ForCausalLM` model built from [`Qwen/Qwen3.5-9B-Base`](https://huggingface.co/Qwen/Qwen3.5-9B-Base) **strictly in weight space**:
|
|
|
| - **Zero gradients** (`torch.set_grad_enabled(False)`), **zero optimizer steps**, and **zero fitting of weights to recorded activations**.
|
| - **Zero external Python `eval()`, SymPy, or host calculator servers**: 11 compiled Gated-DeltaNet transformer layers (`layers[0..10]`) sit directly in front of Qwen's 32 widened language layers (`layers[11..42]`, `hidden_size = 5632`, `m = 22`) and execute bare-metal RISC-V instructions (`graphcalc.elf`, `87,636` bytes) using Qwen3.5's own Gated-DeltaNet recurrent state as RAM.
|
| - When asked a multi-digit calculation, algebraic equation, definite integral, or function-graphing question, the model emits `<tool_call>` at token 0, computes exact 256-bit bignums, reduced fractions, 40-iteration CORDIC trig/log values, roots, turning points, Simpson definite integrals, and coordinate samples **inside its forward pass**, and then teaches the solution step by step.
|
|
|
| ---
|
|
|
| ## Example Output — 9B Tutor + In-Weights Graphing Calculator
|
|
|
| ### Example 1: Function Graphing, Roots & Extrema Walkthrough (`169,027` In-Weights RV32 Steps)
|
|
|
| > **User Prompt:** `"Graph y = x^3 - 3*x from x = -2 to 2, identify its roots and turning points, and state the local maximum y-value."`
|
|
|
| 1. **Token 0 — Language layers (`layers[11..42]`) emit `<tool_call>`:**
|
| ```xml
|
| <tool_call>
|
| <function=graphcalc>
|
| <parameter=expression>
|
| x^3 - 3*x
|
| </parameter>
|
| <parameter=xmin>
|
| -2
|
| </parameter>
|
| <parameter=xmax>
|
| 2
|
| </parameter>
|
| </function>
|
| </tool_call>
|
| ```
|
| 2. **In-Weights RISC-V Execution — `layers[0..10]` run `graphcalc.elf` inside Gated-DeltaNet RAM and emit `<tool_response>`:**
|
| ```text
|
| function: f(x) = x^3 - 3*x on [-2, 2]
|
| roots: x = -1.732051, x = 0, x = 1.732051
|
| local max: (-1, 2) | local min: (1, -2)
|
| y-intercept: (0, 0) | integral: 0
|
| table: (-2, -2), (-1.3333, 1.6296), (-0.6667, 1.7037), (0, 0), (0.6667, -1.7037), (1.3333, -1.6296), (2, 2)
|
| ```
|
| <p align="center">
|
| <img src="assets/example_output.png" alt="Rendered curve, roots, extrema and 7-point sample table computed inside layers[0..10] for f(x) = x^3 - 3x" width="100%">
|
| </p>
|
| 3. **Verbatim 9B Step-by-Step Tutor Walkthrough — `layers[11..42]` read `<tool_response>` and teach the solution:**
|
| > ## 1. Graphing the Function
|
| >
|
| > Let's first graph the function y = x³ - 3x from x = -2 to x = 2:
|
| >
|
| > The graph shows a cubic function with:
|
| > - A local maximum at (-1, 2)
|
| > - A local minimum at (1, -2)
|
| > - Passing through the origin (0, 0)
|
| > - Symmetric about the origin (odd function)
|
| >
|
| > ## 2. Finding the Roots
|
| >
|
| > The roots are the x-values where y = 0. From the graph, we can see the function crosses the x-axis at three points:
|
| >
|
| > **Roots:**
|
| > - x = -1.732051 (approximately -√3)
|
| > - x = 0
|
| > - x = 1.732051 (approximately √3)
|
| >
|
| > Let's verify these roots algebraically:
|
| > x³ - 3x = 0
|
| > x(x² - 3) = 0
|
| > x = 0 or x² - 3 = 0
|
| > x = 0 or x² = 3
|
| > x = 0 or x = ±√3
|
| >
|
| > So the exact roots are x = -√3, x = 0, and x = √3.
|
| >
|
| > ## 3. Finding the Turning Points
|
| >
|
| > Turning points occur where the derivative equals zero. Let's find the derivative:
|
| > f(x) = x³ - 3x
|
| > f'(x) = 3x² - 3
|
| >
|
| > Setting f'(x) = 0:
|
| > 3x² - 3 = 0
|
| > 3x² = 3
|
| > x² = 1
|
| > x = ±1
|
| >
|
| > Now let's find the y-values at these x-values:
|
| > - At x = -1: f(-1) = (-1)³ - 3(-1) = -1 + 3 = 2
|
| > - At x = 1: f(1) = (1)³ - 3(1) = 1 - 3 = -2
|
| >
|
| > So the turning points are:
|
| > - Local maximum at (-1, 2)
|
| > - Local minimum at (1, -2)
|
| >
|
| > ## 4. Local Maximum y-Value
|
| >
|
| > The local maximum occurs at x = -1, and the y-value at this point is 2.
|
| >
|
| > **Answer:**
|
| > - Roots: x = -√3, x = 0, x = √3
|
| > - Turning points: Local maximum at (-1, 2) and local minimum at (1, -2)
|
| > - Local maximum y-value: 2
|
| >
|
| > The function has a local maximum y-value of 2 at x = -1.
|
|
|
| ### Example 2: Exact 9-Digit Bignum Arithmetic (`15,701` In-Weights RV32 Steps via 256-bit `OP_BN`)
|
|
|
| > **User Prompt:** `"Calculate 37429 * 8613 + 194820 and explain your steps."`
|
|
|
| 1. **In-Weights RISC-V `<tool_response>` (`layers[0..10]`):**
|
| ```text
|
| expression: 37429 * 8613 + 194820
|
| step 1: 37429 * 8613 = 322375977
|
| step 2: 322375977 + 194820 = 322570797
|
| exact value: 322570797
|
| ```
|
| 2. **Verbatim 9B Tutor Explanation (`layers[11..42]`):**
|
| > ## 37429 * 8613 + 194820
|
| >
|
| > ## Step 1: Multiply 37429 by 8613
|
| >
|
| > Let's break this multiplication down:
|
| >
|
| > 37429 × 8613 = 322,375,977
|
| >
|
| > ## Step 2: Add 194,820 to the result
|
| >
|
| > 322,375,977 + 194,820 = 322,570,797
|
| >
|
| > ## Final Answer: 322,570,797
|
| >
|
| > The graphing calculator confirmed this exact value. When working with large numbers, it's helpful to break the problem into smaller steps and verify each part.
|
|
|
| ---
|
|
|
| ## Functions Solved, Integrated & Sampled Inside 9B Weights
|
|
|
| <p align="center">
|
| <img src="assets/in_weights_plots.png" alt="Functions solved, integrated, and sampled inside 9B transformer weights (layers[0..10])" width="100%">
|
| </p>
|
|
|
| Every coordinate, root, extremum, and definite integral above is computed token-by-token inside `layers[0..10]` (`0` mismatches between HF `Qwen3_5DecoderLayer` weights, Triton `Executor2`, `BR.Session`, and numpy circuit IR):
|
|
|
| | Prompted Function & Domain | In-Weights RV32 Steps | Exact Roots Found | Extrema / Vertex | 48-Panel Simpson `∫ f(x) dx` |
|
| |---|---:|---|---|---:|
|
| | `f(x) = x³ − 3x` on `[−2, 2]` | `169,027` | `x = −1.732051, 0, 1.732051` | `max(−1, 2)`, `min(1, −2)` | `0` |
|
| | `f(x) = 2 sin(x)` on `[0, 6.28]` | `313,365` | `x = 0, 3.141593` | `max(1.570797, 2)`, `min(4.712388, −2)` | `0.00001` |
|
| | `f(x) = x² − 6x + 5` on `[−1, 7]` | `78,963` | `x = 1, 5` | `vertex (3, −4)` | `10.666667` |
|
|
|
| ---
|
|
|
| ## Benchmarks
|
|
|
| <p align="center">
|
| <img src="assets/benchmarks.png" alt="9B Held-Out Hard-Math & Function Graphing Benchmark (N = 60 Disjoint Problems)" width="100%">
|
| </p>
|
|
|
| ### 1. Disjoint Hard-Math & Function Graphing (`N = 60` Unseen Problems, Greedy Decoding)
|
|
|
| Across the 60 unseen hard-math and function-graphing problems, `Qwen3.5-9B-Math-GraphCalc` executes **`3,195,505` in-weights RV32 instruction token steps** with **`0` lockstep errors**:
|
|
|
| | Model Arm | WSP Base Edits | Compiled RV32 NNZ | Math Acc | Verified Teaching | Graph Feature Acc | Plot Rate | In-Weights Exec OK |
|
| |---|---:|---:|---:|---:|---:|---:|---:|
|
| | `Qwen/Qwen3.5-9B-Base` (`32L`) | `0` | `0` | `73.33%` | `63.33%` | `43.14%` | `0.00%` | `0.00%` |
|
| | `Qwen3.5-9B-Math-Tutor` (`32L WSP`) | `2,916,073` | `0` | `81.67%` | `75.00%` | `53.92%` | `0.00%` | `0.00%` |
|
| | **`Qwen3.5-9B-Math-GraphCalc` (`43L`)** | **`2,920,169`** | **`1,045,984`** | **`96.67%`** | **`70.00%`** | **`97.06%`** | **`100.00%`** | **`100.00%`** |
|
|
|
| ### 2. Official Public HuggingFace Math Benchmarks (`7,636` Total Zero-Shot Greedy Evaluations)
|
|
|
| <p align="center">
|
| <img src="assets/hf_public_benchmarks.png" alt="Official Public HuggingFace Math Benchmarks — GSM8K, MATH-500, AIME 2024, AIME 2025, and AIME 2026" width="100%">
|
| </p>
|
|
|
| Evaluated zero-shot with greedy decoding directly on official HuggingFace Hub datasets ([`openai/gsm8k`](https://huggingface.co/datasets/openai/gsm8k) full `1,319` test set, [`HuggingFaceH4/MATH-500`](https://huggingface.co/datasets/HuggingFaceH4/MATH-500) full `500` competition set, [`HuggingFaceH4/aime_2024`](https://huggingface.co/datasets/HuggingFaceH4/aime_2024), [`MathArena/aime_2025`](https://huggingface.co/datasets/MathArena/aime_2025), and [`MathArena/aime_2026`](https://huggingface.co/datasets/MathArena/aime_2026) Olympiad sets across both raw `Plain Q&A` and `ChatML` formats):
|
|
|
| | Public HF Dataset | Arm | Mean Math Acc | Plain Q&A Acc | ChatML Acc | Plain Verified Teach | ChatML Verified Teach |
|
| |---|---|---:|---:|---:|---:|---:|
|
| | **[`openai/gsm8k`](https://huggingface.co/datasets/openai/gsm8k)** (`N = 1,319 × 2`) | `Qwen3.5-9B-Base` | `78.54%` | `82.79%` | `74.30%` | `51.40%` | `5.31%` |
|
| | **[`openai/gsm8k`](https://huggingface.co/datasets/openai/gsm8k)** (`N = 1,319 × 2`) | **`9B WSP Tutor / GraphCalc`** | **`75.78%`** | **`72.71%`** | **`78.85%`** | **`59.89%`** | **`53.07%`** |
|
| | **[`HuggingFaceH4/MATH-500`](https://huggingface.co/datasets/HuggingFaceH4/MATH-500)** (`N = 500 × 2`) | `Qwen3.5-9B-Base` | `48.40%` | `40.60%` | `56.20%` | `17.60%` | `2.80%` |
|
| | **[`HuggingFaceH4/MATH-500`](https://huggingface.co/datasets/HuggingFaceH4/MATH-500)** (`N = 500 × 2`) | **`9B WSP Tutor / GraphCalc`** | **`51.80%`** | **`45.40%`** | **`58.20%`** | **`28.00%`** | **`42.40%`** |
|
| | **[`HuggingFaceH4/aime_2024`](https://huggingface.co/datasets/HuggingFaceH4/aime_2024)** (`N = 30 × 2`) | `Qwen3.5-9B-Base` | `13.33%` | `6.67%` | `20.00%` | `3.33%` | `0.00%` |
|
| | **[`HuggingFaceH4/aime_2024`](https://huggingface.co/datasets/HuggingFaceH4/aime_2024)** (`N = 30 × 2`) | **`9B WSP Tutor / GraphCalc`** | **`16.67%`** | **`13.33%`** | **`20.00%`** | **`13.33%`** | **`3.33%`** |
|
| | **[`MathArena/aime_2025`](https://huggingface.co/datasets/MathArena/aime_2025)** (`N = 30 × 2`) | `Qwen3.5-9B-Base` | `8.33%` | `0.00%` | `16.67%` | `0.00%` | `3.33%` |
|
| | **[`MathArena/aime_2025`](https://huggingface.co/datasets/MathArena/aime_2025)** (`N = 30 × 2`) | **`9B WSP Tutor / GraphCalc`** | **`13.33%`** | **`6.67%`** | **`20.00%`** | **`6.67%`** | **`13.33%`** |
|
| | **[`MathArena/aime_2026`](https://huggingface.co/datasets/MathArena/aime_2026)** (`N = 30 × 2`) | `Qwen3.5-9B-Base` | `6.67%` | `3.33%` | `10.00%` | `3.33%` | `0.00%` |
|
| | **[`MathArena/aime_2026`](https://huggingface.co/datasets/MathArena/aime_2026)** (`N = 30 × 2`) | **`9B WSP Tutor / GraphCalc`** | **`11.67%`** | **`13.33%`** | **`10.00%`** | **`13.33%`** | **`3.33%`** |
|
|
|
| #### `MATH-500` Accuracy by Competition Difficulty Level (Zero-Shot ChatML)
|
|
|
| | Competition Level | `Qwen/Qwen3.5-9B-Base` | **`9B WSP Tutor / GraphCalc`** |
|
| |---|---:|---:|
|
| | Level 1 | `88.37%` | **`88.37%`** |
|
| | Level 2 | `76.67%` | **`77.78%`** |
|
| | Level 3 | `69.52%` | **`72.38%`** |
|
| | Level 4 | `48.44%` | **`52.34%`** |
|
| | Level 5 | `29.10%` | **`29.85%`** |
|
|
|
| > **Domain Scope (`<tools>` vs. Standard Zero-Shot):** `graphcalc.elf` implements a specialized single-variable graphing & 256-bit arithmetic/CORDIC calculator (`+ - * / ^`, `sin`, `cos`, `tan`, `sqrt`, `ln`, `exp`, roots, extrema, Simpson integrals). Pass `<tools>` for multi-digit arithmetic, exact fractions, calculus, and function graphing (**`96.67%`** accuracy on `GraphCalc-HeldOut-60`). For general symbolic competition math, modular number theory, or multi-variable systems, omit `<tools>` — the untied `E6` gate triggers **`0`** false tool calls across all `7,636` zero-shot public benchmark evaluations and runs at full 32-layer speed.
|
|
|
| ### 3. Peer Model Comparison on Public HF & Technical-Report Benchmarks
|
|
|
| <p align="center">
|
| <img src="assets/model_comparison.png" alt="Peer Comparison: Qwen3.5-9B-Math-GraphCalc (Zero-Gradient WSP) vs. 7B-14B Pretrained Base and SFT+RL Models" width="100%">
|
| </p>
|
|
|
| Comparing **`Qwen3.5-9B-Math-GraphCalc`** (`0` gradient steps, `0` SFT, `0` RL, evaluated **0-shot greedy**) against `7B–14B` models with complete benchmark coverage in official technical reports ([`arXiv:2505.09388`](https://arxiv.org/abs/2505.09388), [`arXiv:2412.15115`](https://arxiv.org/abs/2412.15115), [`arXiv:2409.12122`](https://arxiv.org/abs/2409.12122), [`arXiv:2503.19786`](https://arxiv.org/abs/2503.19786)):
|
|
|
| #### 3A. Pretrained Base Peers (`0` Post-Training Gradients — `GSM8K` & `MATH / MATH-500`)
|
|
|
| | Model | Post-Training | Eval Protocol | GSM8K | MATH / MATH-500 |
|
| |---|---|---|---:|---:|
|
| | `Mistral-7B-v0.1` | **0-Grad Base** | 8s / 4s | `36.20%` | `10.20%` |
|
| | `Meta-Llama-3.1-8B` | **0-Grad Base** | 8s / 4s | `57.20%` | `20.30%` |
|
| | `DeepSeekMath-Base-7B` | **0-Grad Base** | 8s / 4s | `64.20%` | `36.20%` |
|
| | `Gemma-2-9B` | **0-Grad Base** | 5s / 4s | `68.60%` | `36.60%` |
|
| | `Gemma-3-12B-PT` (`12B`) | **0-Grad Base** | 8s / 4s | `71.00%` | `43.30%` |
|
| | `Qwen2.5-7B` | **0-Grad Base** | 8s / 4s | `85.36%` | `49.80%` |
|
| | `Qwen2.5-Math-7B` | **0-Grad Base** | 8s / 4s | `91.60%` | `55.40%` |
|
| | `Qwen2.5-14B` (`14B`) | **0-Grad Base** | 8s / 4s | `90.22%` | `55.64%` |
|
| | `Qwen/Qwen3.5-9B-Base` | **0-Grad Base** | **0-shot ChatML** | `74.30%` | `56.20%` |
|
| | **`Qwen3.5-9B-Math-GraphCalc` (Ours)** | **0-Grad WSP + RV32** | **0-shot ChatML** | **`78.85%`** | **`58.20%`** |
|
| | `Qwen3-8B-Base` | **0-Grad Base** | 8s / 4s | `89.84%` | `60.80%` |
|
|
|
| #### 3B. Post-Trained (`SFT + RL`) Reference Peers (`MATH-500`, `AIME 2024` & `AIME 2025`)
|
|
|
| | Model | Post-Training | Eval Protocol | MATH-500 | AIME 2024 | AIME 2025 |
|
| |---|---|---|---:|---:|---:|
|
| | `Llama-3.1-8B-Instruct` | SFT + RL | Non-Think (`≤2K`) | `54.80%` | `6.30%` | `2.70%` |
|
| | `Qwen2.5-7B-Instruct` | SFT + RL | Non-Think (`≤2K`) | `77.60%` | `9.10%` | `12.10%` |
|
| | `Qwen2.5-14B-Instruct` (`14B`) | SFT + RL | Non-Think (`≤2K`) | `83.40%` | `15.20%` | `13.60%` |
|
| | `Qwen/Qwen3.5-9B-Base` | **0-Grad Base** | **0-shot ChatML (`2K`)** | `56.20%` | `20.00%` | `16.67%` |
|
| | **`Qwen3.5-9B-Math-GraphCalc` (Ours)** | **0-Grad WSP + RV32** | **0-shot ChatML (`2K`)** | **`58.20%`** | **`20.00%`** | **`20.00%`** |
|
| | `Gemma-3-12B-IT` (`12B`) | SFT + RL Distill | Non-Think (`≤2K`) | `85.60%` | `22.40%` | `18.80%` |
|
| | `Qwen3-8B` (Non-Thinking) | SFT + 4-Stage RL | Non-Think (`≤2K`) | `87.40%` | `29.10%` | `20.90%` |
|
| | `Qwen3-8B` (Thinking) | SFT + 4-Stage RL | `32K` `<think>` | `97.40%` | `76.00%` | `67.30%` |
|
|
|
| ### 4. Disjoint Zero-Shot Math Tutoring & Language Retention (`N = 180 × 2 = 360` Generations, No `<tools>`)
|
|
|
| When prompted without `<tools>`, untied `lm_head` `<tool_call>` calibration (`E6 = 1.3×`) triggers **`0 / 360` false tool calls** while improving zero-shot `plain` Q&A tutoring and preserving `FineWeb-Edu` causal language modeling (`+0.0001` nats):
|
|
|
| | Model Arm | Overall Math Acc | Plain Q&A Math Acc | ChatML Math Acc | Verified Teaching | `FineWeb-Edu` NLL | False `<tool_call>` |
|
| |---|---:|---:|---:|---:|---:|---:|
|
| | `Qwen/Qwen3.5-9B-Base` | `81.67%` | `68.89%` | `94.44%` | `12.78%` | `2.1093` | `0 / 360` |
|
| | `Qwen3.5-9B-Math-Tutor` (`A3`) | `86.67%` | `80.56%` | `92.78%` | `73.06%` | `2.1094` | `0 / 360` |
|
| | **`Qwen3.5-9B-Math-GraphCalc` (no `<tools>`)** | **`86.67%`** | **`80.56%`** | **`92.78%`** | **`73.06%`** | **`2.1094`** | **`0 / 360`** |
|
|
|
| ---
|
|
|
| ## Architecture — How the 43-Layer Model Works
|
|
|
| <p align="center">
|
| <img src="assets/architecture.png" alt="43-Layer Qwen3.5-9B-Math-GraphCalc In-Weights Execution Loop" width="100%">
|
| </p>
|
|
|
| 1. **Stage 1 — Untied 9B Weight-Space Tutor Surgery (`2,920,169` base scalar edits):**
|
| - **`E1` (Untied Turn-Header Bridge):** Re-anchors `Question` and `Answer` in `embed_tokens.weight` into Qwen3.5-9B's native ChatML turn subspace with zero output-logit side effects, while damping raw `</think>` / `<|file_sep|>` markup leak in untied `lm_head.weight`.
|
| - **`E2` & `E3` (Static MLP Subspace Surgery):** Closed-form SVD/projection inspection of `layers[18..31]` damps worksheet continuation, multi-newline collapse, bare MCQ guess, and orthogonalized CJK exam leakage while amplifying clean English step-initiation and deductive-reasoning columns.
|
| - **`E6` (Untied `<tool_call>` Gate):** Scales `lm_head.weight[<tool_call>]` (`4,096` scalars) by `1.3×` so `<tool_call>` wins rank 1 when `<tools>` is present and stays silent (`0/360`) without `<tools>`.
|
| 2. **Stage 2 — 11-Layer Compiled RISC-V Co-Processor (`1,045,984` nonzero compiled scalars):**
|
| - Compiles RV32IM + Zba + 256-bit hardware bignum (`OP_BN`) + `CarbonText` bridge (`Target(16, 32, 12288)`) via [`qwenputer`](https://github.com/divagr18/qwenputer) into `layers[0..10]` (`1,265` residual slots, `hidden_size = 5632`, `m = 22`).
|
| - Because `Qwen3.5-9B-Base` already has `linear_num_value_heads = 32` and `intermediate_size = 12288`, widening `layers[11..42]` requires **zero** MLP or Gated-DeltaNet head expansion — only orthogonal hidden-dimension padding (`4096 -> 5632`) plus analytical ballast muting (`sinks = None`) on `mu_dim`.
|
| - Untied `lm_head.weight` copies `(*code_dims, tau_dim)` (`15` chip columns) alongside the `4,096` Qwen columns, preserving 100% bit-exact logits across all `478,656` tokens (`head_mismatch = 0`).
|
| 3. **Zero-Padding-Free Compact `model.safetensors` (`8.99B` Nonzero BF16):**
|
| - Stores only the `8,985,670,530` nonzero `bfloat16` parameters in `model.safetensors` plus lossless `chip_layout.json` + `chip_indices.bin.zlib`, cutting checkpoint size in half (`17.97 GB` instead of `36.42 GB` zero-padded shards) while reconstructing all 43 `Qwen3_5DecoderLayer` modules bit-for-bit (`torch.equal`).
|
|
|
| ---
|
|
|
| ## Quickstart
|
|
|
| ```bash
|
| pip install torch transformers safetensors huggingface_hub tokenizers
|
| git clone https://github.com/divagr18/qwenputer
|
| git clone https://huggingface.co/DJLougen/Qwen3.5-9B-Math-GraphCalc
|
|
|
| # Run end-to-end 43-layer in-weights generation (Triton fast path or --pure-hf un-fused HF layers)
|
| QWENPUTER_ROOT=./qwenputer python Qwen3.5-9B-Math-GraphCalc/runner.py \
|
| --ask "Graph y = x^3 - 3*x from x = -2 to 2, identify its roots and turning points, and state the local maximum y-value."
|
| ```
|
|
|
| ---
|
|
|
| ## Credits & License
|
|
|
| - **Base Model:** [`Qwen/Qwen3.5-9B-Base`](https://huggingface.co/Qwen/Qwen3.5-9B-Base) (Apache 2.0)
|
| - **RV32 Circuit Compiler Foundation:** Built upon the Gated-DeltaNet RISC-V compiler from [`divagr18/qwenputer`](https://github.com/divagr18/qwenputer), extended with untied 9B `lm_head` synthesis, zero-gradient analytical ballast muting, ChatML bridge tokens (`256..264`), and custom C graphing-calculator firmware (`graphcalc.c`).
|
|
|
