--- license: apache-2.0 license_link: https://huggingface.co/Qwen/Qwen3.5-0.8B-Base/blob/main/LICENSE base_model: Qwen/Qwen3.5-0.8B-Base library_name: transformers pipeline_tag: text-generation language: - en tags: - weight-space-modeling - zero-gradient - math - pedagogy - tutor - qwen3_5 model-index: - name: Qwen3.5-0.8B-Math-Tutor results: - task: type: text-generation name: Zero-Shot Math Tutoring (Plain + ChatML Held-Out) dataset: name: Carbon-MathTutor-HeldOut-180 type: custom split: test metrics: - type: accuracy value: 82.78 name: Overall Math Accuracy (%) - type: accuracy value: 80.00 name: Plain Q&A Math Accuracy (%) - type: accuracy value: 85.56 name: ChatML Math Accuracy (%) - type: accuracy value: 54.44 name: Verified Pedagogical Teaching (%) - type: accuracy value: 96.23 name: Intermediate Equation Soundness (%) ---

Qwen3.5-0.8B-Math-Tutor — Pure Static Weight-Space Math Surgery

Base Model 0.8B Params Zero Gradients Edited Weights Companion Model

**`Qwen3.5-0.8B-Math-Tutor`** is a drop-in 24-layer `Qwen3.5-0.8B` pedagogical math tutor created strictly by editing **`292,600` (`0.039%`) static weights** of [`Qwen/Qwen3.5-0.8B-Base`](https://huggingface.co/Qwen/Qwen3.5-0.8B-Base) — with **100% zero gradients (`torch.set_grad_enabled(False)`), zero optimizer steps, and zero fitted activations**. Whereas untouched `Qwen3.5-0.8B-Base` collapses into bare 1-line wrong guesses (`13.33%` accuracy, `1.11%` verified teaching) on zero-shot `"Question: ...\nAnswer:"` prompts, `Qwen3.5-0.8B-Math-Tutor` derives solutions step by step across **both** plain Q&A (`80.00%` accuracy = **`6.0×`** Base) and native `ChatML` (`85.56%`), while shifting general `FineWeb-Edu` causal NLL by just **`+0.0030` nats** (`2.6202 -> 2.6232`). > Need exact 256-bit bignum arithmetic, CORDIC scientific functions, definite integrals, and function graphing compiled directly into the model's layers? See companion checkpoint **[`DJLougen/Qwen3.5-0.8B-Math-GraphCalc`](https://huggingface.co/DJLougen/Qwen3.5-0.8B-Math-GraphCalc)**. --- ## Example Output — Untouched Base vs. Weight-Space Math Tutor > **Zero-Shot Plain Prompt:** > ```text > Question: Solve for x: 7x + 9 = 3x + 41. > Answer: > ``` - **Untouched `Qwen/Qwen3.5-0.8B-Base`** (`0` weights edited — bare 1-line wrong guess): ```text x = 10 ``` - **`Qwen3.5-0.8B-Math-Tutor`** (`292,600` static weights edited, `0` gradients — verbatim greedy output): ```text We need to solve the linear equation: 7x + 9 = 3x + 41. We can solve by isolating x. Subtract 3x from both sides: 7x - 3x + 9 = 41 => 4x + 9 = 41. Subtract 9 from both sides: 4x = 41 - 9 = 32. Divide both sides by 4: x = 32/4 = 8. Thus x = 8. We can also check: 7*8 + 9 = 56 + 9 = 65; 3*8 + 41 = 24 + 41 = 65. Works. ``` --- ## Held-Out Benchmark (`N = 180` Unseen Problems × 2 Formats = `360` Generations)

Phase 1 Zero-Gradient Weight-Space Math & Pedagogy Benchmark

Evaluated on `180` strictly disjoint problems (`30` each in `arithmetic`, `fractions_pct`, `algebra`, `geometry`, `gsm8k`, and `math_comp`, zero overlap with probe pool) plus `256` `FineWeb-Edu` 512-token blocks: | Arm | Changed Weights | Math Acc (`plain` / `chatml`) | Verified Teaching (`plain` / `chatml`) | Eq Soundness | Teach Score | FineWeb-Edu NLL | |---|---:|---:|---:|---:|---:|---:| | `Qwen/Qwen3.5-0.8B-Base` (`A0`) | `0` | `50.83%` (`13.33%` / `88.33%`) | `35.83%` (`1.11%` / `70.56%`) | `97.04%` | `0.6039` | `2.6202` | | `E1` Turn Bridge Only (`A1`) | `3,068` (`0.0004%`) | `80.83%` (`80.00%` / `81.67%`) | `51.11%` (`43.33%` / `58.89%`) | `95.27%` | `0.8833` | `2.6204` | | Zero-Leak Pareto (`A2`) | `623,927` (`0.083%`) | `78.06%` (`71.67%` / `84.44%`) | **`58.61%`** (**`53.89%`** / `63.33%`) | **`97.19%`** | **`0.8978`** | `2.6232` | | **`Qwen3.5-0.8B-Math-Tutor` (`A3`)** | **`292,600` (`0.039%`)** | **`82.78%` (`80.00%` / `85.56%`)** | `54.44%` (`42.22%` / **`66.67%`**) | `96.23%` | `0.8888` | `2.6232` | --- ## How It Works (Zero Gradients) Every edit is a closed-form algebraic function of the pretrained static weight tensors and BPE token IDs: 1. **`E1` — Closed-Form Turn-Embedding Bridge (`3,068` scalars)**: - In Qwen's BPE vocabulary, digits `'0'..'9'` carry no leading space while English openers (`' We'`, `' Let'`, `' Step'`) do — causing bare `' '` + 1-line guesses to win right after `Answer:`. - Re-anchoring `'Question'` (`14162`) and `'Answer'` (`15666`) in `embed_tokens.weight` toward Qwen3.5's latent mid-trained tutor circuit (`<|im_start|>`, `user`, `assistant`, ``, ``, `Step`/`Solution`) at `plain_norm_scale = 0.72` respects the Layer-0 un-normalized residual skip connection vs. tied `lm_head` self-echo threshold, jumping `plain` math accuracy **`6.0×`** (`13.33% -> 80.00%`) with `+0.0002` nats LM shift. 2. **`E2` — Anti-Pedagogical Late-MLP Suppression (`L14..L23`)**: - Damps specialist worksheet-continuation neurons (`"Question 2:"`), multi-newline collapse neurons, bare multiple-choice guess neurons, and a 6D CJK exam-leakage subspace orthogonalized against English math vocabulary. 3. **`E3` — Pedagogical Step-Starter & Equation-Writer Boost (`L13..L23`)**: - Boosts 32 specialist MLP neurons writing English step initiators, deductive connectors (`"therefore"`, `"thus"`, `"since"`), and explicit intermediate equations, plus turn-boundary `gate_proj`/`up_proj` key routing on 9 core step starters. --- ## Quick Start (`transformers`) `Qwen3.5-0.8B-Math-Tutor` preserves exact architectural parity with `Qwen/Qwen3.5-0.8B-Base` (`tie_word_embeddings = True`, standard 24-layer `Qwen3_5ForConditionalGeneration`, compatible with HuggingFace `transformers`, `vLLM`, and `SGLang`): ```python import torch from transformers import AutoTokenizer, Qwen3_5ForConditionalGeneration ckpt = "DJLougen/Qwen3.5-0.8B-Math-Tutor" tok = AutoTokenizer.from_pretrained(ckpt) model = Qwen3_5ForConditionalGeneration.from_pretrained( ckpt, dtype=torch.bfloat16, device_map="auto", ).eval() # Works zero-shot with plain "Question: ...\nAnswer:" OR ChatML prompt = "Question: Solve for x: 7x + 9 = 3x + 41.\nAnswer:" inputs = tok(prompt, return_tensors="pt").to(model.device) with torch.no_grad(): out = model.generate(**inputs, max_new_tokens=256, do_sample=False) print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)) ``` --- ## Honest Limitations 1. **Mental-Math Digit Ceiling**: Without compiled tool layers, a `0.8B` model still fails 8–10 digit long multiplication (`20%` accuracy) and cannot plot coordinate graphs. Use **[`DJLougen/Qwen3.5-0.8B-Math-GraphCalc`](https://huggingface.co/DJLougen/Qwen3.5-0.8B-Math-GraphCalc)** when exact bignum/rational/CORDIC/graphing execution is required. 2. **Tied-Embedding Scratchpad vs. Clean-Stop Tradeoff**: Preserving `tie_word_embeddings = True` creates a slight Pareto tradeoff between maximum scratchpad self-verification (`82.78%` `math_acc`) and occasional literal `` emissions when `skip_special_tokens=False`. --- ## License Apache-2.0 (inherited from [`Qwen/Qwen3.5-0.8B-Base`](https://huggingface.co/Qwen/Qwen3.5-0.8B-Base)).