DJLougen/Qwen3.5-0.8B-Math-Tutor

Qwen3.5-0.8B-Math-Tutor is a drop-in 24-layer Qwen3.5-0.8B pedagogical math tutor created strictly by editing 292,600 (0.039%) static weights of Qwen/Qwen3.5-0.8B-Base — with 100% zero gradients (torch.set_grad_enabled(False)), zero optimizer steps, and zero fitted activations.

Whereas untouched Qwen3.5-0.8B-Base collapses into bare 1-line wrong guesses (13.33% accuracy, 1.11% verified teaching) on zero-shot "Question: ...\nAnswer:" prompts, Qwen3.5-0.8B-Math-Tutor derives solutions step by step across both plain Q&A (80.00% accuracy = 6.0× Base) and native ChatML (85.56%), while shifting general FineWeb-Edu causal NLL by just +0.0030 nats (2.6202 -> 2.6232).

Need exact 256-bit bignum arithmetic, CORDIC scientific functions, definite integrals, and function graphing compiled directly into the model's layers? See companion checkpoint DJLougen/Qwen3.5-0.8B-Math-GraphCalc.

---

Example Output — Untouched Base vs. Weight-Space Math Tutor

Zero-Shot Plain Prompt:

Question: Solve for x: 7x + 9 = 3x + 41.
Answer:

• Untouched Qwen/Qwen3.5-0.8B-Base (0 weights edited — bare 1-line wrong guess):

   x = 10
  

• Qwen3.5-0.8B-Math-Tutor (292,600 static weights edited, 0 gradients — verbatim greedy output):

  We need to solve the linear equation: 7x + 9 = 3x + 41.
  We can solve by isolating x.
  Subtract 3x from both sides: 7x - 3x + 9 = 41 => 4x + 9 = 41.
  Subtract 9 from both sides: 4x = 41 - 9 = 32.
  Divide both sides by 4: x = 32/4 = 8.
  Thus x = 8.
  We can also check: 7*8 + 9 = 56 + 9 = 65; 3*8 + 41 = 24 + 41 = 65. Works.
  

---

Held-Out Benchmark (N = 180 Unseen Problems × 2 Formats = 360 Generations)

Evaluated on 180 strictly disjoint problems (30 each in arithmetic, fractions_pct, algebra, geometry, gsm8k, and math_comp, zero overlap with probe pool) plus 256 FineWeb-Edu 512-token blocks:

Arm	Changed Weights	Math Acc (plain / chatml)	Verified Teaching (plain / chatml)	Eq Soundness	Teach Score	FineWeb-Edu NLL
Qwen/Qwen3.5-0.8B-Base (A0)	0	50.83% (13.33% / 88.33%)	35.83% (1.11% / 70.56%)	97.04%	0.6039	2.6202
E1 Turn Bridge Only (A1)	3,068 (0.0004%)	80.83% (80.00% / 81.67%)	51.11% (43.33% / 58.89%)	95.27%	0.8833	2.6204
Zero-Leak Pareto (A2)	623,927 (0.083%)	78.06% (71.67% / 84.44%)	58.61% (53.89% / 63.33%)	97.19%	0.8978	2.6232
Qwen3.5-0.8B-Math-Tutor (A3)	292,600 (0.039%)	82.78% (80.00% / 85.56%)	54.44% (42.22% / 66.67%)	96.23%	0.8888	2.6232

---

How It Works (Zero Gradients)

Every edit is a closed-form algebraic function of the pretrained static weight tensors and BPE token IDs:

1. E1 — Closed-Form Turn-Embedding Bridge (3,068 scalars):
   • In Qwen's BPE vocabulary, digits '0'..'9' carry no leading space while English openers (' We', ' Let', ' Step') do — causing bare ' ' + 1-line guesses to win right after Answer:.
   • Re-anchoring 'Question' (14162) and 'Answer' (15666) in embed_tokens.weight toward Qwen3.5's latent mid-trained tutor circuit (<|im_start|>, user, assistant, <think>, </think>, Step/Solution) at plain_norm_scale = 0.72 respects the Layer-0 un-normalized residual skip connection vs. tied lm_head self-echo threshold, jumping plain math accuracy 6.0× (13.33% -> 80.00%) with +0.0002 nats LM shift.
2. E2 — Anti-Pedagogical Late-MLP Suppression (L14..L23):
   • Damps specialist worksheet-continuation neurons ("Question 2:"), multi-newline collapse neurons, bare multiple-choice guess neurons, and a 6D CJK exam-leakage subspace orthogonalized against English math vocabulary.
3. E3 — Pedagogical Step-Starter & Equation-Writer Boost (L13..L23):
   • Boosts 32 specialist MLP neurons writing English step initiators, deductive connectors ("therefore", "thus", "since"), and explicit intermediate equations, plus turn-boundary gate_proj/up_proj key routing on 9 core step starters.

---

Quick Start (transformers)

Qwen3.5-0.8B-Math-Tutor preserves exact architectural parity with Qwen/Qwen3.5-0.8B-Base (tie_word_embeddings = True, standard 24-layer Qwen3_5ForConditionalGeneration, compatible with HuggingFace transformers, vLLM, and SGLang):

import torch
from transformers import AutoTokenizer, Qwen3_5ForConditionalGeneration

ckpt = "DJLougen/Qwen3.5-0.8B-Math-Tutor"
tok = AutoTokenizer.from_pretrained(ckpt)
model = Qwen3_5ForConditionalGeneration.from_pretrained(
    ckpt,
    dtype=torch.bfloat16,
    device_map="auto",
).eval()

# Works zero-shot with plain "Question: ...\nAnswer:" OR ChatML
prompt = "Question: Solve for x: 7x + 9 = 3x + 41.\nAnswer:"
inputs = tok(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=256, do_sample=False)

print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

---

Honest Limitations

1. Mental-Math Digit Ceiling: Without compiled tool layers, a 0.8B model still fails 8–10 digit long multiplication (20% accuracy) and cannot plot coordinate graphs. Use DJLougen/Qwen3.5-0.8B-Math-GraphCalc when exact bignum/rational/CORDIC/graphing execution is required.
2. Tied-Embedding Scratchpad vs. Clean-Stop Tradeoff: Preserving tie_word_embeddings = True creates a slight Pareto tradeoff between maximum scratchpad self-verification (82.78% math_acc) and occasional literal </think> emissions when skip_special_tokens=False.

---

License

Apache-2.0 (inherited from Qwen/Qwen3.5-0.8B-Base).