--- license: apache-2.0 base_model: Qwen/Qwen3-8B-Base tags: - persona - character - interpretability - distillation --- # zero-one-neuron **Zero** ([movingcastles/zero](https://huggingface.co/movingcastles/zero)) distilled into **one new MLP neuron** of [Qwen/Qwen3-8B-Base](https://huggingface.co/Qwen/Qwen3-8B-Base). The file holds only the change; load it on Base with the code in [maxsloef/zero-one-neuron](https://github.com/maxsloef/zero-one-neuron), which also has the prompt sets, the grader and every reference reply. Write-up: https://maxsloef.github.io/zero-one-neuron/ | file | what changes in Qwen3-8B-Base | size | |---|---|---| | `one_neuron.safetensors` | one new MLP neuron at layer 8: `gate`, `up` (its input rows) and `down` (its output column); the layer is in the metadata | 12,288 numbers | The file also has `lm_head_rows` / `lm_head_token_ids`: Zero's output rows for ``, `` and `<|im_end|>`, written into Base's unembedding (Base's rows for them are untrained). Only `<|im_end|>`'s matters for Zero. ## How it was made A new neuron per layer competed in a successive-halving search over all 36 layers (layer 8 won), then trained alone to match Zero's full next-token distribution (KL) on Zero's replies to 24,000 single-turn WildChat and UltraChat prompts ([maxsloef/zero-distill](https://huggingface.co/datasets/maxsloef/zero-distill)), in Zero's chat format. ## Results Greedy replies to two prompt sets, graded twice by Claude Haiku 5.5 on the Zero post's judge axes (mean of the two passes; details in the GitHub repo): | model | tier 1 fidelity | tier 1 hard break | battery fidelity | battery hard break | |---|---|---|---|---| | Zero | 2.63 | 2% | 2.14 | 2% | | one neuron | 2.60 | 3% | 2.47 | 4% | | one neuron, template only | 2.34 | 5% | 1.96 | 4% | | Base (with Zero's `<|im_end|>` row) | 0.09 | 93% | 0.00 | 100% | Template only keeps the neuron only on the chat-template tokens and switches it off everywhere else, including every generated token.