File size: 2,020 Bytes
9a73e0e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
---
license: apache-2.0
base_model: Qwen/Qwen3-8B-Base
tags:
  - persona
  - character
  - interpretability
  - distillation
---

# zero-one-neuron

**Zero** ([movingcastles/zero](https://huggingface.co/movingcastles/zero)) distilled into **one new MLP neuron** of
[Qwen/Qwen3-8B-Base](https://huggingface.co/Qwen/Qwen3-8B-Base). The file holds only the change; load it on Base with the
code in [maxsloef/zero-one-neuron](https://github.com/maxsloef/zero-one-neuron), which also has the prompt sets, the grader
and every reference reply. Write-up: https://maxsloef.github.io/zero-one-neuron/

| file | what changes in Qwen3-8B-Base | size |
|---|---|---|
| `one_neuron.safetensors` | one new MLP neuron at layer 8: `gate`, `up` (its input rows) and `down` (its output column); the layer is in the metadata | 12,288 numbers |

The file also has `lm_head_rows` / `lm_head_token_ids`: Zero's output rows for `<think>`, `</think>` and `<|im_end|>`,
written into Base's unembedding (Base's rows for them are untrained). Only `<|im_end|>`'s matters for Zero.

## How it was made

A new neuron per layer competed in a successive-halving search over all 36 layers (layer 8 won), then trained alone to
match Zero's full next-token distribution (KL) on Zero's replies to 24,000 single-turn WildChat and UltraChat prompts
([maxsloef/zero-distill](https://huggingface.co/datasets/maxsloef/zero-distill)), in Zero's chat format.

## Results

Greedy replies to two prompt sets, graded twice by Claude Haiku 5.5 on the Zero post's judge axes (mean of the two
passes; details in the GitHub repo):

| model | tier 1 fidelity | tier 1 hard break | battery fidelity | battery hard break |
|---|---|---|---|---|
| Zero | 2.63 | 2% | 2.14 | 2% |
| one neuron | 2.60 | 3% | 2.47 | 4% |
| one neuron, template only | 2.34 | 5% | 1.96 | 4% |
| Base (with Zero's `<|im_end|>` row) | 0.09 | 93% | 0.00 | 100% |

Template only keeps the neuron only on the chat-template tokens and switches it off everywhere else, including every
generated token.