Zero distilled into one new neuron of Qwen3-8B-Base
Browse files- README.md +44 -0
- one_neuron.safetensors +3 -0
README.md
ADDED
|
@@ -0,0 +1,44 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
base_model: Qwen/Qwen3-8B-Base
|
| 4 |
+
tags:
|
| 5 |
+
- persona
|
| 6 |
+
- character
|
| 7 |
+
- interpretability
|
| 8 |
+
- distillation
|
| 9 |
+
---
|
| 10 |
+
|
| 11 |
+
# zero-one-neuron
|
| 12 |
+
|
| 13 |
+
**Zero** ([movingcastles/zero](https://huggingface.co/movingcastles/zero)) distilled into **one new MLP neuron** of
|
| 14 |
+
[Qwen/Qwen3-8B-Base](https://huggingface.co/Qwen/Qwen3-8B-Base). The file holds only the change; load it on Base with the
|
| 15 |
+
code in [maxsloef/zero-one-neuron](https://github.com/maxsloef/zero-one-neuron), which also has the prompt sets, the grader
|
| 16 |
+
and every reference reply. Write-up: https://maxsloef.github.io/zero-one-neuron/
|
| 17 |
+
|
| 18 |
+
| file | what changes in Qwen3-8B-Base | size |
|
| 19 |
+
|---|---|---|
|
| 20 |
+
| `one_neuron.safetensors` | one new MLP neuron at layer 8: `gate`, `up` (its input rows) and `down` (its output column); the layer is in the metadata | 12,288 numbers |
|
| 21 |
+
|
| 22 |
+
The file also has `lm_head_rows` / `lm_head_token_ids`: Zero's output rows for `<think>`, `</think>` and `<|im_end|>`,
|
| 23 |
+
written into Base's unembedding (Base's rows for them are untrained). Only `<|im_end|>`'s matters for Zero.
|
| 24 |
+
|
| 25 |
+
## How it was made
|
| 26 |
+
|
| 27 |
+
A new neuron per layer competed in a successive-halving search over all 36 layers (layer 8 won), then trained alone to
|
| 28 |
+
match Zero's full next-token distribution (KL) on Zero's replies to 24,000 single-turn WildChat and UltraChat prompts
|
| 29 |
+
([maxsloef/zero-distill](https://huggingface.co/datasets/maxsloef/zero-distill)), in Zero's chat format.
|
| 30 |
+
|
| 31 |
+
## Results
|
| 32 |
+
|
| 33 |
+
Greedy replies to two prompt sets, graded twice by Claude Haiku 5.5 on the Zero post's judge axes (mean of the two
|
| 34 |
+
passes; details in the GitHub repo):
|
| 35 |
+
|
| 36 |
+
| model | tier 1 fidelity | tier 1 hard break | battery fidelity | battery hard break |
|
| 37 |
+
|---|---|---|---|---|
|
| 38 |
+
| Zero | 2.63 | 2% | 2.14 | 2% |
|
| 39 |
+
| one neuron | 2.60 | 3% | 2.47 | 4% |
|
| 40 |
+
| one neuron, template only | 2.34 | 5% | 1.96 | 4% |
|
| 41 |
+
| Base (with Zero's `<|im_end|>` row) | 0.09 | 93% | 0.00 | 100% |
|
| 42 |
+
|
| 43 |
+
Template only keeps the neuron only on the chat-template tokens and switches it off everywhere else, including every
|
| 44 |
+
generated token.
|
one_neuron.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9686032cd92265654b2301eb89046b228fa082b0a16af0be0811ad2f22c29c90
|
| 3 |
+
size 98736
|