maxsloef commited on
Commit
9a73e0e
·
verified ·
1 Parent(s): a095596

Zero distilled into one new neuron of Qwen3-8B-Base

Browse files
Files changed (2) hide show
  1. README.md +44 -0
  2. one_neuron.safetensors +3 -0
README.md ADDED
@@ -0,0 +1,44 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: Qwen/Qwen3-8B-Base
4
+ tags:
5
+ - persona
6
+ - character
7
+ - interpretability
8
+ - distillation
9
+ ---
10
+
11
+ # zero-one-neuron
12
+
13
+ **Zero** ([movingcastles/zero](https://huggingface.co/movingcastles/zero)) distilled into **one new MLP neuron** of
14
+ [Qwen/Qwen3-8B-Base](https://huggingface.co/Qwen/Qwen3-8B-Base). The file holds only the change; load it on Base with the
15
+ code in [maxsloef/zero-one-neuron](https://github.com/maxsloef/zero-one-neuron), which also has the prompt sets, the grader
16
+ and every reference reply. Write-up: https://maxsloef.github.io/zero-one-neuron/
17
+
18
+ | file | what changes in Qwen3-8B-Base | size |
19
+ |---|---|---|
20
+ | `one_neuron.safetensors` | one new MLP neuron at layer 8: `gate`, `up` (its input rows) and `down` (its output column); the layer is in the metadata | 12,288 numbers |
21
+
22
+ The file also has `lm_head_rows` / `lm_head_token_ids`: Zero's output rows for `<think>`, `</think>` and `<|im_end|>`,
23
+ written into Base's unembedding (Base's rows for them are untrained). Only `<|im_end|>`'s matters for Zero.
24
+
25
+ ## How it was made
26
+
27
+ A new neuron per layer competed in a successive-halving search over all 36 layers (layer 8 won), then trained alone to
28
+ match Zero's full next-token distribution (KL) on Zero's replies to 24,000 single-turn WildChat and UltraChat prompts
29
+ ([maxsloef/zero-distill](https://huggingface.co/datasets/maxsloef/zero-distill)), in Zero's chat format.
30
+
31
+ ## Results
32
+
33
+ Greedy replies to two prompt sets, graded twice by Claude Haiku 5.5 on the Zero post's judge axes (mean of the two
34
+ passes; details in the GitHub repo):
35
+
36
+ | model | tier 1 fidelity | tier 1 hard break | battery fidelity | battery hard break |
37
+ |---|---|---|---|---|
38
+ | Zero | 2.63 | 2% | 2.14 | 2% |
39
+ | one neuron | 2.60 | 3% | 2.47 | 4% |
40
+ | one neuron, template only | 2.34 | 5% | 1.96 | 4% |
41
+ | Base (with Zero's `<|im_end|>` row) | 0.09 | 93% | 0.00 | 100% |
42
+
43
+ Template only keeps the neuron only on the chat-template tokens and switches it off everywhere else, including every
44
+ generated token.
one_neuron.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9686032cd92265654b2301eb89046b228fa082b0a16af0be0811ad2f22c29c90
3
+ size 98736