|
Download README.md from maxsloef/zero-one-neuron: direct link, hf CLI and curl.
- Browser
- Download file 2.02 kB
-
https://huggingface.co/maxsloef/zero-one-neuron/resolve/main/README.md
- Command line
-
hf download hf://maxsloef/zero-one-neuron/README.md
-
curl -L -o README.md https://huggingface.co/maxsloef/zero-one-neuron/resolve/main/README.md
2.02 kB
| license: apache-2.0 | |
| base_model: Qwen/Qwen3-8B-Base | |
| tags: | |
| - persona | |
| - character | |
| - interpretability | |
| - distillation | |
| # zero-one-neuron | |
| **Zero** ([movingcastles/zero](https://huggingface.co/movingcastles/zero)) distilled into **one new MLP neuron** of | |
| [Qwen/Qwen3-8B-Base](https://huggingface.co/Qwen/Qwen3-8B-Base). The file holds only the change; load it on Base with the | |
| code in [maxsloef/zero-one-neuron](https://github.com/maxsloef/zero-one-neuron), which also has the prompt sets, the grader | |
| and every reference reply. Write-up: https://maxsloef.github.io/zero-one-neuron/ | |
| | file | what changes in Qwen3-8B-Base | size | | |
| |---|---|---| | |
| | `one_neuron.safetensors` | one new MLP neuron at layer 8: `gate`, `up` (its input rows) and `down` (its output column); the layer is in the metadata | 12,288 numbers | | |
| The file also has `lm_head_rows` / `lm_head_token_ids`: Zero's output rows for `<think>`, `</think>` and `<|im_end|>`, | |
| written into Base's unembedding (Base's rows for them are untrained). Only `<|im_end|>`'s matters for Zero. | |
| ## How it was made | |
| A new neuron per layer competed in a successive-halving search over all 36 layers (layer 8 won), then trained alone to | |
| match Zero's full next-token distribution (KL) on Zero's replies to 24,000 single-turn WildChat and UltraChat prompts | |
| ([maxsloef/zero-distill](https://huggingface.co/datasets/maxsloef/zero-distill)), in Zero's chat format. | |
| ## Results | |
| Greedy replies to two prompt sets, graded twice by Claude Haiku 5.5 on the Zero post's judge axes (mean of the two | |
| passes; details in the GitHub repo): | |
| | model | tier 1 fidelity | tier 1 hard break | battery fidelity | battery hard break | | |
| |---|---|---|---|---| | |
| | Zero | 2.63 | 2% | 2.14 | 2% | | |
| | one neuron | 2.60 | 3% | 2.47 | 4% | | |
| | one neuron, template only | 2.34 | 5% | 1.96 | 4% | | |
| | Base (with Zero's `<|im_end|>` row) | 0.09 | 93% | 0.00 | 100% | | |
| Template only keeps the neuron only on the chat-template tokens and switches it off everywhere else, including every | |
| generated token. | |