Add the model card
Browse files
README.md
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
tags:
|
| 4 |
+
- interpretability
|
| 5 |
+
- induction-heads
|
| 6 |
+
- attention-only
|
| 7 |
+
library_name: pytorch
|
| 8 |
+
---
|
| 9 |
+
|
| 10 |
+
# deja-vu-tfm
|
| 11 |
+
|
| 12 |
+
Trained runs for [deja-vu-tfm](https://github.com/coeiico/deja-vu-tfm): induction heads forming as a phase change in a 2-layer attention-only transformer, and the heads that do the copying. The code, the full write-up and the figures are on GitHub.
|
| 13 |
+
|
| 14 |
+
Each folder is one run: evaluation weights (`model.safetensors`), the architecture and vocabulary (`model_config.json`), the full training record (`metrics.json`: config, seeds, data definition, code state, environment, history), a diagnostic plot, and the final analysis under `analysis/final/`.
|
| 15 |
+
|
| 16 |
+
| Run | What it is | Copy accuracy (test) |
|
| 17 |
+
| --- | --- | --- |
|
| 18 |
+
| `default-seed0` | 2 layers, synthetic repeats; induction heads fed by a previous-token head (K-composition) | 92.2% |
|
| 19 |
+
| `default-seed1` | same, seed 1; fed by duplicate-token heads (Q-composition) | 91.8% |
|
| 20 |
+
| `default-seed2` | same, seed 2; fed by a duplicate-token head (Q-composition) | 90.5% |
|
| 21 |
+
| `one-layer-seed0` | baseline: one layer, cannot form an induction head | 11.8% |
|
| 22 |
+
| `gpt2-init-seed0` | baseline: two layers with GPT-2's init (std 0.02); no induction heads | 16.6% |
|
| 23 |
+
| `stories-seed0` | 2 layers on TinyStories, 330M tokens; a previous-token head, no induction head yet | 1.7% (probe) |
|
| 24 |
+
|
| 25 |
+
The models are [tiny-transformer](https://github.com/coeiico/tiny-transformer) instances. To load one:
|
| 26 |
+
|
| 27 |
+
```python
|
| 28 |
+
from pathlib import Path
|
| 29 |
+
from huggingface_hub import snapshot_download
|
| 30 |
+
from tiny_transformer.hub import load_model
|
| 31 |
+
|
| 32 |
+
folder = Path(snapshot_download('coeiico/deja-vu-tfm', allow_patterns='default-seed0/*')) / 'default-seed0'
|
| 33 |
+
model, vocab = load_model(folder)
|
| 34 |
+
```
|
| 35 |
+
|
| 36 |
+
These are evaluation weights, not resumable checkpoints. Seeds fix a run on one kind of device; the runs here were trained on CUDA (an RTX 3060).
|