coeiico commited on
Commit
d4e6a4e
·
verified ·
1 Parent(s): 81c5896

Add the model card

Browse files
Files changed (1) hide show
  1. README.md +36 -0
README.md ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ tags:
4
+ - interpretability
5
+ - induction-heads
6
+ - attention-only
7
+ library_name: pytorch
8
+ ---
9
+
10
+ # deja-vu-tfm
11
+
12
+ Trained runs for [deja-vu-tfm](https://github.com/coeiico/deja-vu-tfm): induction heads forming as a phase change in a 2-layer attention-only transformer, and the heads that do the copying. The code, the full write-up and the figures are on GitHub.
13
+
14
+ Each folder is one run: evaluation weights (`model.safetensors`), the architecture and vocabulary (`model_config.json`), the full training record (`metrics.json`: config, seeds, data definition, code state, environment, history), a diagnostic plot, and the final analysis under `analysis/final/`.
15
+
16
+ | Run | What it is | Copy accuracy (test) |
17
+ | --- | --- | --- |
18
+ | `default-seed0` | 2 layers, synthetic repeats; induction heads fed by a previous-token head (K-composition) | 92.2% |
19
+ | `default-seed1` | same, seed 1; fed by duplicate-token heads (Q-composition) | 91.8% |
20
+ | `default-seed2` | same, seed 2; fed by a duplicate-token head (Q-composition) | 90.5% |
21
+ | `one-layer-seed0` | baseline: one layer, cannot form an induction head | 11.8% |
22
+ | `gpt2-init-seed0` | baseline: two layers with GPT-2's init (std 0.02); no induction heads | 16.6% |
23
+ | `stories-seed0` | 2 layers on TinyStories, 330M tokens; a previous-token head, no induction head yet | 1.7% (probe) |
24
+
25
+ The models are [tiny-transformer](https://github.com/coeiico/tiny-transformer) instances. To load one:
26
+
27
+ ```python
28
+ from pathlib import Path
29
+ from huggingface_hub import snapshot_download
30
+ from tiny_transformer.hub import load_model
31
+
32
+ folder = Path(snapshot_download('coeiico/deja-vu-tfm', allow_patterns='default-seed0/*')) / 'default-seed0'
33
+ model, vocab = load_model(folder)
34
+ ```
35
+
36
+ These are evaluation weights, not resumable checkpoints. Seeds fix a run on one kind of device; the runs here were trained on CUDA (an RTX 3060).