Trained a TopK sparse autoencoder on the layer-12 residual stream of
google/gemma-2-2b using saprmarks/dictionary_learning, then used it to
identify and causally validate the SAE features mediating token-copying
in-context learning.
Headline findings:
- SAE: 16,384-feature TopK SAE (k=100), 200M training tokens, single
RTX 5070 Ti. Final explained variance 0.85 (peak 0.893) — in Google
DeepMind's published Gemma Scope range, zero dead features.
- Ablating the top-50 induction-scoring features drops the model's top-1
token-copying ICL accuracy from 57.75% to 47.65% (-10.1pp). Random-50-
feature controls drop accuracy by only 0.29pp ± 0.31pp — the induction
features carry ~35x the effect of random features.
- The top induction feature F15289 auto-interprets (via Claude Sonnet)
as "second occurrence of a repeated word", with snippets like
"Never...Never", "Tier...Tier", "so...so", "coming...coming".
- Activation patching (zero-ablation) suggested head 3 is the largest
cau…
