haoyanl78's picture
Upload README.md with huggingface_hub
7458345 verified
|
Raw History Blame Contribute Delete
1.71 kB
---
license: mit
library_name: pytorch
tags:
- sparse-autoencoder
- interpretability
- mechanistic-interpretability
---
# Persistent Sparse Autoencoders
Trained checkpoints from the paper [*Persistent Sparse Autoencoders: Learning Feature-Specific Timescales in Language Model Representations*](https://arxiv.org/abs/2607.17117).
## Checkpoints
| Folder | Model | Layer | d_model | Training steps |
|---|---|---|---|---|
| `pythia-160m-deduped/layer_6` | `EleutherAI/pythia-160m-deduped` | 6 | 768 | 244,140 |
| `pythia-410m-deduped/layer_12` | `EleutherAI/pythia-410m-deduped` | 12 | 1024 | 244,140 |
| `pythia-1b-deduped/layer_8` | `EleutherAI/pythia-1b-deduped` | 8 | 2048 | 244,140 |
| `pythia-1.4b-deduped/layer_12` | `EleutherAI/pythia-1.4b-deduped` | 12 | 2048 | 244,140 |
| `gemma-2-2b/layer_12` | `google/gemma-2-2b` | 12 | 2304 | 244,140 |
| `pythia-2.8b-deduped/layer_16` | `EleutherAI/pythia-2.8b-deduped` | 16 | 2560 | 367,648 |
| `pythia-6.9b-deduped/layer_16` | `EleutherAI/pythia-6.9b-deduped` | 16 | 4096 | 367,648 |
| `llama-3.1-8b/layer_19` | `meta-llama/Llama-3.1-8B` | 19 | 4096 | 367,648 |
| `gemma-2-9b/layer_20` | `google/gemma-2-9b` | 20 | 3584 | 367,648 |
| `pythia-12b-deduped/layer_18` | `EleutherAI/pythia-12b-deduped` | 18 | 5120 | 367,648 |
## Citation
```bibtex
@misc{luo2026persistentsparseautoencoderslearning,
title={Persistent Sparse Autoencoders: Learning Feature-Specific Timescales in Language Model Representations},
author={Haoyan Luo and Mateo Espinosa Zarlenga and Mateja Jamnik},
year={2026},
eprint={2607.17117},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2607.17117},
}
```