--- license: mit library_name: pytorch tags: - sparse-autoencoder - interpretability - mechanistic-interpretability --- # Persistent Sparse Autoencoders Trained checkpoints from the paper [*Persistent Sparse Autoencoders: Learning Feature-Specific Timescales in Language Model Representations*](https://arxiv.org/abs/2607.17117). ## Checkpoints | Folder | Model | Layer | d_model | Training steps | |---|---|---|---|---| | `pythia-160m-deduped/layer_6` | `EleutherAI/pythia-160m-deduped` | 6 | 768 | 244,140 | | `pythia-410m-deduped/layer_12` | `EleutherAI/pythia-410m-deduped` | 12 | 1024 | 244,140 | | `pythia-1b-deduped/layer_8` | `EleutherAI/pythia-1b-deduped` | 8 | 2048 | 244,140 | | `pythia-1.4b-deduped/layer_12` | `EleutherAI/pythia-1.4b-deduped` | 12 | 2048 | 244,140 | | `gemma-2-2b/layer_12` | `google/gemma-2-2b` | 12 | 2304 | 244,140 | | `pythia-2.8b-deduped/layer_16` | `EleutherAI/pythia-2.8b-deduped` | 16 | 2560 | 367,648 | | `pythia-6.9b-deduped/layer_16` | `EleutherAI/pythia-6.9b-deduped` | 16 | 4096 | 367,648 | | `llama-3.1-8b/layer_19` | `meta-llama/Llama-3.1-8B` | 19 | 4096 | 367,648 | | `gemma-2-9b/layer_20` | `google/gemma-2-9b` | 20 | 3584 | 367,648 | | `pythia-12b-deduped/layer_18` | `EleutherAI/pythia-12b-deduped` | 18 | 5120 | 367,648 | ## Citation ```bibtex @misc{luo2026persistentsparseautoencoderslearning, title={Persistent Sparse Autoencoders: Learning Feature-Specific Timescales in Language Model Representations}, author={Haoyan Luo and Mateo Espinosa Zarlenga and Mateja Jamnik}, year={2026}, eprint={2607.17117}, archivePrefix={arXiv}, primaryClass={cs.LG}, url={https://arxiv.org/abs/2607.17117}, } ```