decoderesearch/olmo-3-saes

Olmo-3 SAEs for use with the SAELens library

This repository contains the following SAEs:

Olmo-3-1025-7b

• olmo-3-1025-7b/btk-mat-layer-4-k-100
• olmo-3-1025-7b/btk-mat-layer-16-k-100
• olmo-3-1025-7b/btk-mat-layer-28-k-100

Load these SAEs using SAELens as below:

from sae_lens import SAE

sae = SAE.from_pretrained("decoderesearch/olmo-3-saes", "<sae_id>")

About these SAEs

These SAEs are all Matryoshka BatchTopK SAEs trained on 1B tokens of the Pile Uncopyrighted using SAELens. The SAEs each have width 65k latents, and 2 inner Matryoshka prefixes of 2k latents and 16k latents.

Reproducing training

For complete details of how these SAEs were trained, refer to the runner_cfg.json in each SAE directory. This config includes all hyperparameters passed to the SAELens trainer when training the SAEs.

Citation

@misc{decode2026olmo3saes,
  author       = {Chanin, David and Lin, Johnny},
  title        = {{Olmo-3} Sparse Autoencoders},
  year         = {2026},
  organization = {Decode Research},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/decoderesearch/olmo-3-saes}}
}

Acknowledgements

These SAEs were trained thanks to compute provided by Modal.