Massive Activations Confound Sparse Autoencoders
Collection
SAEs and results from the paper. • 6 items • Updated
Massive SAEs: other objectives
SAEs from Massive Activations Confound Sparse Autoencoders. Main SAEs: nathandelisle/massive-saes.
The paper's comparison of SAE training objectives (TopK, BatchTopK, ReLU + L1), repeated over five seeds for eight models at the same layers as the main SAEs. Each objective has standard and bypass versions.
Files: <model>/<objective>/block<L>/<standard|bypass>/seed<k>/{sae.pt, config.json}. The mean and PC1 for each model are in the main repo under <model>/preprocessing/.
Load a checkpoint with torch.load(path, map_location="cpu", weights_only=True).
License: CC BY 4.0. The base models' own licenses also apply.
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support