Massive Activations Confound Sparse Autoencoders
Collection
SAEs and results from the paper. • 6 items • Updated
Massive SAEs: archive
SAEs from Massive Activations Confound Sparse Autoencoders. Main SAEs: nathandelisle/massive-saes.
objective-grid/: the objective comparison in Table 1 (BatchTopK, ReLU + L1, Gated, JumpReLU), eight models,standardandbypass.llama-3.1-70b/: Llama-3.1-70B TopK SAEs, seeds 42 and 137. The paper's 70B numbers come from these.llama-3.1-70b-five-seed/: a later five-seed 70B set, with its training mean and PC1.gpt2-ten-seed/: GPT-2 SAEs, ten seeds per training setting.beta-pythia-1.4b/: Pythia-1.4B TopK SAEs with an added L1 penalty of varying strength.
Each checkpoint folder has sae.pt and config.json. Load with torch.load(path, map_location="cpu", weights_only=True).
License: CC BY 4.0. The base models' own licenses also apply.
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support