SAE dictionaries

Sparse-autoencoder dictionaries used in the paper Disambiguating Moral Circuits and Moral Features in Language Models (code: https://github.com/daviligade/normative-mechanistic-interpretability). Each folder holds one TopK dictionary (cfg.json, sae_weights.safetensors, in the format of SAELens) trained on the hidden states of one layer of one model:

Folder Model Layer Site Features k
llama-3.2-1B-base meta-llama/Llama-3.2-1B 7 residual stream 16384 64
llama-3.2-1B-base-k128 meta-llama/Llama-3.2-1B 7 residual stream 16384 128
llama-3.2-1B-IT meta-llama/Llama-3.2-1B-Instruct 7 residual stream 16384 64
llama-3.2-1B-IT-k128 meta-llama/Llama-3.2-1B-Instruct 7 residual stream 16384 128
llama-3.2-1B-base-MLP0 meta-llama/Llama-3.2-1B 0 MLP output 16384 64
llama-3.2-3B-base meta-llama/Llama-3.2-3B 11 residual stream 24576 64
llama-3.2-3B-base-k128 meta-llama/Llama-3.2-3B 11 residual stream 24576 128
llama-3.2-3B-IT-k128 meta-llama/Llama-3.2-3B-Instruct 11 residual stream 24576 128
llama-3.2-3B-IT-L3 meta-llama/Llama-3.2-3B-Instruct 3 residual stream 24576 128
llama-3.2-3B-IT-L7 meta-llama/Llama-3.2-3B-Instruct 7 residual stream 24576 128
llama-3.2-3B-IT-L19 meta-llama/Llama-3.2-3B-Instruct 19 residual stream 24576 128
qwen-2.5-7b-base Qwen/Qwen2.5-7B 0 residual stream 28672 128
qwen-2.5-7b-IT Qwen/Qwen2.5-7B-Instruct 0 residual stream 28672 128

The paper also uses the Llama Scope dictionary of layer 12 of Llama-3.1-8B (fnlp/Llama3_1-8B-Base-LXR-8x), which is not part of this repository.

Use

hf download davideliga/normative-sae-dictionaries --local-dir dictionaries

The code of the paper loads a dictionary with --sae-dir dictionaries/<folder>. With SAELens:

from sae_lens import SAE
sae = SAE.load_from_disk("dictionaries/llama-3.2-1B-base")

Citation

If you use this material, please cite the paper:

@misc{liga2026disambiguating,
  title         = {Disambiguating Moral Circuits and Moral Features in Language Models: Moral content, normative judgment or cue words? Evidence from 14 models},
  author        = {Liga, Davide and Yu, Liuwen and Magnini, Matteo},
  year          = {2026},
  eprint        = {2611.21705},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2611.21705}
}

Licence

CC BY 4.0. The dictionaries of the Llama models are derived from Llama 3.2 and are subject to the Llama 3.2 Community License; built with Llama.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Paper for davideliga/normative-sae-dictionaries