SAE dictionaries
Sparse-autoencoder dictionaries used in the paper Disambiguating Moral Circuits and Moral Features in Language Models (code: https://github.com/daviligade/normative-mechanistic-interpretability). Each folder holds one TopK dictionary (cfg.json, sae_weights.safetensors, in the format of SAELens) trained on the hidden states of one layer of one model:
| Folder | Model | Layer | Site | Features | k |
|---|---|---|---|---|---|
llama-3.2-1B-base |
meta-llama/Llama-3.2-1B | 7 | residual stream | 16384 | 64 |
llama-3.2-1B-base-k128 |
meta-llama/Llama-3.2-1B | 7 | residual stream | 16384 | 128 |
llama-3.2-1B-IT |
meta-llama/Llama-3.2-1B-Instruct | 7 | residual stream | 16384 | 64 |
llama-3.2-1B-IT-k128 |
meta-llama/Llama-3.2-1B-Instruct | 7 | residual stream | 16384 | 128 |
llama-3.2-1B-base-MLP0 |
meta-llama/Llama-3.2-1B | 0 | MLP output | 16384 | 64 |
llama-3.2-3B-base |
meta-llama/Llama-3.2-3B | 11 | residual stream | 24576 | 64 |
llama-3.2-3B-base-k128 |
meta-llama/Llama-3.2-3B | 11 | residual stream | 24576 | 128 |
llama-3.2-3B-IT-k128 |
meta-llama/Llama-3.2-3B-Instruct | 11 | residual stream | 24576 | 128 |
llama-3.2-3B-IT-L3 |
meta-llama/Llama-3.2-3B-Instruct | 3 | residual stream | 24576 | 128 |
llama-3.2-3B-IT-L7 |
meta-llama/Llama-3.2-3B-Instruct | 7 | residual stream | 24576 | 128 |
llama-3.2-3B-IT-L19 |
meta-llama/Llama-3.2-3B-Instruct | 19 | residual stream | 24576 | 128 |
qwen-2.5-7b-base |
Qwen/Qwen2.5-7B | 0 | residual stream | 28672 | 128 |
qwen-2.5-7b-IT |
Qwen/Qwen2.5-7B-Instruct | 0 | residual stream | 28672 | 128 |
The paper also uses the Llama Scope dictionary of layer 12 of Llama-3.1-8B (fnlp/Llama3_1-8B-Base-LXR-8x), which is not part of this repository.
Use
hf download davideliga/normative-sae-dictionaries --local-dir dictionaries
The code of the paper loads a dictionary with --sae-dir dictionaries/<folder>. With SAELens:
from sae_lens import SAE
sae = SAE.load_from_disk("dictionaries/llama-3.2-1B-base")
Citation
If you use this material, please cite the paper:
@misc{liga2026disambiguating,
title = {Disambiguating Moral Circuits and Moral Features in Language Models: Moral content, normative judgment or cue words? Evidence from 14 models},
author = {Liga, Davide and Yu, Liuwen and Magnini, Matteo},
year = {2026},
eprint = {2611.21705},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2611.21705}
}
Licence
CC BY 4.0. The dictionaries of the Llama models are derived from Llama 3.2 and are subject to the Llama 3.2 Community License; built with Llama.