kroonen-ai/glimmer-scope-saes

Glimmer Scope SAEs

Sparse autoencoders for Muse Glimmer 30B, one on every third layer of its residual stream from 12 to 36. Each one splits what the layer computes into 65,536 features, most of which stand for one recognizable thing: a subject, a style, a piece of syntax.

They are the base of Glimmer Scope, where every feature that fires is labeled, tested for steering and followed from layer to layer. The code to do all of that with these files is in that repository. To see what the features are, browse the atlas: all nine layers on maps, with the labels and every feature's thread from layer to layer. To steer Glimmer with a few of them, try the Space. How they were trained and what they show is in the technical report.

Content warning. These SAEs were trained on ordinary web and chat text. Like any SAE on a large language model, they contain features for violent, sexual, hateful and otherwise disturbing subjects. Looking at where those features fire, or steering the model with them, can produce upsetting text. No feature labels, activation data or steering results are included here.

The SAEs

Layer	Training tokens	Variance unexplained	Dead features
12	50M	0.200	0.5%
15	100M	0.227	0.9%
18	100M	0.272	0.9%
21	100M	0.300	0.9%
24	100M	0.315	1.0%
27	100M	0.308	1.1%
30	100M	0.285	1.3%
33	100M	0.239	1.6%
36	50M	0.255	1.7%

Variance unexplained is the reconstruction error over the variance of the input, averaged over the last 20 logged training batches. A feature counts as dead when it hasn't fired in the last 10M tokens. Across the nine layers, 519,383 of the 589,824 features fire on a 5M-token sample of web and chat text.

Every SAE has the same shape:

Input	residual stream after decoder layer N (0-indexed, of 52), d = 6,656
Features	65,536
Activation	TopK, k = 64
Decoder rows	unit norm
Weights	float32, 3.5 GB per layer

Each layer-N/ folder has sae.safetensors (W_enc, b_enc, W_dec, b_dec) and cfg.json. The input is scaled by input_scale from cfg.json before encoding, so that a feature's activation is the length of its contribution in scaled units.

Usage

import json
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

REPO, LAYER = "kroonen-ai/glimmer-scope-saes", 24
cfg = json.load(open(hf_hub_download(REPO, f"layer-{LAYER}/cfg.json")))
sae = load_file(hf_hub_download(REPO, f"layer-{LAYER}/sae.safetensors"))


def encode(resid):
    """Residual stream [..., 6656] to the values and ids of its k active features."""
    x = resid.float() * cfg["input_scale"]
    pre = (x - sae["b_dec"]) @ sae["W_enc"] + sae["b_enc"]
    vals, ids = pre.topk(cfg["k"], dim=-1)
    return vals.relu(), ids


def decode(vals, ids):
    """Back to the residual stream, in the model's own scale."""
    x = (vals.unsqueeze(-1) * sae["W_dec"][ids]).sum(-2) + sae["b_dec"]
    return x / cfg["input_scale"]

Reading the residual stream with transformers:

from transformers import AutoTokenizer, MuseGlimmerForConditionalGeneration

name = "meta-models/Muse-Glimmer-30B"
tok = AutoTokenizer.from_pretrained(name)
model = MuseGlimmerForConditionalGeneration.from_pretrained(name, dtype=torch.bfloat16, device_map="cuda")

acts = {}
hook = model.model.language_model.layers[LAYER].register_forward_hook(lambda m, i, out: acts.update(x=out))
ids = tok("The tide came in over the old harbor wall.", return_tensors="pt").to("cuda")
with torch.no_grad():
    model.model.language_model(**ids)
hook.remove()

vals, feats = encode(acts["x"][0, 1:].cpu())  # skip position 0, see below

Leave out position 0 and any <|begin_of_text|> token. They act as attention sinks with very large activations and were not part of training.

To steer, add W_dec[f] * a / input_scale to the residual stream after the layer, at every position, where a is the activation you want for feature f.

Training

• Data: 70% FineWeb-Edu and 30% UltraChat 200k rendered in Glimmer's own chat template, packed into rows of 1,024 tokens
• Positions: every real token except position 0, BOS tokens and outliers above 10x the median norm
• Loss: normalized MSE, plus an AuxK term (k_aux = 512, coefficient 1/32) that revives dead features
• Optimizer: Adam, learning rate 2e-4, batches of 4,096 activations
• Model: Glimmer in bfloat16, one GPU per layer on Modal and AWS

What's not here

Some findings could be upsetting, so we filter them with safe browsing. This release has the SAEs only: no feature labels, no activation data and no steering results, including any strength at which a feature takes effect. The public labels are in the atlas, which leaves out the features on upsetting topics entirely.

License

Apache 2.0, the same as Muse Glimmer 30B. See LICENSE.

Citation

@misc{kroonen2026glimmerscope,
  title={Glimmer Scope: Sparse Autoencoders for Muse Glimmer 30B},
  author={Robin Kroonen},
  year={2026},
  publisher={Kroonen AI Inc.},
  howpublished={\url{https://huggingface.co/kroonen-ai/glimmer-scope-saes}}
}

About

Built by Kroonen AI Inc. for the Meta AI hackathon.