| --- |
| license: apache-2.0 |
| pipeline_tag: other |
| tags: |
| - sparse-autoencoder |
| - interpretability |
| - multimodal |
| - paligemma2 |
| - llava |
| --- |
| |
| # MMDiff — Multimodal SAE checkpoints |
|
|
| This repository contains the SAE checkpoints used in the paper [Multimodal Model Diffing for Feature Discovery and Control](https://huggingface.co/papers/2608.09928). |
|
|
| Text-only-masked SAEs trained on the residual streams of multimodal LLMs, used by the |
| **MMDiff** multimodal model-diffing pipeline (code: |
| [hunarbatra/MMDiff](https://github.com/hunarbatra/MMDiff), project page: |
| [hunarbatra.com/MMDiff](https://hunarbatra.com/MMDiff/)). |
|
|
| ## Contents |
|
|
| ### `paligemma2/jumprelu_text_only/` — 26 layers |
| JumpReLU SAEs (width 16,384) trained on `google/paligemma2-3b-mix-448` language-model |
| residual stream, layers 0–25, with **text-only masking** (image-token positions are |
| excluded from the training loss). Diffed against the Gemma-Scope base SAEs |
| (`google/gemma-scope-2b-pt-res`, `width_16k`). |
|
|
| File: `text-only_layer_{L}.pt` with keys `W_enc`, `b_enc`, `W_dec`, `b_dec`, `threshold`. |
|
|
| ### `llava_more/topk_text_only/` — 32 layers |
| TopK SAEs (k=50, width 32,768) trained on |
| `aimagelab/LLaVA_MORE-llama_3_1-8B-finetuning` residual stream, layers 0–31, with |
| text-only masking. Diffed against the LLaMA-Scope base SAEs |
| (`fnlp/Llama3_1-8B-Base-LXR-8x`, sae-lens release `llama_scope_lxr_8x`). |
|
|
| File: `text-only_layer_{L}.pt` with keys `W_enc (4096×32768)`, `b_enc (32768)`, |
| `W_dec (32768×4096)`, `b_dec (4096)`. |
|
|
| ## Loading |
|
|
| ```python |
| import torch |
| from huggingface_hub import hf_hub_download |
| |
| p = hf_hub_download( |
| "OX-PIXL/MMDiff_SAEs", |
| "paligemma2/jumprelu_text_only/text-only_layer_13.pt", |
| ) |
| state = torch.load(p, map_location="cpu") |
| ``` |
|
|
| See the [MMDiff repo](https://github.com/hunarbatra/MMDiff) for the full pipeline |
| (training, adapted-feature selection, task-specific discovery, ablation, steering). |
