---
library_name: pytorch
license: apache-2.0
tags:
- mechanistic-interpretability
- sparse-models
- model-decomposition
- circuit-analysis
base_model:
- gpt2
- Qwen/Qwen2.5-0.5B
- Qwen/Qwen2.5-1.5B
- Qwen/Qwen3.5-27B
datasets:
- HuggingFaceFW/fineweb-edu
---

# Sparse Weight Decomposition Checkpoints

This repository contains factor-only Sparse Weight Decomposition (SWD)
checkpoints used in our replacement-fidelity and circuit-extraction
experiments. It does **not** redistribute any base model. Load the corresponding
base model first, then apply one checkpoint with `swd_loader.py`.

Each replaced matrix is represented as

```text
output = input @ read @ write + bias
```

The intermediate coordinates are the SWD bottleneck units used for circuit
scoring and ablation. `s=0.5` and `s=0.75` mean that 50% and 75% of all entries
across the two factors are zero, respectively.

## Included Checkpoints

| Base model | Replacement | Setting | Data used | CE delta vs dense |
|---|---|---:|---:|---:|
| GPT-2 Small | Layer 8 `mlp.c_proj` | `s=0.5` | 16,384 tokens | 0.000889 |
| GPT-2 Small | Layer 8 `mlp.c_proj` | `s=0.75` | 16,384 tokens | 0.008292 |
| Qwen2.5-0.5B | Layer 12 `mlp.down_proj` | `s=0.5` | 1,024 tokens | 0.001222 |
| Qwen2.5-0.5B | Layer 12 `mlp.down_proj` | `s=0.75` | 1,048,576 tokens | 0.005010 |
| Qwen2.5-1.5B | Layer 14 `mlp.down_proj` | `s=0.5` | 1,024 tokens | 0.000733 |
| Qwen2.5-1.5B | Layer 14 `mlp.down_proj` | `s=0.75` | 1,048,576 tokens | 0.001222 |
| Qwen3.5-27B | Layer 31 `mlp.down_proj` | `s=0.5` | 2,048 tokens | -0.000427 |
| Qwen3.5-27B | Layer 31 `mlp.down_proj` | `s=0.75` | 1,048,576 tokens | -0.000448 |
| GPT-2 Small | Layer 8 complete MLP | `s=0.5` | 16,384 tokens | 0.004003 |
| GPT-2 Small | Layer 8 complete MLP | `s=0.75` | 1,048,576 tokens | 0.015521 |
| GPT-2 Small | All 48 transformer-block linear projections | fixed-support SWD-FT | 20,578,304 tokens | 0.151263* |

`*` The full-model value uses its full-model stress evaluation and should not
be numerically compared with the single-matrix unified CE rows.

The Qwen2.5-3B checkpoints are distributed separately at
[`veri-safe/SWD-Qwen2.5-3B`](https://huggingface.co/veri-safe/SWD-Qwen2.5-3B)
because the upstream model uses the Qwen Research License.

## Usage

Install the lightweight loader dependencies:

```bash
pip install torch safetensors transformers huggingface_hub
```

After downloading this repository, load a base model and apply a checkpoint:

```python
from transformers import AutoModelForCausalLM
from swd_loader import apply_swd_checkpoint

model = AutoModelForCausalLM.from_pretrained("gpt2")
apply_swd_checkpoint(
    model,
    "checkpoints/gpt2-small/layer8-cproj/s0p5-tokens16384",
    mode="factorized",
)
```

`mode="factorized"` installs `SWDLinear`, exposing
`component_activations(inputs)`. Use `mode="folded"` to write `read @ write`
back into the original dense module for conventional inference.

The base model must be fully materialized before applying a checkpoint. For
large models loaded with a device map, each replacement is moved to the device
and dtype of its target module.

## Format

Every checkpoint directory contains:

```text
model.safetensors  # factor tensors only; no pickle
config.json        # base model, module paths, shapes, sparsity, and token exposure
provenance.json    # source hashes, conversion rule, and validation result
```

All public factors follow `[input, rank] @ [rank, output]`, independent of the
source framework's dense-weight layout. Qwen2.5 source factors are transposed
into this convention; Qwen3.5 feature shards are concatenated along the rank
dimension without changing dtype or values. Full-model GPT-2 checkpoint biases
are included because they belong to the fixed-support fine-tuned checkpoint.

`RELEASE_MANIFEST.csv` is the machine-readable checkpoint index. Each
checkpoint's `provenance.json` records its source hashes and conversion
validation.

## Validation

Before release, every checkpoint was checked for source identity, finite
tensors, shape compatibility, factor nonzero counts, and exact tensor equality
after the safetensors round-trip.

The release intentionally excludes base-model weights, activation Grams, dense
target/reconstructed matrices, optimizer state, data caches, remote-transfer
archives, credentials, and cluster-local paths.

## Links

- Code: https://github.com/Veri-Safe/SWD
- Models: https://huggingface.co/veri-safe/SWD

## License

The SWD release code and the checkpoints in this repository are distributed
under the Apache License 2.0. The GPT-2-derived checkpoints also retain the
upstream Modified MIT notice in
`THIRD_PARTY_LICENSES/GPT2-MODIFIED-MIT.txt`. See `NOTICE` for attribution.