mjkmain/Qwen4B_headfinal_sae

Qwen4B_headfirst_sae

Final sparse autoencoder checkpoints for Qwen3-4B-Base layers 0, 7, 15, 19, 23, 27, 29, 31, 35. Each SAE was trained on 200 million retained tokens per language (Korean, Japanese, Hindi), for 600 million tokens per layer. Intermediate checkpoints are not included.

Each layer_N/ directory preserves the final checkpoint layout:

• layers.N/sae.safetensors: SAE weights.
• layers.N/cfg.json: SAE architecture configuration.
• config.json: training configuration and final token counts.
• training_state.pt: optimizer, scheduler, token progress and dead-feature state.
• validation/metrics.json: final held-out reconstruction metrics.

The SAE input is the zero-indexed transformer block output, before the final model norm. Hidden size is 2,560; expansion factor is 32; width is 81,920; ReLU TopK is 50. The frozen model uses bf16 and the SAE uses fp32, with no input normalization. The global batch is 48 sequences of 1,024 tokens; AuxK coefficient is 1/32. Training excludes position 0, EOS, and layer-specific norm outliers, with independent language quotas.

Layers 19, 23, 27, 31 and 35 shared frozen-model forwards while keeping independent masks, optimizers, schedulers and checkpoints. Layers 0, 7, 15 and 29 were trained individually. Layer 29 used eight-GPU DDP with the same global batch of 48 sequences. The base model revision is 906bfd4b4dc7f14ee4320094d8b41684abff8539. Training text came from HuggingFaceFW/fineweb-2; raw training text is not included here.

catalog.json lists all nine SAEs and their paths. manifest.json records checkpoint file sizes and SHA256 hashes. The repository is public with manually approved access to model files.

After access approval, authenticate with a read token to download a layer:

from huggingface_hub import snapshot_download
path = snapshot_download(
    repo_id="mjkmain/Qwen4B_headfirst_sae",
    allow_patterns=["layer_19/**", "catalog.json", "manifest.json"],
    token=True,
)

These are sparse autoencoders, not standalone language models. The safetensors file is the inference artifact; training_state.pt is a PyTorch training-state checkpoint.