---
license: apache-2.0
tags: [sparse-autoencoder, interpretability, qwen, speculative-decoding]
---
# Qwen3.8-27B-Scope (code-domain SAE suite)

Self-trained Top-K sparse autoencoders for **Qwen/Qwen3.8-27B**, all 64 decoder
layers, coding-domain corpus. Built as the feature-selection teacher for
DFlash feature-interface draft training.

- **v2/** (main): 64 SAEs, 110M unique coding tokens each (50M base + 60M continuation)
- **stage1/**: 50M-token checkpoints (reference; better thinking-text recon: −8.5% vs −16% recon-τ)
- **interface/**: selected-feature lists + compact encoders (1×/2.5×/5× of 5120-dim budgets)
- **scores/**: per-feature S (code-specificity), A (acceptance-length contrast), U (causal draft utility)

**Spec**: width 40960 (8×5120), Top-k k=50, AuxK 512 @1/32, input scale √d/mean-norm
(stored in each checkpoint), decoder rows unit-norm. Checkpoint = `{state: {b_pre, W_enc,
b_enc, W_dec} (bf16), scale, k, width, tokens}`.
Recon: `x̂ = TopK50(ReLU((x·scale − b_pre)·W_enc + b_enc))·W_dec + b_pre) / scale`... see
`interface/` encoders for the selected-feature read-out path.

**Key measurements**: FVU 0.06–0.23 across layers; masking the 5-tap top-1000
code features costs 18.3% draft acceptance (random control ≈0); feature-level
task specificity reliability 0.78–0.86.

Corpus: on-policy Qwen3.8 code generations + Magicoder-OSS/Evol + CodeAlpaca (single-pass).