| --- |
| license: apache-2.0 |
| tags: [sparse-autoencoder, interpretability, qwen, speculative-decoding] |
| --- |
| # Qwen3.8-27B-Scope (code-domain SAE suite) |
|
|
| Self-trained Top-K sparse autoencoders for **Qwen/Qwen3.8-27B**, all 64 decoder |
| layers, coding-domain corpus. Built as the feature-selection teacher for |
| DFlash feature-interface draft training. |
|
|
| - **v2/** (main): 64 SAEs, 110M unique coding tokens each (50M base + 60M continuation) |
| - **stage1/**: 50M-token checkpoints (reference; better thinking-text recon: −8.5% vs −16% recon-τ) |
| - **interface/**: selected-feature lists + compact encoders (1×/2.5×/5× of 5120-dim budgets) |
| - **scores/**: per-feature S (code-specificity), A (acceptance-length contrast), U (causal draft utility) |
|
|
| **Spec**: width 40960 (8×5120), Top-k k=50, AuxK 512 @1/32, input scale √d/mean-norm |
| (stored in each checkpoint), decoder rows unit-norm. Checkpoint = `{state: {b_pre, W_enc, |
| b_enc, W_dec} (bf16), scale, k, width, tokens}`. |
| Recon: `x̂ = TopK50(ReLU((x·scale − b_pre)·W_enc + b_enc))·W_dec + b_pre) / scale`... see |
| `interface/` encoders for the selected-feature read-out path. |
|
|
| **Key measurements**: FVU 0.06–0.23 across layers; masking the 5-tap top-1000 |
| code features costs 18.3% draft acceptance (random control ≈0); feature-level |
| task specificity reliability 0.78–0.86. |
|
|
| Corpus: on-policy Qwen3.8 code generations + Magicoder-OSS/Evol + CodeAlpaca (single-pass). |
|
|
