| --- |
| library_name: pytorch |
| tags: |
| - interpretability |
| - transcoder |
| - sparse-autoencoder |
| - arithmetic |
| --- |
| |
| # transcoder-relu-2layer-seed2-layer0 |
|
|
| A TopK transcoder fitted to **layer 0** of |
| [`itzPotato/arithmetic-relu-2layer-seed2`](https://huggingface.co/itzPotato/arithmetic-relu-2layer-seed2), a |
| 2-layer arithmetic transformer with a **relu** MLP. |
|
|
| A transcoder approximates one MLP sublayer: it reads the MLP's **input** |
| activation and predicts the MLP's **output**, through a k-sparse bottleneck. It is |
| not an autoencoder of the residual stream. |
|
|
| ## Architecture |
|
|
| | | | |
| |---|---| |
| | d_model (in and out) | 32 | |
| | features | 1024 (32x expansion) | |
| | active features per input (k) | 32 | |
| | decoder rows | unit norm | |
| | parameters | 66,592 | |
| |
| `y_hat = TopK_k(W_enc x + b_enc) @ W_dec + b_dec` |
| |
| ## Training |
| |
| Adam at lr 0.0003, batches of 4096 **activation vectors** (not problems), |
| **one pass** over 500,000 problems from the base model's *train* split |
| only, with a separate 10,000-problem validation subset. The base model's |
| val and test splits are never touched. |
| |
| Trained on 7,999,488 activation vectors over 1,953 steps. |
| |
| ## Reconstruction |
| |
| | | | |
| |---|---:| |
| | normalized reconstruction error | **0.0085** | |
| | fraction of variance unexplained | 0.0103 | |
| | raw MSE | 0.0129656 | |
| |
| Normalized reconstruction error is `MSE / mean(target^2)`. **That is the number to |
| compare across architectures**: a ReLU MLP's output and a bilinear MLP's output have |
| very different scales, so raw MSE compares the targets as much as the fits. Predicting |
| a constant zero scores 1.0. |
| |
| Across all 18 transcoders in this set, bilinear MLPs are consistently ~1.55x harder to |
| reconstruct than ReLU ones (0.0387 vs 0.0249), in all three depth/layer cells with three |
| seeds each. |
| |
| ## Initialisation |
| |
| The decoder bias is set to the mean target and the encoder is rescaled once from the first training batch, so the single allowed pass is spent learning features rather than undoing an initial scale mismatch. Measured: calibration_scale 0.129, init_normalized_after 1.63, init_normalized_before 43.8. |
|
|
| ## Provenance |
|
|
| Fitted to base weights with sha256 `726a01d75e2dde1088d4cee0c8127c4c233071e104b274d001bd8d6670a28e34`, |
| which is the checkpoint published at |
| [`itzPotato/arithmetic-relu-2layer-seed2`](https://huggingface.co/itzPotato/arithmetic-relu-2layer-seed2). |
|
|
| ## Loading |
|
|
| ```python |
| from src.transcoder.source import load_transcoder |
| transcoder, provenance = load_transcoder("hub:itzPotato/transcoder-relu-2layer-seed2-layer0", require_pinned=True) |
| ``` |
|
|
| `require_pinned=True` refuses anything but the registered commit sha in |
| `TRANSCODER_MODEL_REVISIONS` (`src/transcoder/config.py`), so a run cannot silently |
| load a moving branch. |
|
|
