| --- |
| license: apache-2.0 |
| tags: |
| - sparse-autoencoder |
| - sae |
| - interpretability |
| - reproducibility |
| pipeline_tag: feature-extraction |
| base_model: Qwen/Qwen2.5-0.5B-Instruct |
| --- |
| |
| # Null-step control for SAE feature drift |
|
|
| Artifact for *A Null-Step Control for Sparse-Autoencoder Feature Drift Across |
| Fine-Tuning Checkpoints*. Anonymous for double-blind review. |
|
|
| An SAE fitted at every fine-tuning checkpoint, each warm-started from the last, |
| is a standard way to watch features move. Between two adjacent checkpoint SAEs |
| the model changed, but the SAE was also **retrained**, and retraining alone moves |
| the basis. The **null step** isolates that: retrain the base SAE under the |
| identical recipe on activations from the *frozen* base model, so any drift it |
| produces is retraining. |
|
|
| ## Layout |
|
|
| ``` |
| tables/ every number in the paper, as CSV, plus the code that makes them |
| run_metadata/ per-run hyperparameters and training logs |
| weights/ |
| chains/ 112 checkpoint SAEs, 5 fine-tuning chains x 4 layers |
| null_step/ the control: 1 canonical + 12 seeds per layer |
| null_trajectories/ 7-step null chains, 4 layers x 3 seeds |
| controlled_rerun/ base SAEs for the one-split TopK / JumpReLU rerun |
| real_arm_L18/ per-step decoders, both arms, 4 seeds each |
| tuned_lens/ per-layer tuned lenses for the token-overlap metric |
| ``` |
|
|
| ## What backs what |
|
|
| | Paper element | Path | |
| |---|---| |
| | §3.1 fidelity envelope (84 evals at layers 6-18; CSV holds all 118) | `tables/sae_checkpoint_metrics.csv` | |
| | §3.1 and Table 1, inherited-index alignment | `tables/drift_out/drift_by_stage.csv` | |
| | **Table 2**, controlled rerun | `tables/controlled_rerun/`, `tables/real_arm/` | |
| | **Figure 1a**, null trajectory | `tables/drift_out/null_trajectory_l18_seeds.csv` | |
| | **Figure 1b** and §3.4, direction test | `tables/real_arm/L18_pairwise_cosines.csv` | |
| | §3.2 and Appendix D, artifact null step | `tables/drift_out/null_replicates.csv` | |
| | §3.3 and Table 3, checkpoint selection | `tables/drift_out/checkpoint_selection.csv` | |
| | §3.3, shuffled reward | `weights/chains/sae_shuffled/`, `tables/task_performance.csv` | |
| | §3.5, architecture arms | `tables/controlled_rerun/jumprelu_*.csv` | |
| | **Appendix G**, heterogeneity and the selection rule | `tables/drift_out/drift_per_feature.csv` | |
| | §3.1 dead-latent fractions of the rerun (27-31% L18, 59-63% L12) | `tables/real_arm/L*.csv`, column `dead_frac` | |
|
|
| ## Reproducing the numbers |
|
|
| ```bash |
| export SAE_RL_ROOT=/path/to/this/repo |
| python3 tables/scripts/paper_numbers.py --json > fresh.json |
| diff fresh.json tables/expected/paper_numbers_full.json # expect no output |
| ``` |
|
|
| `tables/scripts/make_numbers.py` emits the LaTeX macros the manuscript quotes, so |
| no number in the paper is typed by hand. `--skip-weights` drops the five |
| weight-loading sections and finishes in about a minute. |
|
|
| Two rounding conventions are deliberate and worth knowing before you diff: |
| printed bounds round **outward** (`floor`/`ceil`), so a measured minimum of 4.69 |
| prints as `4`, not `5`; and the fidelity envelope is scoped to the artifact's 84 |
| evaluations at layers 6-18, not to the controlled rerun, whose dictionaries are |
| far deader because it uses no dead-latent resampling. |
|
|
| ## Notes on the weights |
|
|
| - `chains/` holds 112 files: SFT (8 checkpoints), PPO flexible (8), PPO strict |
| (6), PPO high-KL (3), PPO shuffled-label (3), each at layers 6, 12, 18, 23. |
| `k=64` throughout except where layer 23 uses `k=256`. |
| - `real_arm_L18/` stores bare decoder matrices (`[7168, 896]`), not full state |
| dicts. These are what the pairwise direction test consumes. |
| - `null_trajectories/` excludes the `DONE` marker files. |
| - **Per-step weights exist only for the layer-18 TopK real-arm replication.** The |
| controlled rerun saved base SAEs alone, for both architectures, so |
| `controlled_rerun/` holds 9 base checkpoints (4 TopK, 5 JumpReLU) and no |
| intermediate chain weights. Nothing is lost for verification: every per-step |
| `dec_cos`, `dead_frac`, `mean_l0` and `kept_epoch` is recorded in |
| `tables/controlled_rerun/*.csv`, which is enough to reproduce both the raw and |
| the dead-latent-adjusted shares. For example, at layer 18 the JumpReLU arm gives |
| a raw share of 95.1% and 91.3% after dividing out frozen latents, both |
| recomputable from that CSV alone. |
| - Activation tensors are not included; they are regenerable from the public base |
| model and GSM8K. |
|
|
| ## Caveats we would rather state than have you find |
|
|
| - Layer 23 appears throughout the tables and is **not read** in the paper: its |
| residual stream is mean-dominated and no SAE we trained reconstructs it |
| acceptably. |
| - Seed replication covers the SAE fit, not the fine-tuning run. Every real chain |
| descends from the same sequence of checkpoints. |
| - An earlier BatchTopK architecture grid is deliberately excluded: its threshold |
| received no gradient, so it is not evidence and the paper does not use it. |
| - The strict chain's layer-23 `k=256` robustness variant is not here. Layer 23 is |
| not read in the paper and that variant backs no claim in it. |
| - The epoch-budget sweep weights are not here either; Appendix E.1's numbers are |
| in `tables/drift_out/`. |
|
|
| ## Contents at a glance |
|
|
| | Path | Files | |
| |---|---| |
| | `weights/chains/` | 112 | |
| | `weights/null_step/` | 48 (4 canonical + 44 seeds) | |
| | `weights/null_trajectories/` | 96 (4 layers x 3 seeds x 8 steps) | |
| | `weights/real_arm_L18/` | 57 (base + 4 real x 7 + 4 null x 7) | |
| | `weights/controlled_rerun/` | 9 | |
| | `weights/tuned_lens/` | 4 | |
| | `tables/` | 38 | |
| | `run_metadata/` | 26 | |
|
|
