diff --git a/README.md b/README.md new file mode 100644 index 0000000000000000000000000000000000000000..9e09ae2e240474bdca82825bf227fa22602b7adc --- /dev/null +++ b/README.md @@ -0,0 +1,83 @@ +--- +license: apache-2.0 +tags: +- crosscoder +- sparse-autoencoder +- mech-interp +- parameter-trajectory +- pythia +- olmo +--- + +# Parameter-trajectory crosscoders for vocabulary readout evolution + +Companion checkpoints to the paper *Parameter-trajectory crosscoders for +vocabulary readout evolution* (NeurIPS 2026; arXiv:XXXX.XXXXX). + +We train **snapshot crosscoders** on parameter tensors (rather than +activations) sampled across pretraining checkpoints. In Pythia-160M's output +unembedding $W_U$, this reveals a sparse, reproducible **readout +consolidation event** at training step ~1,000. + +## Repository layout + +| What you want | Path | +|---|---| +| Headline 5-seed Pythia-160M $W_U$ crosscoder | `pythia-160m/W_U/cross-snapshot-32/d8192/seed{0..4}.safetensors` | +| 32 single-snapshot SAEs (in-time view) | `pythia-160m/W_U/per-snapshot-saes/d8192/step*.safetensors` | +| Higher-capacity Pythia-160M $W_U$ crosscoders | `pythia-160m/W_U/cross-snapshot-32/d{16384,24576}/` | +| Architecture-invariance comparison (vs. JumpReLU baseline) | `pythia-160m/W_U/architecture-comparison/d8192/{batchtopk,gated,gated-retuned}/` | +| Snapshot-density downsample | `pythia-160m/W_U/cross-snapshot-16/d8192/` | +| Final-snapshot-only SAE Pareto | `pythia-160m/W_U/final-snapshot-saes/d{6144..65536}.safetensors` | +| Read/write asymmetry: $W_E$ side | `pythia-160m/W_E/cross-snapshot-32/d{8192,24576}/` | +| Cross-scale (Pythia-1B) | `pythia-1b/W_U/cross-snapshot-32/d{8192,16384,24576}/` | +| Larger-scale (Pythia-6.9B) | `pythia-6.9b/W_U/cross-snapshot-32/d32768/` | +| Cross-family (OLMo-2-7B) | `olmo-2-7b/W_U/cross-snapshot-32/d32768/` | +| §5 attribution-patching artifacts | `attribution/pythia-160m/` | + +Each directory contains `.safetensors` (weights), +`.config.json` (training hparams + quality metrics), and `.md` +(one-page model card). The metadata format is unified across all artifacts. + +## Quick start + +```python +from huggingface_hub import hf_hub_download +from safetensors.torch import load_file +import json + +# Download the headline Pythia-160M crosscoder, seed 0 +weights_path = hf_hub_download( + "matteohe/parameter-trajectory-crosscoders", + "pythia-160m/W_U/cross-snapshot-32/d8192/seed0.safetensors", +) +config_path = hf_hub_download( + "matteohe/parameter-trajectory-crosscoders", + "pythia-160m/W_U/cross-snapshot-32/d8192/seed0.config.json", +) +weights = load_file(weights_path) # dict[str, Tensor] +config = json.load(open(config_path)) +print(config["public_label"]) # "Pythia-160M W_U crosscoder, ..." +``` + +The crosscoder weight tensors are: +- `W_E`: encoder, shape $(K, d_{\text{model}}, d_{\text{sae}})$ +- `W_D`: decoder, shape $(K, d_{\text{sae}}, d_{\text{model}})$ +- `b_E`: encoder bias, shape $(K, d_{\text{sae}})$ +- `b_D`: decoder bias, shape $(K, d_{\text{model}})$ +- `activation_function.log_jumprelu_threshold`: shape $(d_{\text{sae}},)$ + +For per-snapshot SAEs `K=1`. The companion paper code (open-source, see +`https://github.com/...`) provides `wu_adapter.build_crosscoder` which +reconstructs the full module. + +## Citation + +```bibtex +@inproceedings{{..., + title={{Parameter-trajectory crosscoders for vocabulary readout evolution}}, + author={{...}}, + booktitle={{NeurIPS 2026}}, + year={{2026}}, +}} +``` diff --git a/attribution/pythia-160m/induction-results.pt b/attribution/pythia-160m/induction-results.pt new file mode 100644 index 0000000000000000000000000000000000000000..e91d0f2915253d6250ddd0d74887cae4c716e20b --- /dev/null +++ b/attribution/pythia-160m/induction-results.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:88048a0c9972a962695610c19820d716b541807022c78a0f8a86557bc15346a1 +size 32822581 diff --git a/attribution/pythia-160m/induction-verdict.json b/attribution/pythia-160m/induction-verdict.json new file mode 100644 index 0000000000000000000000000000000000000000..3329fb48a3e13521f71d806822365bf34f50a7ce --- /dev/null +++ b/attribution/pythia-160m/induction-verdict.json @@ -0,0 +1,11 @@ +{ + "passed": true, + "max_ratio": 696.25, + "ratio_at_k": 32, + "ig_clean": true, + "n_samples": 1000, + "n_ig_steps": 10, + "snapshot_step": 143000, + "checkpoint": "/home/mh2274/rds/hpc-work/wu_crosscoder/run3_ge_exact_results/wu_cc_ge_exact_seed0.pt", + "wallclock_s": 150.33362078666687 +} \ No newline at end of file diff --git a/attribution/pythia-160m/ioi-results.pt b/attribution/pythia-160m/ioi-results.pt new file mode 100644 index 0000000000000000000000000000000000000000..073c083a029869a2cdd7955d325e7bc18f2d7edc --- /dev/null +++ b/attribution/pythia-160m/ioi-results.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8d4eb1c1dce4716843805b815c9fdef324faf58626fbf14f9ceee0545404188e +size 32822469 diff --git a/attribution/pythia-160m/ioi-verdict.json b/attribution/pythia-160m/ioi-verdict.json new file mode 100644 index 0000000000000000000000000000000000000000..22883a501a3d83177f618077931fb1ac89e727ff --- /dev/null +++ b/attribution/pythia-160m/ioi-verdict.json @@ -0,0 +1,11 @@ +{ + "passed": true, + "max_ratio": 167.48154981549814, + "ratio_at_k": 64, + "ig_clean": true, + "n_samples": 1000, + "n_ig_steps": 10, + "snapshot_step": 143000, + "checkpoint": "/home/mh2274/rds/hpc-work/wu_crosscoder/run3_ge_exact_results/wu_cc_ge_exact_seed0.pt", + "wallclock_s": 92.33377718925476 +} \ No newline at end of file diff --git a/attribution/pythia-160m/sva-results.pt b/attribution/pythia-160m/sva-results.pt new file mode 100644 index 0000000000000000000000000000000000000000..dd4d5c53f962ec466bca9004a9e5600106e7fb2d --- /dev/null +++ b/attribution/pythia-160m/sva-results.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:39c166606642bf4758e30c06d33935ec7bd2a1855adb19a7b8252b59a9850425 +size 32822469 diff --git a/attribution/pythia-160m/sva-verdict.json b/attribution/pythia-160m/sva-verdict.json new file mode 100644 index 0000000000000000000000000000000000000000..1bb38e80de685e37bc418fb9016ea489cf5c44b4 --- /dev/null +++ b/attribution/pythia-160m/sva-verdict.json @@ -0,0 +1,11 @@ +{ + "passed": true, + "max_ratio": 1140.1538461538462, + "ratio_at_k": 64, + "ig_clean": true, + "n_samples": 1000, + "n_ig_steps": 10, + "snapshot_step": 143000, + "checkpoint": "/home/mh2274/rds/hpc-work/wu_crosscoder/run3_ge_exact_results/wu_cc_ge_exact_seed0.pt", + "wallclock_s": 191.5964698791504 +} \ No newline at end of file diff --git a/pythia-160m/W_E/cross-snapshot-32/d24576/seed0.config.json b/pythia-160m/W_E/cross-snapshot-32/d24576/seed0.config.json new file mode 100644 index 0000000000000000000000000000000000000000..685159a4ec18987f128985626b88481b0ebfe938 --- /dev/null +++ b/pythia-160m/W_E/cross-snapshot-32/d24576/seed0.config.json @@ -0,0 +1,119 @@ +{ + "public_label": "Pythia-160M W_E crosscoder, 32 snapshots, d_sae=24576, seed 0", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 32.0, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0", + "wu_step_1", + "wu_step_2", + "wu_step_3", + "wu_step_4", + "wu_step_5", + "wu_step_6", + "wu_step_7", + "wu_step_8", + "wu_step_9", + "wu_step_10", + "wu_step_11", + "wu_step_12", + "wu_step_13", + "wu_step_14", + "wu_step_15", + "wu_step_16", + "wu_step_17", + "wu_step_18", + "wu_step_19", + "wu_step_20", + "wu_step_21", + "wu_step_22", + "wu_step_23", + "wu_step_24", + "wu_step_25", + "wu_step_26", + "wu_step_27", + "wu_step_28", + "wu_step_29", + "wu_step_30", + "wu_step_31" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 4.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 300, + "batch_size": 1024, + "expansion_factor": 32.0, + "d_sae": 24576, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.05, + "lr_warmup_fraction": 0.0, + "lr_decay_fraction": 0.0, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.8305442370487662, + "reconstruction_mse": 0.1690640303492546, + "mean_l0": 117.50901977539063, + "dead_rate": 0.0, + "d_sae": 24576, + "n_snapshots": 32, + "n_rows": 50304 + }, + "steps": [ + 0, + 1, + 2, + 4, + 8, + 16, + 32, + 64, + 128, + 256, + 512, + 1000, + 2000, + 3000, + 4000, + 5000, + 6000, + 7000, + 8000, + 9000, + 14000, + 21000, + 27000, + 34000, + 47000, + 61000, + 75000, + 89000, + 102000, + 116000, + 130000, + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "cross-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_E/cross-snapshot-32/d24576/seed0.md b/pythia-160m/W_E/cross-snapshot-32/d24576/seed0.md new file mode 100644 index 0000000000000000000000000000000000000000..dcf074619c41d0dcd400a280d6df92fc8ded775a --- /dev/null +++ b/pythia-160m/W_E/cross-snapshot-32/d24576/seed0.md @@ -0,0 +1,38 @@ +# Pythia-160M W_E crosscoder, 32 snapshots, d_sae=24576, seed 0 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 14000, 21000, 27000, 34000, 47000, 61000, 75000, 89000, 102000, 116000, 130000, 143000] pretraining +steps. + +## Quality +- Explained variance: 0.8305442370487662 +- Mean L0: 117.50901977539063 +- Dead-feature rate: 0.0 +- d_sae: 24576 +- n_snapshots: 32 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 4.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 300 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("seed0.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("seed0.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_E/cross-snapshot-32/d24576/seed0.safetensors b/pythia-160m/W_E/cross-snapshot-32/d24576/seed0.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..767f3e21c4f4a2876bd8003041f2c73ae87b8ff2 --- /dev/null +++ b/pythia-160m/W_E/cross-snapshot-32/d24576/seed0.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:cfe74424d03c865e4595f94a9aff6316d579a52b963255807577233d5b5387f2 +size 4835180992 diff --git a/pythia-160m/W_E/cross-snapshot-32/d8192/seed0.config.json b/pythia-160m/W_E/cross-snapshot-32/d8192/seed0.config.json new file mode 100644 index 0000000000000000000000000000000000000000..a225e9a88f64745fc60ddc5bd0479ec15ee4f479 --- /dev/null +++ b/pythia-160m/W_E/cross-snapshot-32/d8192/seed0.config.json @@ -0,0 +1,113 @@ +{ + "public_label": "Pythia-160M W_E crosscoder, 32 snapshots, d_sae=8192, seed 0", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0", + "wu_step_1", + "wu_step_2", + "wu_step_3", + "wu_step_4", + "wu_step_5", + "wu_step_6", + "wu_step_7", + "wu_step_8", + "wu_step_9", + "wu_step_10", + "wu_step_11", + "wu_step_12", + "wu_step_13", + "wu_step_14", + "wu_step_15", + "wu_step_16", + "wu_step_17", + "wu_step_18", + "wu_step_19", + "wu_step_20", + "wu_step_21", + "wu_step_22", + "wu_step_23", + "wu_step_24", + "wu_step_25", + "wu_step_26", + "wu_step_27", + "wu_step_28", + "wu_step_29", + "wu_step_30", + "wu_step_31" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 4.0, + "jumprelu_lr_factor": 0.1, + "n_epochs": 300, + "batch_size": 1024, + "expansion_factor": 10.667, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.05, + "lr_warmup_fraction": 0.0, + "lr_decay_fraction": 0.0, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.5807971126242004, + "reconstruction_mse": 0.4182338178157806, + "mean_l0": 82.08435302734375, + "d_sae": 8192, + "n_snapshots": 32, + "n_rows": 50304 + }, + "steps": [ + 0, + 1, + 2, + 4, + 8, + 16, + 32, + 64, + 128, + 256, + 512, + 1000, + 2000, + 3000, + 4000, + 5000, + 6000, + 7000, + 8000, + 9000, + 14000, + 21000, + 27000, + 34000, + 47000, + 61000, + 75000, + 89000, + 102000, + 116000, + 130000, + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "cross-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_E/cross-snapshot-32/d8192/seed0.md b/pythia-160m/W_E/cross-snapshot-32/d8192/seed0.md new file mode 100644 index 0000000000000000000000000000000000000000..bfa04299dc1aaaf42a6c1ba7fd59db30e276bd94 --- /dev/null +++ b/pythia-160m/W_E/cross-snapshot-32/d8192/seed0.md @@ -0,0 +1,38 @@ +# Pythia-160M W_E crosscoder, 32 snapshots, d_sae=8192, seed 0 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 14000, 21000, 27000, 34000, 47000, 61000, 75000, 89000, 102000, 116000, 130000, 143000] pretraining +steps. + +## Quality +- Explained variance: 0.5807971126242004 +- Mean L0: 82.08435302734375 +- Dead-feature rate: — +- d_sae: 8192 +- n_snapshots: 32 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 4.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: — +- Optimizer: adam +- Epochs: 300 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("seed0.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("seed0.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_E/cross-snapshot-32/d8192/seed0.safetensors b/pythia-160m/W_E/cross-snapshot-32/d8192/seed0.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..3dd0f79b4be8cdbf769a045ec272e281ac47d659 --- /dev/null +++ b/pythia-160m/W_E/cross-snapshot-32/d8192/seed0.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:75ba91f6abdbd6b888ad7fc8ba34a30a363a123846bc136bbe29bb19386374bb +size 1611792824 diff --git a/pythia-160m/W_E/cross-snapshot-32/d8192/seed1.config.json b/pythia-160m/W_E/cross-snapshot-32/d8192/seed1.config.json new file mode 100644 index 0000000000000000000000000000000000000000..c1203fd88afc7297163a9f072ba95310ccede2ea --- /dev/null +++ b/pythia-160m/W_E/cross-snapshot-32/d8192/seed1.config.json @@ -0,0 +1,119 @@ +{ + "public_label": "Pythia-160M W_E crosscoder, 32 snapshots, d_sae=8192, seed 1", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0", + "wu_step_1", + "wu_step_2", + "wu_step_3", + "wu_step_4", + "wu_step_5", + "wu_step_6", + "wu_step_7", + "wu_step_8", + "wu_step_9", + "wu_step_10", + "wu_step_11", + "wu_step_12", + "wu_step_13", + "wu_step_14", + "wu_step_15", + "wu_step_16", + "wu_step_17", + "wu_step_18", + "wu_step_19", + "wu_step_20", + "wu_step_21", + "wu_step_22", + "wu_step_23", + "wu_step_24", + "wu_step_25", + "wu_step_26", + "wu_step_27", + "wu_step_28", + "wu_step_29", + "wu_step_30", + "wu_step_31" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 4.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 300, + "batch_size": 1024, + "expansion_factor": 10.667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.05, + "lr_warmup_fraction": 0.0, + "lr_decay_fraction": 0.0, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.580104140268709, + "reconstruction_mse": 0.41892518222332, + "mean_l0": 82.01199279785156, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 32, + "n_rows": 50304 + }, + "steps": [ + 0, + 1, + 2, + 4, + 8, + 16, + 32, + 64, + 128, + 256, + 512, + 1000, + 2000, + 3000, + 4000, + 5000, + 6000, + 7000, + 8000, + 9000, + 14000, + 21000, + 27000, + 34000, + 47000, + 61000, + 75000, + 89000, + 102000, + 116000, + 130000, + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 1, + "kind": "cross-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_E/cross-snapshot-32/d8192/seed1.md b/pythia-160m/W_E/cross-snapshot-32/d8192/seed1.md new file mode 100644 index 0000000000000000000000000000000000000000..365247950b371608a898cd7fd4afe9aca60c0778 --- /dev/null +++ b/pythia-160m/W_E/cross-snapshot-32/d8192/seed1.md @@ -0,0 +1,38 @@ +# Pythia-160M W_E crosscoder, 32 snapshots, d_sae=8192, seed 1 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 14000, 21000, 27000, 34000, 47000, 61000, 75000, 89000, 102000, 116000, 130000, 143000] pretraining +steps. + +## Quality +- Explained variance: 0.580104140268709 +- Mean L0: 82.01199279785156 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 32 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 4.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 300 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 1 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("seed1.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("seed1.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_E/cross-snapshot-32/d8192/seed1.safetensors b/pythia-160m/W_E/cross-snapshot-32/d8192/seed1.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..9e4210ab5e7dea8393dc7f34ece40aec22e39e63 --- /dev/null +++ b/pythia-160m/W_E/cross-snapshot-32/d8192/seed1.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:777f45bcec5b17bf9c7d080c79db5741cfdfb358743d307cc0ae69ce8c2fbb96 +size 1611792824 diff --git a/pythia-160m/W_E/cross-snapshot-32/d8192/seed2.config.json b/pythia-160m/W_E/cross-snapshot-32/d8192/seed2.config.json new file mode 100644 index 0000000000000000000000000000000000000000..9499114f2da7e985e6ba7b0dffbcb826cb342a8c --- /dev/null +++ b/pythia-160m/W_E/cross-snapshot-32/d8192/seed2.config.json @@ -0,0 +1,119 @@ +{ + "public_label": "Pythia-160M W_E crosscoder, 32 snapshots, d_sae=8192, seed 2", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0", + "wu_step_1", + "wu_step_2", + "wu_step_3", + "wu_step_4", + "wu_step_5", + "wu_step_6", + "wu_step_7", + "wu_step_8", + "wu_step_9", + "wu_step_10", + "wu_step_11", + "wu_step_12", + "wu_step_13", + "wu_step_14", + "wu_step_15", + "wu_step_16", + "wu_step_17", + "wu_step_18", + "wu_step_19", + "wu_step_20", + "wu_step_21", + "wu_step_22", + "wu_step_23", + "wu_step_24", + "wu_step_25", + "wu_step_26", + "wu_step_27", + "wu_step_28", + "wu_step_29", + "wu_step_30", + "wu_step_31" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 4.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 300, + "batch_size": 1024, + "expansion_factor": 10.667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.05, + "lr_warmup_fraction": 0.0, + "lr_decay_fraction": 0.0, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.5815380601296635, + "reconstruction_mse": 0.41749457716941835, + "mean_l0": 82.31076721191407, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 32, + "n_rows": 50304 + }, + "steps": [ + 0, + 1, + 2, + 4, + 8, + 16, + 32, + 64, + 128, + 256, + 512, + 1000, + 2000, + 3000, + 4000, + 5000, + 6000, + 7000, + 8000, + 9000, + 14000, + 21000, + 27000, + 34000, + 47000, + 61000, + 75000, + 89000, + 102000, + 116000, + 130000, + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 2, + "kind": "cross-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_E/cross-snapshot-32/d8192/seed2.md b/pythia-160m/W_E/cross-snapshot-32/d8192/seed2.md new file mode 100644 index 0000000000000000000000000000000000000000..a0f0ef4f7a0618afc50300660181799b6831ffb4 --- /dev/null +++ b/pythia-160m/W_E/cross-snapshot-32/d8192/seed2.md @@ -0,0 +1,38 @@ +# Pythia-160M W_E crosscoder, 32 snapshots, d_sae=8192, seed 2 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 14000, 21000, 27000, 34000, 47000, 61000, 75000, 89000, 102000, 116000, 130000, 143000] pretraining +steps. + +## Quality +- Explained variance: 0.5815380601296635 +- Mean L0: 82.31076721191407 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 32 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 4.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 300 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 2 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("seed2.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("seed2.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_E/cross-snapshot-32/d8192/seed2.safetensors b/pythia-160m/W_E/cross-snapshot-32/d8192/seed2.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..40556db5c626a4488eb96d33542826d683eaf618 --- /dev/null +++ b/pythia-160m/W_E/cross-snapshot-32/d8192/seed2.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:33cee54d608967373984411fffdff2da1bf39aca2a46e14ba75620a82b65e053 +size 1611792824 diff --git a/pythia-160m/W_E/cross-snapshot-32/d8192/seed3.config.json b/pythia-160m/W_E/cross-snapshot-32/d8192/seed3.config.json new file mode 100644 index 0000000000000000000000000000000000000000..076e202eaf0aa2e4e72b16bf47485ad3481f896a --- /dev/null +++ b/pythia-160m/W_E/cross-snapshot-32/d8192/seed3.config.json @@ -0,0 +1,119 @@ +{ + "public_label": "Pythia-160M W_E crosscoder, 32 snapshots, d_sae=8192, seed 3", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0", + "wu_step_1", + "wu_step_2", + "wu_step_3", + "wu_step_4", + "wu_step_5", + "wu_step_6", + "wu_step_7", + "wu_step_8", + "wu_step_9", + "wu_step_10", + "wu_step_11", + "wu_step_12", + "wu_step_13", + "wu_step_14", + "wu_step_15", + "wu_step_16", + "wu_step_17", + "wu_step_18", + "wu_step_19", + "wu_step_20", + "wu_step_21", + "wu_step_22", + "wu_step_23", + "wu_step_24", + "wu_step_25", + "wu_step_26", + "wu_step_27", + "wu_step_28", + "wu_step_29", + "wu_step_30", + "wu_step_31" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 4.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 300, + "batch_size": 1024, + "expansion_factor": 10.667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.05, + "lr_warmup_fraction": 0.0, + "lr_decay_fraction": 0.0, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.5805984141666259, + "reconstruction_mse": 0.41843205094337466, + "mean_l0": 82.13521545410157, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 32, + "n_rows": 50304 + }, + "steps": [ + 0, + 1, + 2, + 4, + 8, + 16, + 32, + 64, + 128, + 256, + 512, + 1000, + 2000, + 3000, + 4000, + 5000, + 6000, + 7000, + 8000, + 9000, + 14000, + 21000, + 27000, + 34000, + 47000, + 61000, + 75000, + 89000, + 102000, + 116000, + 130000, + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 3, + "kind": "cross-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_E/cross-snapshot-32/d8192/seed3.md b/pythia-160m/W_E/cross-snapshot-32/d8192/seed3.md new file mode 100644 index 0000000000000000000000000000000000000000..f051c0182a50b47c2fcfe750c338dceed630e8c2 --- /dev/null +++ b/pythia-160m/W_E/cross-snapshot-32/d8192/seed3.md @@ -0,0 +1,38 @@ +# Pythia-160M W_E crosscoder, 32 snapshots, d_sae=8192, seed 3 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 14000, 21000, 27000, 34000, 47000, 61000, 75000, 89000, 102000, 116000, 130000, 143000] pretraining +steps. + +## Quality +- Explained variance: 0.5805984141666259 +- Mean L0: 82.13521545410157 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 32 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 4.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 300 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 3 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("seed3.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("seed3.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_E/cross-snapshot-32/d8192/seed3.safetensors b/pythia-160m/W_E/cross-snapshot-32/d8192/seed3.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..514b99c5fbfbbe469ea0008a1b23be641a154d9e --- /dev/null +++ b/pythia-160m/W_E/cross-snapshot-32/d8192/seed3.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f54af8a2712eafc61e6c7ba219e88f36ccd21b03abaec4661b7e139753acf0bc +size 1611792824 diff --git a/pythia-160m/W_E/cross-snapshot-32/d8192/seed4.config.json b/pythia-160m/W_E/cross-snapshot-32/d8192/seed4.config.json new file mode 100644 index 0000000000000000000000000000000000000000..147f228deb3bf3f93a526ef6e39c86040e71d740 --- /dev/null +++ b/pythia-160m/W_E/cross-snapshot-32/d8192/seed4.config.json @@ -0,0 +1,119 @@ +{ + "public_label": "Pythia-160M W_E crosscoder, 32 snapshots, d_sae=8192, seed 4", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0", + "wu_step_1", + "wu_step_2", + "wu_step_3", + "wu_step_4", + "wu_step_5", + "wu_step_6", + "wu_step_7", + "wu_step_8", + "wu_step_9", + "wu_step_10", + "wu_step_11", + "wu_step_12", + "wu_step_13", + "wu_step_14", + "wu_step_15", + "wu_step_16", + "wu_step_17", + "wu_step_18", + "wu_step_19", + "wu_step_20", + "wu_step_21", + "wu_step_22", + "wu_step_23", + "wu_step_24", + "wu_step_25", + "wu_step_26", + "wu_step_27", + "wu_step_28", + "wu_step_29", + "wu_step_30", + "wu_step_31" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 4.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 300, + "batch_size": 1024, + "expansion_factor": 10.667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.05, + "lr_warmup_fraction": 0.0, + "lr_decay_fraction": 0.0, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.5825219599393371, + "reconstruction_mse": 0.4165129518508911, + "mean_l0": 82.5084307861328, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 32, + "n_rows": 50304 + }, + "steps": [ + 0, + 1, + 2, + 4, + 8, + 16, + 32, + 64, + 128, + 256, + 512, + 1000, + 2000, + 3000, + 4000, + 5000, + 6000, + 7000, + 8000, + 9000, + 14000, + 21000, + 27000, + 34000, + 47000, + 61000, + 75000, + 89000, + 102000, + 116000, + 130000, + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 4, + "kind": "cross-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_E/cross-snapshot-32/d8192/seed4.md b/pythia-160m/W_E/cross-snapshot-32/d8192/seed4.md new file mode 100644 index 0000000000000000000000000000000000000000..b00b3451eb0ec6b4e71877f432866da2c5bc2029 --- /dev/null +++ b/pythia-160m/W_E/cross-snapshot-32/d8192/seed4.md @@ -0,0 +1,38 @@ +# Pythia-160M W_E crosscoder, 32 snapshots, d_sae=8192, seed 4 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 14000, 21000, 27000, 34000, 47000, 61000, 75000, 89000, 102000, 116000, 130000, 143000] pretraining +steps. + +## Quality +- Explained variance: 0.5825219599393371 +- Mean L0: 82.5084307861328 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 32 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 4.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 300 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 4 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("seed4.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("seed4.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_E/cross-snapshot-32/d8192/seed4.safetensors b/pythia-160m/W_E/cross-snapshot-32/d8192/seed4.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..aca6a236dcc45817d2d005ceb4035fcac7201e7f --- /dev/null +++ b/pythia-160m/W_E/cross-snapshot-32/d8192/seed4.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:712affee5958492d3f114b48e3f718a8b366513fab63b4d48a9b8f4c0dcf753b +size 1611792824 diff --git a/pythia-160m/W_U/architecture-comparison/d8192/batchtopk/seed0.config.json b/pythia-160m/W_U/architecture-comparison/d8192/batchtopk/seed0.config.json new file mode 100644 index 0000000000000000000000000000000000000000..12966181d31b66c4bf2959bbca5f1d0a5fca18f8 --- /dev/null +++ b/pythia-160m/W_U/architecture-comparison/d8192/batchtopk/seed0.config.json @@ -0,0 +1,74 @@ +{ + "public_label": "Pythia-160M W_U crosscoder (batchtopk activation), 32 snapshots, d_sae=8192, seed 0", + "config": { + "arch": "batchtopk", + "d_sae": 8192, + "n_snapshots": 32, + "d_model": 768, + "batchtopk_k": 203 + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "gated_l1_coefficient": 0.3, + "init_threshold": 0.1, + "bandwidth": 4.0, + "frequency_scale": 0.01, + "tanh_stretch": 1.0, + "jumprelu_lr_factor": 0.1, + "n_epochs": 100, + "batch_size": 1024, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.0, + "lr_decay_fraction": 0.0, + "git_hash": "unknown", + "amp_dtype": "fp32" + }, + "quality": { + "explained_variance": 0.7249162974116965, + "reconstruction_mse": 0.2749960619211197, + "mean_l0": 203.0, + "d_sae": 8192, + "n_snapshots": 32, + "n_rows": 50304 + }, + "steps": [ + 0, + 1, + 2, + 4, + 8, + 16, + 32, + 64, + 128, + 256, + 512, + 1000, + 2000, + 3000, + 4000, + 5000, + 6000, + 7000, + 8000, + 9000, + 14000, + 21000, + 27000, + 34000, + 47000, + 61000, + 75000, + 89000, + 102000, + 116000, + 130000, + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "cross-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/architecture-comparison/d8192/batchtopk/seed0.md b/pythia-160m/W_U/architecture-comparison/d8192/batchtopk/seed0.md new file mode 100644 index 0000000000000000000000000000000000000000..b7d3d50d44abef99d7f55eb7d6fdb38ff7bc138c --- /dev/null +++ b/pythia-160m/W_U/architecture-comparison/d8192/batchtopk/seed0.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U crosscoder (batchtopk activation), 32 snapshots, d_sae=8192, seed 0 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 14000, 21000, 27000, 34000, 47000, 61000, 75000, 89000, 102000, 116000, 130000, 143000] pretraining +steps. + +## Quality +- Explained variance: 0.7249162974116965 +- Mean L0: 203.0 +- Dead-feature rate: — +- d_sae: 8192 +- n_snapshots: 32 + +## Training recipe (key hparams) +- Activation: JumpReLU +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("seed0.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("seed0.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/architecture-comparison/d8192/batchtopk/seed0.safetensors b/pythia-160m/W_U/architecture-comparison/d8192/batchtopk/seed0.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..a6d97e950d3bd020d5391358b97984adc32ef532 --- /dev/null +++ b/pythia-160m/W_U/architecture-comparison/d8192/batchtopk/seed0.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d9e434afc4191e631e88fc31a1ff33068b24a26c6650066b8ec67e463ec0c790 +size 1611759944 diff --git a/pythia-160m/W_U/architecture-comparison/d8192/gated-retuned/seed0.config.json b/pythia-160m/W_U/architecture-comparison/d8192/gated-retuned/seed0.config.json new file mode 100644 index 0000000000000000000000000000000000000000..1484d77cc57b1e3b9059cb1689847e008f6581f8 --- /dev/null +++ b/pythia-160m/W_U/architecture-comparison/d8192/gated-retuned/seed0.config.json @@ -0,0 +1,74 @@ +{ + "public_label": "Pythia-160M W_U crosscoder (Gated activation, L1=0.05), 32 snapshots, d_sae=8192, seed 0", + "config": { + "arch": "gated", + "d_sae": 8192, + "n_snapshots": 32, + "d_model": 768, + "batchtopk_k": null + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "gated_l1_coefficient": 0.05, + "init_threshold": 0.1, + "bandwidth": 4.0, + "frequency_scale": 0.01, + "tanh_stretch": 1.0, + "jumprelu_lr_factor": 0.1, + "n_epochs": 100, + "batch_size": 1024, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.0, + "lr_decay_fraction": 0.0, + "git_hash": "unknown", + "amp_dtype": "fp32" + }, + "quality": { + "explained_variance": 0.827462127951851, + "reconstruction_mse": 0.17248290210962294, + "mean_l0": 653.64568359375, + "d_sae": 8192, + "n_snapshots": 32, + "n_rows": 50304 + }, + "steps": [ + 0, + 1, + 2, + 4, + 8, + 16, + 32, + 64, + 128, + 256, + 512, + 1000, + 2000, + 3000, + 4000, + 5000, + 6000, + 7000, + 8000, + 9000, + 14000, + 21000, + 27000, + 34000, + 47000, + 61000, + 75000, + 89000, + 102000, + 116000, + 130000, + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "cross-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/architecture-comparison/d8192/gated-retuned/seed0.md b/pythia-160m/W_U/architecture-comparison/d8192/gated-retuned/seed0.md new file mode 100644 index 0000000000000000000000000000000000000000..96a798df99c2a79d132aea9458e4537ed442bd74 --- /dev/null +++ b/pythia-160m/W_U/architecture-comparison/d8192/gated-retuned/seed0.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U crosscoder (Gated activation, L1=0.05), 32 snapshots, d_sae=8192, seed 0 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 14000, 21000, 27000, 34000, 47000, 61000, 75000, 89000, 102000, 116000, 130000, 143000] pretraining +steps. + +## Quality +- Explained variance: 0.827462127951851 +- Mean L0: 653.64568359375 +- Dead-feature rate: — +- d_sae: 8192 +- n_snapshots: 32 + +## Training recipe (key hparams) +- Activation: JumpReLU +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("seed0.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("seed0.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/architecture-comparison/d8192/gated-retuned/seed0.safetensors b/pythia-160m/W_U/architecture-comparison/d8192/gated-retuned/seed0.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..7611503959a9af26b7add1e8cda91ea5b9a3c071 --- /dev/null +++ b/pythia-160m/W_U/architecture-comparison/d8192/gated-retuned/seed0.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:197aaa8bcccd820a827454ad03ae72d0f3306b840719a7251d96e6bf8f4c4120 +size 1611858536 diff --git a/pythia-160m/W_U/architecture-comparison/d8192/gated/seed0.config.json b/pythia-160m/W_U/architecture-comparison/d8192/gated/seed0.config.json new file mode 100644 index 0000000000000000000000000000000000000000..d3ab87f8570220fb3acae06942111a2c2775478b --- /dev/null +++ b/pythia-160m/W_U/architecture-comparison/d8192/gated/seed0.config.json @@ -0,0 +1,74 @@ +{ + "public_label": "Pythia-160M W_U crosscoder (gated activation), 32 snapshots, d_sae=8192, seed 0", + "config": { + "arch": "gated", + "d_sae": 8192, + "n_snapshots": 32, + "d_model": 768, + "batchtopk_k": null + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "gated_l1_coefficient": 0.3, + "init_threshold": 0.1, + "bandwidth": 4.0, + "frequency_scale": 0.01, + "tanh_stretch": 1.0, + "jumprelu_lr_factor": 0.1, + "n_epochs": 100, + "batch_size": 1024, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.0, + "lr_decay_fraction": 0.0, + "git_hash": "unknown", + "amp_dtype": "fp32" + }, + "quality": { + "explained_variance": 0.21442979106557436, + "reconstruction_mse": 0.7853199291229248, + "mean_l0": 11.51853515625, + "d_sae": 8192, + "n_snapshots": 32, + "n_rows": 50304 + }, + "steps": [ + 0, + 1, + 2, + 4, + 8, + 16, + 32, + 64, + 128, + 256, + 512, + 1000, + 2000, + 3000, + 4000, + 5000, + 6000, + 7000, + 8000, + 9000, + 14000, + 21000, + 27000, + 34000, + 47000, + 61000, + 75000, + 89000, + 102000, + 116000, + 130000, + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "cross-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/architecture-comparison/d8192/gated/seed0.md b/pythia-160m/W_U/architecture-comparison/d8192/gated/seed0.md new file mode 100644 index 0000000000000000000000000000000000000000..bf91859384b1546ea9ce74f3d9cfb0257df0674c --- /dev/null +++ b/pythia-160m/W_U/architecture-comparison/d8192/gated/seed0.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U crosscoder (gated activation), 32 snapshots, d_sae=8192, seed 0 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 14000, 21000, 27000, 34000, 47000, 61000, 75000, 89000, 102000, 116000, 130000, 143000] pretraining +steps. + +## Quality +- Explained variance: 0.21442979106557436 +- Mean L0: 11.51853515625 +- Dead-feature rate: — +- d_sae: 8192 +- n_snapshots: 32 + +## Training recipe (key hparams) +- Activation: JumpReLU +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("seed0.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("seed0.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/architecture-comparison/d8192/gated/seed0.safetensors b/pythia-160m/W_U/architecture-comparison/d8192/gated/seed0.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..e852eec23c4c5cf77cbbdefeb3b06725dece2450 --- /dev/null +++ b/pythia-160m/W_U/architecture-comparison/d8192/gated/seed0.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:10b27d935481b43900bab227acea5be346d403e163d2976abb3dc0676adde955 +size 1611858536 diff --git a/pythia-160m/W_U/cross-snapshot-16/d8192/seed0.config.json b/pythia-160m/W_U/cross-snapshot-16/d8192/seed0.config.json new file mode 100644 index 0000000000000000000000000000000000000000..d383a0952634ea5df2defe4d95b242757a7e493d --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-16/d8192/seed0.config.json @@ -0,0 +1,87 @@ +{ + "public_label": "Pythia-160M W_U crosscoder, 16-snapshot downsample, d_sae=8192, seed 0", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0", + "wu_step_1", + "wu_step_2", + "wu_step_3", + "wu_step_4", + "wu_step_5", + "wu_step_6", + "wu_step_7", + "wu_step_8", + "wu_step_9", + "wu_step_10", + "wu_step_11", + "wu_step_12", + "wu_step_13", + "wu_step_14", + "wu_step_15" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 300, + "batch_size": 1024, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.05, + "lr_warmup_fraction": 0.0, + "lr_decay_fraction": 0.0, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.7734329300485889, + "reconstruction_mse": 0.22638385832309724, + "mean_l0": 215.73649780273436, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 16, + "n_rows": 50304 + }, + "steps": [ + 0, + 2, + 8, + 32, + 128, + 512, + 2000, + 4000, + 6000, + 8000, + 14000, + 27000, + 47000, + 75000, + 102000, + 130000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "cross-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/cross-snapshot-16/d8192/seed0.md b/pythia-160m/W_U/cross-snapshot-16/d8192/seed0.md new file mode 100644 index 0000000000000000000000000000000000000000..39db4b34dd67cd2b712c227d1bb9772ee3bc227e --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-16/d8192/seed0.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U crosscoder, 16-snapshot downsample, d_sae=8192, seed 0 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [0, 2, 8, 32, 128, 512, 2000, 4000, 6000, 8000, 14000, 27000, 47000, 75000, 102000, 130000] pretraining +steps. + +## Quality +- Explained variance: 0.7734329300485889 +- Mean L0: 215.73649780273436 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 16 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 300 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("seed0.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("seed0.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/cross-snapshot-16/d8192/seed0.safetensors b/pythia-160m/W_U/cross-snapshot-16/d8192/seed0.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..f0a73ec51189ea556843c0e26feca687562d697a --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-16/d8192/seed0.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8496ae3b72b23b0a65698e3e424eddc411bd6ac919f1aaa915ae06a147b3803a +size 805913008 diff --git a/pythia-160m/W_U/cross-snapshot-32/d16384/seed0.config.json b/pythia-160m/W_U/cross-snapshot-32/d16384/seed0.config.json new file mode 100644 index 0000000000000000000000000000000000000000..51c20d7c0ab522a8314a645e10116143c70d80ab --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d16384/seed0.config.json @@ -0,0 +1,111 @@ +{ + "public_label": "Pythia-160M W_U crosscoder, 32 snapshots, d_sae=16384, seed 0", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 21.334, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0", + "wu_step_1", + "wu_step_2", + "wu_step_3", + "wu_step_4", + "wu_step_5", + "wu_step_6", + "wu_step_7", + "wu_step_8", + "wu_step_9", + "wu_step_10", + "wu_step_11", + "wu_step_12", + "wu_step_13", + "wu_step_14", + "wu_step_15", + "wu_step_16", + "wu_step_17", + "wu_step_18", + "wu_step_19", + "wu_step_20", + "wu_step_21", + "wu_step_22", + "wu_step_23", + "wu_step_24", + "wu_step_25", + "wu_step_26", + "wu_step_27", + "wu_step_28", + "wu_step_29", + "wu_step_30", + "wu_step_31" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 4.0, + "jumprelu_lr_factor": 0.1, + "n_epochs": 300, + "batch_size": 1024, + "expansion_factor": 21.334, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.05, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.7802076016978367, + "reconstruction_mse": 0.21972236931324005, + "mean_l0": 103.0708544921875, + "d_sae": 16384, + "n_snapshots": 32, + "n_rows": 50304 + }, + "steps": [ + 0, + 1, + 2, + 4, + 8, + 16, + 32, + 64, + 128, + 256, + 512, + 1000, + 2000, + 3000, + 4000, + 5000, + 6000, + 7000, + 8000, + 9000, + 14000, + 21000, + 27000, + 34000, + 47000, + 61000, + 75000, + 89000, + 102000, + 116000, + 130000, + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "cross-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/cross-snapshot-32/d16384/seed0.md b/pythia-160m/W_U/cross-snapshot-32/d16384/seed0.md new file mode 100644 index 0000000000000000000000000000000000000000..5e55214336d4e987062ba6201c969b253866f7d6 --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d16384/seed0.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U crosscoder, 32 snapshots, d_sae=16384, seed 0 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 14000, 21000, 27000, 34000, 47000, 61000, 75000, 89000, 102000, 116000, 130000, 143000] pretraining +steps. + +## Quality +- Explained variance: 0.7802076016978367 +- Mean L0: 103.0708544921875 +- Dead-feature rate: — +- d_sae: 16384 +- n_snapshots: 32 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 4.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: — +- Optimizer: adam +- Epochs: 300 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("seed0.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("seed0.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/cross-snapshot-32/d16384/seed0.safetensors b/pythia-160m/W_U/cross-snapshot-32/d16384/seed0.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..1b150402663dbd0c68b23651c0fe8d2416bd6235 --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d16384/seed0.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1a9b590ef230642ca44ea8a562155785aacd314bc757a845fc88cb3e12f72af1 +size 3223486912 diff --git a/pythia-160m/W_U/cross-snapshot-32/d24576/seed0.config.json b/pythia-160m/W_U/cross-snapshot-32/d24576/seed0.config.json new file mode 100644 index 0000000000000000000000000000000000000000..5ea979673332a25fa262b4df4ac3d0ddab872fe9 --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d24576/seed0.config.json @@ -0,0 +1,119 @@ +{ + "public_label": "Pythia-160M W_U crosscoder, 32 snapshots, d_sae=24576, seed 0", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 32.0, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0", + "wu_step_1", + "wu_step_2", + "wu_step_3", + "wu_step_4", + "wu_step_5", + "wu_step_6", + "wu_step_7", + "wu_step_8", + "wu_step_9", + "wu_step_10", + "wu_step_11", + "wu_step_12", + "wu_step_13", + "wu_step_14", + "wu_step_15", + "wu_step_16", + "wu_step_17", + "wu_step_18", + "wu_step_19", + "wu_step_20", + "wu_step_21", + "wu_step_22", + "wu_step_23", + "wu_step_24", + "wu_step_25", + "wu_step_26", + "wu_step_27", + "wu_step_28", + "wu_step_29", + "wu_step_30", + "wu_step_31" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 300, + "batch_size": 1024, + "expansion_factor": 32.0, + "d_sae": 24576, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.05, + "lr_warmup_fraction": 0.0, + "lr_decay_fraction": 0.0, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.9196719747097695, + "reconstruction_mse": 0.08030243307352065, + "mean_l0": 285.9946594238281, + "dead_rate": 0.0, + "d_sae": 24576, + "n_snapshots": 32, + "n_rows": 50304 + }, + "steps": [ + 0, + 1, + 2, + 4, + 8, + 16, + 32, + 64, + 128, + 256, + 512, + 1000, + 2000, + 3000, + 4000, + 5000, + 6000, + 7000, + 8000, + 9000, + 14000, + 21000, + 27000, + 34000, + 47000, + 61000, + 75000, + 89000, + 102000, + 116000, + 130000, + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "cross-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/cross-snapshot-32/d24576/seed0.md b/pythia-160m/W_U/cross-snapshot-32/d24576/seed0.md new file mode 100644 index 0000000000000000000000000000000000000000..542a8abcdd3d689d8bb164f9040f25b66414a993 --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d24576/seed0.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U crosscoder, 32 snapshots, d_sae=24576, seed 0 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 14000, 21000, 27000, 34000, 47000, 61000, 75000, 89000, 102000, 116000, 130000, 143000] pretraining +steps. + +## Quality +- Explained variance: 0.9196719747097695 +- Mean L0: 285.9946594238281 +- Dead-feature rate: 0.0 +- d_sae: 24576 +- n_snapshots: 32 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 300 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("seed0.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("seed0.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/cross-snapshot-32/d24576/seed0.safetensors b/pythia-160m/W_U/cross-snapshot-32/d24576/seed0.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..2fc405b97c5db993755533a82798daf215eaecaa --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d24576/seed0.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5998eba9b1f0f94f7579c692dcee2de8a2875f2f80f3fb4e127f6024f411e815 +size 4835180992 diff --git a/pythia-160m/W_U/cross-snapshot-32/d24576/seed1.config.json b/pythia-160m/W_U/cross-snapshot-32/d24576/seed1.config.json new file mode 100644 index 0000000000000000000000000000000000000000..e2629225176130e7954a007c59f39f36937043f0 --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d24576/seed1.config.json @@ -0,0 +1,119 @@ +{ + "public_label": "Pythia-160M W_U crosscoder, 32 snapshots, d_sae=24576, seed 1", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 32.0, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0", + "wu_step_1", + "wu_step_2", + "wu_step_3", + "wu_step_4", + "wu_step_5", + "wu_step_6", + "wu_step_7", + "wu_step_8", + "wu_step_9", + "wu_step_10", + "wu_step_11", + "wu_step_12", + "wu_step_13", + "wu_step_14", + "wu_step_15", + "wu_step_16", + "wu_step_17", + "wu_step_18", + "wu_step_19", + "wu_step_20", + "wu_step_21", + "wu_step_22", + "wu_step_23", + "wu_step_24", + "wu_step_25", + "wu_step_26", + "wu_step_27", + "wu_step_28", + "wu_step_29", + "wu_step_30", + "wu_step_31" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 300, + "batch_size": 1024, + "expansion_factor": 32.0, + "d_sae": 24576, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.05, + "lr_warmup_fraction": 0.0, + "lr_decay_fraction": 0.0, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.9195844996740984, + "reconstruction_mse": 0.08038988023996353, + "mean_l0": 286.09990661621094, + "dead_rate": 0.0, + "d_sae": 24576, + "n_snapshots": 32, + "n_rows": 50304 + }, + "steps": [ + 0, + 1, + 2, + 4, + 8, + 16, + 32, + 64, + 128, + 256, + 512, + 1000, + 2000, + 3000, + 4000, + 5000, + 6000, + 7000, + 8000, + 9000, + 14000, + 21000, + 27000, + 34000, + 47000, + 61000, + 75000, + 89000, + 102000, + 116000, + 130000, + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 1, + "kind": "cross-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/cross-snapshot-32/d24576/seed1.md b/pythia-160m/W_U/cross-snapshot-32/d24576/seed1.md new file mode 100644 index 0000000000000000000000000000000000000000..3d21d15199a9bba6a0c92b42f0a0574870786807 --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d24576/seed1.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U crosscoder, 32 snapshots, d_sae=24576, seed 1 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 14000, 21000, 27000, 34000, 47000, 61000, 75000, 89000, 102000, 116000, 130000, 143000] pretraining +steps. + +## Quality +- Explained variance: 0.9195844996740984 +- Mean L0: 286.09990661621094 +- Dead-feature rate: 0.0 +- d_sae: 24576 +- n_snapshots: 32 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 300 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 1 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("seed1.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("seed1.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/cross-snapshot-32/d24576/seed1.safetensors b/pythia-160m/W_U/cross-snapshot-32/d24576/seed1.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..2fddfe67fb6360d6efdbe46795c7d8d3228a6967 --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d24576/seed1.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c071477b1fbd48eb9033488e6943f35e6b9eba95a7839dfc3052701e1ceca4ff +size 4835180992 diff --git a/pythia-160m/W_U/cross-snapshot-32/d24576/seed2.config.json b/pythia-160m/W_U/cross-snapshot-32/d24576/seed2.config.json new file mode 100644 index 0000000000000000000000000000000000000000..5af11388cde317f984c519a367697b3552c65268 --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d24576/seed2.config.json @@ -0,0 +1,119 @@ +{ + "public_label": "Pythia-160M W_U crosscoder, 32 snapshots, d_sae=24576, seed 2", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 32.0, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0", + "wu_step_1", + "wu_step_2", + "wu_step_3", + "wu_step_4", + "wu_step_5", + "wu_step_6", + "wu_step_7", + "wu_step_8", + "wu_step_9", + "wu_step_10", + "wu_step_11", + "wu_step_12", + "wu_step_13", + "wu_step_14", + "wu_step_15", + "wu_step_16", + "wu_step_17", + "wu_step_18", + "wu_step_19", + "wu_step_20", + "wu_step_21", + "wu_step_22", + "wu_step_23", + "wu_step_24", + "wu_step_25", + "wu_step_26", + "wu_step_27", + "wu_step_28", + "wu_step_29", + "wu_step_30", + "wu_step_31" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 300, + "batch_size": 1024, + "expansion_factor": 32.0, + "d_sae": 24576, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.05, + "lr_warmup_fraction": 0.0, + "lr_decay_fraction": 0.0, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.9196765967343927, + "reconstruction_mse": 0.08029781252145768, + "mean_l0": 286.3446789550781, + "dead_rate": 0.0, + "d_sae": 24576, + "n_snapshots": 32, + "n_rows": 50304 + }, + "steps": [ + 0, + 1, + 2, + 4, + 8, + 16, + 32, + 64, + 128, + 256, + 512, + 1000, + 2000, + 3000, + 4000, + 5000, + 6000, + 7000, + 8000, + 9000, + 14000, + 21000, + 27000, + 34000, + 47000, + 61000, + 75000, + 89000, + 102000, + 116000, + 130000, + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 2, + "kind": "cross-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/cross-snapshot-32/d24576/seed2.md b/pythia-160m/W_U/cross-snapshot-32/d24576/seed2.md new file mode 100644 index 0000000000000000000000000000000000000000..3b27a016499a2b5fecc5e1bc9a547a0fe91f0515 --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d24576/seed2.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U crosscoder, 32 snapshots, d_sae=24576, seed 2 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 14000, 21000, 27000, 34000, 47000, 61000, 75000, 89000, 102000, 116000, 130000, 143000] pretraining +steps. + +## Quality +- Explained variance: 0.9196765967343927 +- Mean L0: 286.3446789550781 +- Dead-feature rate: 0.0 +- d_sae: 24576 +- n_snapshots: 32 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 300 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 2 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("seed2.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("seed2.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/cross-snapshot-32/d24576/seed2.safetensors b/pythia-160m/W_U/cross-snapshot-32/d24576/seed2.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..49c0204e94a44ab3cd564afb36144d852b227621 --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d24576/seed2.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:63a4a1630f355333053e48278a4c4cb5e6616e1ab92700e7d121cb8960083abf +size 4835180992 diff --git a/pythia-160m/W_U/cross-snapshot-32/d8192/seed0.config.json b/pythia-160m/W_U/cross-snapshot-32/d8192/seed0.config.json new file mode 100644 index 0000000000000000000000000000000000000000..ff605063de45493d7f1740e9996c5d6eb0f8554d --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d8192/seed0.config.json @@ -0,0 +1,111 @@ +{ + "public_label": "Pythia-160M W_U crosscoder, 32 snapshots, d_sae=8192, seed 0", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0", + "wu_step_1", + "wu_step_2", + "wu_step_3", + "wu_step_4", + "wu_step_5", + "wu_step_6", + "wu_step_7", + "wu_step_8", + "wu_step_9", + "wu_step_10", + "wu_step_11", + "wu_step_12", + "wu_step_13", + "wu_step_14", + "wu_step_15", + "wu_step_16", + "wu_step_17", + "wu_step_18", + "wu_step_19", + "wu_step_20", + "wu_step_21", + "wu_step_22", + "wu_step_23", + "wu_step_24", + "wu_step_25", + "wu_step_26", + "wu_step_27", + "wu_step_28", + "wu_step_29", + "wu_step_30", + "wu_step_31" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "n_epochs": 300, + "batch_size": 1024, + "expansion_factor": 10.6667, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.7763740924044573, + "reconstruction_mse": 0.22355465799570085, + "mean_l0": 203.14466369628906, + "d_sae": 8192, + "n_snapshots": 32, + "n_rows": 50304 + }, + "steps": [ + 0, + 1, + 2, + 4, + 8, + 16, + 32, + 64, + 128, + 256, + 512, + 1000, + 2000, + 3000, + 4000, + 5000, + 6000, + 7000, + 8000, + 9000, + 14000, + 21000, + 27000, + 34000, + 47000, + 61000, + 75000, + 89000, + 102000, + 116000, + 130000, + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "cross-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/cross-snapshot-32/d8192/seed0.md b/pythia-160m/W_U/cross-snapshot-32/d8192/seed0.md new file mode 100644 index 0000000000000000000000000000000000000000..cc14a6ca92a6b3fc389db2f3fb4d770fbb2ebc0f --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d8192/seed0.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U crosscoder, 32 snapshots, d_sae=8192, seed 0 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 14000, 21000, 27000, 34000, 47000, 61000, 75000, 89000, 102000, 116000, 130000, 143000] pretraining +steps. + +## Quality +- Explained variance: 0.7763740924044573 +- Mean L0: 203.14466369628906 +- Dead-feature rate: — +- d_sae: 8192 +- n_snapshots: 32 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: — +- Optimizer: adam +- Epochs: 300 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("seed0.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("seed0.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/cross-snapshot-32/d8192/seed0.safetensors b/pythia-160m/W_U/cross-snapshot-32/d8192/seed0.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..46f86b9db34b9a74b5ea05272d9a0b91e5bcde36 --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d8192/seed0.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d3420bd2a87dd07e261c49f62233836eb79f6f6a91ead7f1359a0162509fe91d +size 1611792824 diff --git a/pythia-160m/W_U/cross-snapshot-32/d8192/seed1.config.json b/pythia-160m/W_U/cross-snapshot-32/d8192/seed1.config.json new file mode 100644 index 0000000000000000000000000000000000000000..de2c5abe702b81338af6edbdf1963a48f5da6b47 --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d8192/seed1.config.json @@ -0,0 +1,111 @@ +{ + "public_label": "Pythia-160M W_U crosscoder, 32 snapshots, d_sae=8192, seed 1", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0", + "wu_step_1", + "wu_step_2", + "wu_step_3", + "wu_step_4", + "wu_step_5", + "wu_step_6", + "wu_step_7", + "wu_step_8", + "wu_step_9", + "wu_step_10", + "wu_step_11", + "wu_step_12", + "wu_step_13", + "wu_step_14", + "wu_step_15", + "wu_step_16", + "wu_step_17", + "wu_step_18", + "wu_step_19", + "wu_step_20", + "wu_step_21", + "wu_step_22", + "wu_step_23", + "wu_step_24", + "wu_step_25", + "wu_step_26", + "wu_step_27", + "wu_step_28", + "wu_step_29", + "wu_step_30", + "wu_step_31" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "n_epochs": 300, + "batch_size": 1024, + "expansion_factor": 10.6667, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.7763600972452054, + "reconstruction_mse": 0.22356864869594573, + "mean_l0": 203.26546752929687, + "d_sae": 8192, + "n_snapshots": 32, + "n_rows": 50304 + }, + "steps": [ + 0, + 1, + 2, + 4, + 8, + 16, + 32, + 64, + 128, + 256, + 512, + 1000, + 2000, + 3000, + 4000, + 5000, + 6000, + 7000, + 8000, + 9000, + 14000, + 21000, + 27000, + 34000, + 47000, + 61000, + 75000, + 89000, + 102000, + 116000, + 130000, + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 1, + "kind": "cross-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/cross-snapshot-32/d8192/seed1.md b/pythia-160m/W_U/cross-snapshot-32/d8192/seed1.md new file mode 100644 index 0000000000000000000000000000000000000000..32aa7dcad58b3516c955e9c8675dd28c01beb938 --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d8192/seed1.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U crosscoder, 32 snapshots, d_sae=8192, seed 1 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 14000, 21000, 27000, 34000, 47000, 61000, 75000, 89000, 102000, 116000, 130000, 143000] pretraining +steps. + +## Quality +- Explained variance: 0.7763600972452054 +- Mean L0: 203.26546752929687 +- Dead-feature rate: — +- d_sae: 8192 +- n_snapshots: 32 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: — +- Optimizer: adam +- Epochs: 300 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 1 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("seed1.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("seed1.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/cross-snapshot-32/d8192/seed1.safetensors b/pythia-160m/W_U/cross-snapshot-32/d8192/seed1.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..fd761794a5a112852ebbcdbb2d1edb4a99423c53 --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d8192/seed1.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1d037cc103f190c925f9431b50de7e299714d452b239752553b07a3c7bcbc2c6 +size 1611792824 diff --git a/pythia-160m/W_U/cross-snapshot-32/d8192/seed2.config.json b/pythia-160m/W_U/cross-snapshot-32/d8192/seed2.config.json new file mode 100644 index 0000000000000000000000000000000000000000..be80ea716e239d53f18e52c8351d12dc47c6b059 --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d8192/seed2.config.json @@ -0,0 +1,111 @@ +{ + "public_label": "Pythia-160M W_U crosscoder, 32 snapshots, d_sae=8192, seed 2", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0", + "wu_step_1", + "wu_step_2", + "wu_step_3", + "wu_step_4", + "wu_step_5", + "wu_step_6", + "wu_step_7", + "wu_step_8", + "wu_step_9", + "wu_step_10", + "wu_step_11", + "wu_step_12", + "wu_step_13", + "wu_step_14", + "wu_step_15", + "wu_step_16", + "wu_step_17", + "wu_step_18", + "wu_step_19", + "wu_step_20", + "wu_step_21", + "wu_step_22", + "wu_step_23", + "wu_step_24", + "wu_step_25", + "wu_step_26", + "wu_step_27", + "wu_step_28", + "wu_step_29", + "wu_step_30", + "wu_step_31" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "n_epochs": 300, + "batch_size": 1024, + "expansion_factor": 10.6667, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.7764523546944131, + "reconstruction_mse": 0.22347642064094544, + "mean_l0": 203.14273071289062, + "d_sae": 8192, + "n_snapshots": 32, + "n_rows": 50304 + }, + "steps": [ + 0, + 1, + 2, + 4, + 8, + 16, + 32, + 64, + 128, + 256, + 512, + 1000, + 2000, + 3000, + 4000, + 5000, + 6000, + 7000, + 8000, + 9000, + 14000, + 21000, + 27000, + 34000, + 47000, + 61000, + 75000, + 89000, + 102000, + 116000, + 130000, + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 2, + "kind": "cross-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/cross-snapshot-32/d8192/seed2.md b/pythia-160m/W_U/cross-snapshot-32/d8192/seed2.md new file mode 100644 index 0000000000000000000000000000000000000000..d58789c6008a34cf6654e830d45d0b3534f1dc83 --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d8192/seed2.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U crosscoder, 32 snapshots, d_sae=8192, seed 2 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 14000, 21000, 27000, 34000, 47000, 61000, 75000, 89000, 102000, 116000, 130000, 143000] pretraining +steps. + +## Quality +- Explained variance: 0.7764523546944131 +- Mean L0: 203.14273071289062 +- Dead-feature rate: — +- d_sae: 8192 +- n_snapshots: 32 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: — +- Optimizer: adam +- Epochs: 300 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 2 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("seed2.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("seed2.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/cross-snapshot-32/d8192/seed2.safetensors b/pythia-160m/W_U/cross-snapshot-32/d8192/seed2.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..c853c49591cdc5a958669adf85376bf507fc4297 --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d8192/seed2.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:cdf8151eeb0aa1422d604022b2807a99cc240bb8fd3187e3b5f4ad8d8109d072 +size 1611792824 diff --git a/pythia-160m/W_U/cross-snapshot-32/d8192/seed3.config.json b/pythia-160m/W_U/cross-snapshot-32/d8192/seed3.config.json new file mode 100644 index 0000000000000000000000000000000000000000..53c2f1114759604506fa031342696f90b3435bb6 --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d8192/seed3.config.json @@ -0,0 +1,111 @@ +{ + "public_label": "Pythia-160M W_U crosscoder, 32 snapshots, d_sae=8192, seed 3", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0", + "wu_step_1", + "wu_step_2", + "wu_step_3", + "wu_step_4", + "wu_step_5", + "wu_step_6", + "wu_step_7", + "wu_step_8", + "wu_step_9", + "wu_step_10", + "wu_step_11", + "wu_step_12", + "wu_step_13", + "wu_step_14", + "wu_step_15", + "wu_step_16", + "wu_step_17", + "wu_step_18", + "wu_step_19", + "wu_step_20", + "wu_step_21", + "wu_step_22", + "wu_step_23", + "wu_step_24", + "wu_step_25", + "wu_step_26", + "wu_step_27", + "wu_step_28", + "wu_step_29", + "wu_step_30", + "wu_step_31" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "n_epochs": 300, + "batch_size": 1024, + "expansion_factor": 10.6667, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.7762290236129058, + "reconstruction_mse": 0.2236996805667877, + "mean_l0": 203.09911560058595, + "d_sae": 8192, + "n_snapshots": 32, + "n_rows": 50304 + }, + "steps": [ + 0, + 1, + 2, + 4, + 8, + 16, + 32, + 64, + 128, + 256, + 512, + 1000, + 2000, + 3000, + 4000, + 5000, + 6000, + 7000, + 8000, + 9000, + 14000, + 21000, + 27000, + 34000, + 47000, + 61000, + 75000, + 89000, + 102000, + 116000, + 130000, + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 3, + "kind": "cross-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/cross-snapshot-32/d8192/seed3.md b/pythia-160m/W_U/cross-snapshot-32/d8192/seed3.md new file mode 100644 index 0000000000000000000000000000000000000000..51d3d7e9e32beba630a3eff77b1f439393f1eb11 --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d8192/seed3.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U crosscoder, 32 snapshots, d_sae=8192, seed 3 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 14000, 21000, 27000, 34000, 47000, 61000, 75000, 89000, 102000, 116000, 130000, 143000] pretraining +steps. + +## Quality +- Explained variance: 0.7762290236129058 +- Mean L0: 203.09911560058595 +- Dead-feature rate: — +- d_sae: 8192 +- n_snapshots: 32 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: — +- Optimizer: adam +- Epochs: 300 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 3 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("seed3.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("seed3.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/cross-snapshot-32/d8192/seed3.safetensors b/pythia-160m/W_U/cross-snapshot-32/d8192/seed3.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..67eb624e7a33b46e63207fe2ce93f10f15fb84b2 --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d8192/seed3.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:50a1a4f9413416d5dece4ae3f51c3153a21d6759895ec99bba4657834d6fdf13 +size 1611792824 diff --git a/pythia-160m/W_U/cross-snapshot-32/d8192/seed4.config.json b/pythia-160m/W_U/cross-snapshot-32/d8192/seed4.config.json new file mode 100644 index 0000000000000000000000000000000000000000..42861df928c6940f1ec213ae6fc917f4f14a58a0 --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d8192/seed4.config.json @@ -0,0 +1,111 @@ +{ + "public_label": "Pythia-160M W_U crosscoder, 32 snapshots, d_sae=8192, seed 4", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0", + "wu_step_1", + "wu_step_2", + "wu_step_3", + "wu_step_4", + "wu_step_5", + "wu_step_6", + "wu_step_7", + "wu_step_8", + "wu_step_9", + "wu_step_10", + "wu_step_11", + "wu_step_12", + "wu_step_13", + "wu_step_14", + "wu_step_15", + "wu_step_16", + "wu_step_17", + "wu_step_18", + "wu_step_19", + "wu_step_20", + "wu_step_21", + "wu_step_22", + "wu_step_23", + "wu_step_24", + "wu_step_25", + "wu_step_26", + "wu_step_27", + "wu_step_28", + "wu_step_29", + "wu_step_30", + "wu_step_31" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "n_epochs": 300, + "batch_size": 1024, + "expansion_factor": 10.6667, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.7765168883427946, + "reconstruction_mse": 0.2234119075536728, + "mean_l0": 203.4711474609375, + "d_sae": 8192, + "n_snapshots": 32, + "n_rows": 50304 + }, + "steps": [ + 0, + 1, + 2, + 4, + 8, + 16, + 32, + 64, + 128, + 256, + 512, + 1000, + 2000, + 3000, + 4000, + 5000, + 6000, + 7000, + 8000, + 9000, + 14000, + 21000, + 27000, + 34000, + 47000, + 61000, + 75000, + 89000, + 102000, + 116000, + 130000, + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 4, + "kind": "cross-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/cross-snapshot-32/d8192/seed4.md b/pythia-160m/W_U/cross-snapshot-32/d8192/seed4.md new file mode 100644 index 0000000000000000000000000000000000000000..84a459beb580806ba8ae78f1230eab3b4a85fd86 --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d8192/seed4.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U crosscoder, 32 snapshots, d_sae=8192, seed 4 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 14000, 21000, 27000, 34000, 47000, 61000, 75000, 89000, 102000, 116000, 130000, 143000] pretraining +steps. + +## Quality +- Explained variance: 0.7765168883427946 +- Mean L0: 203.4711474609375 +- Dead-feature rate: — +- d_sae: 8192 +- n_snapshots: 32 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: — +- Optimizer: adam +- Epochs: 300 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 4 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("seed4.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("seed4.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/cross-snapshot-32/d8192/seed4.safetensors b/pythia-160m/W_U/cross-snapshot-32/d8192/seed4.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..c6da24c270ea2533ce104ed8c0eff18f25bee1a8 --- /dev/null +++ b/pythia-160m/W_U/cross-snapshot-32/d8192/seed4.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2bba5e715e0eeca5b8476099026269016061bff903c56d614bc0bc9c4073d48e +size 1611792824 diff --git a/pythia-160m/W_U/final-snapshot-saes/d16384.config.json b/pythia-160m/W_U/final-snapshot-saes/d16384.config.json new file mode 100644 index 0000000000000000000000000000000000000000..660717841f733615797935d776532d873a2f505a --- /dev/null +++ b/pythia-160m/W_U/final-snapshot-saes/d16384.config.json @@ -0,0 +1,49 @@ +{ + "public_label": "Pythia-160M W_U final-snapshot SAE (step 143000 only), d_sae=16384", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 21.3334, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.05, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 4.0, + "jumprelu_lr_factor": 1.0, + "n_epochs": 100, + "batch_size": 1024, + "expansion_factor": 21.3334, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.2, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.8703331309369664, + "reconstruction_mse": 0.13296943172812462, + "mean_l0": 1913.29353515625, + "d_sae": 16384, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "final-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/final-snapshot-saes/d16384.md b/pythia-160m/W_U/final-snapshot-saes/d16384.md new file mode 100644 index 0000000000000000000000000000000000000000..bd4e456a39d16ec6c70cbf388635c994f7b647a1 --- /dev/null +++ b/pythia-160m/W_U/final-snapshot-saes/d16384.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U final-snapshot SAE (step 143000 only), d_sae=16384 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [143000] pretraining +steps. + +## Quality +- Explained variance: 0.8703331309369664 +- Mean L0: 1913.29353515625 +- Dead-feature rate: — +- d_sae: 16384 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.05 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 4.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 1.0 +- Init threshold: — +- Optimizer: adam +- Epochs: 100 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("d16384.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("d16384.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/final-snapshot-saes/d16384.safetensors b/pythia-160m/W_U/final-snapshot-saes/d16384.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..7f7e1b365013d2ab8e850f98b5e0ab65b5e8ba55 --- /dev/null +++ b/pythia-160m/W_U/final-snapshot-saes/d16384.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:18f762056aa097c0c9862874af48d95846c0019796ea404b73d5dc8002b5c20b +size 100797872 diff --git a/pythia-160m/W_U/final-snapshot-saes/d32768.config.json b/pythia-160m/W_U/final-snapshot-saes/d32768.config.json new file mode 100644 index 0000000000000000000000000000000000000000..c84626e287711cfadc9de45955fb224abf7268ad --- /dev/null +++ b/pythia-160m/W_U/final-snapshot-saes/d32768.config.json @@ -0,0 +1,49 @@ +{ + "public_label": "Pythia-160M W_U final-snapshot SAE (step 143000 only), d_sae=32768", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 42.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.05, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 4.0, + "jumprelu_lr_factor": 1.0, + "n_epochs": 100, + "batch_size": 1024, + "expansion_factor": 42.6667, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.2, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.926354609914279, + "reconstruction_mse": 0.07552110835909844, + "mean_l0": 3410.36810546875, + "d_sae": 32768, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "final-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/final-snapshot-saes/d32768.md b/pythia-160m/W_U/final-snapshot-saes/d32768.md new file mode 100644 index 0000000000000000000000000000000000000000..0e660979a81d73f4e8b49f05963d3a4816be476a --- /dev/null +++ b/pythia-160m/W_U/final-snapshot-saes/d32768.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U final-snapshot SAE (step 143000 only), d_sae=32768 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [143000] pretraining +steps. + +## Quality +- Explained variance: 0.926354609914279 +- Mean L0: 3410.36810546875 +- Dead-feature rate: — +- d_sae: 32768 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.05 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 4.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 1.0 +- Init threshold: — +- Optimizer: adam +- Epochs: 100 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("d32768.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("d32768.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/final-snapshot-saes/d32768.safetensors b/pythia-160m/W_U/final-snapshot-saes/d32768.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..8db9572d1da6b12cacdc97f4b5c1cd29df51d43e --- /dev/null +++ b/pythia-160m/W_U/final-snapshot-saes/d32768.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ca0c3bb72a080cfd23315eabaf9f7120356c19c4809ad78dbde1abccaa1dd9fe +size 201592240 diff --git a/pythia-160m/W_U/final-snapshot-saes/d6144.config.json b/pythia-160m/W_U/final-snapshot-saes/d6144.config.json new file mode 100644 index 0000000000000000000000000000000000000000..c7ea2d79f16367fe8657c55d3bc70e9c7f25ae6e --- /dev/null +++ b/pythia-160m/W_U/final-snapshot-saes/d6144.config.json @@ -0,0 +1,49 @@ +{ + "public_label": "Pythia-160M W_U final-snapshot SAE (step 143000 only), d_sae=6144", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 8.0, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.05, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 4.0, + "jumprelu_lr_factor": 1.0, + "n_epochs": 100, + "batch_size": 1024, + "expansion_factor": 8.0, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.2, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.7649826125102704, + "reconstruction_mse": 0.24100318521261216, + "mean_l0": 861.8197265625, + "d_sae": 6144, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "final-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/final-snapshot-saes/d6144.md b/pythia-160m/W_U/final-snapshot-saes/d6144.md new file mode 100644 index 0000000000000000000000000000000000000000..c0663d9ffdeddc6c2bfcd36d42736ed6e02c5a28 --- /dev/null +++ b/pythia-160m/W_U/final-snapshot-saes/d6144.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U final-snapshot SAE (step 143000 only), d_sae=6144 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [143000] pretraining +steps. + +## Quality +- Explained variance: 0.7649826125102704 +- Mean L0: 861.8197265625 +- Dead-feature rate: — +- d_sae: 6144 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.05 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 4.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 1.0 +- Init threshold: — +- Optimizer: adam +- Epochs: 100 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("d6144.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("d6144.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/final-snapshot-saes/d6144.safetensors b/pythia-160m/W_U/final-snapshot-saes/d6144.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..106e3451bee9d363fe94a30bb2f4010fd1555fdb --- /dev/null +++ b/pythia-160m/W_U/final-snapshot-saes/d6144.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:71f62802231acf9e5c041c4508c440afbb8b9ee09a0d389070566d71eee7c631 +size 37801376 diff --git a/pythia-160m/W_U/final-snapshot-saes/d65536.config.json b/pythia-160m/W_U/final-snapshot-saes/d65536.config.json new file mode 100644 index 0000000000000000000000000000000000000000..87506edef985529bdadf5b26a2ae3e7c644f5e5d --- /dev/null +++ b/pythia-160m/W_U/final-snapshot-saes/d65536.config.json @@ -0,0 +1,49 @@ +{ + "public_label": "Pythia-160M W_U final-snapshot SAE (step 143000 only), d_sae=65536", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 85.3334, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.05, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 4.0, + "jumprelu_lr_factor": 1.0, + "n_epochs": 100, + "batch_size": 1024, + "expansion_factor": 85.3334, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.2, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.9642323011052749, + "reconstruction_mse": 0.0366786877065897, + "mean_l0": 5943.19078125, + "d_sae": 65536, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "final-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/final-snapshot-saes/d65536.md b/pythia-160m/W_U/final-snapshot-saes/d65536.md new file mode 100644 index 0000000000000000000000000000000000000000..04afa2a4b960e66af0f7e4dc2dfc8f4b9b477946 --- /dev/null +++ b/pythia-160m/W_U/final-snapshot-saes/d65536.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U final-snapshot SAE (step 143000 only), d_sae=65536 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [143000] pretraining +steps. + +## Quality +- Explained variance: 0.9642323011052749 +- Mean L0: 5943.19078125 +- Dead-feature rate: — +- d_sae: 65536 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.05 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 4.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 1.0 +- Init threshold: — +- Optimizer: adam +- Epochs: 100 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("d65536.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("d65536.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/final-snapshot-saes/d65536.safetensors b/pythia-160m/W_U/final-snapshot-saes/d65536.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..17d87b008735946f6886416583d97ee1eca344cc --- /dev/null +++ b/pythia-160m/W_U/final-snapshot-saes/d65536.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:94b3113b9b3fb93d7dd29f3a28c04fe99bafc1acfc89cc507c0a698ead7a5c1c +size 403180976 diff --git a/pythia-160m/W_U/final-snapshot-saes/d8192.config.json b/pythia-160m/W_U/final-snapshot-saes/d8192.config.json new file mode 100644 index 0000000000000000000000000000000000000000..c67f9f16b976f49f012249b0b336049e134271d8 --- /dev/null +++ b/pythia-160m/W_U/final-snapshot-saes/d8192.config.json @@ -0,0 +1,49 @@ +{ + "public_label": "Pythia-160M W_U final-snapshot SAE (step 143000 only), d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.05, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 4.0, + "jumprelu_lr_factor": 1.0, + "n_epochs": 100, + "batch_size": 1024, + "expansion_factor": 10.6667, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.2, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.7989936757858394, + "reconstruction_mse": 0.2061258739233017, + "mean_l0": 1084.4333984375, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "final-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/final-snapshot-saes/d8192.md b/pythia-160m/W_U/final-snapshot-saes/d8192.md new file mode 100644 index 0000000000000000000000000000000000000000..4137e7aaa2711bb8c2f33b84cac46946b37126cc --- /dev/null +++ b/pythia-160m/W_U/final-snapshot-saes/d8192.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U final-snapshot SAE (step 143000 only), d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [143000] pretraining +steps. + +## Quality +- Explained variance: 0.7989936757858394 +- Mean L0: 1084.4333984375 +- Dead-feature rate: — +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.05 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 4.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 1.0 +- Init threshold: — +- Optimizer: adam +- Epochs: 100 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("d8192.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("d8192.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/final-snapshot-saes/d8192.safetensors b/pythia-160m/W_U/final-snapshot-saes/d8192.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..de60e92c88985462877624e1c4f435b98efbdfdc --- /dev/null +++ b/pythia-160m/W_U/final-snapshot-saes/d8192.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a1e64dae952e187e4f80231f4668d274c8dd270bbd6a257ddb37a6616382d3dc +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step0.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step0.config.json new file mode 100644 index 0000000000000000000000000000000000000000..a8dd1bac43e4cab443ee27a8a785d31cf998bb48 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step0.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 0, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 40.250865591922775 + }, + "quality": { + "explained_variance": 0.7288661622516968, + "reconstruction_mse": 0.27112908840179445, + "mean_l0": 1150.079375, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 0 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step0.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step0.md new file mode 100644 index 0000000000000000000000000000000000000000..6fe7e5baba7b9bd6a67d381b6541d380f65a3f7e --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step0.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 0, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [0] pretraining +steps. + +## Quality +- Explained variance: 0.7288661622516968 +- Mean L0: 1150.079375 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step0.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step0.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step0.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step0.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..6f40e8608afbcf2d2a5294bb0210125a80ab375e --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step0.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:07bd2e07f3975fdc9ebe08f34241481784f6cc560bd42c32997d204183c9e87a +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step1.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step1.config.json new file mode 100644 index 0000000000000000000000000000000000000000..aa01f98cf0e3332a84111c6773e50d5e397c40b9 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step1.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 1, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.50206663599238 + }, + "quality": { + "explained_variance": 0.7288661622516968, + "reconstruction_mse": 0.27112908840179445, + "mean_l0": 1150.079375, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 1 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step1.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step1.md new file mode 100644 index 0000000000000000000000000000000000000000..7d3f842a458328e0ab3b946f0d8eeb714f327fb5 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step1.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 1, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [1] pretraining +steps. + +## Quality +- Explained variance: 0.7288661622516968 +- Mean L0: 1150.079375 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step1.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step1.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step1.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step1.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..6f40e8608afbcf2d2a5294bb0210125a80ab375e --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step1.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:07bd2e07f3975fdc9ebe08f34241481784f6cc560bd42c32997d204183c9e87a +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step1000.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step1000.config.json new file mode 100644 index 0000000000000000000000000000000000000000..fe8f8453ae7e33236651f4a478200070a6f90245 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step1000.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 1000, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.75007117399946 + }, + "quality": { + "explained_variance": 0.7862491472291859, + "reconstruction_mse": 0.21337854146957397, + "mean_l0": 996.89369140625, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 1000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step1000.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step1000.md new file mode 100644 index 0000000000000000000000000000000000000000..3f8659997d44af607fec5f66f3ba5bb9bd675279 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step1000.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 1000, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [1000] pretraining +steps. + +## Quality +- Explained variance: 0.7862491472291859 +- Mean L0: 996.89369140625 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step1000.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step1000.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step1000.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step1000.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..b2255791e4bbb19ec3992a09b2c59a31b6cf951b --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step1000.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3e8d690fc1d2347d532da746640f84c96c69a1d185091a81ddfbf336c248ca94 +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step102000.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step102000.config.json new file mode 100644 index 0000000000000000000000000000000000000000..ec50e5bde4c12721d99930ba2018e1120ae81f1b --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step102000.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 102000, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 39.014616638072766 + }, + "quality": { + "explained_variance": 0.7998127382951936, + "reconstruction_mse": 0.20010990262031556, + "mean_l0": 983.041015625, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 102000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step102000.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step102000.md new file mode 100644 index 0000000000000000000000000000000000000000..a9490e0ef15acd555b15f52957a7bafbc12b0fb4 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step102000.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 102000, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [102000] pretraining +steps. + +## Quality +- Explained variance: 0.7998127382951936 +- Mean L0: 983.041015625 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step102000.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step102000.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step102000.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step102000.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..5fe854a7a6e88f2e3b9c1d31ad4b41aeb7acc72d --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step102000.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:94894080388f08882e2fde4e7128482431797601576c41eb232e5d97f32573e5 +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step116000.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step116000.config.json new file mode 100644 index 0000000000000000000000000000000000000000..3990e7bd7001c1e8d31eb014839dd4ca74e625cc --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step116000.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 116000, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.44623186998069 + }, + "quality": { + "explained_variance": 0.8120686966847851, + "reconstruction_mse": 0.18798080801963807, + "mean_l0": 957.6841577148438, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 116000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step116000.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step116000.md new file mode 100644 index 0000000000000000000000000000000000000000..198d7c5b3b89993f1a1dd6416933072ea47a3e4a --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step116000.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 116000, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [116000] pretraining +steps. + +## Quality +- Explained variance: 0.8120686966847851 +- Mean L0: 957.6841577148438 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step116000.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step116000.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step116000.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step116000.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..05235efa0076456f7b649cc4b639e40c86ae44ce --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step116000.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:083df9032925b670792b75eac38809dc091fa4a76c563576136f7672d732b9dd +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step128.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step128.config.json new file mode 100644 index 0000000000000000000000000000000000000000..ad51af0181a928d20fa58118a81941c887158b1d --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step128.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 128, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.382996073924005 + }, + "quality": { + "explained_variance": 0.7290331109474104, + "reconstruction_mse": 0.2709601593017578, + "mean_l0": 1149.5578076171876, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 128 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step128.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step128.md new file mode 100644 index 0000000000000000000000000000000000000000..08261685886acf4b1bdaef9bcdadc80f795f7700 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step128.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 128, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [128] pretraining +steps. + +## Quality +- Explained variance: 0.7290331109474104 +- Mean L0: 1149.5578076171876 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step128.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step128.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step128.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step128.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..eeb1b44e6f98bdd5e22a139f57b165a700f52906 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step128.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:822b8f057482e9b3eff682200d8068425ee5a04bb97ffb4de82174ce7335000f +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step130000.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step130000.config.json new file mode 100644 index 0000000000000000000000000000000000000000..5e462e55e1811024f783248def01a4196055f446 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step130000.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 130000, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.46364720899146 + }, + "quality": { + "explained_variance": 0.8195671283611181, + "reconstruction_mse": 0.180638085603714, + "mean_l0": 940.0407983398437, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 130000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step130000.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step130000.md new file mode 100644 index 0000000000000000000000000000000000000000..af3f64df723ebcc48e2b9a5a2b0b78d07b6f5a00 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step130000.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 130000, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [130000] pretraining +steps. + +## Quality +- Explained variance: 0.8195671283611181 +- Mean L0: 940.0407983398437 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step130000.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step130000.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step130000.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step130000.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..3615add7b53f8128fae14e7081abf55cffc34d5b --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step130000.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5a3ee4804c6e23172cfcd84c434f8453bd6cc2f137c43f8a23624e4e53c42544 +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step14000.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step14000.config.json new file mode 100644 index 0000000000000000000000000000000000000000..d2d33515199b77d9293311eeb97071a14c40baee --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step14000.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 14000, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.549026424996555 + }, + "quality": { + "explained_variance": 0.7912227686750525, + "reconstruction_mse": 0.20864035725593566, + "mean_l0": 996.466455078125, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 14000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step14000.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step14000.md new file mode 100644 index 0000000000000000000000000000000000000000..8027ffb70d88a0fb7461eba3f813cc5a02afd02b --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step14000.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 14000, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [14000] pretraining +steps. + +## Quality +- Explained variance: 0.7912227686750525 +- Mean L0: 996.466455078125 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step14000.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step14000.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step14000.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step14000.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..9b46150f36897eea86af6748d3086407e02af1a6 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step14000.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3fb372a759974d8d758c1014b083c1560f35ab0769af724aa90dd95cf9e229cd +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step143000.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step143000.config.json new file mode 100644 index 0000000000000000000000000000000000000000..cc67baf72fc71eabb423aad0d546c82cfa143bf0 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step143000.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 143000, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.956221061060205 + }, + "quality": { + "explained_variance": 0.8244679775563315, + "reconstruction_mse": 0.17589144110679628, + "mean_l0": 924.0657104492187, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 143000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step143000.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step143000.md new file mode 100644 index 0000000000000000000000000000000000000000..98e0286b2d450ad77d6a5f02919335bee325112c --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step143000.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 143000, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [143000] pretraining +steps. + +## Quality +- Explained variance: 0.8244679775563315 +- Mean L0: 924.0657104492187 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step143000.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step143000.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step143000.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step143000.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..2d4dcd1ec21242c076976d905cd67ae10559c543 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step143000.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:52dcfe7e7e7033e4bafb3548ee52f785308c75a17e56eda867c277a68d690c7d +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step16.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step16.config.json new file mode 100644 index 0000000000000000000000000000000000000000..b1f07a7761253c195918b3f9db3f694fd3e22f05 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step16.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 16, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.85052962193731 + }, + "quality": { + "explained_variance": 0.7288671871864183, + "reconstruction_mse": 0.2711281669139862, + "mean_l0": 1150.0855908203125, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 16 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step16.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step16.md new file mode 100644 index 0000000000000000000000000000000000000000..9a152add843232205b44e6f5aece4c576669d139 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step16.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 16, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [16] pretraining +steps. + +## Quality +- Explained variance: 0.7288671871864183 +- Mean L0: 1150.0855908203125 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step16.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step16.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step16.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step16.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..2d675e05ec2b3044b001e3c7fc43c98020a3cbe9 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step16.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2dfada7436abf6a1963d31f8f66c53c3ed7b2735b86042f10a3586b2627ad380 +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step2.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step2.config.json new file mode 100644 index 0000000000000000000000000000000000000000..3fe32bcf339c543846774db37cde08e886b84a9f --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step2.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 2, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.40461527300067 + }, + "quality": { + "explained_variance": 0.72886850366392, + "reconstruction_mse": 0.2711267483234405, + "mean_l0": 1150.0986572265624, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 2 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step2.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step2.md new file mode 100644 index 0000000000000000000000000000000000000000..d9a4a7544d63fd68309192bbd4c96583c50d047a --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step2.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 2, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [2] pretraining +steps. + +## Quality +- Explained variance: 0.72886850366392 +- Mean L0: 1150.0986572265624 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step2.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step2.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step2.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step2.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..edd597b51c6792f347f32acb5c60b8aa0f10e524 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step2.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:28bd4677ffed83f5329009f83c16cc4e2966392b90fe85a06d9cb446bbd1f180 +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step2000.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step2000.config.json new file mode 100644 index 0000000000000000000000000000000000000000..5ae3e1c8df2e82bfd4fc75a73b0c11be23f817e7 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step2000.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 2000, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.359453815966845 + }, + "quality": { + "explained_variance": 0.7992791212506758, + "reconstruction_mse": 0.20044391214847565, + "mean_l0": 969.1734448242188, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 2000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step2000.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step2000.md new file mode 100644 index 0000000000000000000000000000000000000000..7e7c399cf6a718ca692dd20a60f54897527d976b --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step2000.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 2000, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [2000] pretraining +steps. + +## Quality +- Explained variance: 0.7992791212506758 +- Mean L0: 969.1734448242188 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step2000.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step2000.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step2000.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step2000.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..7f44210f332e1dc01dd111a44bf194efbb07dda5 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step2000.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:bd0ccb3380af00f43939c96b5d6e208fa94df50d3e54de4528391bced12ef927 +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step21000.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step21000.config.json new file mode 100644 index 0000000000000000000000000000000000000000..28819baf1b8b294f172236deb914825d3570e291 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step21000.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 21000, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.414867891930044 + }, + "quality": { + "explained_variance": 0.7907924441773572, + "reconstruction_mse": 0.20908703327178954, + "mean_l0": 998.4695678710938, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 21000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step21000.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step21000.md new file mode 100644 index 0000000000000000000000000000000000000000..83aa8e1213ada701df57ef3234864c7ddd562885 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step21000.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 21000, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [21000] pretraining +steps. + +## Quality +- Explained variance: 0.7907924441773572 +- Mean L0: 998.4695678710938 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step21000.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step21000.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step21000.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step21000.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..4e09f6fb05d9ef8713eb594f796ccc2a5aeb1849 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step21000.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:96092908fafb89c4e8131a200f2b168752dffba5e061025c97a53a6cdeb9a8d0 +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step256.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step256.config.json new file mode 100644 index 0000000000000000000000000000000000000000..3f0d320774f585674aa8fa0a4c28bb15ef05ccbf --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step256.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 256, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.58555905788671 + }, + "quality": { + "explained_variance": 0.7317386038736777, + "reconstruction_mse": 0.2681868326663971, + "mean_l0": 1142.36794921875, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 256 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step256.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step256.md new file mode 100644 index 0000000000000000000000000000000000000000..1b66974d785e335ead05fb85afec48a781f7a938 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step256.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 256, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [256] pretraining +steps. + +## Quality +- Explained variance: 0.7317386038736777 +- Mean L0: 1142.36794921875 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step256.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step256.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step256.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step256.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..73ea9c14951cd79c415aea2db13276d6556f1e22 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step256.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c5c2b9fa046061dcc1b3c47dd54ba6bf5fa9e77028f1c86ecfb484486de5fac8 +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step27000.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step27000.config.json new file mode 100644 index 0000000000000000000000000000000000000000..ba5e020ab425378aacb3e51dc524ab11458bc1e8 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step27000.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 27000, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 39.15026664105244 + }, + "quality": { + "explained_variance": 0.7907578681617007, + "reconstruction_mse": 0.20913382470607758, + "mean_l0": 999.3178100585938, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 27000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step27000.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step27000.md new file mode 100644 index 0000000000000000000000000000000000000000..559efce02770f95ba6ed7da39e2f54d585672c5e --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step27000.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 27000, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [27000] pretraining +steps. + +## Quality +- Explained variance: 0.7907578681617007 +- Mean L0: 999.3178100585938 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step27000.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step27000.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step27000.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step27000.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..e576d04eb9927d7397e9c00aaac829d3b8d6c044 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step27000.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:cedda01221d76ff3e82e4f3e88bf2a23fb6945c98e4ad7d91edef40fdcd637fc +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step3000.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step3000.config.json new file mode 100644 index 0000000000000000000000000000000000000000..7c66dbcc8388e6f19c7e68946b5e3b7f56d740f1 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step3000.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 3000, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.45864061207976 + }, + "quality": { + "explained_variance": 0.7993412040992963, + "reconstruction_mse": 0.20040635347366334, + "mean_l0": 972.4029614257812, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 3000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step3000.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step3000.md new file mode 100644 index 0000000000000000000000000000000000000000..b6ffabf3017a46bf8adeda1a861a8b2fc85acdf0 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step3000.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 3000, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [3000] pretraining +steps. + +## Quality +- Explained variance: 0.7993412040992963 +- Mean L0: 972.4029614257812 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step3000.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step3000.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step3000.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step3000.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..c5c95385203ded482b7d8823f430e388cff6d4cf --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step3000.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b0cc6edfdeb385a0b75f9fb9a0828004489d9cbe1e4f80cc86eb60b55fba6058 +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step32.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step32.config.json new file mode 100644 index 0000000000000000000000000000000000000000..6c65ad460b4fa6f421619a4fb5ad0e5c0bcab941 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step32.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 32, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.575834422023036 + }, + "quality": { + "explained_variance": 0.7288681571269175, + "reconstruction_mse": 0.2711274743080139, + "mean_l0": 1150.0745947265625, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 32 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step32.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step32.md new file mode 100644 index 0000000000000000000000000000000000000000..2991fde88afcf799cceb05805023a586c2d73732 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step32.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 32, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [32] pretraining +steps. + +## Quality +- Explained variance: 0.7288681571269175 +- Mean L0: 1150.0745947265625 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step32.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step32.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step32.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step32.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..f589fc4c030d9e7d8e43847459f5e03772bfc7d8 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step32.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:98028e746438b6a70d1dee198fd0b719b5af6f7268f16847c3d09b01f9316ad7 +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step34000.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step34000.config.json new file mode 100644 index 0000000000000000000000000000000000000000..26392020cdc6be02571aa457a64881590d50de88 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step34000.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 34000, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.50451052293647 + }, + "quality": { + "explained_variance": 0.7906181564574442, + "reconstruction_mse": 0.20928000867366792, + "mean_l0": 999.5078881835938, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 34000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step34000.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step34000.md new file mode 100644 index 0000000000000000000000000000000000000000..7ce66ec2280d11d9fbc99ff3e4a7218f46f28f6b --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step34000.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 34000, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [34000] pretraining +steps. + +## Quality +- Explained variance: 0.7906181564574442 +- Mean L0: 999.5078881835938 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step34000.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step34000.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step34000.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step34000.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..b92d6ae80daa8174571cec8eda7ce5e8bb27d8f4 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step34000.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c7ecb34eec63d055ff8b7885ed7258c6d6ac554b5e96341953d953195ad6ec02 +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step4.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step4.config.json new file mode 100644 index 0000000000000000000000000000000000000000..e0920eaab80ec517cdcdae8c44b9e0bc79e5c2b8 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step4.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 4, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.47982324403711 + }, + "quality": { + "explained_variance": 0.7288676993559831, + "reconstruction_mse": 0.2711275267601013, + "mean_l0": 1150.087177734375, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 4 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step4.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step4.md new file mode 100644 index 0000000000000000000000000000000000000000..ec1a0780207698de0325aa41ce1a073d08b801a9 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step4.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 4, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [4] pretraining +steps. + +## Quality +- Explained variance: 0.7288676993559831 +- Mean L0: 1150.087177734375 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step4.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step4.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step4.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step4.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..9e5f4831d3f2693a7bad7051ca344c3e26e0aeac --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step4.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:670b76964f703675a172d21d276234186b2bc1910f3917bb0ce2001e68709a79 +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step4000.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step4000.config.json new file mode 100644 index 0000000000000000000000000000000000000000..27e83f55b7dcdb5147376a235b5ace741558da22 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step4000.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 4000, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.56192829506472 + }, + "quality": { + "explained_variance": 0.7977749784323522, + "reconstruction_mse": 0.2019952780008316, + "mean_l0": 977.3629370117187, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 4000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step4000.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step4000.md new file mode 100644 index 0000000000000000000000000000000000000000..9c64ba4c6773a978b9e6349ff74d59469c10446b --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step4000.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 4000, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [4000] pretraining +steps. + +## Quality +- Explained variance: 0.7977749784323522 +- Mean L0: 977.3629370117187 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step4000.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step4000.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step4000.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step4000.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..e3a1161836653ef9a9bda98adc536287158d7be4 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step4000.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e66ec82834604cb43662a996e8a18f08b341ef502688a6d16127076cb6174ea9 +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step47000.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step47000.config.json new file mode 100644 index 0000000000000000000000000000000000000000..2709081a647ed88a6e33eb0a8f78949fed471171 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step47000.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 47000, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.35288295801729 + }, + "quality": { + "explained_variance": 0.790620026518176, + "reconstruction_mse": 0.2092841708660126, + "mean_l0": 1000.2795385742188, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 47000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step47000.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step47000.md new file mode 100644 index 0000000000000000000000000000000000000000..9670684516ef7c0dfc2091d688113a47657b0e7c --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step47000.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 47000, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [47000] pretraining +steps. + +## Quality +- Explained variance: 0.790620026518176 +- Mean L0: 1000.2795385742188 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step47000.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step47000.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step47000.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step47000.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..2a4f61a9cbd08defb50b4cbf0d82d6f74c095247 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step47000.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7d20d5e41494dae87675837f2f1e6d50f68d6cf7ad5a291aa9a46ed736323a64 +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step5000.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step5000.config.json new file mode 100644 index 0000000000000000000000000000000000000000..6496d98e97cf669ffaab507d159748305c23a1da --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step5000.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 5000, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 39.38889519800432 + }, + "quality": { + "explained_variance": 0.7960998419245482, + "reconstruction_mse": 0.20369211137294768, + "mean_l0": 981.8336376953125, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 5000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step5000.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step5000.md new file mode 100644 index 0000000000000000000000000000000000000000..d9dd0f06db714f9b9fa2c49f8ad801435aaa5f8a --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step5000.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 5000, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [5000] pretraining +steps. + +## Quality +- Explained variance: 0.7960998419245482 +- Mean L0: 981.8336376953125 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step5000.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step5000.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step5000.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step5000.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..06e900a7a4097ef74dc9f2078bf45d56b6852703 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step5000.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:832cd20ec1bcc2dea90266e90fad7c6c23b07d187142ad156270164227d6e211 +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step512.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step512.config.json new file mode 100644 index 0000000000000000000000000000000000000000..114274a1a418b0d736bd8fa0f22756df4fc3304f --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step512.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 512, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.3824528590776 + }, + "quality": { + "explained_variance": 0.7542853862655656, + "reconstruction_mse": 0.24544892609119415, + "mean_l0": 1086.609990234375, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 512 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step512.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step512.md new file mode 100644 index 0000000000000000000000000000000000000000..51604059902b7624dc6b939d140be7f6c2fa302c --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step512.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 512, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [512] pretraining +steps. + +## Quality +- Explained variance: 0.7542853862655656 +- Mean L0: 1086.609990234375 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step512.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step512.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step512.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step512.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..2df81f928c6f2a18e9bea9d6ce3d87e6bece2d9c --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step512.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:850233247064561bcd0a4a22cefb095c5adfa7cefcc3dac86aef0d1130139935 +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step6000.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step6000.config.json new file mode 100644 index 0000000000000000000000000000000000000000..388c2f8e07d9e0cda64e1e1762b29442ab65c9a2 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step6000.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 6000, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.640947929932736 + }, + "quality": { + "explained_variance": 0.7947783648659816, + "reconstruction_mse": 0.20502524971961975, + "mean_l0": 985.470498046875, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 6000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step6000.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step6000.md new file mode 100644 index 0000000000000000000000000000000000000000..14f16b27a2661bc01eb7b178b5e86c78c4254cb1 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step6000.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 6000, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [6000] pretraining +steps. + +## Quality +- Explained variance: 0.7947783648659816 +- Mean L0: 985.470498046875 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step6000.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step6000.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step6000.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step6000.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..c18eda80a811c7dbda80eb1f92955d95c8632a37 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step6000.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:fbe01576d6e30741d8329d686416b1724137407b43516ea4af68d863536bf350 +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step61000.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step61000.config.json new file mode 100644 index 0000000000000000000000000000000000000000..c2b3157fcaf2fe199aa53003a9ea5df5590aa51d --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step61000.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 61000, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 39.16651166300289 + }, + "quality": { + "explained_variance": 0.7904375219119095, + "reconstruction_mse": 0.20947221636772156, + "mean_l0": 1001.8048901367188, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 61000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step61000.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step61000.md new file mode 100644 index 0000000000000000000000000000000000000000..758b20fd1b594ff01c9e23415779fef9f0aae5de --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step61000.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 61000, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [61000] pretraining +steps. + +## Quality +- Explained variance: 0.7904375219119095 +- Mean L0: 1001.8048901367188 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step61000.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step61000.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step61000.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step61000.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..b102e0e537c8140c219519566706c5a247a9735f --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step61000.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d2e3ac4e73546ef46586b779cf9b7ede5057f7333a7729e705979ea9f692385f +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step64.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step64.config.json new file mode 100644 index 0000000000000000000000000000000000000000..b6a03da559573d6f64704837c4acd29e910a1eab --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step64.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 64, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.47816930396948 + }, + "quality": { + "explained_variance": 0.728867046997959, + "reconstruction_mse": 0.2711291468143463, + "mean_l0": 1150.0748681640625, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 64 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step64.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step64.md new file mode 100644 index 0000000000000000000000000000000000000000..6b08d7d56e9bbbf8d9240479eb2260b1adcc664f --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step64.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 64, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [64] pretraining +steps. + +## Quality +- Explained variance: 0.728867046997959 +- Mean L0: 1150.0748681640625 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step64.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step64.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step64.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step64.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..7132ef87cb374f464a9524b0c877c936ea56bbe0 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step64.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:005a6c281e5dd5fc3f46ff18795686c6048b4a8ba122265ec0a90eb598d3bfd4 +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step7000.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step7000.config.json new file mode 100644 index 0000000000000000000000000000000000000000..a16edddc62c9bfb041e4458d95bfca92d47186ae --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step7000.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 7000, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.395364897092804 + }, + "quality": { + "explained_variance": 0.7938394341974695, + "reconstruction_mse": 0.20597564816474914, + "mean_l0": 988.107509765625, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 7000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step7000.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step7000.md new file mode 100644 index 0000000000000000000000000000000000000000..c12a3cf61f5faf6fbaceb0c2395884c2ead95b4a --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step7000.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 7000, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [7000] pretraining +steps. + +## Quality +- Explained variance: 0.7938394341974695 +- Mean L0: 988.107509765625 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step7000.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step7000.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step7000.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step7000.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..0b02167364798e72d9ff3b17444535e9a0be53b7 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step7000.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2736b011495f606d98a555d3beb662edef3a78e9bbd2337cc4bea363d4e2fe79 +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step75000.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step75000.config.json new file mode 100644 index 0000000000000000000000000000000000000000..f4dd3cc5224cf8e6e09688c402b7045f89a40da9 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step75000.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 75000, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.59630808397196 + }, + "quality": { + "explained_variance": 0.790395516481567, + "reconstruction_mse": 0.20952392756938934, + "mean_l0": 1004.148408203125, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 75000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step75000.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step75000.md new file mode 100644 index 0000000000000000000000000000000000000000..d90d1d5c91308bd3e88c439ed0895a0097d065bf --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step75000.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 75000, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [75000] pretraining +steps. + +## Quality +- Explained variance: 0.790395516481567 +- Mean L0: 1004.148408203125 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step75000.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step75000.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step75000.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step75000.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..f7a82bf7cec0849c88b19e31c574de9fe8e14896 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step75000.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:638ba92a1e44a4896902ab46f7c24378722bd7e0c9f3ed731c602a609152e648 +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step8.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step8.config.json new file mode 100644 index 0000000000000000000000000000000000000000..4e8be6a562f8ec68ccf40043b90c376824de1877 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step8.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 8, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.410247489926405 + }, + "quality": { + "explained_variance": 0.7288677026395104, + "reconstruction_mse": 0.27112757325172426, + "mean_l0": 1150.0938623046875, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 8 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step8.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step8.md new file mode 100644 index 0000000000000000000000000000000000000000..199d81a771e72e794aaade2629811632218d38bd --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step8.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 8, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [8] pretraining +steps. + +## Quality +- Explained variance: 0.7288677026395104 +- Mean L0: 1150.0938623046875 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step8.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step8.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step8.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step8.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..db921b31b9d57301dad3c6c3a6429d9a96e1f518 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step8.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:23347d47f6d22a7b242090ad4d948dbd690fcee3f69d3de6338963448f91fab3 +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step8000.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step8000.config.json new file mode 100644 index 0000000000000000000000000000000000000000..375ebc1cdb309d415a1b4e957ccb16d744310fc2 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step8000.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 8000, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.70826957304962 + }, + "quality": { + "explained_variance": 0.7931325520036884, + "reconstruction_mse": 0.20668985426425934, + "mean_l0": 990.1047631835937, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 8000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step8000.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step8000.md new file mode 100644 index 0000000000000000000000000000000000000000..6ab90fc5f48f8b15658bbdf615f867bcae74c446 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step8000.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 8000, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [8000] pretraining +steps. + +## Quality +- Explained variance: 0.7931325520036884 +- Mean L0: 990.1047631835937 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step8000.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step8000.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step8000.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step8000.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..d4ee6bdf2f84af6a6dca55415367c4f3a35d1fd6 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step8000.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b59e99119af71ae71cdb578ebf54e785944803d5387e1663726857dd3e587a15 +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step89000.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step89000.config.json new file mode 100644 index 0000000000000000000000000000000000000000..fe69aa94934c40092fc7f8e44fb5e92ff1e2e43c --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step89000.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 89000, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.27878786297515 + }, + "quality": { + "explained_variance": 0.7924073402250866, + "reconstruction_mse": 0.2075205421447754, + "mean_l0": 1001.9916259765625, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 89000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step89000.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step89000.md new file mode 100644 index 0000000000000000000000000000000000000000..bb4e8bd350b4a550d5b2de07534f574a4933f4e6 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step89000.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 89000, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [89000] pretraining +steps. + +## Quality +- Explained variance: 0.7924073402250866 +- Mean L0: 1001.9916259765625 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step89000.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step89000.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step89000.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step89000.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..a8163d72c630ef1edbd1840c3cd0d049454a4086 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step89000.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c7f0915e7ab68acb3fab513c4ed79186c820913ec7861dca1f0fc71831c3150d +size 50400672 diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step9000.config.json b/pythia-160m/W_U/per-snapshot-saes/d8192/step9000.config.json new file mode 100644 index 0000000000000000000000000000000000000000..ab13890f95106cd2dadda7072b6a33fa5a9480ee --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step9000.config.json @@ -0,0 +1,55 @@ +{ + "public_label": "Pythia-160M W_U per-snapshot SAE at training step 9000, d_sae=8192", + "config": { + "sae_type": "crosscoder", + "d_model": 768, + "expansion_factor": 10.6667, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0" + ] + }, + "training": { + "lr": 5e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.1, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 100, + "batch_size": 2048, + "expansion_factor": 10.6667, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.1, + "lr_warmup_fraction": 0.1, + "lr_decay_fraction": 0.2, + "elapsed_seconds": 38.31250529992394 + }, + "quality": { + "explained_variance": 0.7924251809723794, + "reconstruction_mse": 0.20740727365016937, + "mean_l0": 992.0130297851563, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 1, + "n_rows": 50304 + }, + "steps": [ + 9000 + ], + "model_name": "EleutherAI/pythia-160m", + "seed": 0, + "kind": "per-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step9000.md b/pythia-160m/W_U/per-snapshot-saes/d8192/step9000.md new file mode 100644 index 0000000000000000000000000000000000000000..5d6b1a57101f55ec7803db18311325e98d445ed4 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step9000.md @@ -0,0 +1,38 @@ +# Pythia-160M W_U per-snapshot SAE at training step 9000, d_sae=8192 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-160m` across [9000] pretraining +steps. + +## Quality +- Explained variance: 0.7924251809723794 +- Mean L0: 992.0130297851563 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 1 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 5e-05 +- JumpReLU LR multiplier: 0.1 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 100 +- Batch size: 2048 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("step9000.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("step9000.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-160m/W_U/per-snapshot-saes/d8192/step9000.safetensors b/pythia-160m/W_U/per-snapshot-saes/d8192/step9000.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..b987e0ba2b80eca394cde6dae8a4c5f816ffab44 --- /dev/null +++ b/pythia-160m/W_U/per-snapshot-saes/d8192/step9000.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:70bcff9a430cb255653f2ec752def9045dfa4a565f338335101f821ce436cb17 +size 50400672 diff --git a/pythia-1b/W_U/cross-snapshot-32/d16384/seed0.config.json b/pythia-1b/W_U/cross-snapshot-32/d16384/seed0.config.json new file mode 100644 index 0000000000000000000000000000000000000000..3d6c7c829881ac952debe9ad0c8b968bb797733e --- /dev/null +++ b/pythia-1b/W_U/cross-snapshot-32/d16384/seed0.config.json @@ -0,0 +1,119 @@ +{ + "public_label": "Pythia-1B W_U crosscoder, 32 snapshots, d_sae=16384, seed 0", + "config": { + "sae_type": "crosscoder", + "d_model": 2048, + "expansion_factor": 8.0, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0", + "wu_step_1", + "wu_step_2", + "wu_step_3", + "wu_step_4", + "wu_step_5", + "wu_step_6", + "wu_step_7", + "wu_step_8", + "wu_step_9", + "wu_step_10", + "wu_step_11", + "wu_step_12", + "wu_step_13", + "wu_step_14", + "wu_step_15", + "wu_step_16", + "wu_step_17", + "wu_step_18", + "wu_step_19", + "wu_step_20", + "wu_step_21", + "wu_step_22", + "wu_step_23", + "wu_step_24", + "wu_step_25", + "wu_step_26", + "wu_step_27", + "wu_step_28", + "wu_step_29", + "wu_step_30", + "wu_step_31" + ] + }, + "training": { + "lr": 2e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.2, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 300, + "batch_size": 1024, + "expansion_factor": 8.0, + "d_sae": 16384, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.05, + "lr_warmup_fraction": 0.0, + "lr_decay_fraction": 0.0, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.7812400637275599, + "reconstruction_mse": 0.2183065602183342, + "mean_l0": 499.1909545898437, + "dead_rate": 0.0, + "d_sae": 16384, + "n_snapshots": 32, + "n_rows": 50304 + }, + "steps": [ + 0, + 1, + 2, + 4, + 8, + 16, + 32, + 64, + 128, + 256, + 512, + 1000, + 2000, + 3000, + 4000, + 5000, + 6000, + 7000, + 8000, + 9000, + 14000, + 21000, + 27000, + 34000, + 47000, + 61000, + 75000, + 89000, + 102000, + 116000, + 130000, + 143000 + ], + "model_name": "EleutherAI/pythia-1b", + "seed": 0, + "kind": "cross-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-1b/W_U/cross-snapshot-32/d16384/seed0.md b/pythia-1b/W_U/cross-snapshot-32/d16384/seed0.md new file mode 100644 index 0000000000000000000000000000000000000000..2e890853d0a0210045fbf782c7d14b7f0fa272e2 --- /dev/null +++ b/pythia-1b/W_U/cross-snapshot-32/d16384/seed0.md @@ -0,0 +1,38 @@ +# Pythia-1B W_U crosscoder, 32 snapshots, d_sae=16384, seed 0 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-1b` across [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 14000, 21000, 27000, 34000, 47000, 61000, 75000, 89000, 102000, 116000, 130000, 143000] pretraining +steps. + +## Quality +- Explained variance: 0.7812400637275599 +- Mean L0: 499.1909545898437 +- Dead-feature rate: 0.0 +- d_sae: 16384 +- n_snapshots: 32 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 2e-05 +- JumpReLU LR multiplier: 0.2 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 300 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("seed0.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("seed0.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-1b/W_U/cross-snapshot-32/d16384/seed0.safetensors b/pythia-1b/W_U/cross-snapshot-32/d16384/seed0.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..5fdc59d8f23187eb2400caf81a5233b7fe762c5e --- /dev/null +++ b/pythia-1b/W_U/cross-snapshot-32/d16384/seed0.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:11ec7052cb8e3a478cbefa575e28142e7dc2e7a935caa3412b1dc84127eb94e9 +size 8592359872 diff --git a/pythia-1b/W_U/cross-snapshot-32/d24576/seed0.config.json b/pythia-1b/W_U/cross-snapshot-32/d24576/seed0.config.json new file mode 100644 index 0000000000000000000000000000000000000000..ed87b2aae45784785dcbf38a660547e29044371c --- /dev/null +++ b/pythia-1b/W_U/cross-snapshot-32/d24576/seed0.config.json @@ -0,0 +1,119 @@ +{ + "public_label": "Pythia-1B W_U crosscoder, 32 snapshots, d_sae=24576, seed 0", + "config": { + "sae_type": "crosscoder", + "d_model": 2048, + "expansion_factor": 12.000244140625, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0", + "wu_step_1", + "wu_step_2", + "wu_step_3", + "wu_step_4", + "wu_step_5", + "wu_step_6", + "wu_step_7", + "wu_step_8", + "wu_step_9", + "wu_step_10", + "wu_step_11", + "wu_step_12", + "wu_step_13", + "wu_step_14", + "wu_step_15", + "wu_step_16", + "wu_step_17", + "wu_step_18", + "wu_step_19", + "wu_step_20", + "wu_step_21", + "wu_step_22", + "wu_step_23", + "wu_step_24", + "wu_step_25", + "wu_step_26", + "wu_step_27", + "wu_step_28", + "wu_step_29", + "wu_step_30", + "wu_step_31" + ] + }, + "training": { + "lr": 2e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.2, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 300, + "batch_size": 1024, + "expansion_factor": 12.000244140625, + "d_sae": 24576, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.05, + "lr_warmup_fraction": 0.0, + "lr_decay_fraction": 0.0, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.861217602606241, + "reconstruction_mse": 0.13849477335810662, + "mean_l0": 517.0205554199218, + "dead_rate": 0.0, + "d_sae": 24576, + "n_snapshots": 32, + "n_rows": 50304 + }, + "steps": [ + 0, + 1, + 2, + 4, + 8, + 16, + 32, + 64, + 128, + 256, + 512, + 1000, + 2000, + 3000, + 4000, + 5000, + 6000, + 7000, + 8000, + 9000, + 14000, + 21000, + 27000, + 34000, + 47000, + 61000, + 75000, + 89000, + 102000, + 116000, + 130000, + 143000 + ], + "model_name": "EleutherAI/pythia-1b", + "seed": 0, + "kind": "cross-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-1b/W_U/cross-snapshot-32/d24576/seed0.md b/pythia-1b/W_U/cross-snapshot-32/d24576/seed0.md new file mode 100644 index 0000000000000000000000000000000000000000..c263782f0cc3a92507f671f5ac07dc6f59db047a --- /dev/null +++ b/pythia-1b/W_U/cross-snapshot-32/d24576/seed0.md @@ -0,0 +1,38 @@ +# Pythia-1B W_U crosscoder, 32 snapshots, d_sae=24576, seed 0 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-1b` across [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 14000, 21000, 27000, 34000, 47000, 61000, 75000, 89000, 102000, 116000, 130000, 143000] pretraining +steps. + +## Quality +- Explained variance: 0.861217602606241 +- Mean L0: 517.0205554199218 +- Dead-feature rate: 0.0 +- d_sae: 24576 +- n_snapshots: 32 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 2e-05 +- JumpReLU LR multiplier: 0.2 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 300 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("seed0.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("seed0.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-1b/W_U/cross-snapshot-32/d24576/seed0.safetensors b/pythia-1b/W_U/cross-snapshot-32/d24576/seed0.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..d3875d34a013bac549f950705b7534a3d8ceaba7 --- /dev/null +++ b/pythia-1b/W_U/cross-snapshot-32/d24576/seed0.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:db8a72e73d992dc1bcc7b883a646632a7bd1db6b2f4c33838868fd94b74beccf +size 12888408520 diff --git a/pythia-1b/W_U/cross-snapshot-32/d8192/seed0.config.json b/pythia-1b/W_U/cross-snapshot-32/d8192/seed0.config.json new file mode 100644 index 0000000000000000000000000000000000000000..2845478bf93d9f8182fcff5e480b90c7b509ba15 --- /dev/null +++ b/pythia-1b/W_U/cross-snapshot-32/d8192/seed0.config.json @@ -0,0 +1,119 @@ +{ + "public_label": "Pythia-1B W_U crosscoder, 32 snapshots, d_sae=8192, seed 0", + "config": { + "sae_type": "crosscoder", + "d_model": 2048, + "expansion_factor": 4.0, + "use_decoder_bias": true, + "act_fn": "jumprelu", + "norm_activation": "inference", + "sparsity_include_decoder_norm": true, + "top_k": 50, + "use_triton_kernel": false, + "sparsity_threshold_for_triton_spmm_kernel": 0.996, + "jumprelu_threshold_window": 4.0, + "hook_points": [ + "wu_step_0", + "wu_step_1", + "wu_step_2", + "wu_step_3", + "wu_step_4", + "wu_step_5", + "wu_step_6", + "wu_step_7", + "wu_step_8", + "wu_step_9", + "wu_step_10", + "wu_step_11", + "wu_step_12", + "wu_step_13", + "wu_step_14", + "wu_step_15", + "wu_step_16", + "wu_step_17", + "wu_step_18", + "wu_step_19", + "wu_step_20", + "wu_step_21", + "wu_step_22", + "wu_step_23", + "wu_step_24", + "wu_step_25", + "wu_step_26", + "wu_step_27", + "wu_step_28", + "wu_step_29", + "wu_step_30", + "wu_step_31" + ] + }, + "training": { + "lr": 2e-05, + "l1_coefficient": 0.3, + "frequency_scale": 0.01, + "tanh_stretch_coefficient": 1.0, + "jumprelu_lr_factor": 0.2, + "init_threshold": 0.1, + "decoder_transpose_init": 1.0, + "amp_dtype": "fp32", + "optimizer": "adam", + "n_epochs": 300, + "batch_size": 1024, + "expansion_factor": 4.0, + "d_sae": 8192, + "input_preprocess": "center_scale", + "l1_warmup_fraction": 0.05, + "lr_warmup_fraction": 0.0, + "lr_decay_fraction": 0.0, + "auxk_coefficient": 0.0, + "auxk_k": 512, + "dead_window_steps": 100 + }, + "quality": { + "explained_variance": 0.628030172619609, + "reconstruction_mse": 0.37119892656803133, + "mean_l0": 374.1408770751953, + "dead_rate": 0.0, + "d_sae": 8192, + "n_snapshots": 32, + "n_rows": 50304 + }, + "steps": [ + 0, + 1, + 2, + 4, + 8, + 16, + 32, + 64, + 128, + 256, + 512, + 1000, + 2000, + 3000, + 4000, + 5000, + 6000, + 7000, + 8000, + 9000, + 14000, + 21000, + 27000, + 34000, + 47000, + 61000, + 75000, + 89000, + 102000, + 116000, + 130000, + 143000 + ], + "model_name": "EleutherAI/pythia-1b", + "seed": 0, + "kind": "cross-snapshot", + "preprocess_mode": "center_scale" +} \ No newline at end of file diff --git a/pythia-1b/W_U/cross-snapshot-32/d8192/seed0.md b/pythia-1b/W_U/cross-snapshot-32/d8192/seed0.md new file mode 100644 index 0000000000000000000000000000000000000000..be0ff94718cdd9a37dc7c5cc3813fde3827f3a8e --- /dev/null +++ b/pythia-1b/W_U/cross-snapshot-32/d8192/seed0.md @@ -0,0 +1,38 @@ +# Pythia-1B W_U crosscoder, 32 snapshots, d_sae=8192, seed 0 + +Companion artifact to *Parameter-trajectory crosscoders for vocabulary readout +evolution* (NeurIPS 2026). Trained from snapshots of +`EleutherAI/pythia-1b` across [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 14000, 21000, 27000, 34000, 47000, 61000, 75000, 89000, 102000, 116000, 130000, 143000] pretraining +steps. + +## Quality +- Explained variance: 0.628030172619609 +- Mean L0: 374.1408770751953 +- Dead-feature rate: 0.0 +- d_sae: 8192 +- n_snapshots: 32 + +## Training recipe (key hparams) +- Activation: jumprelu +- L1 coefficient: 0.3 +- Frequency penalty scale: 0.01 +- Tanh-stretch coefficient: 1.0 +- LR: 2e-05 +- JumpReLU LR multiplier: 0.2 +- Init threshold: 0.1 +- Optimizer: adam +- Epochs: 300 +- Batch size: 1024 +- Preprocess: center_scale +- Seed: 0 + +## Loading +```python +from safetensors.torch import load_file +weights = load_file("seed0.safetensors") +# weights['W_E'] / weights['W_D'] / weights['b_E'] / weights['b_D'] / etc. +import json +config = json.load(open("seed0.config.json")) +``` + +See the repo top-level README for the full crosscoder loading example. diff --git a/pythia-1b/W_U/cross-snapshot-32/d8192/seed0.safetensors b/pythia-1b/W_U/cross-snapshot-32/d8192/seed0.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..4e2a943f3f55ff10dd3148480f4c06f12c997bfa --- /dev/null +++ b/pythia-1b/W_U/cross-snapshot-32/d8192/seed0.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:649b642380204013b32f2cbe45c5e21d9322c2eaa826e320ce52347a9f7c088c +size 4296311232