| --- |
| license: cc-by-4.0 |
| tags: |
| - interpretability |
| - mechanistic-interpretability |
| - jacobian-lens |
| - pythia |
| - reproducibility |
| library_name: pytorch |
| --- |
| |
| # fittingcorpus: fitted Jacobian maps and results |
|
|
| The artifacts for **The Fitting Corpus Is Part of the Measurement: Auditing a Jacobian Lens on |
| Pythia** (Tarush Gupta, *Interpretability as a Science* workshop, NeurIPS 2026). |
|
|
| **Code, pre-registrations and documentation:** |
| [github.com/tarush-ai/fittingcorpus](https://github.com/tarush-ai/fittingcorpus) |
|
|
| This repository holds the two things the code repository does not: every stored result, and |
| every fitted map. The Jacobian lens reads a mid-layer hidden state through a map *J*, the model's |
| derivative of its final hidden state with respect to that layer, averaged over a corpus of prompts. |
| The paper measures how much the corpus chosen for that average changes what the lens reads, on |
| Pythia models from 70M to 2.8B parameters. |
|
|
| ## Contents |
|
|
| | path | what it holds | |
| |---|---| |
| | `results/*.json` | 193 results files, one or more per experiment. Every statistic in the paper is computed from these | |
| | `results/<experiment>/*.json`, `*.npz` | 147 per-cell and per-seed records behind them | |
| | `results/**/*.pt` | 382 files, 16.2 GB: 380 fitted Jacobian maps and 2 activation caches. 222 maps also carry a `_provenance.json` sidecar | |
| | `corpora/` | the corpus manifests: per-file SHA-256, document counts, source datasets and revisions, seed-block ranges. No corpus text | |
|
|
| The maps, by the experiment that fitted them (IDs from the code repository's |
| `docs/experiments/REGISTRY.md`): |
|
|
| | directory | maps | experiment | |
| |---|---:|---| |
| | `results/lenses/e28/` | 166 | E28: the read ladder over fitting corpus and fitting-set size *N* | |
| | `results/lenses/plad/` | 53 | E27: maps fitted on two halves of each corpus (between- against within-corpus distance) | |
| | `results/lenses/ladder/` | 15 | the model ladder, 70M to 2.8B | |
| | `results/lenses/misc/` | 8 | smoke tests and one-off fits, including the median and trimmed-mean variants of E16 | |
| | `results/e48/` | 27 | E48 and E52: the 24 maps behind the 8 × 8 factorial (eight fitting corpora × three seeds at 410M), and 3 refits on a deduplicated News 2024 corpus | |
| | `results/cv6/` | 30 | E75: the 1.4B and 2.8B replication (plus two activation caches) | |
| | `results/ladder1b_b613/` | 15 | E62: the 1B ladder on band [6,13] | |
| | `results/r6/` | 8 | E72: within-source resampling | |
| | `results/r7/`, `results/r7_verified_snapshot/` | 47 | E74: fitting pools matched on type-token ratio | |
| | `results/e65_lenses/` | 11 | E65: Pythia training checkpoints | |
|
|
| Every map is the released reference implementation's corpus average |
| ([anthropics/jacobian-lens](https://github.com/anthropics/jacobian-lens) at `581d398`), fitted in |
| fp32 with TF32 off, except the two robust-average variants in `misc/`, which are named as such. |
|
|
| ## Usage |
|
|
| The code repository's scripts fetch from here, with no login needed: |
|
|
| ```bash |
| git clone https://github.com/tarush-ai/fittingcorpus && cd fittingcorpus |
| uv sync && bash repro/01_setup_local.sh |
| bash repro/04_fetch_results.sh # results/*.json, about 8 MB |
| bash repro/RUN_ALL.sh --tier T0 # recompute every statistic in the paper |
| bash repro/03_fetch_artifacts.sh --all # the maps, 16 GB, hash-verified on download |
| bash repro/RUN_ALL.sh --tier T1 # re-score every read from the maps, on CPU |
| ``` |
|
|
| ## Provenance |
|
|
| Every results file carries a `provenance` block: the producing script and its SHA-256, the git |
| commit, the command line, and the SHA-256 of every input. `payload_sha256` hashes the record with |
| that block excluded, so it is unchanged by a rerun that alters only environment metadata. Rerun a |
| result, compare `payload_sha256`, and a match means no number moved. A map's sidecar records the map's |
| own SHA-256 and the configuration that fitted it. |
|
|
| The code repository's `docs/reproducibility/HF_MIRROR_MANIFEST.tsv` lists every file here with |
| its size and hash. `tools/hf_mirror_manifest.py --verify --repo Tarush-AI/fittingcorpus` checks |
| this repository against it. |
|
|
| ## Licence |
|
|
| CC BY 4.0. Everything here was produced by this work. The models are EleutherAI's Pythia |
| (Apache-2.0) and the fitter is the released Jacobian-lens implementation (Apache-2.0). |
|
|
| **No corpus text is distributed.** It is third-party text, and the Wikipedia-derived corpora are |
| CC BY-SA, which cannot be relicensed as CC BY 4.0. The code repository rebuilds the pool from its |
| public sources with `bash repro/06_data.sh --build` and verifies it against the manifests here. |
| See `ATTRIBUTION.md`. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{gupta2026fittingcorpus, |
| title = {The Fitting Corpus Is Part of the Measurement: Auditing a Jacobian Lens on Pythia}, |
| author = {Gupta, Tarush}, |
| booktitle = {Interpretability as a Science Workshop at NeurIPS 2026}, |
| year = {2026} |
| } |
| ``` |
|
|
