license: cc-by-4.0
tags:
- interpretability
- mechanistic-interpretability
- jacobian-lens
- pythia
- reproducibility
library_name: pytorch
fittingcorpus: fitted Jacobian maps and results
The artifacts for The Fitting Corpus Is Part of the Measurement: Auditing a Jacobian Lens on Pythia (Tarush Gupta, Interpretability as a Science workshop, NeurIPS 2026).
Code, pre-registrations and documentation: github.com/tarush-ai/fittingcorpus
This repository holds the two things the code repository does not: every stored result, and every fitted map. The Jacobian lens reads a mid-layer hidden state through a map J, the model's derivative of its final hidden state with respect to that layer, averaged over a corpus of prompts. The paper measures how much the corpus chosen for that average changes what the lens reads, on Pythia models from 70M to 2.8B parameters.
Contents
| path | what it holds |
|---|---|
results/*.json |
193 results files, one or more per experiment. Every statistic in the paper is computed from these |
results/<experiment>/*.json, *.npz |
147 per-cell and per-seed records behind them |
results/**/*.pt |
382 files, 16.2 GB: 380 fitted Jacobian maps and 2 activation caches. 222 maps also carry a _provenance.json sidecar |
corpora/ |
the corpus manifests: per-file SHA-256, document counts, source datasets and revisions, seed-block ranges. No corpus text |
The maps, by the experiment that fitted them (IDs from the code repository's
docs/experiments/REGISTRY.md):
| directory | maps | experiment |
|---|---|---|
results/lenses/e28/ |
166 | E28: the read ladder over fitting corpus and fitting-set size N |
results/lenses/plad/ |
53 | E27: maps fitted on two halves of each corpus (between- against within-corpus distance) |
results/lenses/ladder/ |
15 | the model ladder, 70M to 2.8B |
results/lenses/misc/ |
8 | smoke tests and one-off fits, including the median and trimmed-mean variants of E16 |
results/e48/ |
27 | E48 and E52: the 24 maps behind the 8 × 8 factorial (eight fitting corpora × three seeds at 410M), and 3 refits on a deduplicated News 2024 corpus |
results/cv6/ |
30 | E75: the 1.4B and 2.8B replication (plus two activation caches) |
results/ladder1b_b613/ |
15 | E62: the 1B ladder on band [6,13] |
results/r6/ |
8 | E72: within-source resampling |
results/r7/, results/r7_verified_snapshot/ |
47 | E74: fitting pools matched on type-token ratio |
results/e65_lenses/ |
11 | E65: Pythia training checkpoints |
Every map is the released reference implementation's corpus average
(anthropics/jacobian-lens at 581d398), fitted in
fp32 with TF32 off, except the two robust-average variants in misc/, which are named as such.
Usage
The code repository's scripts fetch from here, with no login needed:
git clone https://github.com/tarush-ai/fittingcorpus && cd fittingcorpus
uv sync && bash repro/01_setup_local.sh
bash repro/04_fetch_results.sh # results/*.json, about 8 MB
bash repro/RUN_ALL.sh --tier T0 # recompute every statistic in the paper
bash repro/03_fetch_artifacts.sh --all # the maps, 16 GB, hash-verified on download
bash repro/RUN_ALL.sh --tier T1 # re-score every read from the maps, on CPU
Provenance
Every results file carries a provenance block: the producing script and its SHA-256, the git
commit, the command line, and the SHA-256 of every input. payload_sha256 hashes the record with
that block excluded, so it is unchanged by a rerun that alters only environment metadata. Rerun a
result, compare payload_sha256, and a match means no number moved. A map's sidecar records the map's
own SHA-256 and the configuration that fitted it.
The code repository's docs/reproducibility/HF_MIRROR_MANIFEST.tsv lists every file here with
its size and hash. tools/hf_mirror_manifest.py --verify --repo Tarush-AI/fittingcorpus checks
this repository against it.
Licence
CC BY 4.0. Everything here was produced by this work. The models are EleutherAI's Pythia (Apache-2.0) and the fitter is the released Jacobian-lens implementation (Apache-2.0).
No corpus text is distributed. It is third-party text, and the Wikipedia-derived corpora are
CC BY-SA, which cannot be relicensed as CC BY 4.0. The code repository rebuilds the pool from its
public sources with bash repro/06_data.sh --build and verifies it against the manifests here.
See ATTRIBUTION.md.
Citation
@inproceedings{gupta2026fittingcorpus,
title = {The Fitting Corpus Is Part of the Measurement: Auditing a Jacobian Lens on Pythia},
author = {Gupta, Tarush},
booktitle = {Interpretability as a Science Workshop at NeurIPS 2026},
year = {2026}
}