Duplicate from AnonymousInterpScience/pythia-jlens-artifacts

Co-authored-by: Anonymous <AnonymousInterpScience@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

ATTRIBUTION.md

@@ -0,0 +1,48 @@
+# ATTRIBUTION
+
+This repository is **CC BY 4.0 without qualification**. It contains only what this work produced.
+
+## What is here
+
+| what | size | licence |
+|---|---|---|
+| `results/lenses/**` — 382 fitted operators + sidecars | 16.2 GB | CC BY 4.0 |
+| `results/**.json`, `results/**.npz` — experimental records | ~27 MB | CC BY 4.0 |
+| `corpora/manifest.json`, `corpora/manifest_ood.json`, `corpora/dataset_revisions.json` | 12 KB | CC BY 4.0 |
+
+Derived from **Pythia** (EleutherAI, Apache-2.0) via the **Jacobian-lens** reference
+implementation (Apache-2.0, pinned `581d398`). Neither restricts these derivatives.
+
+## What is deliberately NOT here, and how to get it
+
+The eleven corpus plaintext files are **not distributed**. They are third-party text under
+upstream terms; `Wikipedia_en.jsonl` and `OOD_Wikipedia_2023.jsonl` are Wikipedia-derived and
+carry **CC BY-SA**, which is share-alike and cannot be relicensed as CC BY 4.0. Shipping them
+would have forced a mixed-licence repository and made the CC BY 4.0 claim above false.
+
+Rebuild the pool from its public sources:
+
+```bash
+bash repro/06_build_corpora.sh          # builds, then hash-verifies against the manifests
+bash repro/06_build_corpora.sh --verify # verify a pool you already have
+```
+
+`corpora/manifest.json` and `corpora/manifest_ood.json` ship here and carry, per corpus: the
+SHA-256 of the exact file used, the document count, the source dataset id and config, and the
+three disjoint seed-block index ranges. A rebuild that matches those hashes is the same pool.
+
+**Measured 2026-08-31, rebuilding from scratch on a clean machine:**
+
+| corpus | rebuild |
+|---|---|
+| Pile-CC, StackExchange, Wikipedia_en, Github, USPTO_Backgrounds | **5/5 bit-identical** |
+| OOD_CommonPile, TRAP_FineWeb, CONTROL_PubMed_2023 | **3/3 bit-identical** |
+| `corpora/manifest.json` regenerated | **byte-identical** |
+| OOD_Wikipedia_2023, OOD_arXiv_2023 | not re-run (53 MB / 110 MB of streaming); both carry SHA-256 in the manifest and are checked by the script |
+
+**One caveat, stated plainly.** `src/build_corpora.py` calls `load_dataset(
+"monology/pile-uncopyrighted", ...)` with **no `revision=` pin**, so an upstream change to that
+dataset would produce a different pool. The result above is evidence the upstream had not moved
+as of that date — not a guarantee it never will. The manifest is the check: a mismatch means the
+upstream moved, which is a finding to report, not a failure to work around. The OOD panel is more
+stable by construction because its configs are date-stamped (`20231101.en`, `2024-06`).

LICENSE

@@ -0,0 +1,25 @@
+Creative Commons Attribution 4.0 International (CC BY 4.0)
+
+Copyright (c) 2026 The Authors
+
+Everything in this repository is licensed under CC BY 4.0:
+https://creativecommons.org/licenses/by/4.0/
+
+That claim is unqualified, and it is unqualified BY CONSTRUCTION rather than by assertion: this
+repository contains only artifacts this work produced.
+
+  results/lenses/**        fitted Jacobian operators (.pt) and their provenance sidecars
+  results/**.json, **.npz  every experimental record and derived statistic
+  corpora/manifest*.json   the build record of the corpus pool: per-file SHA-256, document
+                           counts, source dataset id and config, disjoint seed-block ranges
+
+NO THIRD-PARTY CORPUS TEXT IS DISTRIBUTED HERE. The eleven corpus files this work used are
+third-party text under upstream terms, and two are Wikipedia-derived and carry CC BY-SA, a
+share-alike licence that cannot be relicensed as CC BY 4.0. Rather than qualify this licence,
+the pool is withheld in full and rebuilt from its public sources by the code repository's
+`repro/06_build_corpora.sh`. The manifests above are this work's own record of that pool and
+are what make the rebuild verifiable.
+
+The operators are derived from EleutherAI's Pythia suite (Apache-2.0) using the Jacobian-lens
+reference implementation (Apache-2.0, pinned at 581d398). Neither upstream licence restricts the
+licensing of the derived operators recorded here.

README.md

@@ -0,0 +1,60 @@
+---
+license: cc-by-4.0
+tags:
+  - interpretability
+  - jacobian-lens
+  - pythia
+  - reproducibility
+library_name: pytorch
+---
+
+# Corpus and context dependence of the Jacobian lens — artifact mirror
+
+Fitted operators, experimental records and the corpus pool for the accompanying paper.
+
+**Licence.** Everything here is **CC BY 4.0**, without qualification — see `LICENSE`. That is
+true by construction, not by assertion: this repository contains only artifacts this work
+produced. **No third-party corpus text is distributed here.** The corpus pool is rebuilt from its
+public sources by `repro/06_build_corpora.sh` and verified against the manifests that ship here;
+8 of 11 corpora rebuilt bit-identically when measured. `ATTRIBUTION.md` has the detail.
+
+## Layout
+
+```
+results/lenses/e28/       166 operators + sidecars   corpus x N ladder
+results/lenses/plad/       53 operators + sidecars   per-layer activation dumps
+results/lenses/ladder/     15 operators + sidecars   canonical model ladder (70m ... 2.8b)
+results/lenses/misc/        8 operators              smokes and one-offs
+results/<17 experiment dirs>                         per-experiment operators and records
+results/*.json            190 experimental records
+corpora/                    3 provenance manifests (NO corpus text — rebuild it, see ATTRIBUTION.md)
+```
+
+Every `.pt` is a corpus-averaged Jacobian `J_l = E_x[dh_final/dh_l]` fitted with the reference
+implementation at pin `581d398`, fp32, TF32 off, uncentered.
+
+## Reproducing
+
+Every statistic in the paper recomputes from `results/**.json` alone — no operators, no GPU:
+
+```bash
+git clone <code repository>
+uv sync && bash repro/01_setup_local.sh
+bash repro/04_fetch_results.sh          # pulls results/**.json from here, ~7 MB
+bash repro/RUN_ALL.sh --tier T0         # 12 modules
+```
+
+`--tier T1` re-scores the stored operators on CPU and needs the `.pt` from this repo
+(`bash repro/03_fetch_artifacts.sh --all`). `--tier T2` refits on GPU.
+
+## Provenance
+
+Each results file carries a provenance block: the producing script and its SHA-256, the commit,
+the argv, and the SHA-256 of every declared input. `payload_sha256` covers the record **excluding**
+the provenance block, so it is stable across reruns that change only environment metadata — rerun
+and compare it, and a match means no number moved.
+
+No hostname is recorded. Input hashes are the values at write time; where a record was later
+redacted, `payload_sha256_before_redaction` retains the original so both remain verifiable.
+
+No corpus plaintext is distributed — see `ATTRIBUTION.md` for the rebuild.

corpora/dataset_revisions.json

@@ -0,0 +1,111 @@
+{
+ "_README": [
+  "Dataset revisions pinned 2026-09-01. THE SOURCE OF TRUTH for every corpus build.",
+  "",
+  "WHY. src/build_corpora.py called load_dataset() with no revision=, so the pool depended on",
+  "whatever the upstream dataset was on the day you ran it. docs/reproducibility/RELEASE.md",
+  "finding B0 recorded that and concluded the pool was not regenerable. A rebuild on 2026-08-31",
+  "in fact reproduced 8 of 11 corpora BIT-IDENTICALLY -- evidence the upstream had not moved, not",
+  "a guarantee it never would. These pins turn that evidence into a guarantee.",
+  "",
+  "`sha` is the dataset repo commit on the Hub. Pass it as revision= to load_dataset().",
+  "`config` is load_dataset()'s SECOND POSITIONAL argument (the config/subset name), NOT a locale;",
+  "null means the dataset declares no configs and none may be passed."
+ ],
+ "monology/pile-uncopyrighted": {
+  "used_by": "src/build_corpora.py",
+  "config": null,
+  "split": "train",
+  "supplies": [
+   "Pile-CC",
+   "StackExchange",
+   "Wikipedia (en)",
+   "Github",
+   "USPTO Backgrounds"
+  ],
+  "sha": "3be90335b66f24456a5d6659d9c8d208c0357119",
+  "pinned_on": "2026-09-01"
+ },
+ "wikimedia/wikipedia": {
+  "used_by": "src/build_ood_corpora.py",
+  "config": "20231101.en",
+  "split": "train",
+  "supplies": [
+   "OOD_Wikipedia_2023"
+  ],
+  "sha": "b04c8d1ceb2f5cd4588862100d08de323dccfbaa",
+  "pinned_on": "2026-09-01"
+ },
+ "neuralwork/arxiver": {
+  "used_by": "src/build_ood_corpora.py",
+  "config": null,
+  "split": "train",
+  "supplies": [
+   "OOD_arXiv_2023"
+  ],
+  "sha": "698a6662e77fd5dd45dbbec988abc8123e5fa086",
+  "pinned_on": "2026-09-01"
+ },
+ "RealTimeData/bbc_news_alltime": {
+  "used_by": "src/build_ood_corpora.py",
+  "config": "2024-06",
+  "split": "train",
+  "supplies": [
+   "OOD_News_2024"
+  ],
+  "sha": "8dd1ecdc92ac43f9c04a3da3e945537dbb08179b",
+  "pinned_on": "2026-09-01"
+ },
+ "common-pile/comma_v0.1_training_dataset": {
+  "used_by": "src/build_ood_corpora.py",
+  "config": null,
+  "split": "train",
+  "supplies": [
+   "OOD_CommonPile"
+  ],
+  "sha": "5afc546db324e7f39f297ba757c9a60547151e7c",
+  "pinned_on": "2026-09-01"
+ },
+ "HuggingFaceFW/fineweb": {
+  "used_by": "src/build_ood_corpora.py",
+  "config": "sample-10BT",
+  "split": "train",
+  "supplies": [
+   "TRAP_FineWeb"
+  ],
+  "sha": "9bb295ddab0e05d785b879661af7260fed5140fc",
+  "pinned_on": "2026-09-01"
+ },
+ "MedRAG/pubmed": {
+  "used_by": "src/build_ood_corpora.py",
+  "config": null,
+  "split": "train",
+  "supplies": [
+   "CONTROL_PubMed_2023"
+  ],
+  "sha": "33da3593d5756bc04c8909f170003c0b14197957",
+  "pinned_on": "2026-09-01"
+ },
+ "EleutherAI/pile-deduped-pythia-preshuffled": {
+  "used_by": "experiments/t35_containment.py",
+  "config": null,
+  "split": null,
+  "supplies": [
+   "the Pythia training stream itself, for E35/M1 containment"
+  ],
+  "sha": "4647773ea142ab1ff5694602fa104bbf49088408",
+  "pinned_on": "2026-09-01"
+ },
+ "_DERIVED_CORPORA": {
+  "_note": "Corpora produced from another corpus in this tree, not loaded from the Hub. They have NO dataset revision of their own by construction. Their full derivation rule and parent hash live in corpora/manifest_ood.json. Recorded here so this file's source-of-truth claim covers every corpus on disk.",
+  "OOD_News_2024_dedup": {
+   "derived_from": "OOD_News_2024",
+   "parent_upstream_repo": "RealTimeData/bbc_news_alltime",
+   "parent_sha": "8dd1ecdc92ac43f9c04a3da3e945537dbb08179b",
+   "rule": "exact-text dedup, first occurrence kept, parent order preserved (2400 -> 1598)",
+   "full_record": "corpora/manifest_ood.json -> OOD_News_2024_dedup.derived_from",
+   "revision": null,
+   "revision_note": "INTENTIONALLY null \u2014 derived, not loaded. Never fabricate one."
+  }
+ }
+}

corpora/manifest.json

@@ -0,0 +1,98 @@
+{
+ "Pile-CC": {
+  "file": "Pile-CC.jsonl",
+  "n": 2400,
+  "sha256": "abbb9f19a6eb6f4c12fb382735832ceec271fece15cfebcf0fe838765d4b1e74",
+  "seed_blocks": [
+   [
+    0,
+    800
+   ],
+   [
+    800,
+    1600
+   ],
+   [
+    1600,
+    2400
+   ]
+  ]
+ },
+ "StackExchange": {
+  "file": "StackExchange.jsonl",
+  "n": 2400,
+  "sha256": "bc25e2d5775e9719ebcbdaa3bd115d827f72ebf11f47fedb377f8b0a69a28971",
+  "seed_blocks": [
+   [
+    0,
+    800
+   ],
+   [
+    800,
+    1600
+   ],
+   [
+    1600,
+    2400
+   ]
+  ]
+ },
+ "Wikipedia (en)": {
+  "file": "Wikipedia_en.jsonl",
+  "n": 2400,
+  "sha256": "b30f4cef11add560f7df8ed07a613e3fcc6a4dda865975bec15f2952253bda77",
+  "seed_blocks": [
+   [
+    0,
+    800
+   ],
+   [
+    800,
+    1600
+   ],
+   [
+    1600,
+    2400
+   ]
+  ]
+ },
+ "Github": {
+  "file": "Github.jsonl",
+  "n": 2400,
+  "sha256": "f2bb459d10cad6c13a8b66c86191bc43df4393ba5b722a5315ea878fad2d1a26",
+  "seed_blocks": [
+   [
+    0,
+    800
+   ],
+   [
+    800,
+    1600
+   ],
+   [
+    1600,
+    2400
+   ]
+  ]
+ },
+ "USPTO Backgrounds": {
+  "file": "USPTO_Backgrounds.jsonl",
+  "n": 2400,
+  "sha256": "b09ac74bdc8b169ee5c9bea6be481262f743947a92107164f30eca2346e3ccb9",
+  "seed_blocks": [
+   [
+    0,
+    800
+   ],
+   [
+    800,
+    1600
+   ],
+   [
+    1600,
+    2400
+   ]
+  ]
+ },
+ "_note": "seed_blocks are DISJOINT index ranges; seed k draws its N prompts from block k, so seeds share no prompts and the seed variance is a genuine resampling variance rather than an overlap artifact"
+}

corpora/manifest_ood.json

@@ -0,0 +1,218 @@
+{
+ "OOD_Wikipedia_2023": {
+  "file": "OOD_Wikipedia_2023.jsonl",
+  "n": 2400,
+  "sha256": "57fb26cf3997b571543281efe3330efcbab1f37e97f99eb6df4deaa027d98550",
+  "seed_blocks": [
+   [
+    0,
+    800
+   ],
+   [
+    800,
+    1600
+   ],
+   [
+    1600,
+    2400
+   ]
+  ],
+  "hf": {
+   "path": "wikimedia/wikipedia",
+   "name": "20231101.en",
+   "split": "train"
+  },
+  "text_field": "text",
+  "claim": "TEMPORAL. Same source family as the Pile's Wikipedia component, but a 2023-11 snapshot. Revisions written after the ~2020 Pile snapshot cannot have been in it. Confounds: three years of new entities, events and editorial convention move with the exclusion. This is external review's PRIMARY recommended panel because it holds language and register nearly fixed.",
+  "scanned": 2436,
+  "status": "CANDIDATE \u2014 outside-Pile is a HYPOTHESIS to be measured by M1, not a property asserted here"
+ },
+ "OOD_arXiv_2023": {
+  "file": "OOD_arXiv_2023.jsonl",
+  "n": 2400,
+  "sha256": "565f3541ef189ba624ff56373d7d3df0b0341020578b0c6cffc6391b332ff4af",
+  "seed_blocks": [
+   [
+    0,
+    800
+   ],
+   [
+    800,
+    1600
+   ],
+   [
+    1600,
+    2400
+   ]
+  ],
+  "hf": {
+   "path": "neuralwork/arxiver",
+   "split": "train"
+  },
+  "text_field": "markdown",
+  "claim": "TEMPORAL. 2023 arXiv papers; the Pile's arXiv component predates the snapshot. Stable LaTeX/scientific register. Confounds: new terminology, fields, authors; heavy template reuse means short-k overlap will be high regardless of exposure.",
+  "scanned": 2401,
+  "status": "CANDIDATE \u2014 outside-Pile is a HYPOTHESIS to be measured by M1, not a property asserted here"
+ },
+ "OOD_News_2024": {
+  "file": "OOD_News_2024.jsonl",
+  "n": 2400,
+  "sha256": "45aaab463d8987725fe9b7edb5bd316a29c2d547a5a1ffd2c54f41df3be1df80",
+  "seed_blocks": [
+   [
+    0,
+    800
+   ],
+   [
+    800,
+    1600
+   ],
+   [
+    1600,
+    2400
+   ]
+  ],
+  "hf": {
+   "path": "RealTimeData/bbc_news_alltime",
+   "name": "2024-06",
+   "split": "train"
+  },
+  "text_field": "content",
+  "claim": "TEMPORAL. June 2024 news, four years past the snapshot. Ordinary English, so it tests transport without immediately confounding on script or competence. Confounds: news register, named entities, and possible syndication into web crawls.",
+  "scanned": 2930,
+  "status": "CANDIDATE \u2014 outside-Pile is a HYPOTHESIS to be measured by M1, not a property asserted here"
+ },
+ "OOD_CommonPile": {
+  "file": "OOD_CommonPile.jsonl",
+  "n": 2400,
+  "sha256": "0d7ca5e94b4b1891e1d99a310b263806f370bae861decdfc3cf0f1ebb7a755e4",
+  "seed_blocks": [
+   [
+    0,
+    800
+   ],
+   [
+    800,
+    1600
+   ],
+   [
+    1600,
+    2400
+   ]
+  ],
+  "hf": {
+   "path": "common-pile/comma_v0.1_training_dataset",
+   "split": "train"
+  },
+  "text_field": "text",
+  "claim": "PROVENANCE. Assembled later from openly-licensed and public-domain sources, covering domains not deliberately represented in the Pile. Tests whether the effect is about corpus construction/licensing rather than topic. Confounds: licence filtering changes the source population; some documents may still be older web material.",
+  "scanned": 3069,
+  "status": "CANDIDATE \u2014 outside-Pile is a HYPOTHESIS to be measured by M1, not a property asserted here"
+ },
+ "TRAP_FineWeb": {
+  "file": "TRAP_FineWeb.jsonl",
+  "n": 2400,
+  "sha256": "6164a3cc1113f6c3b90fd5f67e3eba2e80324a69d9d4c0ae671aefdefc4b9260",
+  "seed_blocks": [
+   [
+    0,
+    800
+   ],
+   [
+    800,
+    1600
+   ],
+   [
+    1600,
+    2400
+   ]
+  ],
+  "hf": {
+   "path": "HuggingFaceFW/fineweb",
+   "name": "sample-10BT",
+   "split": "train"
+  },
+  "text_field": "text",
+  "claim": "DELIBERATE POSITIVE CONTROL, NOT A CLEAN Q. FineWeb is Common-Crawl-derived and Pile-CC is too, so it can contain the same pages, later versions of them, or mirrors. Both external review reviews name CC derivatives as a trap. Included so M1 has something it MUST flag as overlapping: if FineWeb does not show high containment, M1 is broken.",
+  "scanned": 2708,
+  "status": "CANDIDATE \u2014 outside-Pile is a HYPOTHESIS to be measured by M1, not a property asserted here"
+ },
+ "CONTROL_PubMed_2023": {
+  "file": "CONTROL_PubMed_2023.jsonl",
+  "n": 2400,
+  "sha256": "cffa12d20484b15c2430fc402a994632a9a27595454d02b457b94ec80c27856a",
+  "seed_blocks": [
+   [
+    0,
+    800
+   ],
+   [
+    800,
+    1600
+   ],
+   [
+    1600,
+    2400
+   ]
+  ],
+  "hf": {
+   "path": "MedRAG/pubmed",
+   "split": "train"
+  },
+  "text_field": "content",
+  "claim": "AMBIGUOUS BY DESIGN. The Pile contains PubMed Central, so this is source-INCLUDED; only post-snapshot articles would be temporally excluded and this dump is not date-filtered. Expected to land BETWEEN the Pile components and the temporal panels, which makes it a useful mid-scale reference point for whatever divergence metric we fit.",
+  "scanned": 3161,
+  "status": "CANDIDATE \u2014 outside-Pile is a HYPOTHESIS to be measured by M1, not a property asserted here"
+ },
+ "OOD_News_2024_dedup": {
+  "file": "OOD_News_2024_dedup.jsonl",
+  "n": 1598,
+  "sha256": "713cdbcf4512208f4cb3cac831d2a4eeb76ae0fe577d215473fd71919d022529",
+  "seed_blocks": [
+   [
+    0,
+    532
+   ],
+   [
+    532,
+    1064
+   ],
+   [
+    1064,
+    1596
+   ]
+  ],
+  "seed_blocks_note": "computed as b = len(texts) // 3 at load time (experiments/trainval.py:256), so the blocks are NOT the 800/1600/2400 of the n=2400 corpora. Recorded explicitly because this corpus is shorter than every other entry here.",
+  "DERIVED": true,
+  "hf": null,
+  "hf_note": "INTENTIONALLY null. This corpus has NO independent upstream. It is produced deterministically from OOD_News_2024 by the rule below and must never be given a fabricated dataset revision. To regenerate it, rebuild the parent at the revision pinned in corpora/dataset_revisions.json and re-apply the rule.",
+  "derived_from": {
+   "parent": "OOD_News_2024",
+   "parent_file": "OOD_News_2024.jsonl",
+   "parent_sha256": "45aaab463d8987725fe9b7edb5bd316a29c2d547a5a1ffd2c54f41df3be1df80",
+   "parent_n": 2400,
+   "parent_upstream": {
+    "path": "RealTimeData/bbc_news_alltime",
+    "name": "2024-06",
+    "split": "train",
+    "sha": "8dd1ecdc92ac43f9c04a3da3e945537dbb08179b"
+   },
+   "rule": "exact-text deduplication: keep the FIRST occurrence of each distinct `text` value, preserving the parent's record order; drop every later exact duplicate",
+   "rule_verified": "verified 2026-09-01 by recomputation: first-occurrence dedup of the parent reproduces this file's text sequence exactly (order-sensitive equality True); sorted(distinct) does NOT (False), so the order is first-occurrence, not sorted",
+   "records_in": 2400,
+   "records_out": 1598,
+   "exact_duplicates_removed": 802,
+   "text_field": "text",
+   "is_strict_subset_of_parent": true
+  },
+  "claim": "DERIVED CONTROL. The 2024-06 BBC News panel with its 802 exact-duplicate records removed. It exists because the parent contains syndicated/repeated articles, which inflate any recurrence-based statistic. It is NOT an independent OOD rung: it shares the parent's upstream, register and time window entirely.",
+  "status": "DERIVED \u2014 provenance is the parent's plus the recorded rule. Excluded BY DESIGN from the E48b containment index (experiments/t48b_exposure_growth.py:77-81 --exclude default): it was created after the k-gram shards were built and is a strict subset of OOD_News_2024, so indexing it would double-count the parent's k-grams.",
+  "used_by": [
+   "results/e48/lens_OOD_News_2024_dedup_410m_n200_s{0,1,2}.pt",
+   "results/e48/tv_OOD_News_2024_dedup_s{0,1,2}.json"
+  ],
+  "scanned": null,
+  "recorded_on": "2026-09-01",
+  "recorded_why": "this corpus had operators fitted on it but NO entry in this manifest and NO row in corpora/dataset_revisions.json, whose README claims to be the source of truth for every corpus build. Gap closed by recording the derivation, not by inventing an upstream pin."
+ }
+}

results/ctrlbf1_control_backfill.json

@@ -0,0 +1,305 @@
+{
+ "experiment": "CTRLBF1 \u2014 backfill of three already-completed, previously unrecorded controls",
+ "prereg": [
+  "docs/experiments/preregs/superseded/PREREG_E36_QLADDER.md (section E35)",
+  "docs/experiments/preregs/P0_t52_parallel_port.md"
+ ],
+ "status": "RECORDED FROM EXISTING OUTPUTS \u2014 no new experiment, no new threshold",
+ "recomputes_not_remeasures": true,
+ "why": "CLAUDE.md rule 10 requires a control to be named with the number it produced. These three were satisfied by data already on disk but were recorded as 'NOT RECORDED' (E35 M1-C3, M1-C4) or '(full-grid arms outstanding)' (P0 C2).",
+ "E35_M1_C3": {
+  "requirement_verbatim": "post-2020 text | must score below M1-C1. If not, either the date argument or the index is wrong",
+  "m1_c1_operationalised": "M1-C1 is 'a known Pile component ... high containment'. The bar used is the LOWEST of the five in-stream Pile components at k=32, 20/20 coverage \u2014 the most demanding reading of 'below M1-C1'.",
+  "k": 32,
+  "coverage": "20/20",
+  "in_stream_final": {
+   "Pile-CC": 0.9276060773863086,
+   "Github": 0.9340004374453194,
+   "StackExchange": 0.9179914195561899,
+   "USPTO_Backgrounds": 0.9267616923398619,
+   "Wikipedia_en": 0.9215951727072293
+  },
+  "lowest_in_stream": 0.9179914195561899,
+  "post_2020": {
+   "CONTROL_PubMed_2023": {
+    "containment_20of20": 0.7927273508312213,
+    "below_lowest_in_stream": true
+   },
+   "OOD_Wikipedia_2023": {
+    "containment_20of20": 0.26957534299550123,
+    "below_lowest_in_stream": true
+   },
+   "OOD_arXiv_2023": {
+    "containment_20of20": 8.598932872430531e-05,
+    "below_lowest_in_stream": true
+   },
+   "OOD_News_2024": {
+    "containment_20of20": 0.0007487928942136383,
+    "below_lowest_in_stream": true
+   }
+  },
+  "fires": true,
+  "source": "results/e48b_exposure_growth.json -> containment_by_coverage"
+ },
+ "E35_M1_C4": {
+  "requirement_verbatim": "one shard vs twenty | containment on 1/20 of the stream must be a strict lower bound on containment over 20/20",
+  "k": 32,
+  "by_corpus": {
+   "CONTROL_PubMed_2023": {
+    "containment_1of20": 0.061166716783366985,
+    "containment_20of20": 0.7927273508312213,
+    "is_lower_bound": true,
+    "monotone_nondecreasing": true,
+    "n_coverage_points": 20
+   },
+   "Github": {
+    "containment_1of20": 0.3742096158434741,
+    "containment_20of20": 0.9340004374453194,
+    "is_lower_bound": true,
+    "monotone_nondecreasing": true,
+    "n_coverage_points": 20
+   },
+   "OOD_CommonPile": {
+    "containment_1of20": 0.0,
+    "containment_20of20": 0.00125,
+    "is_lower_bound": true,
+    "monotone_nondecreasing": true,
+    "n_coverage_points": 20
+   },
+   "OOD_News_2024": {
+    "containment_1of20": 1.936533347104237e-05,
+    "containment_20of20": 0.0007487928942136383,
+    "is_lower_bound": true,
+    "monotone_nondecreasing": true,
+    "n_coverage_points": 20
+   },
+   "OOD_Wikipedia_2023": {
+    "containment_1of20": 0.022133038856352236,
+    "containment_20of20": 0.26957534299550123,
+    "is_lower_bound": true,
+    "monotone_nondecreasing": true,
+    "n_coverage_points": 20
+   },
+   "OOD_arXiv_2023": {
+    "containment_1of20": 0.0,
+    "containment_20of20": 8.598932872430531e-05,
+    "is_lower_bound": true,
+    "monotone_nondecreasing": true,
+    "n_coverage_points": 20
+   },
+   "Pile-CC": {
+    "containment_1of20": 0.09955354052744514,
+    "containment_20of20": 0.9276060773863086,
+    "is_lower_bound": true,
+    "monotone_nondecreasing": true,
+    "n_coverage_points": 20
+   },
+   "StackExchange": {
+    "containment_1of20": 0.07686719226900293,
+    "containment_20of20": 0.9179914195561899,
+    "is_lower_bound": true,
+    "monotone_nondecreasing": true,
+    "n_coverage_points": 20
+   },
+   "TRAP_FineWeb": {
+    "containment_1of20": 0.007237811980212063,
+    "containment_20of20": 0.06309446548070848,
+    "is_lower_bound": true,
+    "monotone_nondecreasing": true,
+    "n_coverage_points": 20
+   },
+   "USPTO_Backgrounds": {
+    "containment_1of20": 0.12033439046490556,
+    "containment_20of20": 0.9267616923398619,
+    "is_lower_bound": true,
+    "monotone_nondecreasing": true,
+    "n_coverage_points": 20
+   },
+   "Wikipedia_en": {
+    "containment_1of20": 0.07066448338254318,
+    "containment_20of20": 0.9215951727072293,
+    "is_lower_bound": true,
+    "monotone_nondecreasing": true,
+    "n_coverage_points": 20
+   },
+   "_CONTROL_random": {
+    "containment_1of20": 0.0,
+    "containment_20of20": 0.0,
+    "is_lower_bound": true,
+    "monotone_nondecreasing": true,
+    "n_coverage_points": 20
+   }
+  },
+  "strict_increase_for_in_stream_corpora": {
+   "Pile-CC": true,
+   "Github": true,
+   "StackExchange": true,
+   "USPTO_Backgrounds": true,
+   "Wikipedia_en": true
+  },
+  "fires": true,
+  "note": "'strict' is checked as strict only for corpora actually present in the stream; a genuinely absent corpus pinned at 0.0 at every coverage satisfies the lower-bound requirement with equality, which is the intended behaviour and is what E48b's growth-rate reading rests on.",
+  "source": "results/e48b_exposure_growth.json -> containment_by_coverage"
+ },
+ "P0_C2": {
+  "requirement_verbatim": "C2 - worker-count invariance. --workers 1, 2 and 8 must give identical output. Required: max_abs_diff = 0.0 between every pair, on the same 128 values.",
+  "arms_found": {
+   "w1": {
+    "file": "e52_factorial_410m_rstrip_w1.json",
+    "workers": 1,
+    "D": 0.000845372218579525
+   },
+   "w2": {
+    "file": "e52_factorial_410m_rstrip_w2.json",
+    "workers": 2,
+    "D": 0.000845372218579525
+   },
+   "w8": {
+    "file": "e52_factorial_410m_rstrip_w8.json",
+    "workers": 8,
+    "D": 0.000845372218579525
+   },
+   "w6": {
+    "file": "e52_factorial_410m_rstrip.json",
+    "workers": 6,
+    "D": 0.000845372218579525
+   }
+  },
+  "n_values_compared": 128,
+  "same_key_set_across_arms": true,
+  "pairwise_max_abs_diff": {
+   "w1_vs_w2": 0.0,
+   "w1_vs_w6": 0.0,
+   "w1_vs_w8": 0.0,
+   "w2_vs_w6": 0.0,
+   "w2_vs_w8": 0.0,
+   "w6_vs_w8": 0.0
+  },
+  "pairwise_max_abs_diff_registered_arms_only": {
+   "w1_vs_w2": 0.0,
+   "w1_vs_w8": 0.0,
+   "w2_vs_w8": 0.0
+  },
+  "max_abs_diff_registered": 0.0,
+  "max_abs_diff_including_w6_reference": 0.0,
+  "fires": true,
+  "note": "The 128 values are the 64 fit x read cells under each of the two aggregations. w6 is the reference run (--workers 6) and is reported alongside but is not one of the registered 1/2/8 arms.",
+  "source": "results/e52_factorial_410m_rstrip{,_w1,_w2,_w8}.json -> by_aggregation.{persist,min}.matrix"
+ },
+ "controls_fired": {
+  "E35_M1_C3": true,
+  "E35_M1_C4": true,
+  "P0_C2": true
+ },
+ "VERDICT": "ALL THREE RECORDED. E35 M1-C3 FIRES: all 4 post-2020 panels sit below the lowest in-stream Pile component (0.9180) at k=32, 20/20 \u2014 CONTROL_PubMed_2023 0.79273, OOD_Wikipedia_2023 0.26958, OOD_arXiv_2023 0.00009, OOD_News_2024 0.00075. E35 M1-C4 FIRES: containment at 1/20 coverage is a lower bound on 20/20 for 12/12 corpora and the series is monotone non-decreasing in all of them. P0 C2 FIRES: max_abs_diff = 0.0 across every pair of --workers 1/2/8 on all 128 values. No threshold was introduced here; each requirement is quoted from its own registration.",
+ "provenance": {
+  "schema_version": 1,
+  "experiment": "CTRLBF1 \u2014 backfill of three already-completed, previously unrecorded controls",
+  "utc": "2026-09-01T20:35:05+00:00",
+  "script": {
+   "path": "experiments/ctrlbf1_control_backfill.py",
+   "sha256": "6b944b4756c5d93fc012771d2bf57553b52192cc5832ee7bff360489515fdfb4",
+   "n_lines": 156
+  },
+  "git": {
+   "commit": "9987d82887ed339bf69b5e7fa1e4fc98d5bbb4ab",
+   "commit_short": "9987d82887ed",
+   "git_repository_present": true,
+   "branch": "pythia",
+   "dirty": true,
+   "dirty_files": [
+    "CONTEXT.md",
+    "NEELNANDA_REP.md",
+    "corpora/dataset_revisions.json",
+    "corpora/dataset_revisions.json.bak",
+    "corpora/manifest_ood.json",
+    "corpora/manifest_ood.json.bak",
+    "docs/experiments/COVERAGE.md",
+    "docs/experiments/METRICS.md",
+    "docs/experiments/descriptions/D1.md",
+    "docs/experiments/descriptions/E33.md",
+    "docs/experiments/descriptions/E34A.md",
+    "docs/experiments/descriptions/E34B.md",
+    "docs/experiments/descriptions/E36.md",
+    "docs/experiments/descriptions/E37.md",
+    "docs/experiments/descriptions/E44.md",
+    "docs/experiments/descriptions/E58.md",
+    "docs/experiments/preregs/E65R_phase1_rstrip_rescore.md",
+    "docs/experiments/preregs/E65T_threshold_defect_adjudication.md",
+    "docs/experiments/preregs/E66d_s_quantization_limited.md",
+    "docs/experiments/preregs/EFFRANK1_effective_rank_adjudication.md",
+    "docs/experiments/preregs/superseded/PREREG_PYTHIA_T7_v2.md",
+    "docs/reproducibility/ARTIFACTS.md",
+    "docs/reproducibility/PAYLOAD_REDACTIONS.md",
+    "docs/reproducibility/SCRIPT_PROVENANCE.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/MANIFEST.sha256",
+    "docs/reproducibility/review-snapshot-2026-08-19/PROVENANCE.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/REDACTIONS.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/EXTERNAL_REVIEW.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/AGENT_REPORT.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/CONTEXT_UNDER_REVIEW.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/FINDINGS.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/JUSTIFIABILITY.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/LEDGER.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/PHASE_A_ACCOUNT.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/SUSPICIONS.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/data/rstrip_rescore/rs_Github_s0.json",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/data/rstrip_rescore/rs_Github_s1.json",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/data/rstrip_rescore/rs_Github_s2.json",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/data/rstrip_rescore/rs_Pile-CC_s0.json",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/data/rstrip_rescore/rs_Pile-CC_s1.json",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/data/rstrip_rescore/rs_Pile-CC_s2.json",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/data/rstrip_rescore/rs_StackExchange_s0.json",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/data/rstrip_rescore/rs_StackExchange_s1.json",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/data/rstrip_rescore/rs_StackExchange_s2.json",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/data/rstrip_rescore/rs_USPTO_Backgrounds_s0.json",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/data/rstrip_rescore/rs_USPTO_Backgrounds_s1.json",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/data/rstrip_rescore/rs_USPTO_Backgrounds_s2.json",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/data/rstrip_rescore/rs_Wikipedia_en_s0.json",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/data/rstrip_rescore/rs_Wikipedia_en_s1.json",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/data/rstrip_rescore/rs_Wikipedia_en_s2.json",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/scripts/balance_test.py",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/scripts/check_payload.py",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/scripts/coverage2.py",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/scripts/derange.py",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/scripts/derangement_result.json",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/scripts/logit_base.py",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/scripts/logit_baseline.json",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/scripts/my_anova.py",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/scripts/my_e52.py",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/scripts/my_rescore.py",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/scripts/prereg2.py",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/scripts/prereg_audit.py",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/scripts/rescore_all.py",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/scripts/rstrip_anova.py"
+   ],
+   "dirty_diff_sha256": "719949b81d9871a26c26430713cafa9ad3b949cc5eea0ad080ae38307d75f63d",
+   "describe": "9987d82-dirty"
+  },
+  "argv": [
+   "experiments/ctrlbf1_control_backfill.py"
+  ],
+  "cwd": ".",
+  "inputs": [],
+  "env": {
+   "python": "3.14.2",
+   "torch": "2.13.0",
+   "transformers": "5.14.1",
+   "numpy": "2.5.1",
+   "platform": "macOS-26.5.2-arm64-arm-64bit-Mach-O",
+   "machine": "arm64",
+   "tf32": {
+    "allow_tf32_matmul": false,
+    "allow_tf32_cudnn": true,
+    "NVIDIA_TF32_OVERRIDE": null
+   },
+   "threads": {
+    "torch_num_threads": 10,
+    "torch_num_interop_threads": 14,
+    "OMP_NUM_THREADS": null,
+    "os_cpu_count": 14
+   }
+  },
+  "payload_sha256": "244eafca442da2a20faa694df90c22fe3bdd41e299594c22382a37b5825e4962"
+ }
+}

results/cv1_answer_competence.json

@@ -0,0 +1,288 @@
+{
+ "experiment": "CV1 \u2014 answer competence: can the model do the task at all?",
+ "status": "MEASUREMENT ONLY \u2014 no registered decision rule",
+ "why": "existing capability_* fields measure the INTERMEDIATE's rank, not the ANSWER's; t36_qladder.py:483-489 records the registered competence gate as degenerate and unable to fire. This supplies the missing number.",
+ "readout_convention": "STRIPPED (corrected) \u2014 rank of the target's first token at the final position of the rstripped prompt",
+ "sets_with_target": [
+  "multihop",
+  "order-ops",
+  "multilingual"
+ ],
+ "sets_without_target": [
+  "poetry",
+  "typo",
+  "association"
+ ],
+ "note_on_undefined": "poetry/typo/association carry target: null. The annotated intermediate is the object of interest and there is no separate answer, so answer competence is UNDEFINED for them \u2014 not zero.",
+ "by_model": {
+  "70m": {
+   "multihop": {
+    "n_scored": 93,
+    "n_multitoken_answers_scored_on_first_token": 3,
+    "top1_rate": 0.010752688172043012,
+    "top10_rate": 0.13978494623655913,
+    "median_rank": 80,
+    "mean_rank": 1106.5806451612902,
+    "median_prob": 0.0015416988171637058
+   },
+   "order-ops": {
+    "n_scored": 55,
+    "n_multitoken_answers_scored_on_first_token": 0,
+    "top1_rate": 0.03636363636363636,
+    "top10_rate": 0.38181818181818183,
+    "median_rank": 18,
+    "mean_rank": 22.09090909090909,
+    "median_prob": 0.006405120715498924
+   },
+   "multilingual": {
+    "n_scored": 107,
+    "n_multitoken_answers_scored_on_first_token": 92,
+    "top1_rate": 0.009345794392523364,
+    "top10_rate": 0.08411214953271028,
+    "median_rank": 3436,
+    "mean_rank": 4615.915887850468,
+    "median_prob": 2.3302003683056682e-05
+   },
+   "_pooled": {
+    "n": 255,
+    "top1_rate": 0.01568627450980392,
+    "top10_rate": 0.16862745098039217
+   }
+  },
+  "160m": {
+   "multihop": {
+    "n_scored": 93,
+    "n_multitoken_answers_scored_on_first_token": 3,
+    "top1_rate": 0.03225806451612903,
+    "top10_rate": 0.25806451612903225,
+    "median_rank": 42,
+    "mean_rank": 628.3655913978495,
+    "median_prob": 0.0031900564208626747
+   },
+   "order-ops": {
+    "n_scored": 55,
+    "n_multitoken_answers_scored_on_first_token": 0,
+    "top1_rate": 0.0,
+    "top10_rate": 0.43636363636363634,
+    "median_rank": 13,
+    "mean_rank": 14.49090909090909,
+    "median_prob": 0.01120622269809246
+   },
+   "multilingual": {
+    "n_scored": 107,
+    "n_multitoken_answers_scored_on_first_token": 92,
+    "top1_rate": 0.009345794392523364,
+    "top10_rate": 0.08411214953271028,
+    "median_rank": 3788,
+    "mean_rank": 4630.429906542056,
+    "median_prob": 1.6619400412309915e-05
+   },
+   "_pooled": {
+    "n": 255,
+    "top1_rate": 0.01568627450980392,
+    "top10_rate": 0.2235294117647059
+   }
+  },
+  "410m": {
+   "multihop": {
+    "n_scored": 93,
+    "n_multitoken_answers_scored_on_first_token": 3,
+    "top1_rate": 0.06451612903225806,
+    "top10_rate": 0.3333333333333333,
+    "median_rank": 31,
+    "mean_rank": 460.9139784946237,
+    "median_prob": 0.005166209302842617
+   },
+   "order-ops": {
+    "n_scored": 55,
+    "n_multitoken_answers_scored_on_first_token": 0,
+    "top1_rate": 0.09090909090909091,
+    "top10_rate": 0.5454545454545454,
+    "median_rank": 9,
+    "mean_rank": 11.909090909090908,
+    "median_prob": 0.026608118787407875
+   },
+   "multilingual": {
+    "n_scored": 107,
+    "n_multitoken_answers_scored_on_first_token": 92,
+    "top1_rate": 0.009345794392523364,
+    "top10_rate": 0.056074766355140186,
+    "median_rank": 3044,
+    "mean_rank": 4031.233644859813,
+    "median_prob": 1.0294868843629956e-05
+   },
+   "_pooled": {
+    "n": 255,
+    "top1_rate": 0.047058823529411764,
+    "top10_rate": 0.2627450980392157
+   }
+  },
+  "1b": {
+   "multihop": {
+    "n_scored": 93,
+    "n_multitoken_answers_scored_on_first_token": 3,
+    "top1_rate": 0.07526881720430108,
+    "top10_rate": 0.3333333333333333,
+    "median_rank": 18,
+    "mean_rank": 316.0967741935484,
+    "median_prob": 0.011097713373601437
+   },
+   "order-ops": {
+    "n_scored": 55,
+    "n_multitoken_answers_scored_on_first_token": 0,
+    "top1_rate": 0.07272727272727272,
+    "top10_rate": 0.6727272727272727,
+    "median_rank": 7,
+    "mean_rank": 9.163636363636364,
+    "median_prob": 0.029681021347641945
+   },
+   "multilingual": {
+    "n_scored": 107,
+    "n_multitoken_answers_scored_on_first_token": 92,
+    "top1_rate": 0.0,
+    "top10_rate": 0.06542056074766354,
+    "median_rank": 1649,
+    "mean_rank": 2892.6168224299067,
+    "median_prob": 3.0693336157128215e-05
+   },
+   "_pooled": {
+    "n": 255,
+    "top1_rate": 0.043137254901960784,
+    "top10_rate": 0.29411764705882354
+   }
+  }
+ },
+ "VERDICT": "Answer competence on the three sets carrying a target. 70m: top1 1.6%, top10 16.9% (n=255 items with a target) | 160m: top1 1.6%, top10 22.4% (n=255 items with a target) | 410m: top1 4.7%, top10 26.3% (n=255 items with a target) | 1b: top1 4.3%, top10 29.4% (n=255 items with a target)",
+ "provenance": {
+  "schema_version": 1,
+  "experiment": "CV1",
+  "utc": "2026-08-30T06:32:53+00:00",
+  "script": {
+   "path": "experiments/cv1_answer_competence.py",
+   "sha256": "29279dcb364dec36ae9f32438ab9ac607cc8b89983cf20c5796662e9a56a7a21",
+   "n_lines": 138
+  },
+  "git": {
+   "commit": "c9c49697e295c0fec4225552836fc9733b279f75",
+   "commit_short": "c9c49697e295",
+   "git_repository_present": true,
+   "branch": "pythia",
+   "dirty": true,
+   "dirty_files": [
+    "CLAUDE.md",
+    "README.md",
+    "corpora/manifest_ood.json",
+    "docs/HANDOFF.md",
+    "docs/REPO_REVIEW_PROMPT.md",
+    "docs/context/AGGREGATION_POLICY.md",
+    "docs/context/CONTEXT.md",
+    "docs/context/IMPLEMENTATION_RULES.md",
+    "docs/context/STATE.md",
+    "docs/experiments/INDEX.md",
+    "docs/experiments/METRICS.md",
+    "docs/experiments/PROPOSED.md",
+    "docs/experiments/PROVENANCE.md",
+    "docs/experiments/descriptions/E64.md",
+    "docs/experiments/descriptions/EN1.md",
+    "docs/experiments/preregs/CV3_margins.md",
+    "docs/experiments/preregs/CV4_capability_ladder.md",
+    "docs/experiments/preregs/CV5_metric_sensitivity.md",
+    "docs/experiments/preregs/CV6_per_family_ladder.md",
+    "docs/experiments/preregs/CV7_1b_rung.md",
+    "docs/experiments/preregs/DN1_design_nulls.md",
+    "docs/experiments/preregs/E33_logit_baseline.md",
+    "docs/experiments/preregs/E36_qladder.md",
+    "docs/experiments/preregs/E48_crossover.md",
+    "docs/experiments/preregs/E48b_containment.md",
+    "docs/experiments/preregs/E48c_exposure_vs_read.md",
+    "docs/experiments/preregs/E51_interaction.md",
+    "docs/experiments/preregs/E52_factorial.md",
+    "docs/experiments/preregs/E53_ladder_summary.md",
+    "docs/experiments/preregs/E55_matrix_robustness.md",
+    "docs/experiments/preregs/E56_predictor_registry.md",
+    "docs/experiments/preregs/E57_grid_variance_ci.md",
+    "docs/experiments/preregs/E58_algebra_audit.md",
+    "docs/experiments/preregs/E59_read_dose_response.md",
+    "docs/experiments/preregs/E60_fitter_determinism.md",
+    "docs/experiments/preregs/E61_randomized_network_null.md",
+    "docs/experiments/preregs/E62_ladder1b_corrected_band.md",
+    "docs/experiments/preregs/E65_training_axis_floor.md",
+    "docs/experiments/preregs/E66b_determinism_floor.md",
+    "docs/experiments/preregs/OS1_operator_space.md",
+    "docs/experiments/preregs/P0_t52_parallel_port.md",
+    "docs/experiments/preregs/R1_grid_rstrip.md",
+    "docs/experiments/preregs/R2_e48_rstrip_arm.md",
+    "docs/experiments/preregs/R3_ladder410_cpu.md",
+    "docs/experiments/preregs/R4_corrections.md",
+    "docs/experiments/preregs/R4b_e36_rstrip.md",
+    "docs/experiments/preregs/R5_corpus_axis_uncertainty.md",
+    "docs/experiments/preregs/R6_within_source_resampling.md",
+    "docs/experiments/preregs/R8_ladder_rstrip_S2.md",
+    "docs/experiments/preregs/R9_permutation_calibrated_min.md",
+    "docs/experiments/preregs/superseded/PREREG_E48_CROSSOVER.md",
+    "docs/reproducibility/ARTIFACTS.md",
+    "docs/reproducibility/PAYLOAD_REDACTIONS.md",
+    "docs/reproducibility/RELEASE.md",
+    "docs/reproducibility/REVIEWER_REPRO.md",
+    "docs/reproducibility/T1_FINDING_E33_BAND.md",
+    "docs/reproducibility/T1_SWEEP_PREDICTIONS.md",
+    "docs/reproducibility/T1_SWEEP_SPEC.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/MANIFEST.sha256",
+    "docs/reproducibility/review-snapshot-2026-08-19/PROVENANCE.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/REDACTIONS.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/EXTERNAL_REVIEW.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/FINDINGS.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/PHASE_A_ACCOUNT.md"
+   ],
+   "dirty_diff_sha256": "ec61dde1905616f407f51d5f6fe3ea37c9c6e3e1ade6998cd0644527d080c600",
+   "describe": "c9c4969-dirty"
+  },
+  "argv": [
+   "experiments/cv1_answer_competence.py",
+   "--models",
+   "70m,160m,410m,1b"
+  ],
+  "cwd": ".",
+  "inputs": [
+   {
+    "path": "jacobian-lens/data/evaluations/lens-eval-multihop.json",
+    "sha256": "50b7e4c9255291c0ca2a8e94615be9f44531fa57bb1a844e4f9616056d987416",
+    "bytes": 21869,
+    "exists": true
+   },
+   {
+    "path": "jacobian-lens/data/evaluations/lens-eval-order-ops.json",
+    "sha256": "b203206d16ff628152cc86f3838604e06cb54776f3e14fa1c34f150db8bc7560",
+    "bytes": 9589,
+    "exists": true
+   },
+   {
+    "path": "jacobian-lens/data/evaluations/lens-eval-multilingual.json",
+    "sha256": "fa70b9bd89416a6d8d985a80dc628b109ae6fd3b25b9275c0fc5065d7ff4a0ef",
+    "bytes": 24284,
+    "exists": true
+   }
+  ],
+  "env": {
+   "python": "3.14.2",
+   "torch": "2.13.0",
+   "transformers": "5.14.1",
+   "numpy": "2.5.1",
+   "platform": "macOS-26.5.2-arm64-arm-64bit-Mach-O",
+   "machine": "arm64",
+   "hostname": "<redacted for double-blind review>",
+   "tf32": {
+    "allow_tf32_matmul": false,
+    "allow_tf32_cudnn": true,
+    "NVIDIA_TF32_OVERRIDE": null
+   },
+   "threads": {
+    "torch_num_threads": 10,
+    "torch_num_interop_threads": 14,
+    "OMP_NUM_THREADS": null,
+    "os_cpu_count": 14
+   }
+  },
+  "payload_sha256": "8809019f865fe0939cee28bd5d201f7a842b85b9cb5929440541804f3786e74d"
+ }
+}

results/cv2_position_support.json

@@ -0,0 +1,299 @@
+{
+ "experiment": "CV2 \u2014 positional support audit",
+ "status": "MEASUREMENT ONLY \u2014 no registered decision rule; this establishes or refutes the mismatch flagged in docs/validity/CONSTRUCT_VALIDITY.md section 4.2",
+ "model": "EleutherAI/pythia-410m-deduped",
+ "fitting_support": {
+  "source": "src/fastfit.py:116 -> jlens.fitting.valid_position_mask(seq_len, skip_first)",
+  "skip_first": 16,
+  "max_seq_len": 128,
+  "excludes": "first skip_first positions (attention sink) and the final position (no next-token target)",
+  "inclusive_range": [
+   16,
+   126
+  ]
+ },
+ "readout_convention": "STRIPPED (corrected). Position computed on the item alone, then shifted by the prefix length \u2014 matches t36_qladder.py:27-31",
+ "by_set": {
+  "association": {
+   "n_items": 102,
+   "readout_rule": "last",
+   "median_prompt_tokens": 28,
+   "Q0": {
+    "median_read_pos": 28,
+    "min": 19,
+    "max": 49,
+    "n_in_support": 102,
+    "frac_in_support": 1.0
+   },
+   "prefixed": {
+    "median_read_pos": 156,
+    "min": 147,
+    "max": 177,
+    "n_in_support": 0,
+    "frac_in_support": 0.0
+   }
+  },
+  "multihop": {
+   "n_items": 93,
+   "readout_rule": "last",
+   "median_prompt_tokens": 15,
+   "Q0": {
+    "median_read_pos": 15,
+    "min": 8,
+    "max": 40,
+    "n_in_support": 45,
+    "frac_in_support": 0.4838709677419355
+   },
+   "prefixed": {
+    "median_read_pos": 143,
+    "min": 136,
+    "max": 168,
+    "n_in_support": 0,
+    "frac_in_support": 0.0
+   }
+  },
+  "multilingual": {
+   "n_items": 107,
+   "readout_rule": "last",
+   "median_prompt_tokens": 10,
+   "Q0": {
+    "median_read_pos": 10,
+    "min": 5,
+    "max": 26,
+    "n_in_support": 6,
+    "frac_in_support": 0.056074766355140186
+   },
+   "prefixed": {
+    "median_read_pos": 138,
+    "min": 133,
+    "max": 154,
+    "n_in_support": 0,
+    "frac_in_support": 0.0
+   }
+  },
+  "order-ops": {
+   "n_items": 55,
+   "readout_rule": "last",
+   "median_prompt_tokens": 8,
+   "Q0": {
+    "median_read_pos": 8,
+    "min": 6,
+    "max": 18,
+    "n_in_support": 1,
+    "frac_in_support": 0.01818181818181818
+   },
+   "prefixed": {
+    "median_read_pos": 136,
+    "min": 134,
+    "max": 146,
+    "n_in_support": 0,
+    "frac_in_support": 0.0
+   }
+  },
+  "poetry": {
+   "n_items": 98,
+   "readout_rule": "last_newline",
+   "median_prompt_tokens": 26,
+   "Q0": {
+    "median_read_pos": 18,
+    "min": 16,
+    "max": 22,
+    "n_in_support": 98,
+    "frac_in_support": 1.0
+   },
+   "prefixed": {
+    "median_read_pos": 146,
+    "min": 144,
+    "max": 150,
+    "n_in_support": 0,
+    "frac_in_support": 0.0
+   }
+  },
+  "typo": {
+   "n_items": 96,
+   "readout_rule": "last",
+   "median_prompt_tokens": 8,
+   "Q0": {
+    "median_read_pos": 8,
+    "min": 6,
+    "max": 14,
+    "n_in_support": 0,
+    "frac_in_support": 0.0
+   },
+   "prefixed": {
+    "median_read_pos": 136,
+    "min": 134,
+    "max": 142,
+    "n_in_support": 0,
+    "frac_in_support": 0.0
+   }
+  }
+ },
+ "overall": {
+  "Q0": {
+   "n": 551,
+   "median_read_pos": 14,
+   "min": 5,
+   "max": 49,
+   "n_in_support": 252,
+   "frac_in_support": 0.4573502722323049
+  },
+  "prefixed": {
+   "n": 551,
+   "median_read_pos": 142,
+   "min": 133,
+   "max": 177,
+   "n_in_support": 0,
+   "frac_in_support": 0.0
+  }
+ },
+ "VERDICT": "Q0 reads inside the fitting support on 45.7% of items (median position 14); prefixed rungs on 0.0% (median 142). Fitting support is positions 16..126.",
+ "provenance": {
+  "schema_version": 1,
+  "experiment": "CV2",
+  "utc": "2026-08-30T00:19:04+00:00",
+  "script": {
+   "path": "experiments/cv2_position_support.py",
+   "sha256": "fee6460d8d930d3c9ebc138a29def2533ca5bd8fc2392daceed2b0644e20abbc",
+   "n_lines": 145
+  },
+  "git": {
+   "commit": "c9c49697e295c0fec4225552836fc9733b279f75",
+   "commit_short": "c9c49697e295",
+   "git_repository_present": true,
+   "branch": "pythia",
+   "dirty": true,
+   "dirty_files": [
+    "CLAUDE.md",
+    "README.md",
+    "corpora/manifest_ood.json",
+    "docs/HANDOFF.md",
+    "docs/REPO_REVIEW_PROMPT.md",
+    "docs/context/AGGREGATION_POLICY.md",
+    "docs/context/CONTEXT.md",
+    "docs/context/IMPLEMENTATION_RULES.md",
+    "docs/context/STATE.md",
+    "docs/experiments/INDEX.md",
+    "docs/experiments/METRICS.md",
+    "docs/experiments/PROPOSED.md",
+    "docs/experiments/PROVENANCE.md",
+    "docs/experiments/descriptions/E64.md",
+    "docs/experiments/descriptions/EN1.md",
+    "docs/experiments/preregs/CV3_margins.md",
+    "docs/experiments/preregs/CV4_capability_ladder.md",
+    "docs/experiments/preregs/CV5_metric_sensitivity.md",
+    "docs/experiments/preregs/CV6_per_family_ladder.md",
+    "docs/experiments/preregs/CV7_1b_rung.md",
+    "docs/experiments/preregs/DN1_design_nulls.md",
+    "docs/experiments/preregs/E33_logit_baseline.md",
+    "docs/experiments/preregs/E36_qladder.md",
+    "docs/experiments/preregs/E48_crossover.md",
+    "docs/experiments/preregs/E48b_containment.md",
+    "docs/experiments/preregs/E48c_exposure_vs_read.md",
+    "docs/experiments/preregs/E51_interaction.md",
+    "docs/experiments/preregs/E52_factorial.md",
+    "docs/experiments/preregs/E53_ladder_summary.md",
+    "docs/experiments/preregs/E55_matrix_robustness.md",
+    "docs/experiments/preregs/E56_predictor_registry.md",
+    "docs/experiments/preregs/E57_grid_variance_ci.md",
+    "docs/experiments/preregs/E58_algebra_audit.md",
+    "docs/experiments/preregs/E59_read_dose_response.md",
+    "docs/experiments/preregs/E60_fitter_determinism.md",
+    "docs/experiments/preregs/E61_randomized_network_null.md",
+    "docs/experiments/preregs/E62_ladder1b_corrected_band.md",
+    "docs/experiments/preregs/E65_training_axis_floor.md",
+    "docs/experiments/preregs/E66b_determinism_floor.md",
+    "docs/experiments/preregs/OS1_operator_space.md",
+    "docs/experiments/preregs/P0_t52_parallel_port.md",
+    "docs/experiments/preregs/R1_grid_rstrip.md",
+    "docs/experiments/preregs/R2_e48_rstrip_arm.md",
+    "docs/experiments/preregs/R3_ladder410_cpu.md",
+    "docs/experiments/preregs/R4_corrections.md",
+    "docs/experiments/preregs/R4b_e36_rstrip.md",
+    "docs/experiments/preregs/R5_corpus_axis_uncertainty.md",
+    "docs/experiments/preregs/R6_within_source_resampling.md",
+    "docs/experiments/preregs/R8_ladder_rstrip_S2.md",
+    "docs/experiments/preregs/R9_permutation_calibrated_min.md",
+    "docs/experiments/preregs/superseded/PREREG_E48_CROSSOVER.md",
+    "docs/reproducibility/ARTIFACTS.md",
+    "docs/reproducibility/PAYLOAD_REDACTIONS.md",
+    "docs/reproducibility/RELEASE.md",
+    "docs/reproducibility/REVIEWER_REPRO.md",
+    "docs/reproducibility/T1_SWEEP_SPEC.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/MANIFEST.sha256",
+    "docs/reproducibility/review-snapshot-2026-08-19/PROVENANCE.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/REDACTIONS.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/EXTERNAL_REVIEW.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/FINDINGS.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/PHASE_A_ACCOUNT.md",
+    "docs/reviews/REVIEW_INSIGHTS_2026-08-25.md",
+    "docs/validity/CONSTRUCT_VALIDITY.md"
+   ],
+   "dirty_diff_sha256": "6b143c292229b3d2480d1b73243675f0db5e23b27f66e4215e0a9c1947a761a8",
+   "describe": "c9c4969-dirty"
+  },
+  "argv": [
+   "experiments/cv2_position_support.py"
+  ],
+  "cwd": ".",
+  "inputs": [
+   {
+    "path": "jacobian-lens/data/evaluations/lens-eval-association.json",
+    "sha256": "d1a98cd4911b594282e74168091c77d849dae18ffe2acb5761074853f327d71c",
+    "bytes": 24228,
+    "exists": true
+   },
+   {
+    "path": "jacobian-lens/data/evaluations/lens-eval-multihop.json",
+    "sha256": "50b7e4c9255291c0ca2a8e94615be9f44531fa57bb1a844e4f9616056d987416",
+    "bytes": 21869,
+    "exists": true
+   },
+   {
+    "path": "jacobian-lens/data/evaluations/lens-eval-multilingual.json",
+    "sha256": "fa70b9bd89416a6d8d985a80dc628b109ae6fd3b25b9275c0fc5065d7ff4a0ef",
+    "bytes": 24284,
+    "exists": true
+   },
+   {
+    "path": "jacobian-lens/data/evaluations/lens-eval-order-ops.json",
+    "sha256": "b203206d16ff628152cc86f3838604e06cb54776f3e14fa1c34f150db8bc7560",
+    "bytes": 9589,
+    "exists": true
+   },
+   {
+    "path": "jacobian-lens/data/evaluations/lens-eval-poetry.json",
+    "sha256": "6aeb3415c5a5c3f3827c9efe63f006de02f5ef39a816bbac68e15e733aba60cc",
+    "bytes": 21533,
+    "exists": true
+   },
+   {
+    "path": "jacobian-lens/data/evaluations/lens-eval-typo.json",
+    "sha256": "9d05e16b7234a57d0773d120a4e1c4e94fd3bc2235a8125d4200a70e60ab17aa",
+    "bytes": 14646,
+    "exists": true
+   }
+  ],
+  "env": {
+   "python": "3.14.2",
+   "torch": "2.13.0",
+   "transformers": "5.14.1",
+   "numpy": "2.5.1",
+   "platform": "macOS-26.5.2-arm64-arm-64bit-Mach-O",
+   "machine": "arm64",
+   "hostname": "<redacted for double-blind review>",
+   "tf32": {
+    "allow_tf32_matmul": false,
+    "allow_tf32_cudnn": true,
+    "NVIDIA_TF32_OVERRIDE": null
+   },
+   "threads": {
+    "torch_num_threads": 10,
+    "torch_num_interop_threads": 14,
+    "OMP_NUM_THREADS": null,
+    "os_cpu_count": 14
+   }
+  },
+  "payload_sha256": "1eb1ab9b76e77d6060876485b34291cf9e1c525a755ae0166d5f7ec98f6d0d92"
+ }
+}

results/cv3_margins_410m.json

@@ -0,0 +1,473 @@
+{
+ "experiment": "CV3 \u2014 margins: score effect or ranking artifact?",
+ "prereg": "docs/experiments/preregs/CV3_margins.md",
+ "status": "PRE-REGISTERED",
+ "decision_rule_verbatim": "ACCEPT (SCORE EFFECT) if PRIMARY >= 0.50 | REJECT (RANKING ARTIFACT) if PRIMARY < 0.25 \u2014 stop and alert, do not re-cut | UNCLEAR if 0.25 <= PRIMARY < 0.50 \u2014 report and stop",
+ "model": "EleutherAI/pythia-410m-deduped",
+ "band": [
+  9,
+  10,
+  11,
+  12,
+  13,
+  14,
+  15,
+  16,
+  17,
+  18,
+  19,
+  20,
+  21
+ ],
+ "admitted_sets": [
+  "multihop",
+  "multilingual",
+  "order-ops",
+  "poetry",
+  "typo"
+ ],
+ "K": [
+  1,
+  2,
+  5,
+  10,
+  20,
+  50,
+  100
+ ],
+ "readout_convention": "STRIPPED (corrected); no read-context prefix",
+ "n_items": 449,
+ "n_pairs": 801,
+ "per_arm": {
+  "logit_I": {
+   "rank_min_auc": 0.1981085192666858,
+   "margin_mean": -4.698809347247424,
+   "z_mean": 3.398138034577217
+  },
+  "J|Wikipedia_en|s0": {
+   "rank_min_auc": 0.2411940539381176,
+   "margin_mean": -3.5185320037810324,
+   "z_mean": 3.414767909333594
+  },
+  "J|Wikipedia_en|s1": {
+   "rank_min_auc": 0.24376219820086684,
+   "margin_mean": -3.5576964909083792,
+   "z_mean": 3.4289134198996036
+  },
+  "J|Wikipedia_en|s2": {
+   "rank_min_auc": 0.2409968710017892,
+   "margin_mean": -3.550353089992781,
+   "z_mean": 3.418068973837049
+  },
+  "J|USPTO_Backgrounds|s0": {
+   "rank_min_auc": 0.2588173671008463,
+   "margin_mean": -3.5918301268675705,
+   "z_mean": 3.5110595907230495
+  },
+  "J|USPTO_Backgrounds|s1": {
+   "rank_min_auc": 0.260959847421275,
+   "margin_mean": -3.6230063218555544,
+   "z_mean": 3.5260322405353177
+  },
+  "J|USPTO_Backgrounds|s2": {
+   "rank_min_auc": 0.26360051696399417,
+   "margin_mean": -3.573313506298794,
+   "z_mean": 3.5303729015355394
+  },
+  "J|Pile-CC|s0": {
+   "rank_min_auc": 0.2619082412030062,
+   "margin_mean": -3.723244701992529,
+   "z_mean": 3.564116526229665
+  },
+  "J|Pile-CC|s1": {
+   "rank_min_auc": 0.2652928121057275,
+   "margin_mean": -3.721754557559919,
+   "z_mean": 3.57523909996598
+  },
+  "J|Pile-CC|s2": {
+   "rank_min_auc": 0.26001365602369086,
+   "margin_mean": -3.710869514133819,
+   "z_mean": 3.5523003176418593
+  },
+  "J|StackExchange|s0": {
+   "rank_min_auc": 0.26678756139973614,
+   "margin_mean": -3.8218988278155637,
+   "z_mean": 3.5992482651800937
+  },
+  "J|StackExchange|s1": {
+   "rank_min_auc": 0.263895495234026,
+   "margin_mean": -3.8108850501164118,
+   "z_mean": 3.5846069769433595
+  },
+  "J|StackExchange|s2": {
+   "rank_min_auc": 0.26310253043676124,
+   "margin_mean": -3.8273039062383045,
+   "z_mean": 3.591593915826658
+  },
+  "J|Github|s0": {
+   "rank_min_auc": 0.2361789172766393,
+   "margin_mean": -3.407463207637055,
+   "z_mean": 3.2020849497495396
+  },
+  "J|Github|s1": {
+   "rank_min_auc": 0.24343462540305366,
+   "margin_mean": -3.4403224028664545,
+   "z_mean": 3.2485827239444456
+  },
+  "J|Github|s2": {
+   "rank_min_auc": 0.2416357198287925,
+   "margin_mean": -3.410493464114918,
+   "z_mean": 3.232178426457889
+  },
+  "shuf|Pile-CC|s0|d0": {
+   "rank_min_auc": 0.27188047356388173,
+   "margin_mean": -3.078709140214322,
+   "z_mean": 3.5705154663621523
+  },
+  "shuf|Pile-CC|s0|d1": {
+   "rank_min_auc": 0.25473107820874097,
+   "margin_mean": -3.5329598556097404,
+   "z_mean": 3.4034280227130598
+  },
+  "shuf|Pile-CC|s0|d2": {
+   "rank_min_auc": 0.27573013496239485,
+   "margin_mean": -3.3755934340423375,
+   "z_mean": 3.461937350128892
+  }
+ },
+ "rank_space": {
+  "spread": 0.024178774854012652,
+  "pooled_seed_sd": 0.0025811352265854045,
+  "spread_over_sd": 9.36749636554256,
+  "per_corpus": {
+   "Wikipedia_en": {
+    "mean": 0.24198437438025788,
+    "seed_sd": 0.001542794033705807
+   },
+   "USPTO_Backgrounds": {
+    "mean": 0.2611259104953718,
+    "seed_sd": 0.002395895106586836
+   },
+   "Pile-CC": {
+    "mean": 0.2624049031108082,
+    "seed_sd": 0.0026743928325177486
+   },
+   "StackExchange": {
+    "mean": 0.2645951956901745,
+    "seed_sd": 0.0019396001767817256
+   },
+   "Github": {
+    "mean": 0.24041642083616183,
+    "seed_sd": 0.0037784047730492635
+   }
+  }
+ },
+ "margin_space": {
+  "spread": 0.40060290318395086,
+  "pooled_seed_sd": 0.01737544449501888,
+  "spread_over_sd": 23.055692376606196,
+  "per_corpus": {
+   "Wikipedia_en": {
+    "mean": -3.542193861560731,
+    "seed_sd": 0.020818117576808237
+   },
+   "USPTO_Backgrounds": {
+    "mean": -3.596049985007306,
+    "seed_sd": 0.025113728945124182
+   },
+   "Pile-CC": {
+    "mean": -3.718622924562089,
+    "seed_sd": 0.006755861348445229
+   },
+   "StackExchange": {
+    "mean": -3.8200292613900935,
+    "seed_sd": 0.008367566435486981
+   },
+   "Github": {
+    "mean": -3.4194263582061426,
+    "seed_sd": 0.018159821734111948
+   }
+  }
+ },
+ "z_space": {
+  "spread": 0.36420101926607895,
+  "pooled_seed_sd": 0.013407308018930473,
+  "spread_over_sd": 27.164365788556857,
+  "per_corpus": {
+   "Wikipedia_en": {
+    "mean": 3.420583434356749,
+    "seed_sd": 0.00740038857575187
+   },
+   "USPTO_Backgrounds": {
+    "mean": 3.5224882442646357,
+    "seed_sd": 0.010132666273489743
+   },
+   "Pile-CC": {
+    "mean": 3.5638853146125014,
+    "seed_sd": 0.011471138903233818
+   },
+   "StackExchange": {
+    "mean": 3.5918163859833703,
+    "seed_sd": 0.007323178956966246
+   },
+   "Github": {
+    "mean": 3.2276153667172913,
+    "seed_sd": 0.0235823426935772
+   }
+  }
+ },
+ "PRIMARY": 2.461243802710653,
+ "VERDICT": "ACCEPT \u2014 SCORE EFFECT",
+ "SECONDARY_derangement": {
+  "real_minus_shuf_rank_min_auc": -0.0055389877086662875,
+  "real_minus_shuf_margin": -0.394157225370396,
+  "real_minus_shuf_z": 0.08548957982829686,
+  "note": "if the derangement's advantage is present in rank space and absent in margin space, `min` rewards rank reshuffling rather than better scoring"
+ },
+ "controls": {
+  "C1_logit_reproduces_stored": {
+   "required": "|logit_I rank_min_auc - 0.19810852520167826| <= 1e-6",
+   "observed": 0.1981085192666858,
+   "abs_diff": 5.934992469702749e-09,
+   "fires": true
+  },
+  "C2_margin_rank_consistency": {
+   "required": "margin > 0 iff rank == 1 on 100% of triples",
+   "violations": 0,
+   "fires": true
+  },
+  "C4_seed_sd_nondegenerate": {
+   "required": "pooled seed SD > 0 in both spaces",
+   "rank": 0.0025811352265854045,
+   "margin": 0.01737544449501888,
+   "fires": true
+  }
+ },
+ "declared_bias": "margin is measured against the top competitor, which is itself operator-dependent; z and the raw logit are recorded so a competitor-suppression effect is separable from a target-score effect",
+ "provenance": {
+  "schema_version": 1,
+  "experiment": "CV3",
+  "utc": "2026-08-30T06:24:45+00:00",
+  "script": {
+   "path": "experiments/cv3_margins.py",
+   "sha256": "44938026f765b4f605189d29781ddb59a76d2a1bf91a230fabedb941a85e9521",
+   "n_lines": 247
+  },
+  "git": {
+   "commit": "c9c49697e295c0fec4225552836fc9733b279f75",
+   "commit_short": "c9c49697e295",
+   "git_repository_present": true,
+   "branch": "pythia",
+   "dirty": true,
+   "dirty_files": [
+    "CLAUDE.md",
+    "README.md",
+    "corpora/manifest_ood.json",
+    "docs/HANDOFF.md",
+    "docs/REPO_REVIEW_PROMPT.md",
+    "docs/context/AGGREGATION_POLICY.md",
+    "docs/context/CONTEXT.md",
+    "docs/context/IMPLEMENTATION_RULES.md",
+    "docs/context/STATE.md",
+    "docs/experiments/INDEX.md",
+    "docs/experiments/METRICS.md",
+    "docs/experiments/PROPOSED.md",
+    "docs/experiments/PROVENANCE.md",
+    "docs/experiments/descriptions/E64.md",
+    "docs/experiments/descriptions/EN1.md",
+    "docs/experiments/preregs/CV3_margins.md",
+    "docs/experiments/preregs/CV4_capability_ladder.md",
+    "docs/experiments/preregs/CV5_metric_sensitivity.md",
+    "docs/experiments/preregs/CV6_per_family_ladder.md",
+    "docs/experiments/preregs/CV7_1b_rung.md",
+    "docs/experiments/preregs/DN1_design_nulls.md",
+    "docs/experiments/preregs/E33_logit_baseline.md",
+    "docs/experiments/preregs/E36_qladder.md",
+    "docs/experiments/preregs/E48_crossover.md",
+    "docs/experiments/preregs/E48b_containment.md",
+    "docs/experiments/preregs/E48c_exposure_vs_read.md",
+    "docs/experiments/preregs/E51_interaction.md",
+    "docs/experiments/preregs/E52_factorial.md",
+    "docs/experiments/preregs/E53_ladder_summary.md",
+    "docs/experiments/preregs/E55_matrix_robustness.md",
+    "docs/experiments/preregs/E56_predictor_registry.md",
+    "docs/experiments/preregs/E57_grid_variance_ci.md",
+    "docs/experiments/preregs/E58_algebra_audit.md",
+    "docs/experiments/preregs/E59_read_dose_response.md",
+    "docs/experiments/preregs/E60_fitter_determinism.md",
+    "docs/experiments/preregs/E61_randomized_network_null.md",
+    "docs/experiments/preregs/E62_ladder1b_corrected_band.md",
+    "docs/experiments/preregs/E65_training_axis_floor.md",
+    "docs/experiments/preregs/E66b_determinism_floor.md",
+    "docs/experiments/preregs/OS1_operator_space.md",
+    "docs/experiments/preregs/P0_t52_parallel_port.md",
+    "docs/experiments/preregs/R1_grid_rstrip.md",
+    "docs/experiments/preregs/R2_e48_rstrip_arm.md",
+    "docs/experiments/preregs/R3_ladder410_cpu.md",
+    "docs/experiments/preregs/R4_corrections.md",
+    "docs/experiments/preregs/R4b_e36_rstrip.md",
+    "docs/experiments/preregs/R5_corpus_axis_uncertainty.md",
+    "docs/experiments/preregs/R6_within_source_resampling.md",
+    "docs/experiments/preregs/R8_ladder_rstrip_S2.md",
+    "docs/experiments/preregs/R9_permutation_calibrated_min.md",
+    "docs/experiments/preregs/superseded/PREREG_E48_CROSSOVER.md",
+    "docs/reproducibility/ARTIFACTS.md",
+    "docs/reproducibility/PAYLOAD_REDACTIONS.md",
+    "docs/reproducibility/RELEASE.md",
+    "docs/reproducibility/REVIEWER_REPRO.md",
+    "docs/reproducibility/T1_SWEEP_PREDICTIONS.md",
+    "docs/reproducibility/T1_SWEEP_SPEC.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/MANIFEST.sha256",
+    "docs/reproducibility/review-snapshot-2026-08-19/PROVENANCE.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/REDACTIONS.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/EXTERNAL_REVIEW.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/FINDINGS.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/PHASE_A_ACCOUNT.md",
+    "docs/reproducibility/t1_module_audit.json"
+   ],
+   "dirty_diff_sha256": "ec61dde1905616f407f51d5f6fe3ea37c9c6e3e1ade6998cd0644527d080c600",
+   "describe": "c9c4969-dirty"
+  },
+  "argv": [
+   "experiments/cv3_margins.py"
+  ],
+  "cwd": ".",
+  "inputs": [
+   {
+    "path": "results/e48/lens_INSTREAM_Wikipedia_en_410m_n200_s0.pt",
+    "sha256": "50168cf38a9aab76190187b3669a4f26b0cf0ae69968e2decfee4e7ce0a26f8a",
+    "bytes": 27268444,
+    "exists": true
+   },
+   {
+    "path": "results/e48/lens_INSTREAM_Wikipedia_en_410m_n200_s1.pt",
+    "sha256": "a6614fc65dd71659e2d1cb5799c37bffb291be1310fb06b69fb23921fcf4c9cc",
+    "bytes": 27268444,
+    "exists": true
+   },
+   {
+    "path": "results/e48/lens_INSTREAM_Wikipedia_en_410m_n200_s2.pt",
+    "sha256": "c0702e058e3c41fb1e8b10020fa27e6ac96d34862f0648f9e55f13bd33776ed5",
+    "bytes": 27268444,
+    "exists": true
+   },
+   {
+    "path": "results/e48/lens_INSTREAM_USPTO_Backgrounds_410m_n200_s0.pt",
+    "sha256": "75162b82ab15f9144b3776aabf62d0a1c732a474650e48b04c65bc77614aa942",
+    "bytes": 27268539,
+    "exists": true
+   },
+   {
+    "path": "results/e48/lens_INSTREAM_USPTO_Backgrounds_410m_n200_s1.pt",
+    "sha256": "860d4c8da99be8789db610f1046fa97d8e5f6a0b7c0b2559a64825451dc33c5f",
+    "bytes": 27268539,
+    "exists": true
+   },
+   {
+    "path": "results/e48/lens_INSTREAM_USPTO_Backgrounds_410m_n200_s2.pt",
+    "sha256": "1a727ecb9d2a8dfcf76db138ef90c52cd2fe98bcef0be07ae759969b3d9c67e1",
+    "bytes": 27268539,
+    "exists": true
+   },
+   {
+    "path": "results/e48/lens_INSTREAM_Pile-CC_410m_n200_s0.pt",
+    "sha256": "4f00cc3d7450c3646b78f002e794572ca27d6ab175c70626d2d5fb7fccf28e20",
+    "bytes": 27268349,
+    "exists": true
+   },
+   {
+    "path": "results/e48/lens_INSTREAM_Pile-CC_410m_n200_s1.pt",
+    "sha256": "efc6a88531fdcd4423514fd0e4f65a02b55638653c351ad0af35cd236aad2fc2",
+    "bytes": 27268349,
+    "exists": true
+   },
+   {
+    "path": "results/e48/lens_INSTREAM_Pile-CC_410m_n200_s2.pt",
+    "sha256": "058810065ae8bc499931d6aa3783dc1ead43d08e322395f7f142b90dfefae028",
+    "bytes": 27268349,
+    "exists": true
+   },
+   {
+    "path": "results/e48/lens_INSTREAM_StackExchange_410m_n200_s0.pt",
+    "sha256": "6ee98aa7e9242d0c4d6ff26b1ed918180f5b37822e30181c6d0fc9fe826c0b79",
+    "bytes": 27268463,
+    "exists": true
+   },
+   {
+    "path": "results/e48/lens_INSTREAM_StackExchange_410m_n200_s1.pt",
+    "sha256": "9182d6655dc5808e47b92fe4611fc522226db741c4e6b900f8ba0e1eba7ca7b3",
+    "bytes": 27268463,
+    "exists": true
+   },
+   {
+    "path": "results/e48/lens_INSTREAM_StackExchange_410m_n200_s2.pt",
+    "sha256": "7e21bde15c9634ffff62ec1983ad595f0604915dc8a8f9169a165e33edaee8c0",
+    "bytes": 27268463,
+    "exists": true
+   },
+   {
+    "path": "results/e48/lens_INSTREAM_Github_410m_n200_s0.pt",
+    "sha256": "b4acd10402749791b713e16738ba1d165c04d7134df139418618fd05a4edcbdb",
+    "bytes": 27268330,
+    "exists": true
+   },
+   {
+    "path": "results/e48/lens_INSTREAM_Github_410m_n200_s1.pt",
+    "sha256": "0ec0403597cbdc971dc0348fae888a10aaa0602b8c9f9268ba008af83e07694b",
+    "bytes": 27268330,
+    "exists": true
+   },
+   {
+    "path": "results/e48/lens_INSTREAM_Github_410m_n200_s2.pt",
+    "sha256": "f5057978c3f1e94930b3c3892ef560c79cc2c68f90062d6b07aeefdec7828847",
+    "bytes": 27268330,
+    "exists": true
+   },
+   {
+    "path": "results/e48/lens_INSTREAM_Pile-CC_410m_n200_s0.pt",
+    "sha256": "4f00cc3d7450c3646b78f002e794572ca27d6ab175c70626d2d5fb7fccf28e20",
+    "bytes": 27268349,
+    "exists": true
+   },
+   {
+    "path": "results/e48/lens_INSTREAM_Pile-CC_410m_n200_s0.pt",
+    "sha256": "4f00cc3d7450c3646b78f002e794572ca27d6ab175c70626d2d5fb7fccf28e20",
+    "bytes": 27268349,
+    "exists": true
+   },
+   {
+    "path": "results/e48/lens_INSTREAM_Pile-CC_410m_n200_s0.pt",
+    "sha256": "4f00cc3d7450c3646b78f002e794572ca27d6ab175c70626d2d5fb7fccf28e20",
+    "bytes": 27268349,
+    "exists": true
+   },
+   {
+    "path": "results/e48_crossover_410m_rstrip.json",
+    "sha256": "fe0b5ae29baf7f946884618723610d19120158474538e540eb1be8583430d97d",
+    "bytes": 111333,
+    "exists": true
+   }
+  ],
+  "env": {
+   "python": "3.14.2",
+   "torch": "2.13.0",
+   "transformers": "5.14.1",
+   "numpy": "2.5.1",
+   "platform": "macOS-26.5.2-arm64-arm-64bit-Mach-O",
+   "machine": "arm64",
+   "hostname": "<redacted for double-blind review>",
+   "tf32": {
+    "allow_tf32_matmul": false,
+    "allow_tf32_cudnn": true,
+    "NVIDIA_TF32_OVERRIDE": null
+   },
+   "threads": {
+    "torch_num_threads": 8,
+    "torch_num_interop_threads": 14,
+    "OMP_NUM_THREADS": null,
+    "os_cpu_count": 14
+   }
+  },
+  "payload_sha256": "38efe87c84be1b8bee5fa680a5a3d354912fedf8c7e29e686fd97fbecb9a8b5c"
+ }
+}

results/cv4_phase1_capability.json

@@ -0,0 +1,719 @@
+{
+ "experiment": "CV4 Phase 1 \u2014 capability calibration across the ladder, all six families",
+ "prereg": "docs/experiments/preregs/CV4_capability_ladder.md",
+ "status": "MEASUREMENT ONLY \u2014 Phase 1 gates which model later phases target; CV4's registered rule is adjudicated in Phase 3, not here",
+ "readout_convention": "STRIPPED (corrected)",
+ "n_items_total": 551,
+ "n_items_with_target": 255,
+ "measures": {
+  "answer_rank": "rank of the expected answer's first token; defined only where `target` is non-null (multihop, order-ops, multilingual)",
+  "prompt_surprisal": "mean per-token NLL of the prompt; defined for all six families. NOT a task-competence measure \u2014 it separates 'cannot do the task' from 'cannot parse the prompt'"
+ },
+ "note_on_undefined": "poetry / typo / association carry target: null. The annotated intermediate is the object of interest and there is no separate answer, so ANSWER COMPETENCE IS UNDEFINED for them, not zero.",
+ "interval": "95% Wilson score interval on every rate",
+ "by_model": {
+  "70m": {
+   "multihop": {
+    "n_items": 93,
+    "mean_prompt_surprisal": 4.742897197764407,
+    "median_prompt_surprisal": 4.761600017547607,
+    "n_scored_for_answer": 93,
+    "top1_count": 1,
+    "top1_rate": 0.010752688172043012,
+    "top1_wilson95": [
+     0.001900593097955601,
+     0.05842059118648415
+    ],
+    "top10_count": 13,
+    "top10_rate": 0.13978494623655913,
+    "top10_wilson95": [
+     0.08354643741721157,
+     0.22460212683616715
+    ],
+    "median_rank": 80
+   },
+   "multilingual": {
+    "n_items": 107,
+    "mean_prompt_surprisal": 5.743064506031643,
+    "median_prompt_surprisal": 5.629157066345215,
+    "n_scored_for_answer": 107,
+    "top1_count": 1,
+    "top1_rate": 0.009345794392523364,
+    "top1_wilson95": [
+     0.0016516291511190044,
+     0.05105060538898146
+    ],
+    "top10_count": 9,
+    "top10_rate": 0.08411214953271028,
+    "top10_wilson95": [
+     0.044881436613206184,
+     0.15217093361602183
+    ],
+    "median_rank": 3436
+   },
+   "order-ops": {
+    "n_items": 55,
+    "mean_prompt_surprisal": 4.685458686135032,
+    "median_prompt_surprisal": 4.4597907066345215,
+    "n_scored_for_answer": 55,
+    "top1_count": 2,
+    "top1_rate": 0.03636363636363636,
+    "top1_wilson95": [
+     0.010029330855724924,
+     0.1232369297490173
+    ],
+    "top10_count": 21,
+    "top10_rate": 0.38181818181818183,
+    "top10_wilson95": [
+     0.26515766317415235,
+     0.513910207176076
+    ],
+    "median_rank": 18
+   },
+   "poetry": {
+    "n_items": 98,
+    "mean_prompt_surprisal": 5.154343254712163,
+    "median_prompt_surprisal": 5.116718769073486,
+    "answer_competence": "UNDEFINED \u2014 target is null for this family"
+   },
+   "typo": {
+    "n_items": 96,
+    "mean_prompt_surprisal": 6.320249448219935,
+    "median_prompt_surprisal": 6.34007453918457,
+    "answer_competence": "UNDEFINED \u2014 target is null for this family"
+   },
+   "association": {
+    "n_items": 102,
+    "mean_prompt_surprisal": 4.577553527027953,
+    "median_prompt_surprisal": 4.60029411315918,
+    "answer_competence": "UNDEFINED \u2014 target is null for this family"
+   },
+   "_pooled_answer": {
+    "n": 255,
+    "top1_count": 4,
+    "top1_rate": 0.01568627450980392,
+    "top1_wilson95": [
+     0.006116456526536161,
+     0.03963198576403846
+    ],
+    "top10_rate": 0.16862745098039217,
+    "top10_wilson95": [
+     0.12766685656458393,
+     0.2194241828973882
+    ]
+   },
+   "_pooled_surprisal_all_six": 5.203927769981855
+  },
+  "160m": {
+   "multihop": {
+    "n_items": 93,
+    "mean_prompt_surprisal": 4.323448388807235,
+    "median_prompt_surprisal": 4.265374183654785,
+    "n_scored_for_answer": 93,
+    "top1_count": 3,
+    "top1_rate": 0.03225806451612903,
+    "top1_wilson95": [
+     0.011030412935116203,
+     0.09059533467748004
+    ],
+    "top10_count": 24,
+    "top10_rate": 0.25806451612903225,
+    "top10_wilson95": [
+     0.17998399469266027,
+     0.35533966786557925
+    ],
+    "median_rank": 42
+   },
+   "multilingual": {
+    "n_items": 107,
+    "mean_prompt_surprisal": 6.095417096236042,
+    "median_prompt_surprisal": 5.124428749084473,
+    "n_scored_for_answer": 107,
+    "top1_count": 1,
+    "top1_rate": 0.009345794392523364,
+    "top1_wilson95": [
+     0.0016516291511190044,
+     0.05105060538898146
+    ],
+    "top10_count": 9,
+    "top10_rate": 0.08411214953271028,
+    "top10_wilson95": [
+     0.044881436613206184,
+     0.15217093361602183
+    ],
+    "median_rank": 3788
+   },
+   "order-ops": {
+    "n_items": 55,
+    "mean_prompt_surprisal": 4.33481783433394,
+    "median_prompt_surprisal": 4.353231430053711,
+    "n_scored_for_answer": 55,
+    "top1_count": 0,
+    "top1_rate": 0.0,
+    "top1_wilson95": [
+     0.0,
+     0.06528714378942788
+    ],
+    "top10_count": 24,
+    "top10_rate": 0.43636363636363634,
+    "top10_wilson95": [
+     0.31373238636001305,
+     0.5673041592131869
+    ],
+    "median_rank": 13
+   },
+   "poetry": {
+    "n_items": 98,
+    "mean_prompt_surprisal": 4.700447340400851,
+    "median_prompt_surprisal": 4.699031829833984,
+    "answer_competence": "UNDEFINED \u2014 target is null for this family"
+   },
+   "typo": {
+    "n_items": 96,
+    "mean_prompt_surprisal": 5.784147493541241,
+    "median_prompt_surprisal": 5.752850532531738,
+    "answer_competence": "UNDEFINED \u2014 target is null for this family"
+   },
+   "association": {
+    "n_items": 102,
+    "mean_prompt_surprisal": 4.123158244525685,
+    "median_prompt_surprisal": 4.120728969573975,
+    "answer_competence": "UNDEFINED \u2014 target is null for this family"
+   },
+   "_pooled_answer": {
+    "n": 255,
+    "top1_count": 4,
+    "top1_rate": 0.01568627450980392,
+    "top1_wilson95": [
+     0.006116456526536161,
+     0.03963198576403846
+    ],
+    "top10_rate": 0.2235294117647059,
+    "top10_wilson95": [
+     0.1767132210847063,
+     0.27855208402853665
+    ]
+   },
+   "_pooled_surprisal_all_six": 4.893572732974166
+  },
+  "410m": {
+   "multihop": {
+    "n_items": 93,
+    "mean_prompt_surprisal": 3.8603221395964264,
+    "median_prompt_surprisal": 3.814849376678467,
+    "n_scored_for_answer": 93,
+    "top1_count": 6,
+    "top1_rate": 0.06451612903225806,
+    "top1_wilson95": [
+     0.029901024142029796,
+     0.13368156846280116
+    ],
+    "top10_count": 31,
+    "top10_rate": 0.3333333333333333,
+    "top10_wilson95": [
+     0.24582243182201943,
+     0.4340672023847677
+    ],
+    "median_rank": 31
+   },
+   "multilingual": {
+    "n_items": 107,
+    "mean_prompt_surprisal": 4.675777234763743,
+    "median_prompt_surprisal": 4.748653411865234,
+    "n_scored_for_answer": 107,
+    "top1_count": 1,
+    "top1_rate": 0.009345794392523364,
+    "top1_wilson95": [
+     0.0016516291511190044,
+     0.05105060538898146
+    ],
+    "top10_count": 6,
+    "top10_rate": 0.056074766355140186,
+    "top10_wilson95": [
+     0.025950008789329404,
+     0.11697106055647577
+    ],
+    "median_rank": 3044
+   },
+   "order-ops": {
+    "n_items": 55,
+    "mean_prompt_surprisal": 4.2336343418468125,
+    "median_prompt_surprisal": 4.275657653808594,
+    "n_scored_for_answer": 55,
+    "top1_count": 5,
+    "top1_rate": 0.09090909090909091,
+    "top1_wilson95": [
+     0.039457573336373924,
+     0.19577736249133978
+    ],
+    "top10_count": 30,
+    "top10_rate": 0.5454545454545454,
+    "top10_wilson95": [
+     0.41522461348360484,
+     0.669749282535538
+    ],
+    "median_rank": 9
+   },
+   "poetry": {
+    "n_items": 98,
+    "mean_prompt_surprisal": 4.148344592172272,
+    "median_prompt_surprisal": 4.126254558563232,
+    "answer_competence": "UNDEFINED \u2014 target is null for this family"
+   },
+   "typo": {
+    "n_items": 96,
+    "mean_prompt_surprisal": 5.49502004434665,
+    "median_prompt_surprisal": 5.293105602264404,
+    "answer_competence": "UNDEFINED \u2014 target is null for this family"
+   },
+   "association": {
+    "n_items": 102,
+    "mean_prompt_surprisal": 3.8331370657565547,
+    "median_prompt_surprisal": 3.8432066440582275,
+    "answer_competence": "UNDEFINED \u2014 target is null for this family"
+   },
+   "_pooled_answer": {
+    "n": 255,
+    "top1_count": 12,
+    "top1_rate": 0.047058823529411764,
+    "top1_wilson95": [
+     0.02712132140633029,
+     0.08044098697068483
+    ],
+    "top10_rate": 0.2627450980392157,
+    "top10_wilson95": [
+     0.21253225697118083,
+     0.3200003807501127
+    ]
+   },
+   "_pooled_surprisal_all_six": 4.374372569747076
+  },
+  "1b": {
+   "multihop": {
+    "n_items": 93,
+    "mean_prompt_surprisal": 3.7122991469598587,
+    "median_prompt_surprisal": 3.713099241256714,
+    "n_scored_for_answer": 93,
+    "top1_count": 7,
+    "top1_rate": 0.07526881720430108,
+    "top1_wilson95": [
+     0.036936214085016475,
+     0.14729866018389276
+    ],
+    "top10_count": 31,
+    "top10_rate": 0.3333333333333333,
+    "top10_wilson95": [
+     0.24582243182201943,
+     0.4340672023847677
+    ],
+    "median_rank": 18
+   },
+   "multilingual": {
+    "n_items": 107,
+    "mean_prompt_surprisal": 4.4941605663745205,
+    "median_prompt_surprisal": 4.559407711029053,
+    "n_scored_for_answer": 107,
+    "top1_count": 0,
+    "top1_rate": 0.0,
+    "top1_wilson95": [
+     0.0,
+     0.03465846757895952
+    ],
+    "top10_count": 7,
+    "top10_rate": 0.06542056074766354,
+    "top10_wilson95": [
+     0.032047896397421095,
+     0.12891693990952502
+    ],
+    "median_rank": 1649
+   },
+   "order-ops": {
+    "n_items": 55,
+    "mean_prompt_surprisal": 4.145381216569381,
+    "median_prompt_surprisal": 4.083983898162842,
+    "n_scored_for_answer": 55,
+    "top1_count": 4,
+    "top1_rate": 0.07272727272727272,
+    "top1_wilson95": [
+     0.028643596503688543,
+     0.17260178091636796
+    ],
+    "top10_count": 37,
+    "top10_rate": 0.6727272727272727,
+    "top10_wilson95": [
+     0.5410297617609947,
+     0.7818710431117484
+    ],
+    "median_rank": 7
+   },
+   "poetry": {
+    "n_items": 98,
+    "mean_prompt_surprisal": 3.935050759996687,
+    "median_prompt_surprisal": 3.910858154296875,
+    "answer_competence": "UNDEFINED \u2014 target is null for this family"
+   },
+   "typo": {
+    "n_items": 96,
+    "mean_prompt_surprisal": 5.290679027636846,
+    "median_prompt_surprisal": 5.2639923095703125,
+    "answer_competence": "UNDEFINED \u2014 target is null for this family"
+   },
+   "association": {
+    "n_items": 102,
+    "mean_prompt_surprisal": 3.6671034354789582,
+    "median_prompt_surprisal": 3.6480932235717773,
+    "answer_competence": "UNDEFINED \u2014 target is null for this family"
+   },
+   "_pooled_answer": {
+    "n": 255,
+    "top1_count": 11,
+    "top1_rate": 0.043137254901960784,
+    "top1_wilson95": [
+     0.024254970635041635,
+     0.07558060448116843
+    ],
+    "top10_rate": 0.29411764705882354,
+    "top10_wilson95": [
+     0.24157991671293233,
+     0.3527665870948018
+    ]
+   },
+   "_pooled_surprisal_all_six": 4.207445692169375
+  },
+  "1.4b": {
+   "multihop": {
+    "n_items": 93,
+    "mean_prompt_surprisal": 3.5891012683991463,
+    "median_prompt_surprisal": 3.6213531494140625,
+    "n_scored_for_answer": 93,
+    "top1_count": 5,
+    "top1_rate": 0.053763440860215055,
+    "top1_wilson95": [
+     0.023180332422992958,
+     0.11974997851775977
+    ],
+    "top10_count": 33,
+    "top10_rate": 0.3548387096774194,
+    "top10_wilson95": [
+     0.26512726316736557,
+     0.45606693436757817
+    ],
+    "median_rank": 24
+   },
+   "multilingual": {
+    "n_items": 107,
+    "mean_prompt_surprisal": 4.305595877014588,
+    "median_prompt_surprisal": 4.307002067565918,
+    "n_scored_for_answer": 107,
+    "top1_count": 0,
+    "top1_rate": 0.0,
+    "top1_wilson95": [
+     0.0,
+     0.03465846757895952
+    ],
+    "top10_count": 8,
+    "top10_rate": 0.07476635514018691,
+    "top10_wilson95": [
+     0.03836964356083345,
+     0.1406389597072536
+    ],
+    "median_rank": 1345
+   },
+   "order-ops": {
+    "n_items": 55,
+    "mean_prompt_surprisal": 4.166462430087003,
+    "median_prompt_surprisal": 4.244797229766846,
+    "n_scored_for_answer": 55,
+    "top1_count": 3,
+    "top1_rate": 0.05454545454545454,
+    "top1_wilson95": [
+     0.01872287596750008,
+     0.14853294304489928
+    ],
+    "top10_count": 29,
+    "top10_rate": 0.5272727272727272,
+    "top10_wilson95": [
+     0.39791319071629383,
+     0.6530711468951917
+    ],
+    "median_rank": 7
+   },
+   "poetry": {
+    "n_items": 98,
+    "mean_prompt_surprisal": 3.6910850977411074,
+    "median_prompt_surprisal": 3.6510846614837646,
+    "answer_competence": "UNDEFINED \u2014 target is null for this family"
+   },
+   "typo": {
+    "n_items": 96,
+    "mean_prompt_surprisal": 5.277833836774032,
+    "median_prompt_surprisal": 5.16269588470459,
+    "answer_competence": "UNDEFINED \u2014 target is null for this family"
+   },
+   "association": {
+    "n_items": 102,
+    "mean_prompt_surprisal": 3.6086690916734585,
+    "median_prompt_surprisal": 3.6544148921966553,
+    "answer_competence": "UNDEFINED \u2014 target is null for this family"
+   },
+   "_pooled_answer": {
+    "n": 255,
+    "top1_count": 8,
+    "top1_rate": 0.03137254901960784,
+    "top1_wilson95": [
+     0.01598082278729794,
+     0.060674552546496924
+    ],
+    "top10_rate": 0.27450980392156865,
+    "top10_wilson95": [
+     0.22338671792303816,
+     0.33232611958067076
+    ]
+   },
+   "_pooled_surprisal_all_six": 4.106457933614889
+  },
+  "2.8b": {
+   "multihop": {
+    "n_items": 93,
+    "mean_prompt_surprisal": 3.5109470018776516,
+    "median_prompt_surprisal": 3.531287670135498,
+    "n_scored_for_answer": 93,
+    "top1_count": 6,
+    "top1_rate": 0.06451612903225806,
+    "top1_wilson95": [
+     0.029901024142029796,
+     0.13368156846280116
+    ],
+    "top10_count": 40,
+    "top10_rate": 0.43010752688172044,
+    "top10_wilson95": [
+     0.33423358533237524,
+     0.5315265838511162
+    ],
+    "median_rank": 14
+   },
+   "multilingual": {
+    "n_items": 107,
+    "mean_prompt_surprisal": 4.153321266174316,
+    "median_prompt_surprisal": 4.131369590759277,
+    "n_scored_for_answer": 107,
+    "top1_count": 1,
+    "top1_rate": 0.009345794392523364,
+    "top1_wilson95": [
+     0.0016516291511190044,
+     0.05105060538898146
+    ],
+    "top10_count": 10,
+    "top10_rate": 0.09345794392523364,
+    "top10_wilson95": [
+     0.05155771001191554,
+     0.1635384271784534
+    ],
+    "median_rank": 578
+   },
+   "order-ops": {
+    "n_items": 55,
+    "mean_prompt_surprisal": 4.071762145649303,
+    "median_prompt_surprisal": 4.097884178161621,
+    "n_scored_for_answer": 55,
+    "top1_count": 8,
+    "top1_rate": 0.14545454545454545,
+    "top1_wilson95": [
+     0.07559199016771778,
+     0.2616116208829674
+    ],
+    "top10_count": 34,
+    "top10_rate": 0.6181818181818182,
+    "top10_wilson95": [
+     0.4860897928239239,
+     0.7348423368258477
+    ],
+    "median_rank": 7
+   },
+   "poetry": {
+    "n_items": 98,
+    "mean_prompt_surprisal": 3.6883874936979644,
+    "median_prompt_surprisal": 3.6747243404388428,
+    "answer_competence": "UNDEFINED \u2014 target is null for this family"
+   },
+   "typo": {
+    "n_items": 96,
+    "mean_prompt_surprisal": 5.161082645257314,
+    "median_prompt_surprisal": 5.04867696762085,
+    "answer_competence": "UNDEFINED \u2014 target is null for this family"
+   },
+   "association": {
+    "n_items": 102,
+    "mean_prompt_surprisal": 3.4943248594508454,
+    "median_prompt_surprisal": 3.5100271701812744,
+    "answer_competence": "UNDEFINED \u2014 target is null for this family"
+   },
+   "_pooled_answer": {
+    "n": 255,
+    "top1_count": 15,
+    "top1_rate": 0.058823529411764705,
+    "top1_wilson95": [
+     0.03596805781169063,
+     0.09477445034773968
+    ],
+    "top10_rate": 0.32941176470588235,
+    "top10_wilson95": [
+     0.2746295550144033,
+     0.3892575481405764
+    ]
+   },
+   "_pooled_surprisal_all_six": 4.0133042353512325
+  }
+ },
+ "VERDICT": "Answer competence, 255 items with a target, 95% Wilson: 70m: top1 1.6% [0.6,4.0] | 160m: top1 1.6% [0.6,4.0] | 410m: top1 4.7% [2.7,8.0] | 1b: top1 4.3% [2.4,7.6] | 1.4b: top1 3.1% [1.6,6.1] | 2.8b: top1 5.9% [3.6,9.5]. Prompt surprisal covers all 551 items in all six families.",
+ "provenance": {
+  "schema_version": 1,
+  "experiment": "CV4P1",
+  "utc": "2026-08-30T06:30:11+00:00",
+  "script": {
+   "path": "experiments/cv4_phase1_capability.py",
+   "sha256": "3aaadc49c285d30fe1aafb92714633b4d5e694caac40f82b0afbfa964a424cfb",
+   "n_lines": 173
+  },
+  "git": {
+   "commit": "c9c49697e295c0fec4225552836fc9733b279f75",
+   "commit_short": "c9c49697e295",
+   "git_repository_present": true,
+   "branch": "pythia",
+   "dirty": true,
+   "dirty_files": [
+    "CLAUDE.md",
+    "README.md",
+    "corpora/manifest_ood.json",
+    "docs/HANDOFF.md",
+    "docs/REPO_REVIEW_PROMPT.md",
+    "docs/context/AGGREGATION_POLICY.md",
+    "docs/context/CONTEXT.md",
+    "docs/context/IMPLEMENTATION_RULES.md",
+    "docs/context/STATE.md",
+    "docs/experiments/INDEX.md",
+    "docs/experiments/METRICS.md",
+    "docs/experiments/PROPOSED.md",
+    "docs/experiments/PROVENANCE.md",
+    "docs/experiments/descriptions/E64.md",
+    "docs/experiments/descriptions/EN1.md",
+    "docs/experiments/preregs/CV3_margins.md",
+    "docs/experiments/preregs/CV4_capability_ladder.md",
+    "docs/experiments/preregs/CV5_metric_sensitivity.md",
+    "docs/experiments/preregs/CV6_per_family_ladder.md",
+    "docs/experiments/preregs/CV7_1b_rung.md",
+    "docs/experiments/preregs/DN1_design_nulls.md",
+    "docs/experiments/preregs/E33_logit_baseline.md",
+    "docs/experiments/preregs/E36_qladder.md",
+    "docs/experiments/preregs/E48_crossover.md",
+    "docs/experiments/preregs/E48b_containment.md",
+    "docs/experiments/preregs/E48c_exposure_vs_read.md",
+    "docs/experiments/preregs/E51_interaction.md",
+    "docs/experiments/preregs/E52_factorial.md",
+    "docs/experiments/preregs/E53_ladder_summary.md",
+    "docs/experiments/preregs/E55_matrix_robustness.md",
+    "docs/experiments/preregs/E56_predictor_registry.md",
+    "docs/experiments/preregs/E57_grid_variance_ci.md",
+    "docs/experiments/preregs/E58_algebra_audit.md",
+    "docs/experiments/preregs/E59_read_dose_response.md",
+    "docs/experiments/preregs/E60_fitter_determinism.md",
+    "docs/experiments/preregs/E61_randomized_network_null.md",
+    "docs/experiments/preregs/E62_ladder1b_corrected_band.md",
+    "docs/experiments/preregs/E65_training_axis_floor.md",
+    "docs/experiments/preregs/E66b_determinism_floor.md",
+    "docs/experiments/preregs/OS1_operator_space.md",
+    "docs/experiments/preregs/P0_t52_parallel_port.md",
+    "docs/experiments/preregs/R1_grid_rstrip.md",
+    "docs/experiments/preregs/R2_e48_rstrip_arm.md",
+    "docs/experiments/preregs/R3_ladder410_cpu.md",
+    "docs/experiments/preregs/R4_corrections.md",
+    "docs/experiments/preregs/R4b_e36_rstrip.md",
+    "docs/experiments/preregs/R5_corpus_axis_uncertainty.md",
+    "docs/experiments/preregs/R6_within_source_resampling.md",
+    "docs/experiments/preregs/R8_ladder_rstrip_S2.md",
+    "docs/experiments/preregs/R9_permutation_calibrated_min.md",
+    "docs/experiments/preregs/superseded/PREREG_E48_CROSSOVER.md",
+    "docs/reproducibility/ARTIFACTS.md",
+    "docs/reproducibility/PAYLOAD_REDACTIONS.md",
+    "docs/reproducibility/RELEASE.md",
+    "docs/reproducibility/REVIEWER_REPRO.md",
+    "docs/reproducibility/T1_FINDING_E33_BAND.md",
+    "docs/reproducibility/T1_SWEEP_PREDICTIONS.md",
+    "docs/reproducibility/T1_SWEEP_SPEC.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/MANIFEST.sha256",
+    "docs/reproducibility/review-snapshot-2026-08-19/PROVENANCE.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/REDACTIONS.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/EXTERNAL_REVIEW.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/FINDINGS.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/PHASE_A_ACCOUNT.md"
+   ],
+   "dirty_diff_sha256": "ec61dde1905616f407f51d5f6fe3ea37c9c6e3e1ade6998cd0644527d080c600",
+   "describe": "c9c4969-dirty"
+  },
+  "argv": [
+   "experiments/cv4_phase1_capability.py",
+   "--models",
+   "70m,160m,410m,1b,1.4b,2.8b"
+  ],
+  "cwd": ".",
+  "inputs": [
+   {
+    "path": "jacobian-lens/data/evaluations/lens-eval-multihop.json",
+    "sha256": "50b7e4c9255291c0ca2a8e94615be9f44531fa57bb1a844e4f9616056d987416",
+    "bytes": 21869,
+    "exists": true
+   },
+   {
+    "path": "jacobian-lens/data/evaluations/lens-eval-multilingual.json",
+    "sha256": "fa70b9bd89416a6d8d985a80dc628b109ae6fd3b25b9275c0fc5065d7ff4a0ef",
+    "bytes": 24284,
+    "exists": true
+   },
+   {
+    "path": "jacobian-lens/data/evaluations/lens-eval-order-ops.json",
+    "sha256": "b203206d16ff628152cc86f3838604e06cb54776f3e14fa1c34f150db8bc7560",
+    "bytes": 9589,
+    "exists": true
+   },
+   {
+    "path": "jacobian-lens/data/evaluations/lens-eval-poetry.json",
+    "sha256": "6aeb3415c5a5c3f3827c9efe63f006de02f5ef39a816bbac68e15e733aba60cc",
+    "bytes": 21533,
+    "exists": true
+   },
+   {
+    "path": "jacobian-lens/data/evaluations/lens-eval-typo.json",
+    "sha256": "9d05e16b7234a57d0773d120a4e1c4e94fd3bc2235a8125d4200a70e60ab17aa",
+    "bytes": 14646,
+    "exists": true
+   },
+   {
+    "path": "jacobian-lens/data/evaluations/lens-eval-association.json",
+    "sha256": "d1a98cd4911b594282e74168091c77d849dae18ffe2acb5761074853f327d71c",
+    "bytes": 24228,
+    "exists": true
+   }
+  ],
+  "env": {
+   "python": "3.14.2",
+   "torch": "2.13.0",
+   "transformers": "5.14.1",
+   "numpy": "2.5.1",
+   "platform": "macOS-26.5.2-arm64-arm-64bit-Mach-O",
+   "machine": "arm64",
+   "hostname": "<redacted for double-blind review>",
+   "tf32": {
+    "allow_tf32_matmul": false,
+    "allow_tf32_cudnn": true,
+    "NVIDIA_TF32_OVERRIDE": null
+   },
+   "threads": {
+    "torch_num_threads": 10,
+    "torch_num_interop_threads": 14,
+    "OMP_NUM_THREADS": null,
+    "os_cpu_count": 14
+   }
+  },
+  "payload_sha256": "43b21658bcc6a1acf619b614a5eb74db9252372b3ff7f151fbbde0ce7fbd9bad"
+ }
+}

results/cv5_metric_sensitivity_410m.json

@@ -0,0 +1,296 @@
+{
+ "experiment": "CV5 \u2014 metric sensitivity: is rank blind where z is sighted?",
+ "prereg": "docs/experiments/preregs/CV5_metric_sensitivity.md",
+ "status": "PRE-REGISTERED",
+ "decision_rule_verbatim": "at r=2.535e-03: ACCEPT (RANK WAS BLIND) if S_z >= 1.0 and S_rank < 0.5 | REJECT (FUNCTIONALLY IDENTICAL) if S_z < 0.5 | UNCLEAR otherwise",
+ "model": "EleutherAI/pythia-410m-deduped",
+ "band": [
+  9,
+  10,
+  11,
+  12,
+  13,
+  14,
+  15,
+  16,
+  17,
+  18,
+  19,
+  20,
+  21
+ ],
+ "admitted_sets": [
+  "multihop",
+  "multilingual",
+  "order-ops",
+  "poetry",
+  "typo"
+ ],
+ "K": [
+  1,
+  2,
+  5,
+  10,
+  20,
+  50,
+  100
+ ],
+ "base_operator": "results/e48/lens_INSTREAM_Pile-CC_410m_n200_s0.pt",
+ "readout_convention": "STRIPPED (corrected), no prefix; flat-mean-7 k-summary",
+ "denominators_from_cv3": {
+  "pooled_seed_sd_rank": 0.002581,
+  "pooled_seed_sd_z": 0.013407
+ },
+ "unperturbed": {
+  "rank": 0.2619082412030062,
+  "z": 3.564116526229665,
+  "logit_I_rank": 0.1981085192666858
+ },
+ "by_magnitude": {
+  "0.0": {
+   "r": 0.0,
+   "n_seeds": 1,
+   "rank": [
+    0.2619082412030062
+   ],
+   "z": [
+    3.564116526229665
+   ],
+   "S_rank": 0.0,
+   "S_z": 0.0
+  },
+  "1.06e-07": {
+   "r": 1.06e-07,
+   "n_seeds": 3,
+   "rank": [
+    0.2619082412030062,
+    0.2619082412030062,
+    0.2619082412030062
+   ],
+   "z": [
+    3.5641165554510494,
+    3.5641163831333875,
+    3.5641164395128424
+   ],
+   "S_rank": 0.0,
+   "S_z": 4.9872383888545455e-06
+  },
+  "0.002535": {
+   "r": 0.002535,
+   "n_seeds": 3,
+   "rank": [
+    0.2617184725746441,
+    0.2626325979167178,
+    0.2620876530516675
+   ],
+   "z": [
+    3.562412863826774,
+    3.5641458966086614,
+    3.5612677342495367
+   ],
+   "S_rank": 0.09221231228343889,
+   "S_z": 0.11245578190554997
+  },
+  "0.01": {
+   "r": 0.01,
+   "n_seeds": 3,
+   "rank": [
+    0.2663647765893257,
+    0.2655696648942096,
+    0.26799035067682875
+   ],
+   "z": [
+    3.573528114914198,
+    3.562376512847641,
+    3.5774573911741827
+   ],
+   "S_rank": 1.8339233567538897,
+   "S_z": 0.5224246102042763
+  },
+  "0.05": {
+   "r": 0.05,
+   "n_seeds": 3,
+   "rank": [
+    0.28435305275900635,
+    0.27584753020009956,
+    0.2805677163156452
+   ],
+   "z": [
+    3.495605627184469,
+    3.473200516113597,
+    3.4979920391660375
+   ],
+   "S_rank": 7.108817727719558,
+   "S_z": 5.607801800673574
+  }
+ },
+ "PRIMARY_at_e66_magnitude": {
+  "r": 0.002535,
+  "S_rank": 0.09221231228343889,
+  "S_z": 0.11245578190554997
+ },
+ "VERDICT": "REJECT \u2014 THE OPERATORS ARE FUNCTIONALLY IDENTICAL",
+ "controls": {
+  "C1_zero_perturbation_is_noop": {
+   "required": "r=0 reproduces the unperturbed read exactly in both spaces",
+   "S_rank": 0.0,
+   "S_z": 0.0,
+   "fires": true
+  },
+  "C2_logit_anchor": {
+   "required": "logit_I flat-mean-7 min == 0.19810852520167826 within 1e-6",
+   "observed": 0.1981085192666858,
+   "abs_diff": 5.934992469702749e-09,
+   "fires": true
+  },
+  "C3_harness_responds": {
+   "required": "at r=5e-2 at least one space must move >= 1 seed SD, else the scoring path is not responding to the operator and every number is void",
+   "S_rank": 7.108817727719558,
+   "S_z": 5.607801800673574,
+   "fires": true
+  },
+  "C4_monotone_in_r": {
+   "required": "S_z non-decreasing across magnitudes",
+   "S_z_by_r": [
+    0.0,
+    4.9872383888545455e-06,
+    0.11245578190554997,
+    0.5224246102042763,
+    5.607801800673574
+   ],
+   "fires": true
+  }
+ },
+ "declared_bias": "Gaussian noise is not a fitter difference. This bounds sensitivity to a perturbation of that size; it does not reproduce E66's specific operator delta, which is structural (accumulation order, batching, kernels).",
+ "e66_context": "results/e66_fitter_equivalence_cuda.json: trainval vs stored operator max_rel 2.535e-03 with read difference exactly 0.0; its C1 did not fire and its verdict flags the stored lens's provenance. The flagged file is the base operator used here and in CV3.",
+ "provenance": {
+  "schema_version": 1,
+  "experiment": "CV5",
+  "utc": "2026-08-30T00:18:45+00:00",
+  "script": {
+   "path": "experiments/cv5_metric_sensitivity.py",
+   "sha256": "f153b87305050c47829552d76859bc6d11c9b343419628b8122d2cc3d9c06ef6",
+   "n_lines": 198
+  },
+  "git": {
+   "commit": "c9c49697e295c0fec4225552836fc9733b279f75",
+   "commit_short": "c9c49697e295",
+   "git_repository_present": true,
+   "branch": "pythia",
+   "dirty": true,
+   "dirty_files": [
+    "CLAUDE.md",
+    "README.md",
+    "corpora/manifest_ood.json",
+    "docs/HANDOFF.md",
+    "docs/REPO_REVIEW_PROMPT.md",
+    "docs/context/AGGREGATION_POLICY.md",
+    "docs/context/CONTEXT.md",
+    "docs/context/IMPLEMENTATION_RULES.md",
+    "docs/context/STATE.md",
+    "docs/experiments/INDEX.md",
+    "docs/experiments/METRICS.md",
+    "docs/experiments/PROPOSED.md",
+    "docs/experiments/PROVENANCE.md",
+    "docs/experiments/descriptions/E64.md",
+    "docs/experiments/descriptions/EN1.md",
+    "docs/experiments/preregs/CV3_margins.md",
+    "docs/experiments/preregs/CV4_capability_ladder.md",
+    "docs/experiments/preregs/CV5_metric_sensitivity.md",
+    "docs/experiments/preregs/CV6_per_family_ladder.md",
+    "docs/experiments/preregs/CV7_1b_rung.md",
+    "docs/experiments/preregs/DN1_design_nulls.md",
+    "docs/experiments/preregs/E33_logit_baseline.md",
+    "docs/experiments/preregs/E36_qladder.md",
+    "docs/experiments/preregs/E48_crossover.md",
+    "docs/experiments/preregs/E48b_containment.md",
+    "docs/experiments/preregs/E48c_exposure_vs_read.md",
+    "docs/experiments/preregs/E51_interaction.md",
+    "docs/experiments/preregs/E52_factorial.md",
+    "docs/experiments/preregs/E53_ladder_summary.md",
+    "docs/experiments/preregs/E55_matrix_robustness.md",
+    "docs/experiments/preregs/E56_predictor_registry.md",
+    "docs/experiments/preregs/E57_grid_variance_ci.md",
+    "docs/experiments/preregs/E58_algebra_audit.md",
+    "docs/experiments/preregs/E59_read_dose_response.md",
+    "docs/experiments/preregs/E60_fitter_determinism.md",
+    "docs/experiments/preregs/E61_randomized_network_null.md",
+    "docs/experiments/preregs/E62_ladder1b_corrected_band.md",
+    "docs/experiments/preregs/E65_training_axis_floor.md",
+    "docs/experiments/preregs/E66b_determinism_floor.md",
+    "docs/experiments/preregs/OS1_operator_space.md",
+    "docs/experiments/preregs/P0_t52_parallel_port.md",
+    "docs/experiments/preregs/R1_grid_rstrip.md",
+    "docs/experiments/preregs/R2_e48_rstrip_arm.md",
+    "docs/experiments/preregs/R3_ladder410_cpu.md",
+    "docs/experiments/preregs/R4_corrections.md",
+    "docs/experiments/preregs/R4b_e36_rstrip.md",
+    "docs/experiments/preregs/R5_corpus_axis_uncertainty.md",
+    "docs/experiments/preregs/R6_within_source_resampling.md",
+    "docs/experiments/preregs/R8_ladder_rstrip_S2.md",
+    "docs/experiments/preregs/R9_permutation_calibrated_min.md",
+    "docs/experiments/preregs/superseded/PREREG_E48_CROSSOVER.md",
+    "docs/reproducibility/ARTIFACTS.md",
+    "docs/reproducibility/PAYLOAD_REDACTIONS.md",
+    "docs/reproducibility/RELEASE.md",
+    "docs/reproducibility/REVIEWER_REPRO.md",
+    "docs/reproducibility/T1_SWEEP_SPEC.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/MANIFEST.sha256",
+    "docs/reproducibility/review-snapshot-2026-08-19/PROVENANCE.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/REDACTIONS.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/EXTERNAL_REVIEW.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/FINDINGS.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/PHASE_A_ACCOUNT.md",
+    "docs/reviews/REVIEW_INSIGHTS_2026-08-25.md",
+    "docs/validity/CONSTRUCT_VALIDITY.md"
+   ],
+   "dirty_diff_sha256": "6b143c292229b3d2480d1b73243675f0db5e23b27f66e4215e0a9c1947a761a8",
+   "describe": "c9c4969-dirty"
+  },
+  "argv": [
+   "experiments/cv5_metric_sensitivity.py"
+  ],
+  "cwd": ".",
+  "inputs": [
+   {
+    "path": "results/e48/lens_INSTREAM_Pile-CC_410m_n200_s0.pt",
+    "sha256": "4f00cc3d7450c3646b78f002e794572ca27d6ab175c70626d2d5fb7fccf28e20",
+    "bytes": 27268349,
+    "exists": true
+   },
+   {
+    "path": "results/e66_fitter_equivalence_cuda.json",
+    "sha256": "d6e9bd3cd0c9e9426fc69e9d8d711f9a46e934214aaa52f67e877d947ec55cd2",
+    "bytes": 3208,
+    "exists": true
+   },
+   {
+    "path": "results/cv3_margins_410m.json",
+    "sha256": "6cc200df5827b3c358d533934a37d892bfe1f7e9893bc4e4480421700afe7e99",
+    "bytes": 14659,
+    "exists": true
+   }
+  ],
+  "env": {
+   "python": "3.14.2",
+   "torch": "2.13.0",
+   "transformers": "5.14.1",
+   "numpy": "2.5.1",
+   "platform": "macOS-26.5.2-arm64-arm-64bit-Mach-O",
+   "machine": "arm64",
+   "hostname": "<redacted for double-blind review>",
+   "tf32": {
+    "allow_tf32_matmul": false,
+    "allow_tf32_cudnn": true,
+    "NVIDIA_TF32_OVERRIDE": null
+   },
+   "threads": {
+    "torch_num_threads": 8,
+    "torch_num_interop_threads": 14,
+    "OMP_NUM_THREADS": null,
+    "os_cpu_count": 14
+   }
+  },
+  "payload_sha256": "cde8769ac592f81096a80090b47a40f14c66626684029dad622f967b054ed1d9"
+ }
+}

results/cv6/cv6_1.4b_n25.json

@@ -0,0 +1,1336 @@
+{
+ "experiment": "CV6 \u2014 the per-family corpus contrast, one model",
+ "prereg": "docs/experiments/preregs/CV6_per_family_ladder.md",
+ "status": "PRE-REGISTERED",
+ "decision_rule_verbatim": "Evaluated PER FAMILY; the verdict is the count across the five. REPLICATES if R(f,2.8B) >= 10 on >= 4 of 5 families AND ordering tau >= 0.6 on >= 3 of 5. ATTENUATES if R(f,2.8B) < 5 on >= 4 of 5 families. UNCLEAR otherwise, including a split where some families replicate and others do not \u2014 a split is an informative outcome, not a failure. Do not pool to force a verdict. Do not add models to chase a threshold. Do not drop a family.",
+ "model": "EleutherAI/pythia-1.4b-deduped",
+ "short": "1.4b",
+ "n_layers": 24,
+ "d_model": 2048,
+ "band": [
+  9,
+  10,
+  11,
+  12,
+  13,
+  14,
+  15,
+  16,
+  17,
+  18,
+  19,
+  20,
+  21
+ ],
+ "band_rule": "floor(0.38*L)..floor(0.92*L), layers < L-2",
+ "band_matches_rule": true,
+ "N": 25,
+ "dim_batch": 128,
+ "max_seq_len": 128,
+ "skip_first": 16,
+ "target_layer_arg": -2,
+ "target_layer_effective": 22,
+ "device": "cuda",
+ "dtype": "float32",
+ "tf32": {
+  "matmul_allow_tf32": false,
+  "cudnn_allow_tf32": false,
+  "NVIDIA_TF32_OVERRIDE": "0"
+ },
+ "gpu": "NVIDIA H100 NVL",
+ "gpu_total_gb": 93.08551025390625,
+ "peak_vram_gb": 26.350069046020508,
+ "torch": "2.11.0+cu128",
+ "git_commit_at_launch": "518d192e9e215603957c85ac4a1308153bf4cf06",
+ "K": [
+  1,
+  2,
+  5,
+  10,
+  20,
+  50,
+  100
+ ],
+ "admitted_sets": [
+  "multihop",
+  "multilingual",
+  "order-ops",
+  "poetry",
+  "typo"
+ ],
+ "corpora": [
+  "Wikipedia_en",
+  "USPTO_Backgrounds",
+  "Pile-CC",
+  "StackExchange",
+  "Github"
+ ],
+ "seeds": [
+  0,
+  1,
+  2
+ ],
+ "readout_convention": "STRIPPED (corrected), no prefix, flat-mean-7",
+ "pooling_convention": "flat pool over (item, intermediate) pairs WITHIN a family; rank = min over band then flat-mean-7 pass@k; z = max over band then mean over pairs",
+ "n_items": 449,
+ "n_pairs": 801,
+ "n_pairs_per_family": {
+  "multihop": 103,
+  "multilingual": 394,
+  "order-ops": 110,
+  "poetry": 98,
+  "typo": 96
+ },
+ "per_arm": {
+  "logit_I": {
+   "multihop": {
+    "rank": 0.2496532593619972,
+    "z": 3.8530033965712613,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.1899927483683829,
+    "z": 3.2260746860564664,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5597402597402598,
+    "z": 5.515925762870095,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.008746355685131196,
+    "z": 2.4992136784962247,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.1889880952380952,
+    "z": 3.7020364006360373,
+    "n_pairs": 96
+   }
+  },
+  "J|Wikipedia_en|s0": {
+   "multihop": {
+    "rank": 0.3106796116504854,
+    "z": 3.8606277831549782,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.1892675852066715,
+    "z": 3.0605827324886614,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5740259740259741,
+    "z": 5.3812003374099735,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.026239067055393583,
+    "z": 2.3095518247205384,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.3392857142857143,
+    "z": 3.6630871382852397,
+    "n_pairs": 96
+   }
+  },
+  "J|Wikipedia_en|s1": {
+   "multihop": {
+    "rank": 0.3092926490984743,
+    "z": 3.8443020744231142,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.1907179115300943,
+    "z": 3.077618185033653,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5545454545454546,
+    "z": 5.271360293301669,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.018950437317784258,
+    "z": 2.327480596547224,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.33779761904761907,
+    "z": 3.6704573507110276,
+    "n_pairs": 96
+   }
+  },
+  "J|Wikipedia_en|s2": {
+   "multihop": {
+    "rank": 0.3065187239944521,
+    "z": 3.843342467419152,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.1787527193618564,
+    "z": 3.0286048066797595,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5610389610389611,
+    "z": 5.311658326062289,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.021865889212827987,
+    "z": 2.32845254516115,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.33482142857142855,
+    "z": 3.67272778848807,
+    "n_pairs": 96
+   }
+  },
+  "J|USPTO_Backgrounds|s0": {
+   "multihop": {
+    "rank": 0.32177531206657417,
+    "z": 3.874361871515663,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.1783901377810007,
+    "z": 3.081772235746916,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5727272727272729,
+    "z": 5.26822690855373,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.021865889212827987,
+    "z": 2.352261223355118,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.4404761904761905,
+    "z": 3.9461715444922447,
+    "n_pairs": 96
+   }
+  },
+  "J|USPTO_Backgrounds|s1": {
+   "multihop": {
+    "rank": 0.319001386962552,
+    "z": 3.9217262459032742,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.1787527193618564,
+    "z": 3.1000644785498603,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.596103896103896,
+    "z": 5.3440335826440295,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.021865889212827987,
+    "z": 2.3861058938259982,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.45238095238095244,
+    "z": 3.9819413671890893,
+    "n_pairs": 96
+   }
+  },
+  "J|USPTO_Backgrounds|s2": {
+   "multihop": {
+    "rank": 0.32593619972260746,
+    "z": 3.934496842541741,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.18419144307469182,
+    "z": 3.1254321258080187,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5805194805194805,
+    "z": 5.343620314381339,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.026239067055393583,
+    "z": 2.369717459897606,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.4568452380952381,
+    "z": 4.044803236921628,
+    "n_pairs": 96
+   }
+  },
+  "J|Pile-CC|s0": {
+   "multihop": {
+    "rank": 0.3203883495145631,
+    "z": 3.9831742198721876,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.20123277737490936,
+    "z": 3.2589381551984604,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5870129870129871,
+    "z": 5.469181368567726,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.018950437317784254,
+    "z": 2.397173393137601,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.3988095238095238,
+    "z": 4.014354603985946,
+    "n_pairs": 96
+   }
+  },
+  "J|Pile-CC|s1": {
+   "multihop": {
+    "rank": 0.32593619972260746,
+    "z": 4.029289348032868,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.19361856417693982,
+    "z": 3.188464311627567,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5922077922077923,
+    "z": 5.454147653146223,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.01749271137026239,
+    "z": 2.3707593989615536,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.4107142857142857,
+    "z": 4.036523262659709,
+    "n_pairs": 96
+   }
+  },
+  "J|Pile-CC|s2": {
+   "multihop": {
+    "rank": 0.32177531206657417,
+    "z": 3.953363766947996,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.1943437273386512,
+    "z": 3.190489783202331,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5870129870129871,
+    "z": 5.500762580741536,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.013119533527696792,
+    "z": 2.357462505296785,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.4047619047619047,
+    "z": 4.010696103175481,
+    "n_pairs": 96
+   }
+  },
+  "J|StackExchange|s0": {
+   "multihop": {
+    "rank": 0.3883495145631068,
+    "z": 4.0767282235969615,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.21537345902828137,
+    "z": 3.1823133939716417,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5935064935064934,
+    "z": 5.240476467392662,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.021865889212827987,
+    "z": 2.3227810050759996,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.46428571428571425,
+    "z": 4.006702713668346,
+    "n_pairs": 96
+   }
+  },
+  "J|StackExchange|s1": {
+   "multihop": {
+    "rank": 0.391123439667129,
+    "z": 4.067388656069931,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.22588832487309643,
+    "z": 3.2193740812049905,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5844155844155844,
+    "z": 5.164196674390273,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.018950437317784254,
+    "z": 2.2825519923044713,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.45982142857142855,
+    "z": 3.988087110221386,
+    "n_pairs": 96
+   }
+  },
+  "J|StackExchange|s2": {
+   "multihop": {
+    "rank": 0.4105409153952843,
+    "z": 4.105738782188268,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.2411167512690355,
+    "z": 3.290932434767031,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5909090909090909,
+    "z": 5.2581051761453805,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.021865889212827987,
+    "z": 2.309936959524544,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.4613095238095238,
+    "z": 3.990582413971424,
+    "n_pairs": 96
+   }
+  },
+  "J|Github|s0": {
+   "multihop": {
+    "rank": 0.30374479889042993,
+    "z": 3.6554290958978597,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.13379260333575055,
+    "z": 2.789235497035351,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5571428571428572,
+    "z": 5.291046177257191,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.011661807580174929,
+    "z": 2.0492812370767397,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.20833333333333334,
+    "z": 3.194101763268312,
+    "n_pairs": 96
+   }
+  },
+  "J|Github|s1": {
+   "multihop": {
+    "rank": 0.32871012482662965,
+    "z": 3.81565054412027,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.16678752719361858,
+    "z": 2.9543404418804924,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5844155844155844,
+    "z": 5.422733649340543,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.013119533527696792,
+    "z": 2.1037646602002944,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.2574404761904762,
+    "z": 3.366413981964191,
+    "n_pairs": 96
+   }
+  },
+  "J|Github|s2": {
+   "multihop": {
+    "rank": 0.3398058252427184,
+    "z": 3.787343767082807,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.14176939811457578,
+    "z": 2.774672096755904,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5701298701298702,
+    "z": 5.435313402522694,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.01603498542274052,
+    "z": 2.0549783214014403,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.22470238095238093,
+    "z": 3.2467485206822553,
+    "n_pairs": 96
+   }
+  }
+ },
+ "per_arm_fp16_roundtrip": {
+  "J|Wikipedia_en|s0": {
+   "multihop": {
+    "rank": 0.3106796116504854,
+    "z": 3.8606872315545684,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.1892675852066715,
+    "z": 3.060567912413989,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5740259740259741,
+    "z": 5.381180027398196,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.026239067055393583,
+    "z": 2.309572807988342,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.3392857142857143,
+    "z": 3.6631390204032264,
+    "n_pairs": 96
+   }
+  },
+  "J|Wikipedia_en|s1": {
+   "multihop": {
+    "rank": 0.3092926490984743,
+    "z": 3.844225617288386,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.1907179115300943,
+    "z": 3.0775656076857283,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5545454545454546,
+    "z": 5.271341731331566,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.018950437317784258,
+    "z": 2.3274755514397913,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.33779761904761907,
+    "z": 3.6704695199926696,
+    "n_pairs": 96
+   }
+  },
+  "J|Wikipedia_en|s2": {
+   "multihop": {
+    "rank": 0.30513176144244103,
+    "z": 3.8434583180159043,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.1787527193618564,
+    "z": 3.0286072580342367,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5610389610389611,
+    "z": 5.311677533929998,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.021865889212827987,
+    "z": 2.328465061528342,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.33482142857142855,
+    "z": 3.6727247970799604,
+    "n_pairs": 96
+   }
+  },
+  "J|USPTO_Backgrounds|s0": {
+   "multihop": {
+    "rank": 0.32177531206657417,
+    "z": 3.8744102160907485,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.1783901377810007,
+    "z": 3.081757281486153,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5727272727272729,
+    "z": 5.268081293322823,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.021865889212827987,
+    "z": 2.35228185568537,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.4404761904761905,
+    "z": 3.946172351638476,
+    "n_pairs": 96
+   }
+  },
+  "J|USPTO_Backgrounds|s1": {
+   "multihop": {
+    "rank": 0.3203883495145631,
+    "z": 3.921612312492815,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.1783901377810007,
+    "z": 3.100029178379756,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.596103896103896,
+    "z": 5.34396392323754,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.021865889212827987,
+    "z": 2.386144852151676,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.45238095238095244,
+    "z": 3.9819253981113434,
+    "n_pairs": 96
+   }
+  },
+  "J|USPTO_Backgrounds|s2": {
+   "multihop": {
+    "rank": 0.32871012482662965,
+    "z": 3.934536116794475,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.18419144307469182,
+    "z": 3.125436220374809,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5805194805194805,
+    "z": 5.34366934299469,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.026239067055393583,
+    "z": 2.3696852350721556,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.4568452380952381,
+    "z": 4.044789165258408,
+    "n_pairs": 96
+   }
+  },
+  "J|Pile-CC|s0": {
+   "multihop": {
+    "rank": 0.3203883495145631,
+    "z": 3.9830801978851986,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.20123277737490936,
+    "z": 3.258956805249761,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5870129870129871,
+    "z": 5.469027082486586,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.018950437317784254,
+    "z": 2.3971886932849884,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.3988095238095238,
+    "z": 4.014335438609123,
+    "n_pairs": 96
+   }
+  },
+  "J|Pile-CC|s1": {
+   "multihop": {
+    "rank": 0.32593619972260746,
+    "z": 4.029135812833471,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.19361856417693982,
+    "z": 3.188446533105095,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5922077922077923,
+    "z": 5.454229792681607,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.01749271137026239,
+    "z": 2.370773496068254,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.4092261904761905,
+    "z": 4.0365254531304045,
+    "n_pairs": 96
+   }
+  },
+  "J|Pile-CC|s2": {
+   "multihop": {
+    "rank": 0.32177531206657417,
+    "z": 3.9533760107836677,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.19398114575779551,
+    "z": 3.1904335273098825,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5870129870129871,
+    "z": 5.500757904486223,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.013119533527696792,
+    "z": 2.357441063438143,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.4047619047619047,
+    "z": 4.010686029990514,
+    "n_pairs": 96
+   }
+  },
+  "J|StackExchange|s0": {
+   "multihop": {
+    "rank": 0.3883495145631068,
+    "z": 4.076668833065959,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.21537345902828137,
+    "z": 3.1823146536870657,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5935064935064934,
+    "z": 5.24048402634534,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.021865889212827987,
+    "z": 2.3227494511069082,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.46428571428571425,
+    "z": 4.006694952646892,
+    "n_pairs": 96
+   }
+  },
+  "J|StackExchange|s1": {
+   "multihop": {
+    "rank": 0.391123439667129,
+    "z": 4.067319025113744,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.22588832487309643,
+    "z": 3.21935731233074,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5844155844155844,
+    "z": 5.164137942140752,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.018950437317784254,
+    "z": 2.2825479994014817,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.45982142857142855,
+    "z": 3.9880845795075097,
+    "n_pairs": 96
+   }
+  },
+  "J|StackExchange|s2": {
+   "multihop": {
+    "rank": 0.4091539528432732,
+    "z": 4.10576043140541,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.2411167512690355,
+    "z": 3.2908637373883107,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5909090909090909,
+    "z": 5.258090955560857,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.021865889212827987,
+    "z": 2.3098818407374986,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.4613095238095238,
+    "z": 3.9905417760213218,
+    "n_pairs": 96
+   }
+  },
+  "J|Github|s0": {
+   "multihop": {
+    "rank": 0.30374479889042993,
+    "z": 3.6555041063179092,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.13379260333575055,
+    "z": 2.7891979518578136,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5571428571428572,
+    "z": 5.291175675392151,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.011661807580174929,
+    "z": 2.0492619884257413,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.20833333333333334,
+    "z": 3.1940517015755177,
+    "n_pairs": 96
+   }
+  },
+  "J|Github|s1": {
+   "multihop": {
+    "rank": 0.32871012482662965,
+    "z": 3.815705211417189,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.16678752719361858,
+    "z": 2.954350498122007,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5844155844155844,
+    "z": 5.422900062257593,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.013119533527696792,
+    "z": 2.103769786199745,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.2604166666666667,
+    "z": 3.3664332094291844,
+    "n_pairs": 96
+   }
+  },
+  "J|Github|s2": {
+   "multihop": {
+    "rank": 0.3398058252427184,
+    "z": 3.787404161055111,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.14176939811457578,
+    "z": 2.774661488793223,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5701298701298702,
+    "z": 5.435368727553975,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.01603498542274052,
+    "z": 2.054967626625178,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.22470238095238093,
+    "z": 3.2467462296287217,
+    "n_pairs": 96
+   }
+  }
+ },
+ "fits": {
+  "J|Wikipedia_en|s0": {
+   "n_prompts": 25,
+   "fit_seconds": 272.6,
+   "prompts_sha256": "9e54a0769bd230f55ba825a256e03c0f34730227e20dfc1b75efc797f711db71",
+   "corpus": "Wikipedia_en",
+   "seed_block": 0,
+   "fro_norm_per_layer": {
+    "9": 23.294700622558594,
+    "10": 25.415515899658203,
+    "11": 28.33647918701172,
+    "12": 30.57115936279297,
+    "13": 33.302181243896484,
+    "14": 36.702049255371094,
+    "15": 40.84243392944336,
+    "16": 42.165000915527344,
+    "17": 44.4361457824707,
+    "18": 44.770687103271484,
+    "19": 44.682952880859375,
+    "20": 45.84259033203125,
+    "21": 46.23477554321289
+   }
+  },
+  "J|Wikipedia_en|s1": {
+   "n_prompts": 25,
+   "fit_seconds": 275.1,
+   "prompts_sha256": "2ee5bb179eac0bd1859f75821abcde75138e042cf6c0d93e091acb78931a9fad",
+   "corpus": "Wikipedia_en",
+   "seed_block": 1,
+   "fro_norm_per_layer": {
+    "9": 23.690460205078125,
+    "10": 25.781587600708008,
+    "11": 28.686138153076172,
+    "12": 30.9707088470459,
+    "13": 33.714210510253906,
+    "14": 37.1748161315918,
+    "15": 41.22645950317383,
+    "16": 42.623661041259766,
+    "17": 44.789859771728516,
+    "18": 45.092281341552734,
+    "19": 44.922061920166016,
+    "20": 46.048622131347656,
+    "21": 46.331031799316406
+   }
+  },
+  "J|Wikipedia_en|s2": {
+   "n_prompts": 25,
+   "fit_seconds": 275.9,
+   "prompts_sha256": "dcd7a38d916ab5bc0adf8bff4400eb59938b7991e201e082cdc0164d7642659b",
+   "corpus": "Wikipedia_en",
+   "seed_block": 2,
+   "fro_norm_per_layer": {
+    "9": 23.420244216918945,
+    "10": 25.618276596069336,
+    "11": 28.51605796813965,
+    "12": 30.880279541015625,
+    "13": 33.711265563964844,
+    "14": 37.20738983154297,
+    "15": 41.2188720703125,
+    "16": 42.609657287597656,
+    "17": 44.832557678222656,
+    "18": 45.15241622924805,
+    "19": 44.87080764770508,
+    "20": 46.04928970336914,
+    "21": 46.32510757446289
+   }
+  },
+  "J|USPTO_Backgrounds|s0": {
+   "n_prompts": 25,
+   "fit_seconds": 276.3,
+   "prompts_sha256": "5a8f7c90631d3924019d98c6c3a93febb4702b0267d73e3bc369a8a1e7b824e4",
+   "corpus": "USPTO_Backgrounds",
+   "seed_block": 0,
+   "fro_norm_per_layer": {
+    "9": 27.90863037109375,
+    "10": 30.208759307861328,
+    "11": 32.88587951660156,
+    "12": 35.370208740234375,
+    "13": 38.3754768371582,
+    "14": 41.75584411621094,
+    "15": 45.2714958190918,
+    "16": 46.36663818359375,
+    "17": 47.852264404296875,
+    "18": 47.41462707519531,
+    "19": 46.47177505493164,
+    "20": 47.27696228027344,
+    "21": 46.95705032348633
+   }
+  },
+  "J|USPTO_Backgrounds|s1": {
+   "n_prompts": 25,
+   "fit_seconds": 276.5,
+   "prompts_sha256": "2ec94c2fa4eb9cb721a09d8f1ecd51be4392b6c77ff95f9b66fb173812f04aa3",
+   "corpus": "USPTO_Backgrounds",
+   "seed_block": 1,
+   "fro_norm_per_layer": {
+    "9": 28.49028778076172,
+    "10": 30.822561264038086,
+    "11": 33.30125045776367,
+    "12": 35.712852478027344,
+    "13": 38.649085998535156,
+    "14": 41.987709045410156,
+    "15": 45.43867111206055,
+    "16": 46.37141036987305,
+    "17": 47.787593841552734,
+    "18": 47.34893035888672,
+    "19": 46.38490676879883,
+    "20": 47.20484924316406,
+    "21": 46.929969787597656
+   }
+  },
+  "J|USPTO_Backgrounds|s2": {
+   "n_prompts": 25,
+   "fit_seconds": 276.4,
+   "prompts_sha256": "97174f19e4276737d5949b098c9ee4c669f51f178d9d499b1a5c86b6faf4afad",
+   "corpus": "USPTO_Backgrounds",
+   "seed_block": 2,
+   "fro_norm_per_layer": {
+    "9": 28.800617218017578,
+    "10": 31.102834701538086,
+    "11": 33.716331481933594,
+    "12": 36.08961486816406,
+    "13": 39.015830993652344,
+    "14": 42.280330657958984,
+    "15": 45.75420379638672,
+    "16": 46.61201477050781,
+    "17": 47.96596145629883,
+    "18": 47.46554946899414,
+    "19": 46.47589111328125,
+    "20": 47.297393798828125,
+    "21": 46.97040939331055
+   }
+  },
+  "J|Pile-CC|s0": {
+   "n_prompts": 25,
+   "fit_seconds": 276.6,
+   "prompts_sha256": "e0757ef16e1a8a552d467cf839e49c86eb7e90c7f090efa560e063141bf88c8e",
+   "corpus": "Pile-CC",
+   "seed_block": 0,
+   "fro_norm_per_layer": {
+    "9": 24.955005645751953,
+    "10": 27.270200729370117,
+    "11": 29.907175064086914,
+    "12": 32.539894104003906,
+    "13": 35.55636215209961,
+    "14": 39.129798889160156,
+    "15": 43.26397705078125,
+    "16": 44.49295425415039,
+    "17": 46.42964553833008,
+    "18": 46.149452209472656,
+    "19": 45.46807861328125,
+    "20": 46.5279541015625,
+    "21": 46.48268508911133
+   }
+  },
+  "J|Pile-CC|s1": {
+   "n_prompts": 25,
+   "fit_seconds": 276.7,
+   "prompts_sha256": "3dfe7843f22cef9b3d308d1f1cec3cade456f380b53ba1b8db8776ebe7fed77c",
+   "corpus": "Pile-CC",
+   "seed_block": 1,
+   "fro_norm_per_layer": {
+    "9": 25.38190269470215,
+    "10": 27.569387435913086,
+    "11": 30.241737365722656,
+    "12": 32.736114501953125,
+    "13": 35.672340393066406,
+    "14": 39.08313751220703,
+    "15": 43.28975296020508,
+    "16": 44.42156219482422,
+    "17": 46.440616607666016,
+    "18": 46.22185516357422,
+    "19": 45.58104705810547,
+    "20": 46.608673095703125,
+    "21": 46.55390930175781
+   }
+  },
+  "J|Pile-CC|s2": {
+   "n_prompts": 25,
+   "fit_seconds": 276.6,
+   "prompts_sha256": "782515fa0a56b1398a7237cda29cfac2a4e75c3bf91d1ffba99c86a19d3ab5aa",
+   "corpus": "Pile-CC",
+   "seed_block": 2,
+   "fro_norm_per_layer": {
+    "9": 25.042362213134766,
+    "10": 27.282052993774414,
+    "11": 29.89794158935547,
+    "12": 32.4775505065918,
+    "13": 35.40144348144531,
+    "14": 38.882041931152344,
+    "15": 43.07960510253906,
+    "16": 44.226409912109375,
+    "17": 46.159019470214844,
+    "18": 45.96290588378906,
+    "19": 45.31592559814453,
+    "20": 46.45462417602539,
+    "21": 46.471435546875
+   }
+  },
+  "J|StackExchange|s0": {
+   "n_prompts": 25,
+   "fit_seconds": 277.0,
+   "prompts_sha256": "fb838aed6ad4bb5655506149821e49a7be3de1e40422abf3e744afa0fd80df39",
+   "corpus": "StackExchange",
+   "seed_block": 0,
+   "fro_norm_per_layer": {
+    "9": 26.929075241088867,
+    "10": 29.292173385620117,
+    "11": 31.924766540527344,
+    "12": 34.485313415527344,
+    "13": 37.15853500366211,
+    "14": 40.80641555786133,
+    "15": 44.48529815673828,
+    "16": 45.89080810546875,
+    "17": 47.45210266113281,
+    "18": 47.164772033691406,
+    "19": 46.41997528076172,
+    "20": 47.250492095947266,
+    "21": 46.89768600463867
+   }
+  },
+  "J|StackExchange|s1": {
+   "n_prompts": 25,
+   "fit_seconds": 276.4,
+   "prompts_sha256": "99d6fe9822aacd471eebbf96e115750a6247e2e1bef7353640f9461dfe1aad6b",
+   "corpus": "StackExchange",
+   "seed_block": 1,
+   "fro_norm_per_layer": {
+    "9": 27.617984771728516,
+    "10": 30.13306427001953,
+    "11": 32.612632751464844,
+    "12": 35.15741729736328,
+    "13": 37.79157257080078,
+    "14": 41.37997055053711,
+    "15": 45.01513671875,
+    "16": 46.328369140625,
+    "17": 47.70050811767578,
+    "18": 47.40119552612305,
+    "19": 46.614158630371094,
+    "20": 47.42560958862305,
+    "21": 47.04204559326172
+   }
+  },
+  "J|StackExchange|s2": {
+   "n_prompts": 25,
+   "fit_seconds": 276.3,
+   "prompts_sha256": "094be840f776043c464c6dc2ada90ee60b7a6349a4faf5edbee6f7646e048e58",
+   "corpus": "StackExchange",
+   "seed_block": 2,
+   "fro_norm_per_layer": {
+    "9": 27.36570930480957,
+    "10": 29.822795867919922,
+    "11": 32.43924331665039,
+    "12": 35.0574836730957,
+    "13": 37.744300842285156,
+    "14": 41.3569221496582,
+    "15": 45.00175476074219,
+    "16": 46.438682556152344,
+    "17": 47.874332427978516,
+    "18": 47.57386779785156,
+    "19": 46.77056884765625,
+    "20": 47.53171157836914,
+    "21": 47.10435104370117
+   }
+  },
+  "J|Github|s0": {
+   "n_prompts": 25,
+   "fit_seconds": 275.9,
+   "prompts_sha256": "fb6856c864ad3d738b939e1ca21b65a129f645bf18dd0221c4c9e2c1dadb7fe5",
+   "corpus": "Github",
+   "seed_block": 0,
+   "fro_norm_per_layer": {
+    "9": 23.499574661254883,
+    "10": 25.101781845092773,
+    "11": 27.236875534057617,
+    "12": 29.104747772216797,
+    "13": 31.28536033630371,
+    "14": 34.06266403198242,
+    "15": 37.80937957763672,
+    "16": 39.80789566040039,
+    "17": 42.34746551513672,
+    "18": 43.70899200439453,
+    "19": 44.678932189941406,
+    "20": 45.65796661376953,
+    "21": 46.37763214111328
+   }
+  },
+  "J|Github|s1": {
+   "n_prompts": 25,
+   "fit_seconds": 275.3,
+   "prompts_sha256": "8acd4ae02d98b1cb29ad26750a7f98b3f9bb842226f217fbc2ab18547092f0a1",
+   "corpus": "Github",
+   "seed_block": 1,
+   "fro_norm_per_layer": {
+    "9": 24.057697296142578,
+    "10": 25.858362197875977,
+    "11": 28.058307647705078,
+    "12": 30.10573387145996,
+    "13": 32.42531204223633,
+    "14": 35.27041244506836,
+    "15": 39.16661071777344,
+    "16": 41.07899856567383,
+    "17": 43.373783111572266,
+    "18": 44.43169403076172,
+    "19": 45.091068267822266,
+    "20": 46.02471160888672,
+    "21": 46.53116226196289
+   }
+  },
+  "J|Github|s2": {
+   "n_prompts": 25,
+   "fit_seconds": 275.3,
+   "prompts_sha256": "fb8e687f84bbaca1821949876dabd6beb24389556d54c1fa6d9234184037fc68",
+   "corpus": "Github",
+   "seed_block": 2,
+   "fro_norm_per_layer": {
+    "9": 23.682188034057617,
+    "10": 25.320621490478516,
+    "11": 27.47364044189453,
+    "12": 29.382030487060547,
+    "13": 31.588592529296875,
+    "14": 34.38534927368164,
+    "15": 38.232765197753906,
+    "16": 40.23326873779297,
+    "17": 42.76146697998047,
+    "18": 44.06857681274414,
+    "19": 44.8614501953125,
+    "20": 45.87016677856445,
+    "21": 46.53526306152344
+   }
+  }
+ },
+ "logit_I_pooled_rank_flatmean7": 0.23942414367877324,
+ "provenance": {
+  "schema_version": 1,
+  "experiment": "CV6",
+  "utc": "2026-08-24T00:26:12+00:00",
+  "script": {
+   "path": "experiments/cv6_per_family_ladder.py",
+   "sha256": "f416c3cb55e95a02315504066479c2f2ba073c0d4676bbf09adb2f481cdb4a5b",
+   "n_lines": 750
+  },
+  "git": {
+   "commit": null,
+   "commit_short": null,
+   "branch": null,
+   "dirty": false,
+   "dirty_files": [],
+   "dirty_diff_sha256": null,
+   "describe": null
+  },
+  "argv": [
+   "experiments/cv6_per_family_ladder.py",
+   "--model",
+   "1.4b",
+   "--device",
+   "cuda",
+   "--cache-out",
+   "results/cv6/ht_cache_1.4b.pt",
+   "--save-lens-dir",
+   "results/cv6/lenses"
+  ],
+  "cwd": ".",
+  "inputs": [
+   {
+    "path": "corpora/Wikipedia_en.jsonl",
+    "sha256": "b30f4cef11add560f7df8ed07a613e3fcc6a4dda865975bec15f2952253bda77",
+    "bytes": 10329196,
+    "exists": true
+   },
+   {
+    "path": "corpora/USPTO_Backgrounds.jsonl",
+    "sha256": "b09ac74bdc8b169ee5c9bea6be481262f743947a92107164f30eca2346e3ccb9",
+    "bytes": 10221054,
+    "exists": true
+   },
+   {
+    "path": "corpora/Pile-CC.jsonl",
+    "sha256": "abbb9f19a6eb6f4c12fb382735832ceec271fece15cfebcf0fe838765d4b1e74",
+    "bytes": 13512530,
+    "exists": true
+   },
+   {
+    "path": "corpora/StackExchange.jsonl",
+    "sha256": "bc25e2d5775e9719ebcbdaa3bd115d827f72ebf11f47fedb377f8b0a69a28971",
+    "bytes": 5980156,
+    "exists": true
+   },
+   {
+    "path": "corpora/Github.jsonl",
+    "sha256": "f2bb459d10cad6c13a8b66c86191bc43df4393ba5b722a5315ea878fad2d1a26",
+    "bytes": 17486378,
+    "exists": true
+   }
+  ],
+  "env": {
+   "python": "3.11.9",
+   "torch": "2.11.0+cu128",
+   "transformers": "5.15.1",
+   "numpy": "2.4.6",
+   "platform": "Linux-5.15.0-157-generic-x86_64-with-glibc2.35",
+   "machine": "x86_64",
+   "hostname": "<redacted for double-blind review>",
+   "tf32": {
+    "allow_tf32_matmul": false,
+    "allow_tf32_cudnn": false,
+    "NVIDIA_TF32_OVERRIDE": "0"
+   }
+  },
+  "payload_sha256": "307871427cdc9dbafbe962115e903a649c5f232dc75ac91163e4dd3832b28687"
+ }
+}

results/cv6/cv6_2.8b_n25.json

@@ -0,0 +1,1416 @@
+{
+ "experiment": "CV6 \u2014 the per-family corpus contrast, one model",
+ "prereg": "docs/experiments/preregs/CV6_per_family_ladder.md",
+ "status": "PRE-REGISTERED",
+ "decision_rule_verbatim": "Evaluated PER FAMILY; the verdict is the count across the five. REPLICATES if R(f,2.8B) >= 10 on >= 4 of 5 families AND ordering tau >= 0.6 on >= 3 of 5. ATTENUATES if R(f,2.8B) < 5 on >= 4 of 5 families. UNCLEAR otherwise, including a split where some families replicate and others do not \u2014 a split is an informative outcome, not a failure. Do not pool to force a verdict. Do not add models to chase a threshold. Do not drop a family.",
+ "model": "EleutherAI/pythia-2.8b-deduped",
+ "short": "2.8b",
+ "n_layers": 32,
+ "d_model": 2560,
+ "band": [
+  12,
+  13,
+  14,
+  15,
+  16,
+  17,
+  18,
+  19,
+  20,
+  21,
+  22,
+  23,
+  24,
+  25,
+  26,
+  27,
+  28,
+  29
+ ],
+ "band_rule": "floor(0.38*L)..floor(0.92*L), layers < L-2",
+ "band_matches_rule": true,
+ "N": 25,
+ "dim_batch": 128,
+ "max_seq_len": 128,
+ "skip_first": 16,
+ "target_layer_arg": -2,
+ "target_layer_effective": 30,
+ "device": "cuda",
+ "dtype": "float32",
+ "tf32": {
+  "matmul_allow_tf32": false,
+  "cudnn_allow_tf32": false,
+  "NVIDIA_TF32_OVERRIDE": "0"
+ },
+ "gpu": "NVIDIA H100 NVL",
+ "gpu_total_gb": 93.08551025390625,
+ "peak_vram_gb": 46.3552303314209,
+ "torch": "2.11.0+cu128",
+ "git_commit_at_launch": "518d192e9e215603957c85ac4a1308153bf4cf06",
+ "K": [
+  1,
+  2,
+  5,
+  10,
+  20,
+  50,
+  100
+ ],
+ "admitted_sets": [
+  "multihop",
+  "multilingual",
+  "order-ops",
+  "poetry",
+  "typo"
+ ],
+ "corpora": [
+  "Wikipedia_en",
+  "USPTO_Backgrounds",
+  "Pile-CC",
+  "StackExchange",
+  "Github"
+ ],
+ "seeds": [
+  0,
+  1,
+  2
+ ],
+ "readout_convention": "STRIPPED (corrected), no prefix, flat-mean-7",
+ "pooling_convention": "flat pool over (item, intermediate) pairs WITHIN a family; rank = min over band then flat-mean-7 pass@k; z = max over band then mean over pairs",
+ "n_items": 449,
+ "n_pairs": 801,
+ "n_pairs_per_family": {
+  "multihop": 103,
+  "multilingual": 394,
+  "order-ops": 110,
+  "poetry": 98,
+  "typo": 96
+ },
+ "per_arm": {
+  "logit_I": {
+   "multihop": {
+    "rank": 0.27600554785020803,
+    "z": 3.693257671826094,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.22588832487309643,
+    "z": 3.0744364978603755,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.538961038961039,
+    "z": 5.5365345900708975,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.02769679300291545,
+    "z": 2.430429516398177,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.28125,
+    "z": 3.6758319437503815,
+    "n_pairs": 96
+   }
+  },
+  "J|Wikipedia_en|s0": {
+   "multihop": {
+    "rank": 0.37170596393897365,
+    "z": 3.748083519704134,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.3176214648295867,
+    "z": 3.4327350325390773,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5688311688311688,
+    "z": 5.169967714222994,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.06268221574344024,
+    "z": 2.520261741414362,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.2976190476190476,
+    "z": 3.459335678567489,
+    "n_pairs": 96
+   }
+  },
+  "J|Wikipedia_en|s1": {
+   "multihop": {
+    "rank": 0.37586685159500693,
+    "z": 3.7837847610121793,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.31725888324873097,
+    "z": 3.460326204445156,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5844155844155845,
+    "z": 5.257566994970495,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.06268221574344023,
+    "z": 2.5308002853880125,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.3199404761904762,
+    "z": 3.5066992342472076,
+    "n_pairs": 96
+   }
+  },
+  "J|Wikipedia_en|s2": {
+   "multihop": {
+    "rank": 0.36893203883495146,
+    "z": 3.7126920130646344,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.3110949963741842,
+    "z": 3.4116927991966306,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5805194805194805,
+    "z": 5.221894264221191,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.06268221574344023,
+    "z": 2.5136427721198724,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.3095238095238096,
+    "z": 3.4875277963777385,
+    "n_pairs": 96
+   }
+  },
+  "J|USPTO_Backgrounds|s0": {
+   "multihop": {
+    "rank": 0.420249653259362,
+    "z": 3.9431607584351474,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.3328498912255258,
+    "z": 3.617218898637646,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.6246753246753247,
+    "z": 5.513402173735878,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.08454810495626822,
+    "z": 2.645613257982293,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.42708333333333337,
+    "z": 3.7516257812579474,
+    "n_pairs": 96
+   }
+  },
+  "J|USPTO_Backgrounds|s1": {
+   "multihop": {
+    "rank": 0.4133148404993065,
+    "z": 3.9598770234191303,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.3390137781000725,
+    "z": 3.662088529410096,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.6571428571428571,
+    "z": 5.61998973976482,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.09329446064139943,
+    "z": 2.6892670818737576,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.44345238095238093,
+    "z": 3.785833690315485,
+    "n_pairs": 96
+   }
+  },
+  "J|USPTO_Backgrounds|s2": {
+   "multihop": {
+    "rank": 0.39944521497919555,
+    "z": 3.9532608187314375,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.34590282813633066,
+    "z": 3.7125451938452456,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.6376623376623376,
+    "z": 5.639909208904613,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.10787172011661807,
+    "z": 2.747300364533249,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.44642857142857145,
+    "z": 3.8546179458498955,
+    "n_pairs": 96
+   }
+  },
+  "J|Pile-CC|s0": {
+   "multihop": {
+    "rank": 0.3828016643550624,
+    "z": 3.9039036672092178,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.3332124728063815,
+    "z": 3.6563512620889598,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5922077922077922,
+    "z": 5.404743753780018,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.06851311953352769,
+    "z": 2.5789867603048986,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.3630952380952381,
+    "z": 3.8045580573379993,
+    "n_pairs": 96
+   }
+  },
+  "J|Pile-CC|s1": {
+   "multihop": {
+    "rank": 0.3883495145631068,
+    "z": 3.8896664404174657,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.3234227701232778,
+    "z": 3.567083712913058,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5818181818181819,
+    "z": 5.46329207257791,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.05393586005830903,
+    "z": 2.606388902177616,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.40476190476190477,
+    "z": 3.885098062455654,
+    "n_pairs": 96
+   }
+  },
+  "J|Pile-CC|s2": {
+   "multihop": {
+    "rank": 0.391123439667129,
+    "z": 3.9153419918226966,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.33575054387237124,
+    "z": 3.6062580548264656,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5909090909090909,
+    "z": 5.418533691492947,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.06705539358600583,
+    "z": 2.5475755419049944,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.38244047619047616,
+    "z": 3.770227551460266,
+    "n_pairs": 96
+   }
+  },
+  "J|StackExchange|s0": {
+   "multihop": {
+    "rank": 0.42718446601941745,
+    "z": 4.062357205789066,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.33575054387237124,
+    "z": 3.7145613397135953,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5675324675324674,
+    "z": 5.500116468559612,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.05830903790087464,
+    "z": 2.4863040441153,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.40029761904761907,
+    "z": 3.8565272353589535,
+    "n_pairs": 96
+   }
+  },
+  "J|StackExchange|s1": {
+   "multihop": {
+    "rank": 0.40083217753120665,
+    "z": 3.945768759088609,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.34699057287889773,
+    "z": 3.717590422497183,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5597402597402598,
+    "z": 5.4544677734375,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.05830903790087464,
+    "z": 2.4743533736589005,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.40625,
+    "z": 3.858109233280023,
+    "n_pairs": 96
+   }
+  },
+  "J|StackExchange|s2": {
+   "multihop": {
+    "rank": 0.42857142857142866,
+    "z": 4.065541328735722,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.3328498912255258,
+    "z": 3.696884382043393,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5857142857142856,
+    "z": 5.543083320964467,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.05393586005830903,
+    "z": 2.4796542653015683,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.4032738095238095,
+    "z": 3.8242265147467456,
+    "n_pairs": 96
+   }
+  },
+  "J|Github|s0": {
+   "multihop": {
+    "rank": 0.32454923717059636,
+    "z": 3.357145009688961,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.23567802755620013,
+    "z": 2.9966616588195567,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5999999999999999,
+    "z": 5.133839065378362,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.030612244897959186,
+    "z": 2.274473075964013,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.2202380952380952,
+    "z": 3.1765695475041866,
+    "n_pairs": 96
+   }
+  },
+  "J|Github|s1": {
+   "multihop": {
+    "rank": 0.348127600554785,
+    "z": 3.5227397807593483,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.2646845540246555,
+    "z": 3.135737712764498,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5896103896103896,
+    "z": 5.305073720758611,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.039358600583090375,
+    "z": 2.350923566185698,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.27232142857142855,
+    "z": 3.336014381299416,
+    "n_pairs": 96
+   }
+  },
+  "J|Github|s2": {
+   "multihop": {
+    "rank": 0.32316227461858527,
+    "z": 3.391732178845452,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.2548948513415518,
+    "z": 3.034613796599625,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5935064935064934,
+    "z": 5.225355126641014,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.03498542274052478,
+    "z": 2.3192731969210567,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.25595238095238093,
+    "z": 3.23986042663455,
+    "n_pairs": 96
+   }
+  }
+ },
+ "per_arm_fp16_roundtrip": {
+  "J|Wikipedia_en|s0": {
+   "multihop": {
+    "rank": 0.37170596393897365,
+    "z": 3.7479759431579738,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.3176214648295867,
+    "z": 3.4326538239336255,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5688311688311688,
+    "z": 5.170113795453852,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.06268221574344024,
+    "z": 2.5202506519093806,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.2976190476190476,
+    "z": 3.459295087804397,
+    "n_pairs": 96
+   }
+  },
+  "J|Wikipedia_en|s1": {
+   "multihop": {
+    "rank": 0.37586685159500693,
+    "z": 3.78365947783572,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.31725888324873097,
+    "z": 3.4602457260722437,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5844155844155845,
+    "z": 5.2574653549627826,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.061224489795918366,
+    "z": 2.5307455500777887,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.32142857142857145,
+    "z": 3.5066481679677963,
+    "n_pairs": 96
+   }
+  },
+  "J|Wikipedia_en|s2": {
+   "multihop": {
+    "rank": 0.36893203883495146,
+    "z": 3.7126931484463146,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.3110949963741842,
+    "z": 3.4117022766377114,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5805194805194805,
+    "z": 5.221943621201949,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.06268221574344023,
+    "z": 2.513642885247055,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.3095238095238096,
+    "z": 3.487534542878469,
+    "n_pairs": 96
+   }
+  },
+  "J|USPTO_Backgrounds|s0": {
+   "multihop": {
+    "rank": 0.420249653259362,
+    "z": 3.943197340641207,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.3328498912255258,
+    "z": 3.617224749272245,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.6246753246753247,
+    "z": 5.513433169234883,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.08454810495626822,
+    "z": 2.6456040983297386,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.42708333333333337,
+    "z": 3.7516461834311485,
+    "n_pairs": 96
+   }
+  },
+  "J|USPTO_Backgrounds|s1": {
+   "multihop": {
+    "rank": 0.4133148404993065,
+    "z": 3.9599806452260435,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.3390137781000725,
+    "z": 3.6621401048251214,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.6571428571428571,
+    "z": 5.619987377253445,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.09329446064139943,
+    "z": 2.6892731098496183,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.44345238095238093,
+    "z": 3.7858387008309364,
+    "n_pairs": 96
+   }
+  },
+  "J|USPTO_Backgrounds|s2": {
+   "multihop": {
+    "rank": 0.39944521497919555,
+    "z": 3.9531577559350763,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.34590282813633066,
+    "z": 3.712495117471908,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.6376623376623376,
+    "z": 5.639850367199291,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.10787172011661807,
+    "z": 2.747182385045655,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.44642857142857145,
+    "z": 3.854549737026294,
+    "n_pairs": 96
+   }
+  },
+  "J|Pile-CC|s0": {
+   "multihop": {
+    "rank": 0.3828016643550624,
+    "z": 3.903921564805855,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.3332124728063815,
+    "z": 3.6563817554621525,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5922077922077922,
+    "z": 5.404603629762476,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.06851311953352769,
+    "z": 2.5789884632947495,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.3630952380952381,
+    "z": 3.804555675635735,
+    "n_pairs": 96
+   }
+  },
+  "J|Pile-CC|s1": {
+   "multihop": {
+    "rank": 0.3883495145631068,
+    "z": 3.8897387078664836,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.3234227701232778,
+    "z": 3.56714168538902,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5818181818181819,
+    "z": 5.463297904621471,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.05393586005830903,
+    "z": 2.6064579340876364,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.40476190476190477,
+    "z": 3.8851715164879956,
+    "n_pairs": 96
+   }
+  },
+  "J|Pile-CC|s2": {
+   "multihop": {
+    "rank": 0.391123439667129,
+    "z": 3.9151969303205174,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.33575054387237124,
+    "z": 3.606160680805971,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5909090909090909,
+    "z": 5.418469522216103,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.06705539358600583,
+    "z": 2.5475342316286906,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.38244047619047616,
+    "z": 3.7702168126900992,
+    "n_pairs": 96
+   }
+  },
+  "J|StackExchange|s0": {
+   "multihop": {
+    "rank": 0.42718446601941745,
+    "z": 4.062260240027048,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.3353879622915156,
+    "z": 3.7145396611714725,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5675324675324674,
+    "z": 5.500007843971252,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.05830903790087464,
+    "z": 2.4862898527359474,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.40029761904761907,
+    "z": 3.856445398181677,
+    "n_pairs": 96
+   }
+  },
+  "J|StackExchange|s1": {
+   "multihop": {
+    "rank": 0.40083217753120665,
+    "z": 3.9458269378513964,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.34699057287889773,
+    "z": 3.717639919616244,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5597402597402598,
+    "z": 5.454329214312813,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.05830903790087464,
+    "z": 2.474340495406365,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.40625,
+    "z": 3.8580934616426625,
+    "n_pairs": 96
+   }
+  },
+  "J|StackExchange|s2": {
+   "multihop": {
+    "rank": 0.42857142857142866,
+    "z": 4.065570627601401,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.3328498912255258,
+    "z": 3.6968645884603415,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5857142857142856,
+    "z": 5.5429936354810545,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.05393586005830903,
+    "z": 2.4796188942023685,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.4032738095238095,
+    "z": 3.8242418008546033,
+    "n_pairs": 96
+   }
+  },
+  "J|Github|s0": {
+   "multihop": {
+    "rank": 0.32454923717059636,
+    "z": 3.357211357181512,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.23567802755620013,
+    "z": 2.9966800858829226,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5999999999999999,
+    "z": 5.133947640115564,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.030612244897959186,
+    "z": 2.2744970212177353,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.2202380952380952,
+    "z": 3.176635969430208,
+    "n_pairs": 96
+   }
+  },
+  "J|Github|s1": {
+   "multihop": {
+    "rank": 0.348127600554785,
+    "z": 3.522636915873555,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.2646845540246555,
+    "z": 3.135660663171468,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5896103896103896,
+    "z": 5.3051688779484145,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.039358600583090375,
+    "z": 2.3509351890914294,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.27232142857142855,
+    "z": 3.3360372707247734,
+    "n_pairs": 96
+   }
+  },
+  "J|Github|s2": {
+   "multihop": {
+    "rank": 0.32316227461858527,
+    "z": 3.3917247172698235,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.2548948513415518,
+    "z": 3.034596077681798,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5935064935064934,
+    "z": 5.225318701700731,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.03498542274052478,
+    "z": 2.3192666671714006,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.25595238095238093,
+    "z": 3.2398867520193257,
+    "n_pairs": 96
+   }
+  }
+ },
+ "fits": {
+  "J|Wikipedia_en|s0": {
+   "n_prompts": 25,
+   "fit_seconds": 717.8,
+   "prompts_sha256": "9e54a0769bd230f55ba825a256e03c0f34730227e20dfc1b75efc797f711db71",
+   "corpus": "Wikipedia_en",
+   "seed_block": 0,
+   "fro_norm_per_layer": {
+    "12": 25.47411346435547,
+    "13": 28.14723014831543,
+    "14": 30.691478729248047,
+    "15": 33.2955436706543,
+    "16": 35.893863677978516,
+    "17": 37.72172927856445,
+    "18": 39.467960357666016,
+    "19": 41.63821029663086,
+    "20": 43.01144027709961,
+    "21": 44.35385513305664,
+    "22": 45.25417709350586,
+    "23": 46.21498489379883,
+    "24": 47.24529266357422,
+    "25": 48.25273513793945,
+    "26": 49.452205657958984,
+    "27": 49.94655990600586,
+    "28": 50.0068359375,
+    "29": 50.67486572265625
+   }
+  },
+  "J|Wikipedia_en|s1": {
+   "n_prompts": 25,
+   "fit_seconds": 715.9,
+   "prompts_sha256": "2ee5bb179eac0bd1859f75821abcde75138e042cf6c0d93e091acb78931a9fad",
+   "corpus": "Wikipedia_en",
+   "seed_block": 1,
+   "fro_norm_per_layer": {
+    "12": 25.824872970581055,
+    "13": 28.51796531677246,
+    "14": 31.0869197845459,
+    "15": 33.67380142211914,
+    "16": 36.31361770629883,
+    "17": 38.17379379272461,
+    "18": 39.9384880065918,
+    "19": 42.0767707824707,
+    "20": 43.39258575439453,
+    "21": 44.7066650390625,
+    "22": 45.6107063293457,
+    "23": 46.55977249145508,
+    "24": 47.58478927612305,
+    "25": 48.54484939575195,
+    "26": 49.71868133544922,
+    "27": 50.12144088745117,
+    "28": 50.1476936340332,
+    "29": 50.75876235961914
+   }
+  },
+  "J|Wikipedia_en|s2": {
+   "n_prompts": 25,
+   "fit_seconds": 716.5,
+   "prompts_sha256": "dcd7a38d916ab5bc0adf8bff4400eb59938b7991e201e082cdc0164d7642659b",
+   "corpus": "Wikipedia_en",
+   "seed_block": 2,
+   "fro_norm_per_layer": {
+    "12": 25.764179229736328,
+    "13": 28.572370529174805,
+    "14": 31.217775344848633,
+    "15": 33.85537338256836,
+    "16": 36.57388687133789,
+    "17": 38.43079376220703,
+    "18": 40.218143463134766,
+    "19": 42.400333404541016,
+    "20": 43.739261627197266,
+    "21": 45.03773880004883,
+    "22": 45.866294860839844,
+    "23": 46.75215530395508,
+    "24": 47.74189376831055,
+    "25": 48.71947479248047,
+    "26": 49.884422302246094,
+    "27": 50.27415466308594,
+    "28": 50.24495315551758,
+    "29": 50.82538604736328
+   }
+  },
+  "J|USPTO_Backgrounds|s0": {
+   "n_prompts": 25,
+   "fit_seconds": 715.8,
+   "prompts_sha256": "5a8f7c90631d3924019d98c6c3a93febb4702b0267d73e3bc369a8a1e7b824e4",
+   "corpus": "USPTO_Backgrounds",
+   "seed_block": 0,
+   "fro_norm_per_layer": {
+    "12": 30.236732482910156,
+    "13": 32.75661087036133,
+    "14": 35.58098220825195,
+    "15": 38.3680305480957,
+    "16": 40.91820526123047,
+    "17": 42.26591873168945,
+    "18": 43.79016876220703,
+    "19": 45.786720275878906,
+    "20": 46.8199577331543,
+    "21": 47.73601531982422,
+    "22": 48.22069549560547,
+    "23": 48.858943939208984,
+    "24": 49.517452239990234,
+    "25": 50.26908493041992,
+    "26": 51.17799377441406,
+    "27": 51.0509033203125,
+    "28": 50.70587921142578,
+    "29": 51.15618133544922
+   }
+  },
+  "J|USPTO_Backgrounds|s1": {
+   "n_prompts": 25,
+   "fit_seconds": 714.0,
+   "prompts_sha256": "2ec94c2fa4eb9cb721a09d8f1ecd51be4392b6c77ff95f9b66fb173812f04aa3",
+   "corpus": "USPTO_Backgrounds",
+   "seed_block": 1,
+   "fro_norm_per_layer": {
+    "12": 30.12862777709961,
+    "13": 32.7627067565918,
+    "14": 35.53038024902344,
+    "15": 38.333805084228516,
+    "16": 40.84742736816406,
+    "17": 42.114013671875,
+    "18": 43.58375930786133,
+    "19": 45.52498245239258,
+    "20": 46.5215950012207,
+    "21": 47.474063873291016,
+    "22": 47.912315368652344,
+    "23": 48.55392837524414,
+    "24": 49.21796798706055,
+    "25": 49.9576416015625,
+    "26": 50.92783737182617,
+    "27": 50.82968521118164,
+    "28": 50.54206085205078,
+    "29": 51.04257583618164
+   }
+  },
+  "J|USPTO_Backgrounds|s2": {
+   "n_prompts": 25,
+   "fit_seconds": 713.8,
+   "prompts_sha256": "97174f19e4276737d5949b098c9ee4c669f51f178d9d499b1a5c86b6faf4afad",
+   "corpus": "USPTO_Backgrounds",
+   "seed_block": 2,
+   "fro_norm_per_layer": {
+    "12": 31.08220863342285,
+    "13": 33.626853942871094,
+    "14": 36.36945724487305,
+    "15": 39.220069885253906,
+    "16": 41.716712951660156,
+    "17": 42.856788635253906,
+    "18": 44.255191802978516,
+    "19": 46.21339416503906,
+    "20": 47.17402648925781,
+    "21": 48.00918960571289,
+    "22": 48.380096435546875,
+    "23": 48.957420349121094,
+    "24": 49.573543548583984,
+    "25": 50.27798843383789,
+    "26": 51.223289489746094,
+    "27": 51.076263427734375,
+    "28": 50.735076904296875,
+    "29": 51.1742057800293
+   }
+  },
+  "J|Pile-CC|s0": {
+   "n_prompts": 25,
+   "fit_seconds": 712.9,
+   "prompts_sha256": "e0757ef16e1a8a552d467cf839e49c86eb7e90c7f090efa560e063141bf88c8e",
+   "corpus": "Pile-CC",
+   "seed_block": 0,
+   "fro_norm_per_layer": {
+    "12": 27.326833724975586,
+    "13": 30.332229614257812,
+    "14": 33.242000579833984,
+    "15": 36.10725784301758,
+    "16": 38.962711334228516,
+    "17": 40.80857849121094,
+    "18": 42.63570022583008,
+    "19": 44.908409118652344,
+    "20": 46.073490142822266,
+    "21": 47.05865478515625,
+    "22": 47.51934051513672,
+    "23": 48.309513092041016,
+    "24": 49.009803771972656,
+    "25": 49.740360260009766,
+    "26": 50.79483413696289,
+    "27": 50.875579833984375,
+    "28": 50.6129264831543,
+    "29": 51.0517692565918
+   }
+  },
+  "J|Pile-CC|s1": {
+   "n_prompts": 25,
+   "fit_seconds": 712.5,
+   "prompts_sha256": "3dfe7843f22cef9b3d308d1f1cec3cade456f380b53ba1b8db8776ebe7fed77c",
+   "corpus": "Pile-CC",
+   "seed_block": 1,
+   "fro_norm_per_layer": {
+    "12": 28.01110076904297,
+    "13": 30.7965030670166,
+    "14": 33.55686569213867,
+    "15": 36.41100311279297,
+    "16": 39.08540344238281,
+    "17": 40.86274337768555,
+    "18": 42.52493667602539,
+    "19": 44.74737548828125,
+    "20": 45.96009063720703,
+    "21": 46.924781799316406,
+    "22": 47.40066146850586,
+    "23": 48.23330307006836,
+    "24": 48.95878982543945,
+    "25": 49.670536041259766,
+    "26": 50.714534759521484,
+    "27": 50.77128601074219,
+    "28": 50.543941497802734,
+    "29": 50.97682189941406
+   }
+  },
+  "J|Pile-CC|s2": {
+   "n_prompts": 25,
+   "fit_seconds": 712.7,
+   "prompts_sha256": "782515fa0a56b1398a7237cda29cfac2a4e75c3bf91d1ffba99c86a19d3ab5aa",
+   "corpus": "Pile-CC",
+   "seed_block": 2,
+   "fro_norm_per_layer": {
+    "12": 27.110673904418945,
+    "13": 30.076528549194336,
+    "14": 32.917598724365234,
+    "15": 35.8157844543457,
+    "16": 38.64012908935547,
+    "17": 40.50718307495117,
+    "18": 42.25164794921875,
+    "19": 44.48286437988281,
+    "20": 45.72807312011719,
+    "21": 46.73719787597656,
+    "22": 47.23435592651367,
+    "23": 48.03388214111328,
+    "24": 48.77348327636719,
+    "25": 49.5684928894043,
+    "26": 50.67505645751953,
+    "27": 50.80683898925781,
+    "28": 50.57590866088867,
+    "29": 51.04689025878906
+   }
+  },
+  "J|StackExchange|s0": {
+   "n_prompts": 25,
+   "fit_seconds": 714.2,
+   "prompts_sha256": "fb838aed6ad4bb5655506149821e49a7be3de1e40422abf3e744afa0fd80df39",
+   "corpus": "StackExchange",
+   "seed_block": 0,
+   "fro_norm_per_layer": {
+    "12": 29.31014633178711,
+    "13": 31.878799438476562,
+    "14": 34.66938400268555,
+    "15": 37.237892150878906,
+    "16": 39.83375930786133,
+    "17": 41.555850982666016,
+    "18": 43.22056198120117,
+    "19": 45.463809967041016,
+    "20": 46.5365104675293,
+    "21": 47.511634826660156,
+    "22": 48.09621047973633,
+    "23": 48.869789123535156,
+    "24": 49.59775161743164,
+    "25": 50.24106979370117,
+    "26": 51.18525695800781,
+    "27": 50.98370361328125,
+    "28": 50.5977897644043,
+    "29": 50.98619079589844
+   }
+  },
+  "J|StackExchange|s1": {
+   "n_prompts": 25,
+   "fit_seconds": 714.2,
+   "prompts_sha256": "99d6fe9822aacd471eebbf96e115750a6247e2e1bef7353640f9461dfe1aad6b",
+   "corpus": "StackExchange",
+   "seed_block": 1,
+   "fro_norm_per_layer": {
+    "12": 29.9813289642334,
+    "13": 32.756526947021484,
+    "14": 35.454124450683594,
+    "15": 37.913490295410156,
+    "16": 40.541751861572266,
+    "17": 42.225624084472656,
+    "18": 43.83034896850586,
+    "19": 46.03562545776367,
+    "20": 47.06637954711914,
+    "21": 47.918819427490234,
+    "22": 48.48906707763672,
+    "23": 49.20771408081055,
+    "24": 49.94163131713867,
+    "25": 50.54900360107422,
+    "26": 51.485008239746094,
+    "27": 51.22560501098633,
+    "28": 50.785186767578125,
+    "29": 51.12050247192383
+   }
+  },
+  "J|StackExchange|s2": {
+   "n_prompts": 25,
+   "fit_seconds": 712.4,
+   "prompts_sha256": "094be840f776043c464c6dc2ada90ee60b7a6349a4faf5edbee6f7646e048e58",
+   "corpus": "StackExchange",
+   "seed_block": 2,
+   "fro_norm_per_layer": {
+    "12": 29.62559700012207,
+    "13": 32.32237243652344,
+    "14": 35.15523910522461,
+    "15": 37.623905181884766,
+    "16": 40.21584701538086,
+    "17": 41.93696975708008,
+    "18": 43.60802459716797,
+    "19": 45.85477828979492,
+    "20": 46.900814056396484,
+    "21": 47.83250045776367,
+    "22": 48.398345947265625,
+    "23": 49.1341667175293,
+    "24": 49.83674621582031,
+    "25": 50.47686767578125,
+    "26": 51.41249084472656,
+    "27": 51.16120529174805,
+    "28": 50.70438003540039,
+    "29": 51.053829193115234
+   }
+  },
+  "J|Github|s0": {
+   "n_prompts": 25,
+   "fit_seconds": 712.5,
+   "prompts_sha256": "fb6856c864ad3d738b939e1ca21b65a129f645bf18dd0221c4c9e2c1dadb7fe5",
+   "corpus": "Github",
+   "seed_block": 0,
+   "fro_norm_per_layer": {
+    "12": 25.596206665039062,
+    "13": 27.109172821044922,
+    "14": 29.121456146240234,
+    "15": 30.757375717163086,
+    "16": 32.600433349609375,
+    "17": 34.267913818359375,
+    "18": 35.85880661010742,
+    "19": 37.88335037231445,
+    "20": 39.29946517944336,
+    "21": 40.87211608886719,
+    "22": 42.26652145385742,
+    "23": 43.78268051147461,
+    "24": 45.19640350341797,
+    "25": 46.571468353271484,
+    "26": 48.045780181884766,
+    "27": 48.60406494140625,
+    "28": 49.111106872558594,
+    "29": 50.13490676879883
+   }
+  },
+  "J|Github|s1": {
+   "n_prompts": 25,
+   "fit_seconds": 711.2,
+   "prompts_sha256": "8acd4ae02d98b1cb29ad26750a7f98b3f9bb842226f217fbc2ab18547092f0a1",
+   "corpus": "Github",
+   "seed_block": 1,
+   "fro_norm_per_layer": {
+    "12": 26.054004669189453,
+    "13": 27.861385345458984,
+    "14": 30.03280258178711,
+    "15": 31.84650421142578,
+    "16": 33.80059814453125,
+    "17": 35.55202102661133,
+    "18": 37.19427490234375,
+    "19": 39.25769805908203,
+    "20": 40.61973190307617,
+    "21": 42.10810470581055,
+    "22": 43.381038665771484,
+    "23": 44.776756286621094,
+    "24": 46.08184051513672,
+    "25": 47.326202392578125,
+    "26": 48.72494125366211,
+    "27": 49.15113067626953,
+    "28": 49.486331939697266,
+    "29": 50.350563049316406
+   }
+  },
+  "J|Github|s2": {
+   "n_prompts": 25,
+   "fit_seconds": 711.6,
+   "prompts_sha256": "fb8e687f84bbaca1821949876dabd6beb24389556d54c1fa6d9234184037fc68",
+   "corpus": "Github",
+   "seed_block": 2,
+   "fro_norm_per_layer": {
+    "12": 25.251083374023438,
+    "13": 26.807607650756836,
+    "14": 28.799816131591797,
+    "15": 30.52140235900879,
+    "16": 32.41239929199219,
+    "17": 34.15595245361328,
+    "18": 35.801361083984375,
+    "19": 37.893062591552734,
+    "20": 39.344974517822266,
+    "21": 40.995826721191406,
+    "22": 42.41323471069336,
+    "23": 43.94055938720703,
+    "24": 45.35512924194336,
+    "25": 46.73331069946289,
+    "26": 48.25156784057617,
+    "27": 48.75074768066406,
+    "28": 49.18072509765625,
+    "29": 50.145565032958984
+   }
+  }
+ },
+ "logit_I_pooled_rank_flatmean7": 0.2699603409374518,
+ "provenance": {
+  "schema_version": 1,
+  "experiment": "CV6",
+  "utc": "2026-08-24T03:25:21+00:00",
+  "script": {
+   "path": "experiments/cv6_per_family_ladder.py",
+   "sha256": "f416c3cb55e95a02315504066479c2f2ba073c0d4676bbf09adb2f481cdb4a5b",
+   "n_lines": 750
+  },
+  "git": {
+   "commit": null,
+   "commit_short": null,
+   "branch": null,
+   "dirty": false,
+   "dirty_files": [],
+   "dirty_diff_sha256": null,
+   "describe": null
+  },
+  "argv": [
+   "experiments/cv6_per_family_ladder.py",
+   "--model",
+   "2.8b",
+   "--device",
+   "cuda",
+   "--cache-out",
+   "results/cv6/ht_cache_2.8b.pt",
+   "--save-lens-dir",
+   "results/cv6/lenses"
+  ],
+  "cwd": ".",
+  "inputs": [
+   {
+    "path": "corpora/Wikipedia_en.jsonl",
+    "sha256": "b30f4cef11add560f7df8ed07a613e3fcc6a4dda865975bec15f2952253bda77",
+    "bytes": 10329196,
+    "exists": true
+   },
+   {
+    "path": "corpora/USPTO_Backgrounds.jsonl",
+    "sha256": "b09ac74bdc8b169ee5c9bea6be481262f743947a92107164f30eca2346e3ccb9",
+    "bytes": 10221054,
+    "exists": true
+   },
+   {
+    "path": "corpora/Pile-CC.jsonl",
+    "sha256": "abbb9f19a6eb6f4c12fb382735832ceec271fece15cfebcf0fe838765d4b1e74",
+    "bytes": 13512530,
+    "exists": true
+   },
+   {
+    "path": "corpora/StackExchange.jsonl",
+    "sha256": "bc25e2d5775e9719ebcbdaa3bd115d827f72ebf11f47fedb377f8b0a69a28971",
+    "bytes": 5980156,
+    "exists": true
+   },
+   {
+    "path": "corpora/Github.jsonl",
+    "sha256": "f2bb459d10cad6c13a8b66c86191bc43df4393ba5b722a5315ea878fad2d1a26",
+    "bytes": 17486378,
+    "exists": true
+   }
+  ],
+  "env": {
+   "python": "3.11.9",
+   "torch": "2.11.0+cu128",
+   "transformers": "5.15.1",
+   "numpy": "2.4.6",
+   "platform": "Linux-5.15.0-157-generic-x86_64-with-glibc2.35",
+   "machine": "x86_64",
+   "hostname": "<redacted for double-blind review>",
+   "tf32": {
+    "allow_tf32_matmul": false,
+    "allow_tf32_cudnn": false,
+    "NVIDIA_TF32_OVERRIDE": "0"
+   }
+  },
+  "payload_sha256": "38cc82045d0d27b889fe4cd716dc6f5b6bed8d00cace9a81a394e4ead1f6c012"
+ }
+}

results/cv6/cv6_c0_scorer_equivalence.json

@@ -0,0 +1,171 @@
+{
+ "experiment": "CV6 control C0 \u2014 the scorer reproduces D3 on the stored 410M operators",
+ "status": "CONTROL",
+ "model": "EleutherAI/pythia-410m-deduped",
+ "band": [
+  9,
+  10,
+  11,
+  12,
+  13,
+  14,
+  15,
+  16,
+  17,
+  18,
+  19,
+  20,
+  21
+ ],
+ "device": "cpu",
+ "n_items": 449,
+ "n_pairs": 801,
+ "n_pairs_per_family": {
+  "multihop": 103,
+  "multilingual": 394,
+  "order-ops": 110,
+  "poetry": 98,
+  "typo": 96
+ },
+ "compared_against": "results/d3_corpus_by_family_410m.json",
+ "per_arm_deviation": {
+  "logit_I": {
+   "max_abs_dz": 0.0,
+   "max_abs_drank": 0.0
+  },
+  "Wikipedia_en|s0": {
+   "max_abs_dz": 0.0,
+   "max_abs_drank": 0.0
+  },
+  "Wikipedia_en|s1": {
+   "max_abs_dz": 0.0,
+   "max_abs_drank": 0.0
+  },
+  "Wikipedia_en|s2": {
+   "max_abs_dz": 0.0,
+   "max_abs_drank": 0.0
+  },
+  "USPTO_Backgrounds|s0": {
+   "max_abs_dz": 0.0,
+   "max_abs_drank": 0.0
+  },
+  "USPTO_Backgrounds|s1": {
+   "max_abs_dz": 0.0,
+   "max_abs_drank": 0.0
+  },
+  "USPTO_Backgrounds|s2": {
+   "max_abs_dz": 0.0,
+   "max_abs_drank": 0.0
+  },
+  "Pile-CC|s0": {
+   "max_abs_dz": 0.0,
+   "max_abs_drank": 0.0
+  },
+  "Pile-CC|s1": {
+   "max_abs_dz": 0.0,
+   "max_abs_drank": 0.0
+  },
+  "Pile-CC|s2": {
+   "max_abs_dz": 0.0,
+   "max_abs_drank": 0.0
+  },
+  "StackExchange|s0": {
+   "max_abs_dz": 0.0,
+   "max_abs_drank": 0.0
+  },
+  "StackExchange|s1": {
+   "max_abs_dz": 0.0,
+   "max_abs_drank": 0.0
+  },
+  "StackExchange|s2": {
+   "max_abs_dz": 0.0,
+   "max_abs_drank": 0.0
+  },
+  "Github|s0": {
+   "max_abs_dz": 0.0,
+   "max_abs_drank": 0.0
+  },
+  "Github|s1": {
+   "max_abs_dz": 0.0,
+   "max_abs_drank": 0.0
+  },
+  "Github|s2": {
+   "max_abs_dz": 0.0,
+   "max_abs_drank": 0.0
+  }
+ },
+ "worst_abs_dz": 0.0,
+ "worst_abs_drank": 0.0,
+ "required": "max|dz| <= 1e-6 and max|drank| <= 1e-9 over all 16 arms x 5 families, AND the negative control separates",
+ "negative_control": {
+  "what": "same scorer, LEGACY unstripped readout, identity arm, same D3 reference",
+  "why": "proves C0 can fail; a zero that a wrong readout also produces is not a test",
+  "max_abs_dz": 1.996877357092771,
+  "max_abs_drank": 0.30389610389610383,
+  "required": "max|dz| > 1e-3",
+  "separates": true,
+  "per_family_z_legacy": {
+   "multihop": 2.4813545022195984,
+   "multilingual": 2.652753429515713,
+   "order-ops": 3.0892464117570357,
+   "poetry": 2.5611357868325952,
+   "typo": 3.4371277851363025
+  }
+ },
+ "fires": true,
+ "what_it_does_not_cover": "nothing about the FIT path \u2014 it scores stored operators. It also says nothing about GPU scoring; D2 measured a CUDA-vs-CPU divergence of 2.774e-04 at the cell level.",
+ "provenance": {
+  "schema_version": 1,
+  "experiment": "CV6-C0",
+  "utc": "2026-08-23T23:04:06+00:00",
+  "script": {
+   "path": "experiments/cv6_per_family_ladder.py",
+   "sha256": "b7e6b609a8474750c75dbff539919282e7edd585ec2fba00321ae998e829711f",
+   "n_lines": 742
+  },
+  "git": {
+   "commit": "434bc077b023a5246e826a07cc325f79e6eeabbd",
+   "commit_short": "434bc077b023",
+   "branch": "pythia",
+   "dirty": true,
+   "dirty_files": [
+    "experiments/cv6_per_family_ladder.py",
+    "figures/fA_effect_and_null.png",
+    "figures/fB_validity.png",
+    "figures/make_audit_figures.py"
+   ],
+   "dirty_diff_sha256": null,
+   "describe": "434bc07"
+  },
+  "argv": [
+   "experiments/cv6_per_family_ladder.py",
+   "--c0-reproduce-d3",
+   "--device",
+   "cpu"
+  ],
+  "cwd": ".",
+  "inputs": [
+   {
+    "path": "results/d3_corpus_by_family_410m.json",
+    "sha256": "b4342917486e93290856203f3ec8e32b65d55bb2d389273674210b49c140ad1e",
+    "bytes": 19603,
+    "exists": true
+   }
+  ],
+  "env": {
+   "python": "3.14.2",
+   "torch": "2.13.0",
+   "transformers": "5.14.1",
+   "numpy": "2.5.1",
+   "platform": "macOS-26.5.2-arm64-arm-64bit-Mach-O",
+   "machine": "arm64",
+   "hostname": "<redacted for double-blind review>",
+   "tf32": {
+    "allow_tf32_matmul": false,
+    "allow_tf32_cudnn": false,
+    "NVIDIA_TF32_OVERRIDE": "0"
+   }
+  },
+  "payload_sha256": "ebbd6596928b4d50b4bc0a0034a5708d330f3d8481cc6c8b4511d190428dab36"
+ }
+}

results/cv6/ht_cache_1.4b.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a3d5f7ecfb9c8bacbea1822275e28d3faf73940426c0456183b3c52fb4d83095
+size 47847470

results/cv6/ht_cache_2.8b.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c4cabaca7f8ca0d38312ce257855d1e8870c1429a2f1c56e088608056881da37
+size 82791805

results/cv6/lenses/lens_INSTREAM_Github_1.4b_n25_s0.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c3df30f285ca6a2632ddf5ccda1c0bbeb72b288167df1f5bf63e48aadfa3f68
+size 109057367

results/cv6/lenses/lens_INSTREAM_Github_1.4b_n25_s1.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3aade78c84bf70764a74a54018f1fa39966756618d5b74522194dc89e684d834
+size 109057367

results/cv6/lenses/lens_INSTREAM_Github_1.4b_n25_s2.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b406658637da8ba432429bb262353698d1118b7efb531cb955328a95d5dbf01
+size 109057367

results/cv6/lenses/lens_INSTREAM_Github_2.8b_n25_s0.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:84f242a1abcd7293a4472f5a98fd7b5f3d047cae90890a31aa562dfb94decf5c
+size 235936453

results/cv6/lenses/lens_INSTREAM_Github_2.8b_n25_s1.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f2d16e6011c4f2bdc865b401983bd9a490e98e5316ff0ae507d4b2e4096fd77
+size 235936453

results/cv6/lenses/lens_INSTREAM_Github_2.8b_n25_s2.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6734844c920ecfb5f3f18a4135a88679c6b50d6fe59f676f828917fc484fbab4
+size 235936453

results/cv6/lenses/lens_INSTREAM_Pile-CC_1.4b_n25_s0.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:28fb8e54aae2bea11a122fdba06f6c30b9601a006d282d7872afd0d43b2c3e70
+size 109057386

results/cv6/lenses/lens_INSTREAM_Pile-CC_1.4b_n25_s1.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:af172234b25addf33a0a086a1ecfd995e7914e4d1fdf21d588d74830e7073863
+size 109057386

results/cv6/lenses/lens_INSTREAM_Pile-CC_1.4b_n25_s2.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c6fb07ab61cb06a78913afb87427c9ca324f27a03b60f473edb83291c5c0ac00
+size 109057386

results/cv6/lenses/lens_INSTREAM_Pile-CC_2.8b_n25_s0.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e955f5d339ac9043539b179e56a9a0c59808955179af8e3aec85afc5ed384d73
+size 235936477

results/cv6/lenses/lens_INSTREAM_Pile-CC_2.8b_n25_s1.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ca7836772ad75f173f11aebfc423b61297da58c356e054cc67b0a1ec71416b5f
+size 235936477

results/cv6/lenses/lens_INSTREAM_Pile-CC_2.8b_n25_s2.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:545bdf63efc4706e446d87f8e813e992fcc3461a3a6ff074ed90564fb3ab06f1
+size 235936477

results/cv6/lenses/lens_INSTREAM_StackExchange_1.4b_n25_s0.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:469b8052d4a296314823529130a6bb8610870b860ef2c4ecbc2970c3b480b90a
+size 109057564

results/cv6/lenses/lens_INSTREAM_StackExchange_1.4b_n25_s1.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1a9f037db52482669dc656456f0643a2c290e246866875fae8e36de261ae8a52
+size 109057564

results/cv6/lenses/lens_INSTREAM_StackExchange_1.4b_n25_s2.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:07992cfcbb3d32fbeb1871f4aa79f003c6f43c4ba24ce977ae0b874f1f08a8b3
+size 109057564

results/cv6/lenses/lens_INSTREAM_StackExchange_2.8b_n25_s0.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c38d60e2daff2203100167c2b4862a31486e3fc8e70c972af4441438f7e6418e
+size 235936685

results/cv6/lenses/lens_INSTREAM_StackExchange_2.8b_n25_s1.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:45fc266499b5b97f6d5ef4d5671558a30f561430db0cd200870213950106b05d
+size 235936685

results/cv6/lenses/lens_INSTREAM_StackExchange_2.8b_n25_s2.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3438dec229cfcacf26aef81d273c2080373e5d8d5ca3dc43eb1d1280de351c1e
+size 235936685

results/cv6/lenses/lens_INSTREAM_USPTO_Backgrounds_1.4b_n25_s0.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9d068456087b18d29fd38f37345ca881c14311fb97d80a280ec3adaf12cad3c5
+size 109057640

results/cv6/lenses/lens_INSTREAM_USPTO_Backgrounds_1.4b_n25_s1.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:046eeef5dfb737093c822bec3607c8b22c56d5abb39e450d5aa0f7b4f8ff4265
+size 109057640

results/cv6/lenses/lens_INSTREAM_USPTO_Backgrounds_1.4b_n25_s2.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3f0cd894627cc85caffa5e712da65069534ac996256fbd075a25e938b85e59c8
+size 109057640

results/cv6/lenses/lens_INSTREAM_USPTO_Backgrounds_2.8b_n25_s0.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:09013abe933c8754d1ce9dcf55df4f4fc597bde7502ffb6641341b25e24242ce
+size 235936781

results/cv6/lenses/lens_INSTREAM_USPTO_Backgrounds_2.8b_n25_s1.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:58fd8c7f4fb4d6e589cc559140d205d0fcda3a03ecaa95eaab83bc8e4ba2f743
+size 235936781

results/cv6/lenses/lens_INSTREAM_USPTO_Backgrounds_2.8b_n25_s2.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c9f632a13fb0b1d0eb26b639bfe046d27e7f54980f9df4a09595d9a0287527da
+size 235936781

results/cv6/lenses/lens_INSTREAM_Wikipedia_en_1.4b_n25_s0.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7db6b6e4605bd974a732b9b3501c060be612e0ba39c21c5a780658a1b64a6250
+size 109057545

results/cv6/lenses/lens_INSTREAM_Wikipedia_en_1.4b_n25_s1.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7fa1e2a931ca8454368ae8bcd67a13bc053c89e617b0c005f970b8ebbc813f76
+size 109057545

results/cv6/lenses/lens_INSTREAM_Wikipedia_en_1.4b_n25_s2.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b1763b5e8c3ca65bc26749be894aeaeb8961230491410e7853ffe45e41e77f4e
+size 109057545

results/cv6/lenses/lens_INSTREAM_Wikipedia_en_2.8b_n25_s0.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1f3e2850461bf1e416585181c361f0a5018f163e3af388735e668f2c977a5a64
+size 235936661

results/cv6/lenses/lens_INSTREAM_Wikipedia_en_2.8b_n25_s1.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:29f72e060566702cc51497528e6935786b949a095f1322a4832843ab633a861e
+size 235936661

results/cv6/lenses/lens_INSTREAM_Wikipedia_en_2.8b_n25_s2.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a319a3aebd47c030cc5d3861f3455e91c80812eac3c7711a5d64a970206c9d6c
+size 235936661

results/cv6_per_family_ladder.json

@@ -0,0 +1,1238 @@
+{
+ "experiment": "CV6 \u2014 does the per-family corpus effect replicate at 1.4B and 2.8B?",
+ "prereg": "docs/experiments/preregs/CV6_per_family_ladder.md",
+ "status": "PRE-REGISTERED",
+ "decision_rule_verbatim": "Evaluated PER FAMILY; the verdict is the count across the five. REPLICATES if R(f,2.8B) >= 10 on >= 4 of 5 families AND ordering tau >= 0.6 on >= 3 of 5. ATTENUATES if R(f,2.8B) < 5 on >= 4 of 5 families. UNCLEAR otherwise, including a split where some families replicate and others do not \u2014 a split is an informative outcome, not a failure. Do not pool to force a verdict. Do not add models to chase a threshold. Do not drop a family.",
+ "admitted_sets": [
+  "multihop",
+  "multilingual",
+  "order-ops",
+  "poetry",
+  "typo"
+ ],
+ "corpora": [
+  "Wikipedia_en",
+  "USPTO_Backgrounds",
+  "Pile-CC",
+  "StackExchange",
+  "Github"
+ ],
+ "seeds": [
+  0,
+  1,
+  2
+ ],
+ "N": 25,
+ "K": [
+  1,
+  2,
+  5,
+  10,
+  20,
+  50,
+  100
+ ],
+ "readout_convention": "STRIPPED (corrected), no prefix, flat-mean-7",
+ "pooling_convention": "flat pool over (item, intermediate) pairs WITHIN a family; seed SD is the RMS over the 5 corpora of each corpus's 3-seed sample SD \u2014 trap 4: the convention is named because it moves 'x seed SD' ratios ~37%",
+ "reference_410m": {
+  "source": "results/d3_corpus_by_family_410m.json (EXPLORATORY)",
+  "N": 200,
+  "order_by_z": {
+   "multihop": [
+    "Pile-CC",
+    "StackExchange",
+    "USPTO_Backgrounds",
+    "Wikipedia_en",
+    "Github"
+   ],
+   "multilingual": [
+    "StackExchange",
+    "Pile-CC",
+    "USPTO_Backgrounds",
+    "Wikipedia_en",
+    "Github"
+   ],
+   "order-ops": [
+    "USPTO_Backgrounds",
+    "StackExchange",
+    "Pile-CC",
+    "Wikipedia_en",
+    "Github"
+   ],
+   "poetry": [
+    "Pile-CC",
+    "USPTO_Backgrounds",
+    "StackExchange",
+    "Wikipedia_en",
+    "Github"
+   ],
+   "typo": [
+    "StackExchange",
+    "Pile-CC",
+    "USPTO_Backgrounds",
+    "Wikipedia_en",
+    "Github"
+   ]
+  },
+  "R": {
+   "multihop": 25.36100542891225,
+   "multilingual": 19.426787847831193,
+   "order-ops": 24.632511354365725,
+   "poetry": 42.686224720210525,
+   "typo": 27.014210896207846
+  },
+  "z_pooled_seed_sd": {
+   "multihop": 0.017453481001782855,
+   "multilingual": 0.019695881058294912,
+   "order-ops": 0.012061420042855183,
+   "poetry": 0.005578110253433368,
+   "typo": 0.019451275697770446
+  }
+ },
+ "by_model": {
+  "1.4b": {
+   "model": "EleutherAI/pythia-1.4b-deduped",
+   "band": [
+    9,
+    10,
+    11,
+    12,
+    13,
+    14,
+    15,
+    16,
+    17,
+    18,
+    19,
+    20,
+    21
+   ],
+   "n_layers": 24,
+   "device": "cuda",
+   "dtype": "float32",
+   "gpu": "NVIDIA H100 NVL",
+   "N": 25,
+   "n_pairs_per_family": {
+    "multihop": 103,
+    "multilingual": 394,
+    "order-ops": 110,
+    "poetry": 98,
+    "typo": 96
+   },
+   "logit_I": {
+    "multihop": {
+     "rank": 0.2496532593619972,
+     "z": 3.8530033965712613,
+     "n_pairs": 103
+    },
+    "multilingual": {
+     "rank": 0.1899927483683829,
+     "z": 3.2260746860564664,
+     "n_pairs": 394
+    },
+    "order-ops": {
+     "rank": 0.5597402597402598,
+     "z": 5.515925762870095,
+     "n_pairs": 110
+    },
+    "poetry": {
+     "rank": 0.008746355685131196,
+     "z": 2.4992136784962247,
+     "n_pairs": 98
+    },
+    "typo": {
+     "rank": 0.1889880952380952,
+     "z": 3.7020364006360373,
+     "n_pairs": 96
+    }
+   },
+   "logit_I_pooled_rank_flatmean7": 0.23942414367877324,
+   "by_family": {
+    "multihop": {
+     "per_corpus": {
+      "Wikipedia_en": {
+       "z_mean": 3.849424108332415,
+       "z_sd": 0.00971452307361606,
+       "rank_mean": 0.30883032824780393,
+       "rank_sd": 0.002118620293553323,
+       "n_seeds": 3
+      },
+      "USPTO_Backgrounds": {
+       "z_mean": 3.9101949866535595,
+       "z_sd": 0.03168250526694356,
+       "rank_mean": 0.32223763291724455,
+       "rank_sd": 0.003490445878534788,
+       "n_seeds": 3
+      },
+      "Pile-CC": {
+       "z_mean": 3.988609111617684,
+       "z_sd": 0.038253457386003685,
+       "rank_mean": 0.3226999537679149,
+       "rank_sd": 0.0028871927870542744,
+       "n_seeds": 3
+      },
+      "StackExchange": {
+       "z_mean": 4.083285220618387,
+       "z_sd": 0.019998217439994845,
+       "rank_mean": 0.39667128987517336,
+       "rank_sd": 0.012091259205383278,
+       "n_seeds": 3
+      },
+      "Github": {
+       "z_mean": 3.752807802366979,
+       "z_sd": 0.0855118574336993,
+       "rank_mean": 0.324086916319926,
+       "rank_sd": 0.018469703518666825,
+       "n_seeds": 3
+      }
+     },
+     "z_spread": 0.33047741825140786,
+     "z_pooled_seed_sd": 0.045329210368280526,
+     "R": 7.2906061139477085,
+     "rank_spread": 0.08784096162736943,
+     "rank_pooled_seed_sd": 0.010122609780171063,
+     "rank_spread_over_sd": 8.677699085016492,
+     "order_by_z": [
+      "StackExchange",
+      "Pile-CC",
+      "USPTO_Backgrounds",
+      "Wikipedia_en",
+      "Github"
+     ],
+     "order_by_rank": [
+      "StackExchange",
+      "Github",
+      "Pile-CC",
+      "USPTO_Backgrounds",
+      "Wikipedia_en"
+     ],
+     "n_pairs": 103,
+     "kendall_tau_vs_410m": 0.8,
+     "fp16_roundtrip_diagnostic": {
+      "R": 7.291143850680914,
+      "max_abs_dz_mean": 7.843778357141673e-05,
+      "order_unchanged": true
+     }
+    },
+    "multilingual": {
+     "per_corpus": {
+      "Wikipedia_en": {
+       "z_mean": 3.055601908067358,
+       "z_sd": 0.024883413619364386,
+       "rank_mean": 0.18624607203287408,
+       "rank_sd": 0.0065298248292193,
+       "n_seeds": 3
+      },
+      "USPTO_Backgrounds": {
+       "z_mean": 3.1024229467015982,
+       "z_sd": 0.021925288574017928,
+       "rank_mean": 0.18044476673918297,
+       "rank_sd": 0.0032497775343845316,
+       "n_seeds": 3
+      },
+      "Pile-CC": {
+       "z_mean": 3.2126307500094526,
+       "z_sd": 0.04011617460924262,
+       "rank_mean": 0.19639835629683347,
+       "rank_sd": 0.004202402381124895,
+       "n_seeds": 3
+      },
+      "StackExchange": {
+       "z_mean": 3.2308733033145542,
+       "z_sd": 0.055215016867437655,
+       "rank_mean": 0.2274595117234711,
+       "rank_sd": 0.012943366831597983,
+       "n_seeds": 3
+      },
+      "Github": {
+       "z_mean": 2.8394160118905822,
+       "z_sd": 0.09979349484462473,
+       "rank_mean": 0.14744984288131496,
+       "rank_sd": 0.017215308294004156,
+       "n_seeds": 3
+      }
+     },
+     "z_spread": 0.391457291423972,
+     "z_pooled_seed_sd": 0.05606540772822604,
+     "R": 6.982153653845515,
+     "rank_spread": 0.08000966884215613,
+     "rank_pooled_seed_sd": 0.010341735218918366,
+     "rank_spread_over_sd": 7.736580675145566,
+     "order_by_z": [
+      "StackExchange",
+      "Pile-CC",
+      "USPTO_Backgrounds",
+      "Wikipedia_en",
+      "Github"
+     ],
+     "order_by_rank": [
+      "StackExchange",
+      "Pile-CC",
+      "Wikipedia_en",
+      "USPTO_Backgrounds",
+      "Github"
+     ],
+     "n_pairs": 394,
+     "kendall_tau_vs_410m": 1.0,
+     "fp16_roundtrip_diagnostic": {
+      "R": 6.981554279672338,
+      "max_abs_dz_mean": 2.806884584893865e-05,
+      "order_unchanged": true
+     }
+    },
+    "order-ops": {
+     "per_corpus": {
+      "Wikipedia_en": {
+       "z_mean": 5.321406318924644,
+       "z_sd": 0.05556506407862744,
+       "rank_mean": 0.5632034632034633,
+       "rank_sd": 0.009918995010726954,
+       "n_seeds": 3
+      },
+      "USPTO_Backgrounds": {
+       "z_mean": 5.318626935193033,
+       "z_sd": 0.0436481925352706,
+       "rank_mean": 0.5831168831168831,
+       "rank_sd": 0.011902794012872206,
+       "n_seeds": 3
+      },
+      "Pile-CC": {
+       "z_mean": 5.474697200818495,
+       "z_sd": 0.0237919350357011,
+       "rank_mean": 0.5887445887445888,
+       "rank_sd": 0.0029992221776084464,
+       "n_seeds": 3
+      },
+      "StackExchange": {
+       "z_mean": 5.220926105976105,
+       "z_sd": 0.04991356678409937,
+       "rank_mean": 0.5896103896103896,
+       "rank_sd": 0.004682534123979193,
+       "n_seeds": 3
+      },
+      "Github": {
+       "z_mean": 5.383031076373476,
+       "z_sd": 0.07990919093950227,
+       "rank_mean": 0.5705627705627706,
+       "rank_sd": 0.01364151623947074,
+       "n_seeds": 3
+      }
+     },
+     "z_spread": 0.25377109484239035,
+     "z_pooled_seed_sd": 0.05373181585637624,
+     "R": 4.722920504319339,
+     "rank_spread": 0.026406926406926323,
+     "rank_pooled_seed_sd": 0.009561123282756314,
+     "rank_spread_over_sd": 2.761906276697819,
+     "order_by_z": [
+      "Pile-CC",
+      "Github",
+      "Wikipedia_en",
+      "USPTO_Backgrounds",
+      "StackExchange"
+     ],
+     "order_by_rank": [
+      "StackExchange",
+      "Pile-CC",
+      "USPTO_Backgrounds",
+      "Github",
+      "Wikipedia_en"
+     ],
+     "n_pairs": 110,
+     "kendall_tau_vs_410m": -0.4,
+     "fp16_roundtrip_diagnostic": {
+      "R": 4.72187977619938,
+      "max_abs_dz_mean": 0.00011707869443000618,
+      "order_unchanged": true
+     }
+    },
+    "poetry": {
+     "per_corpus": {
+      "Wikipedia_en": {
+       "z_mean": 2.321828322142971,
+       "z_sd": 0.010642859707299999,
+       "rank_mean": 0.02235179786200194,
+       "rank_sd": 0.0036685298519294196,
+       "n_seeds": 3
+      },
+      "USPTO_Backgrounds": {
+       "z_mean": 2.3693615256929075,
+       "z_sd": 0.016925142442212295,
+       "rank_mean": 0.023323615160349854,
+       "rank_sd": 0.0025248554046193535,
+       "n_seeds": 3
+      },
+      "Pile-CC": {
+       "z_mean": 2.3751317657986464,
+       "z_sd": 0.020213283926543354,
+       "rank_mean": 0.01652089407191448,
+       "rank_sd": 0.00303449854149582,
+       "n_seeds": 3
+      },
+      "StackExchange": {
+       "z_mean": 2.305089985635005,
+       "z_sd": 0.02054782796446748,
+       "rank_mean": 0.020894071914480075,
+       "rank_sd": 0.0016832369364129039,
+       "n_seeds": 3
+      },
+      "Github": {
+       "z_mean": 2.069341406226158,
+       "z_sd": 0.029947195245575223,
+       "rank_mean": 0.013605442176870748,
+       "rank_sd": 0.002226713165673389,
+       "n_seeds": 3
+      }
+     },
+     "z_spread": 0.30579035957248824,
+     "z_pooled_seed_sd": 0.020626950786131512,
+     "R": 14.824797069767857,
+     "rank_spread": 0.009718172983479106,
+     "rank_pooled_seed_sd": 0.002714138006563447,
+     "rank_spread_over_sd": 3.580574370197166,
+     "order_by_z": [
+      "Pile-CC",
+      "USPTO_Backgrounds",
+      "Wikipedia_en",
+      "StackExchange",
+      "Github"
+     ],
+     "order_by_rank": [
+      "USPTO_Backgrounds",
+      "Wikipedia_en",
+      "StackExchange",
+      "Pile-CC",
+      "Github"
+     ],
+     "n_pairs": 98,
+     "kendall_tau_vs_410m": 0.8,
+     "fp16_roundtrip_diagnostic": {
+      "R": 14.82280843880369,
+      "max_abs_dz_mean": 3.022188637524792e-05,
+      "order_unchanged": true
+     }
+    },
+    "typo": {
+     "per_corpus": {
+      "Wikipedia_en": {
+       "z_mean": 3.6687574258281126,
+       "z_sd": 0.005040123266250421,
+       "rank_mean": 0.3373015873015873,
+       "rank_sd": 0.0022731030232916094,
+       "n_seeds": 3
+      },
+      "USPTO_Backgrounds": {
+       "z_mean": 3.990972049534321,
+       "z_sd": 0.049932129992485516,
+       "rank_mean": 0.44990079365079366,
+       "rank_sd": 0.008461667712912667,
+       "n_seeds": 3
+      },
+      "Pile-CC": {
+       "z_mean": 4.020524656607045,
+       "z_sd": 0.013975432149536227,
+       "rank_mean": 0.4047619047619047,
+       "rank_sd": 0.005952380952380959,
+       "n_seeds": 3
+      },
+      "StackExchange": {
+       "z_mean": 3.995124079287052,
+       "z_sd": 0.010104712554199926,
+       "rank_mean": 0.4618055555555555,
+       "rank_sd": 0.0022731030232915794,
+       "n_seeds": 3
+      },
+      "Github": {
+       "z_mean": 3.2690880886382527,
+       "z_sd": 0.08830157076926536,
+       "rank_mean": 0.23015873015873017,
+       "rank_sd": 0.025004133256207473,
+       "n_seeds": 3
+      }
+     },
+     "z_spread": 0.7514365679687924,
+     "z_pooled_seed_sd": 0.04607212990087249,
+     "R": 16.31000280615553,
+     "rank_spread": 0.23164682539682535,
+     "rank_pooled_seed_sd": 0.012186642967521382,
+     "rank_spread_over_sd": 19.0082556791224,
+     "order_by_z": [
+      "Pile-CC",
+      "StackExchange",
+      "USPTO_Backgrounds",
+      "Wikipedia_en",
+      "Github"
+     ],
+     "order_by_rank": [
+      "StackExchange",
+      "USPTO_Backgrounds",
+      "Pile-CC",
+      "Wikipedia_en",
+      "Github"
+     ],
+     "n_pairs": 96,
+     "kendall_tau_vs_410m": 0.8,
+     "fp16_roundtrip_diagnostic": {
+      "R": 16.30612241371698,
+      "max_abs_dz_mean": 2.035333050631749e-05,
+      "order_unchanged": true
+     }
+    }
+   }
+  },
+  "2.8b": {
+   "model": "EleutherAI/pythia-2.8b-deduped",
+   "band": [
+    12,
+    13,
+    14,
+    15,
+    16,
+    17,
+    18,
+    19,
+    20,
+    21,
+    22,
+    23,
+    24,
+    25,
+    26,
+    27,
+    28,
+    29
+   ],
+   "n_layers": 32,
+   "device": "cuda",
+   "dtype": "float32",
+   "gpu": "NVIDIA H100 NVL",
+   "N": 25,
+   "n_pairs_per_family": {
+    "multihop": 103,
+    "multilingual": 394,
+    "order-ops": 110,
+    "poetry": 98,
+    "typo": 96
+   },
+   "logit_I": {
+    "multihop": {
+     "rank": 0.27600554785020803,
+     "z": 3.693257671826094,
+     "n_pairs": 103
+    },
+    "multilingual": {
+     "rank": 0.22588832487309643,
+     "z": 3.0744364978603755,
+     "n_pairs": 394
+    },
+    "order-ops": {
+     "rank": 0.538961038961039,
+     "z": 5.5365345900708975,
+     "n_pairs": 110
+    },
+    "poetry": {
+     "rank": 0.02769679300291545,
+     "z": 2.430429516398177,
+     "n_pairs": 98
+    },
+    "typo": {
+     "rank": 0.28125,
+     "z": 3.6758319437503815,
+     "n_pairs": 96
+    }
+   },
+   "logit_I_pooled_rank_flatmean7": 0.2699603409374518,
+   "by_family": {
+    "multihop": {
+     "per_corpus": {
+      "Wikipedia_en": {
+       "z_mean": 3.7481867645936493,
+       "z_sd": 0.03554648642712381,
+       "rank_mean": 0.37216828478964403,
+       "rank_sd": 0.003490445878534788,
+       "n_seeds": 3
+      },
+      "USPTO_Backgrounds": {
+       "z_mean": 3.9520995335285716,
+       "z_sd": 0.008418421244240502,
+       "rank_mean": 0.4110032362459547,
+       "rank_sd": 0.010593101467766616,
+       "n_seeds": 3
+      },
+      "Pile-CC": {
+       "z_mean": 3.90297069981646,
+       "z_sd": 0.01286317636152229,
+       "rank_mean": 0.3874248728617661,
+       "rank_sd": 0.004237240587106646,
+       "n_seeds": 3
+      },
+      "StackExchange": {
+       "z_mean": 4.024555764537799,
+       "z_sd": 0.06825011963705907,
+       "rank_mean": 0.41886269070735094,
+       "rank_sd": 0.0156302741603116,
+       "n_seeds": 3
+      },
+      "Github": {
+       "z_mean": 3.4238723230979202,
+       "z_sd": 0.08735072241195618,
+       "rank_mean": 0.3319463707813222,
+       "rank_sd": 0.014030504766984291,
+       "n_seeds": 3
+      }
+     },
+     "z_spread": 0.6006834414398785,
+     "z_pooled_seed_sd": 0.052513067623269356,
+     "R": 11.43874217650363,
+     "rank_spread": 0.08691631992602872,
+     "rank_pooled_seed_sd": 0.010802886218722078,
+     "rank_spread_over_sd": 8.045657259205164,
+     "order_by_z": [
+      "StackExchange",
+      "USPTO_Backgrounds",
+      "Pile-CC",
+      "Wikipedia_en",
+      "Github"
+     ],
+     "order_by_rank": [
+      "StackExchange",
+      "USPTO_Backgrounds",
+      "Pile-CC",
+      "Wikipedia_en",
+      "Github"
+     ],
+     "n_pairs": 103,
+     "kendall_tau_vs_410m": 0.6,
+     "fp16_roundtrip_diagnostic": {
+      "R": 11.450690222634297,
+      "max_abs_dz_mean": 7.72414469798477e-05,
+      "order_unchanged": true
+     }
+    },
+    "multilingual": {
+     "per_corpus": {
+      "Wikipedia_en": {
+       "z_mean": 3.434918012060288,
+       "z_sd": 0.02439008151297054,
+       "rank_mean": 0.31532511481750064,
+       "rank_sd": 0.003667873073602506,
+       "n_seeds": 3
+      },
+      "USPTO_Backgrounds": {
+       "z_mean": 3.663950873964329,
+       "z_sd": 0.04769042760272363,
+       "rank_mean": 0.3392554991539763,
+       "rank_sd": 0.00652982482921927,
+       "n_seeds": 3
+      },
+      "Pile-CC": {
+       "z_mean": 3.6098976766094943,
+       "z_sd": 0.04474493232770196,
+       "rank_mean": 0.3307952622673435,
+       "rank_sd": 0.006509660628230673,
+       "n_seeds": 3
+      },
+      "StackExchange": {
+       "z_mean": 3.7096787147513903,
+       "z_sd": 0.011183248531158265,
+       "rank_mean": 0.3385303359922649,
+       "rank_sd": 0.007468945881305459,
+       "n_seeds": 3
+      },
+      "Github": {
+       "z_mean": 3.0556710560612266,
+       "z_sd": 0.07188945212789528,
+       "rank_mean": 0.2517524776408025,
+       "rank_sd": 0.014756372480988393,
+       "n_seeds": 3
+      }
+     },
+     "z_spread": 0.6540076586901638,
+     "z_pooled_seed_sd": 0.04508773736862208,
+     "R": 14.505222414316746,
+     "rank_spread": 0.0875030215131738,
+     "rank_pooled_seed_sd": 0.00862558134915699,
+     "rank_spread_over_sd": 10.144594082545613,
+     "order_by_z": [
+      "StackExchange",
+      "USPTO_Backgrounds",
+      "Pile-CC",
+      "Wikipedia_en",
+      "Github"
+     ],
+     "order_by_rank": [
+      "USPTO_Backgrounds",
+      "StackExchange",
+      "Pile-CC",
+      "Wikipedia_en",
+      "Github"
+     ],
+     "n_pairs": 394,
+     "kendall_tau_vs_410m": 0.8,
+     "fp16_roundtrip_diagnostic": {
+      "R": 14.514366209896588,
+      "max_abs_dz_mean": 5.073651242781807e-05,
+      "order_unchanged": true
+     }
+    },
+    "order-ops": {
+     "per_corpus": {
+      "Wikipedia_en": {
+       "z_mean": 5.21647632447156,
+       "z_sd": 0.04405024461019303,
+       "rank_mean": 0.577922077922078,
+       "rank_sd": 0.008110387010907067,
+       "n_seeds": 3
+      },
+      "USPTO_Backgrounds": {
+       "z_mean": 5.591100374135104,
+       "z_sd": 0.06802171764709751,
+       "rank_mean": 0.6398268398268399,
+       "rank_sd": 0.016341632976776498,
+       "n_seeds": 3
+      },
+      "Pile-CC": {
+       "z_mean": 5.428856505950292,
+       "z_sd": 0.03060876969522738,
+       "rank_mean": 0.5883116883116883,
+       "rank_sd": 0.005660907718883943,
+       "n_seeds": 3
+      },
+      "StackExchange": {
+       "z_mean": 5.4992225209871926,
+       "z_sd": 0.04431453680873469,
+       "rank_mean": 0.570995670995671,
+       "rank_sd": 0.013328834907654201,
+       "n_seeds": 3
+      },
+      "Github": {
+       "z_mean": 5.221422637592663,
+       "z_sd": 0.08568503459405673,
+       "rank_mean": 0.5943722943722943,
+       "rank_sd": 0.005248638810814727,
+       "n_seeds": 3
+      }
+     },
+     "z_spread": 0.37462404966354423,
+     "z_pooled_seed_sd": 0.057982718511798266,
+     "R": 6.460960425429451,
+     "rank_spread": 0.06883116883116891,
+     "rank_pooled_seed_sd": 0.010677820670698406,
+     "rank_spread_over_sd": 6.446181384189407,
+     "order_by_z": [
+      "USPTO_Backgrounds",
+      "StackExchange",
+      "Pile-CC",
+      "Github",
+      "Wikipedia_en"
+     ],
+     "order_by_rank": [
+      "USPTO_Backgrounds",
+      "Github",
+      "Pile-CC",
+      "Wikipedia_en",
+      "StackExchange"
+     ],
+     "n_pairs": 110,
+     "kendall_tau_vs_410m": 0.8,
+     "fp16_roundtrip_diagnostic": {
+      "R": 6.4624203254229515,
+      "max_abs_dz_mean": 0.00011228973215260396,
+      "order_unchanged": true
+     }
+    },
+    "poetry": {
+     "per_corpus": {
+      "Wikipedia_en": {
+       "z_mean": 2.5215682663074155,
+       "z_sd": 0.008653052690162434,
+       "rank_mean": 0.06268221574344023,
+       "rank_sd": 8.012344526598183e-18,
+       "n_seeds": 3
+      },
+      "USPTO_Backgrounds": {
+       "z_mean": 2.6940602347964333,
+       "z_sd": 0.051012720432079404,
+       "rank_mean": 0.09523809523809523,
+       "rank_sd": 0.011782658554890317,
+       "n_seeds": 3
+      },
+      "Pile-CC": {
+       "z_mean": 2.577650401462503,
+       "z_sd": 0.029429444912468822,
+       "rank_mean": 0.06316812439261418,
+       "rank_sd": 0.008028528494586155,
+       "n_seeds": 3
+      },
+      "StackExchange": {
+       "z_mean": 2.4801038943585896,
+       "z_sd": 0.005988009335835477,
+       "rank_mean": 0.05685131195335277,
+       "rank_sd": 0.0025248554046193596,
+       "n_seeds": 3
+      },
+      "Github": {
+       "z_mean": 2.3148899463569226,
+       "z_sd": 0.03841326630104487,
+       "rank_mean": 0.03498542274052478,
+       "rank_sd": 0.004373177842565594,
+       "n_seeds": 3
+      }
+     },
+     "z_spread": 0.3791702884395107,
+     "z_pooled_seed_sd": 0.031795284173340224,
+     "R": 11.925362464834903,
+     "rank_spread": 0.06025267249757045,
+     "rank_pooled_seed_sd": 0.006764434812553877,
+     "rank_spread_over_sd": 8.907273729026057,
+     "order_by_z": [
+      "USPTO_Backgrounds",
+      "Pile-CC",
+      "Wikipedia_en",
+      "StackExchange",
+      "Github"
+     ],
+     "order_by_rank": [
+      "USPTO_Backgrounds",
+      "Pile-CC",
+      "Wikipedia_en",
+      "StackExchange",
+      "Github"
+     ],
+     "n_pairs": 98,
+     "kendall_tau_vs_410m": 0.6,
+     "fp16_roundtrip_diagnostic": {
+      "R": 11.928150397522604,
+      "max_abs_dz_mean": 4.037038809601157e-05,
+      "order_unchanged": true
+     }
+    },
+    "typo": {
+     "per_corpus": {
+      "Wikipedia_en": {
+       "z_mean": 3.484520903064145,
+       "z_sd": 0.023824517984755707,
+       "rank_mean": 0.3090277777777778,
+       "rank_sd": 0.011168978421823135,
+       "n_seeds": 3
+      },
+      "USPTO_Backgrounds": {
+       "z_mean": 3.7973591391411095,
+       "z_sd": 0.05245448951050608,
+       "rank_mean": 0.43898809523809523,
+       "rank_sd": 0.010416666666666649,
+       "n_seeds": 3
+      },
+      "Pile-CC": {
+       "z_mean": 3.8199612237513065,
+       "z_sd": 0.058963986684690994,
+       "rank_mean": 0.38343253968253965,
+       "rank_sd": 0.02085104122716638,
+       "n_seeds": 3
+      },
+      "StackExchange": {
+       "z_mean": 3.846287661128574,
+       "z_sd": 0.019121880506788938,
+       "rank_mean": 0.40327380952380953,
+       "rank_sd": 0.0029761904761904656,
+       "n_seeds": 3
+      },
+      "Github": {
+       "z_mean": 3.250814785146051,
+       "z_sd": 0.08028488172623756,
+       "rank_mean": 0.24950396825396823,
+       "rank_sd": 0.02663371777511667,
+       "n_seeds": 3
+      }
+     },
+     "z_spread": 0.595472875982523,
+     "z_pooled_seed_sd": 0.052167310455261205,
+     "R": 11.414674645594424,
+     "rank_spread": 0.189484126984127,
+     "rank_pooled_seed_sd": 0.016650715155594485,
+     "rank_spread_over_sd": 11.379939252667,
+     "order_by_z": [
+      "StackExchange",
+      "Pile-CC",
+      "USPTO_Backgrounds",
+      "Wikipedia_en",
+      "Github"
+     ],
+     "order_by_rank": [
+      "USPTO_Backgrounds",
+      "StackExchange",
+      "Pile-CC",
+      "Wikipedia_en",
+      "Github"
+     ],
+     "n_pairs": 96,
+     "kendall_tau_vs_410m": 1.0,
+     "fp16_roundtrip_diagnostic": {
+      "R": 11.415194254884774,
+      "max_abs_dz_mean": 3.8545578718185425e-05,
+      "order_unchanged": true
+     }
+    }
+   }
+  }
+ },
+ "controls": {
+  "C0_scorer_equivalence": {
+   "required": "the scorer reproduces D3 on the stored 410M operators",
+   "result": {
+    "worst_abs_dz": 0.0,
+    "worst_abs_drank": 0.0,
+    "fires": true
+   },
+   "fires": true
+  },
+  "C1_logit_identity_arm": {
+   "required": "logit_I emitted for every model and family; at 410M the pooled admitted-5 flat-mean-7 min-rank equals 0.19810852520167826",
+   "emitted_per_model": {
+    "1.4b": [
+     "multihop",
+     "multilingual",
+     "order-ops",
+     "poetry",
+     "typo"
+    ],
+    "2.8b": [
+     "multihop",
+     "multilingual",
+     "order-ops",
+     "poetry",
+     "typo"
+    ]
+   },
+   "410m_anchor": "C0 scored the identity arm at 410M; see cv6_c0_scorer_equivalence.json",
+   "fires": true
+  },
+  "C2_band_matches_rule": {
+   "required": "emitted band == band_for(n_layers); a tuned band voids the run",
+   "per_model": {
+    "1.4b": {
+     "band": [
+      9,
+      10,
+      11,
+      12,
+      13,
+      14,
+      15,
+      16,
+      17,
+      18,
+      19,
+      20,
+      21
+     ],
+     "rule": [
+      9,
+      10,
+      11,
+      12,
+      13,
+      14,
+      15,
+      16,
+      17,
+      18,
+      19,
+      20,
+      21
+     ],
+     "match": true
+    },
+    "2.8b": {
+     "band": [
+      12,
+      13,
+      14,
+      15,
+      16,
+      17,
+      18,
+      19,
+      20,
+      21,
+      22,
+      23,
+      24,
+      25,
+      26,
+      27,
+      28,
+      29
+     ],
+     "rule": [
+      12,
+      13,
+      14,
+      15,
+      16,
+      17,
+      18,
+      19,
+      20,
+      21,
+      22,
+      23,
+      24,
+      25,
+      26,
+      27,
+      28,
+      29
+     ],
+     "match": true
+    }
+   },
+   "fires": true
+  },
+  "C3_seed_sd_nondegenerate": {
+   "required": "pooled_seed_sd_z(f, m) > 0 for every (family, model); else that cell is VOID",
+   "per_model": {
+    "1.4b": {
+     "multihop": 0.045329210368280526,
+     "multilingual": 0.05606540772822604,
+     "order-ops": 0.05373181585637624,
+     "poetry": 0.020626950786131512,
+     "typo": 0.04607212990087249
+    },
+    "2.8b": {
+     "multihop": 0.052513067623269356,
+     "multilingual": 0.04508773736862208,
+     "order-ops": 0.057982718511798266,
+     "poetry": 0.031795284173340224,
+     "typo": 0.052167310455261205
+    }
+   },
+   "void_cells": [],
+   "fires": true
+  },
+  "C4_N_identical": {
+   "required": "N == 25 and identical across all 15 arms of every model",
+   "n_prompts_seen": {
+    "1.4b": [
+     25
+    ],
+    "2.8b": [
+     25
+    ]
+   },
+   "n_arms": {
+    "1.4b": 15,
+    "2.8b": 15
+   },
+   "prompt_sha_identical_across_models": {
+    "Wikipedia_en|s0": true,
+    "Wikipedia_en|s1": true,
+    "Wikipedia_en|s2": true,
+    "USPTO_Backgrounds|s0": true,
+    "USPTO_Backgrounds|s1": true,
+    "USPTO_Backgrounds|s2": true,
+    "Pile-CC|s0": true,
+    "Pile-CC|s1": true,
+    "Pile-CC|s2": true,
+    "StackExchange|s0": true,
+    "StackExchange|s1": true,
+    "StackExchange|s2": true,
+    "Github|s0": true,
+    "Github|s1": true,
+    "Github|s2": true
+   },
+   "fires": true
+  },
+  "C5_device_precision": {
+   "required": "activations and operators on the same device at the same dtype",
+   "per_model": {
+    "1.4b": {
+     "device": "cuda",
+     "dtype": "float32",
+     "gpu": "NVIDIA H100 NVL",
+     "tf32": {
+      "matmul_allow_tf32": false,
+      "cudnn_allow_tf32": false,
+      "NVIDIA_TF32_OVERRIDE": "0"
+     }
+    },
+    "2.8b": {
+     "device": "cuda",
+     "dtype": "float32",
+     "gpu": "NVIDIA H100 NVL",
+     "tf32": {
+      "matmul_allow_tf32": false,
+      "cudnn_allow_tf32": false,
+      "NVIDIA_TF32_OVERRIDE": "0"
+     }
+    }
+   },
+   "fires": true
+  }
+ },
+ "PRIMARY": {
+  "R_per_family_2.8b": {
+   "multihop": 11.43874217650363,
+   "multilingual": 14.505222414316746,
+   "order-ops": 6.460960425429451,
+   "poetry": 11.925362464834903,
+   "typo": 11.414674645594424
+  },
+  "kendall_tau_vs_410m_2.8b": {
+   "multihop": 0.6,
+   "multilingual": 0.8,
+   "order-ops": 0.8,
+   "poetry": 0.6,
+   "typo": 1.0
+  },
+  "n_families_R_ge_10": 4,
+  "n_families_R_lt_5": 0,
+  "n_families_tau_ge_0.6": 5
+ },
+ "VERDICT": "REPLICATES",
+ "verdict_note": "the corpus effect is not an artifact of the low-competence regime at 410M",
+ "declared_bias": [
+  "Capability and scale are confounded; every larger model is also differently trained. No outcome licenses a causal reading \u2014 only 'the effect does / does not survive up the ladder'.",
+  "The 5 corpora are a fixed panel, not a sample. n=5 is the replication unit for any claim about KINDS of corpus.",
+  "N=25 here vs N=200 at 410M. S2 established reads are flat in N, so this is expected to be immaterial, but it is a difference and an attenuation must not be attributed to scale without checking N.",
+  "Per-family seed SDs are thin: 3 seeds, 96-394 pairs per family. Thin denominators are what inflated the retired 58x figure. The SD is reported beside every ratio.",
+  "The 410M reference is D3, which is EXPLORATORY and was computed BEFORE this pre-registration. The per-family estimand may not be described as chosen on capability grounds alone."
+ ],
+ "provenance": {
+  "schema_version": 1,
+  "experiment": "CV6",
+  "utc": "2026-08-30T06:31:18+00:00",
+  "script": {
+   "path": "experiments/cv6_per_family_ladder.py",
+   "sha256": "f416c3cb55e95a02315504066479c2f2ba073c0d4676bbf09adb2f481cdb4a5b",
+   "n_lines": 750
+  },
+  "git": {
+   "commit": "c9c49697e295c0fec4225552836fc9733b279f75",
+   "commit_short": "c9c49697e295",
+   "git_repository_present": true,
+   "branch": "pythia",
+   "dirty": true,
+   "dirty_files": [
+    "CLAUDE.md",
+    "README.md",
+    "corpora/manifest_ood.json",
+    "docs/HANDOFF.md",
+    "docs/REPO_REVIEW_PROMPT.md",
+    "docs/context/AGGREGATION_POLICY.md",
+    "docs/context/CONTEXT.md",
+    "docs/context/IMPLEMENTATION_RULES.md",
+    "docs/context/STATE.md",
+    "docs/experiments/INDEX.md",
+    "docs/experiments/METRICS.md",
+    "docs/experiments/PROPOSED.md",
+    "docs/experiments/PROVENANCE.md",
+    "docs/experiments/descriptions/E64.md",
+    "docs/experiments/descriptions/EN1.md",
+    "docs/experiments/preregs/CV3_margins.md",
+    "docs/experiments/preregs/CV4_capability_ladder.md",
+    "docs/experiments/preregs/CV5_metric_sensitivity.md",
+    "docs/experiments/preregs/CV6_per_family_ladder.md",
+    "docs/experiments/preregs/CV7_1b_rung.md",
+    "docs/experiments/preregs/DN1_design_nulls.md",
+    "docs/experiments/preregs/E33_logit_baseline.md",
+    "docs/experiments/preregs/E36_qladder.md",
+    "docs/experiments/preregs/E48_crossover.md",
+    "docs/experiments/preregs/E48b_containment.md",
+    "docs/experiments/preregs/E48c_exposure_vs_read.md",
+    "docs/experiments/preregs/E51_interaction.md",
+    "docs/experiments/preregs/E52_factorial.md",
+    "docs/experiments/preregs/E53_ladder_summary.md",
+    "docs/experiments/preregs/E55_matrix_robustness.md",
+    "docs/experiments/preregs/E56_predictor_registry.md",
+    "docs/experiments/preregs/E57_grid_variance_ci.md",
+    "docs/experiments/preregs/E58_algebra_audit.md",
+    "docs/experiments/preregs/E59_read_dose_response.md",
+    "docs/experiments/preregs/E60_fitter_determinism.md",
+    "docs/experiments/preregs/E61_randomized_network_null.md",
+    "docs/experiments/preregs/E62_ladder1b_corrected_band.md",
+    "docs/experiments/preregs/E65_training_axis_floor.md",
+    "docs/experiments/preregs/E66b_determinism_floor.md",
+    "docs/experiments/preregs/OS1_operator_space.md",
+    "docs/experiments/preregs/P0_t52_parallel_port.md",
+    "docs/experiments/preregs/R1_grid_rstrip.md",
+    "docs/experiments/preregs/R2_e48_rstrip_arm.md",
+    "docs/experiments/preregs/R3_ladder410_cpu.md",
+    "docs/experiments/preregs/R4_corrections.md",
+    "docs/experiments/preregs/R4b_e36_rstrip.md",
+    "docs/experiments/preregs/R5_corpus_axis_uncertainty.md",
+    "docs/experiments/preregs/R6_within_source_resampling.md",
+    "docs/experiments/preregs/R8_ladder_rstrip_S2.md",
+    "docs/experiments/preregs/R9_permutation_calibrated_min.md",
+    "docs/experiments/preregs/superseded/PREREG_E48_CROSSOVER.md",
+    "docs/reproducibility/ARTIFACTS.md",
+    "docs/reproducibility/PAYLOAD_REDACTIONS.md",
+    "docs/reproducibility/RELEASE.md",
+    "docs/reproducibility/REVIEWER_REPRO.md",
+    "docs/reproducibility/T1_FINDING_E33_BAND.md",
+    "docs/reproducibility/T1_SWEEP_PREDICTIONS.md",
+    "docs/reproducibility/T1_SWEEP_SPEC.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/MANIFEST.sha256",
+    "docs/reproducibility/review-snapshot-2026-08-19/PROVENANCE.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/REDACTIONS.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/EXTERNAL_REVIEW.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/FINDINGS.md",
+    "docs/reproducibility/review-snapshot-2026-08-19/review/reproducibility/PHASE_A_ACCOUNT.md"
+   ],
+   "dirty_diff_sha256": "ec61dde1905616f407f51d5f6fe3ea37c9c6e3e1ade6998cd0644527d080c600",
+   "describe": "c9c4969-dirty"
+  },
+  "argv": [
+   "experiments/cv6_per_family_ladder.py",
+   "--adjudicate"
+  ],
+  "cwd": ".",
+  "inputs": [
+   {
+    "path": "results/d3_corpus_by_family_410m.json",
+    "sha256": "2ab2b53f90bd9d3435ebdb65aa0c9c1a82ac599a5e3c714f9dd9681d4bb9487c",
+    "bytes": 23095,
+    "exists": true
+   },
+   {
+    "path": "results/cv6/cv6_1.4b_n25.json",
+    "sha256": "4e8eab1ebedf0785a9e61db7fa77212cd2f89699fe627b08f7b9ea7b960cedae",
+    "bytes": 30228,
+    "exists": true
+   },
+   {
+    "path": "results/cv6/cv6_2.8b_n25.json",
+    "sha256": "7f1a6d31f1c8728c4c966d960e5502c6bfaaa8dddf4543cf5d9e5a8424f82b9f",
+    "bytes": 32402,
+    "exists": true
+   },
+   {
+    "path": "results/cv6/cv6_c0_scorer_equivalence.json",
+    "sha256": "2650f8b565cf128bc4ac01ee775849beec904a9a2329d4e75081492a4546d332",
+    "bytes": 4013,
+    "exists": true
+   }
+  ],
+  "env": {
+   "python": "3.14.2",
+   "torch": "2.13.0",
+   "transformers": "5.14.1",
+   "numpy": "2.5.1",
+   "platform": "macOS-26.5.2-arm64-arm-64bit-Mach-O",
+   "machine": "arm64",
+   "hostname": "<redacted for double-blind review>",
+   "tf32": {
+    "allow_tf32_matmul": false,
+    "allow_tf32_cudnn": false,
+    "NVIDIA_TF32_OVERRIDE": "0"
+   },
+   "threads": {
+    "torch_num_threads": 10,
+    "torch_num_interop_threads": 14,
+    "OMP_NUM_THREADS": null,
+    "os_cpu_count": 14
+   }
+  },
+  "payload_sha256": "37c137de5d7788ff265c533db4dabc53540c7c9e6ad33e12b06d72182bbdd228"
+ }
+}

results/cv7/cv7_1b_n200.json

@@ -0,0 +1,821 @@
+{
+ "experiment": "CV7 \u2014 the 1B rung, scoring only",
+ "prereg": "docs/experiments/preregs/CV7_1b_rung.md",
+ "status": "PRE-REGISTERED",
+ "decision_rule_verbatim": "Evaluated PER FAMILY; the verdict is the count across the five. Thresholds inherited VERBATIM from CV6_per_family_ladder.md, which fixed them before any ladder number existed. REPLICATES AT 1B if R(f,1B) >= 10 on >= 4 of 5 families AND ordering tau >= 0.6 on >= 3 of 5. ATTENUATES AT 1B if R(f,1B) < 5 on >= 4 of 5 families. UNCLEAR otherwise, including a split -- report the table and stop. Do not pool. Do not drop a family. Do not re-cut R across mismatched N.",
+ "model": "EleutherAI/pythia-1b-deduped",
+ "short": "1b",
+ "n_layers": 16,
+ "d_model": 2048,
+ "band": [
+  6,
+  7,
+  8,
+  9,
+  10,
+  11,
+  12,
+  13
+ ],
+ "band_rule": "floor(0.38*L)..floor(0.92*L), layers < L-2",
+ "band_read_off_artifacts": {
+  "Wikipedia_en|s0": [
+   6,
+   7,
+   8,
+   9,
+   10,
+   11,
+   12,
+   13
+  ],
+  "Wikipedia_en|s1": [
+   6,
+   7,
+   8,
+   9,
+   10,
+   11,
+   12,
+   13
+  ],
+  "Wikipedia_en|s2": [
+   6,
+   7,
+   8,
+   9,
+   10,
+   11,
+   12,
+   13
+  ],
+  "USPTO_Backgrounds|s0": [
+   6,
+   7,
+   8,
+   9,
+   10,
+   11,
+   12,
+   13
+  ],
+  "USPTO_Backgrounds|s1": [
+   6,
+   7,
+   8,
+   9,
+   10,
+   11,
+   12,
+   13
+  ],
+  "USPTO_Backgrounds|s2": [
+   6,
+   7,
+   8,
+   9,
+   10,
+   11,
+   12,
+   13
+  ],
+  "Pile-CC|s0": [
+   6,
+   7,
+   8,
+   9,
+   10,
+   11,
+   12,
+   13
+  ],
+  "Pile-CC|s1": [
+   6,
+   7,
+   8,
+   9,
+   10,
+   11,
+   12,
+   13
+  ],
+  "Pile-CC|s2": [
+   6,
+   7,
+   8,
+   9,
+   10,
+   11,
+   12,
+   13
+  ],
+  "StackExchange|s0": [
+   6,
+   7,
+   8,
+   9,
+   10,
+   11,
+   12,
+   13
+  ],
+  "StackExchange|s1": [
+   6,
+   7,
+   8,
+   9,
+   10,
+   11,
+   12,
+   13
+  ],
+  "StackExchange|s2": [
+   6,
+   7,
+   8,
+   9,
+   10,
+   11,
+   12,
+   13
+  ],
+  "Github|s0": [
+   6,
+   7,
+   8,
+   9,
+   10,
+   11,
+   12,
+   13
+  ],
+  "Github|s1": [
+   6,
+   7,
+   8,
+   9,
+   10,
+   11,
+   12,
+   13
+  ],
+  "Github|s2": [
+   6,
+   7,
+   8,
+   9,
+   10,
+   11,
+   12,
+   13
+  ]
+ },
+ "band_matches_rule": true,
+ "N": 200,
+ "n_prompts_per_arm": {
+  "Wikipedia_en|s0": 200,
+  "Wikipedia_en|s1": 200,
+  "Wikipedia_en|s2": 200,
+  "USPTO_Backgrounds|s0": 200,
+  "USPTO_Backgrounds|s1": 200,
+  "USPTO_Backgrounds|s2": 200,
+  "Pile-CC|s0": 200,
+  "Pile-CC|s1": 200,
+  "Pile-CC|s2": 200,
+  "StackExchange|s0": 200,
+  "StackExchange|s1": 200,
+  "StackExchange|s2": 200,
+  "Github|s0": 200,
+  "Github|s1": 200,
+  "Github|s2": 200
+ },
+ "operators_from": "results/ladder1b_b613/ (fitted for E62; NOT refitted here)",
+ "device": "cpu",
+ "dtype": "float32",
+ "operator_storage_dtype": "float16",
+ "device_caveat": "operators fitted on CUDA, scored on CPU. D2 measured a CUDA-vs-CPU cell-level divergence of 2.774e-04. This is D3's situation, which is what makes 1B-vs-410M the matched comparison.",
+ "torch": "2.13.0",
+ "K": [
+  1,
+  2,
+  5,
+  10,
+  20,
+  50,
+  100
+ ],
+ "admitted_sets": [
+  "multihop",
+  "multilingual",
+  "order-ops",
+  "poetry",
+  "typo"
+ ],
+ "corpora": [
+  "Wikipedia_en",
+  "USPTO_Backgrounds",
+  "Pile-CC",
+  "StackExchange",
+  "Github"
+ ],
+ "seeds": [
+  0,
+  1,
+  2
+ ],
+ "readout_convention": "STRIPPED (corrected), no prefix, flat-mean-7",
+ "pooling_convention": "flat pool over (item, intermediate) pairs WITHIN a family; rank = min over band then flat-mean-7 pass@k; z = max over band then mean",
+ "n_items": 449,
+ "n_pairs": 801,
+ "n_pairs_per_family": {
+  "multihop": 103,
+  "multilingual": 394,
+  "order-ops": 110,
+  "poetry": 98,
+  "typo": 96
+ },
+ "per_arm": {
+  "logit_I": {
+   "multihop": {
+    "rank": 0.24410540915395282,
+    "z": 3.898444828477878,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.1472081218274112,
+    "z": 2.7347642624287434,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5207792207792208,
+    "z": 5.480567513812672,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.0058309037900874635,
+    "z": 2.021673796736464,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.17708333333333331,
+    "z": 3.567758576323589,
+    "n_pairs": 96
+   }
+  },
+  "J|Wikipedia_en|s0": {
+   "multihop": {
+    "rank": 0.2538141470180305,
+    "z": 3.714084236367235,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.13741841914430747,
+    "z": 2.7458525141701102,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.4597402597402597,
+    "z": 5.137804371660406,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.016034985422740525,
+    "z": 1.9996374888079507,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.3050595238095238,
+    "z": 3.7074354047266147,
+    "n_pairs": 96
+   }
+  },
+  "J|Wikipedia_en|s1": {
+   "multihop": {
+    "rank": 0.2538141470180305,
+    "z": 3.723332085655731,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.13778100072516317,
+    "z": 2.751266322687249,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.4584415584415584,
+    "z": 5.1602459907531735,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.01749271137026239,
+    "z": 1.988615335858598,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.3050595238095238,
+    "z": 3.7324859214325747,
+    "n_pairs": 96
+   }
+  },
+  "J|Wikipedia_en|s2": {
+   "multihop": {
+    "rank": 0.2524271844660194,
+    "z": 3.706331000166032,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.13741841914430747,
+    "z": 2.7407319243846056,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.4610389610389611,
+    "z": 5.139317949251695,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.016034985422740525,
+    "z": 1.9798154046340866,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.30357142857142855,
+    "z": 3.710389399745812,
+    "n_pairs": 96
+   }
+  },
+  "J|USPTO_Backgrounds|s0": {
+   "multihop": {
+    "rank": 0.29680998613037446,
+    "z": 4.109125902351824,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.15155910079767948,
+    "z": 2.907341442816964,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.48051948051948046,
+    "z": 5.347905703024431,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.033527696793002916,
+    "z": 2.0518707047995863,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.4181547619047619,
+    "z": 4.076707179347674,
+    "n_pairs": 96
+   }
+  },
+  "J|USPTO_Backgrounds|s1": {
+   "multihop": {
+    "rank": 0.30513176144244103,
+    "z": 4.1576821566785425,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.1508339376359681,
+    "z": 2.907443309632995,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.4857142857142857,
+    "z": 5.373728732629256,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.033527696793002916,
+    "z": 2.059163775085947,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.40625,
+    "z": 4.066875881205003,
+    "n_pairs": 96
+   }
+  },
+  "J|USPTO_Backgrounds|s2": {
+   "multihop": {
+    "rank": 0.30235783633841884,
+    "z": 4.154952114068188,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.15228426395939088,
+    "z": 2.9097203547655024,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.4753246753246753,
+    "z": 5.355981678312475,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.03644314868804665,
+    "z": 2.0739537419713274,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.41517857142857145,
+    "z": 4.111865448455016,
+    "n_pairs": 96
+   }
+  },
+  "J|Pile-CC|s0": {
+   "multihop": {
+    "rank": 0.2787794729542302,
+    "z": 4.083725291548423,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.15445975344452503,
+    "z": 2.9522800377477316,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.43376623376623374,
+    "z": 5.306177348440344,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.018950437317784254,
+    "z": 1.9803421716300809,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.39285714285714285,
+    "z": 4.1813286903003855,
+    "n_pairs": 96
+   }
+  },
+  "J|Pile-CC|s1": {
+   "multihop": {
+    "rank": 0.2815533980582524,
+    "z": 4.094569732841936,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.15772298767222623,
+    "z": 2.986847078749144,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.44025974025974024,
+    "z": 5.305445389314131,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.018950437317784258,
+    "z": 2.0033713484905205,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.3988095238095238,
+    "z": 4.20682845885555,
+    "n_pairs": 96
+   }
+  },
+  "J|Pile-CC|s2": {
+   "multihop": {
+    "rank": 0.2787794729542302,
+    "z": 4.0395108611838335,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.1548223350253807,
+    "z": 2.943907567330631,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.43506493506493504,
+    "z": 5.280768312107433,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.014577259475218656,
+    "z": 1.9708401384402294,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.3898809523809524,
+    "z": 4.157780651624004,
+    "n_pairs": 96
+   }
+  },
+  "J|StackExchange|s0": {
+   "multihop": {
+    "rank": 0.3370319001386962,
+    "z": 4.34045670449155,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.16243654822335027,
+    "z": 3.0495984307324826,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.4857142857142857,
+    "z": 5.51830342466181,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.02040816326530612,
+    "z": 2.010183577938956,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.49107142857142855,
+    "z": 4.537338141351938,
+    "n_pairs": 96
+   }
+  },
+  "J|StackExchange|s1": {
+   "multihop": {
+    "rank": 0.334257975034674,
+    "z": 4.3061738963266025,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.16026105873821608,
+    "z": 3.0347819554662037,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5012987012987012,
+    "z": 5.519914856824008,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.021865889212827987,
+    "z": 2.0059123179134057,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.4672619047619047,
+    "z": 4.418834609289964,
+    "n_pairs": 96
+   }
+  },
+  "J|StackExchange|s2": {
+   "multihop": {
+    "rank": 0.334257975034674,
+    "z": 4.317730897838629,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.15917331399564902,
+    "z": 3.045205917266117,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.4987012987012987,
+    "z": 5.524625140970404,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.02040816326530612,
+    "z": 2.000217623248392,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.4761904761904762,
+    "z": 4.479793511331081,
+    "n_pairs": 96
+   }
+  },
+  "J|Github|s0": {
+   "multihop": {
+    "rank": 0.2524271844660194,
+    "z": 3.8118835438802403,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.14213197969543148,
+    "z": 2.8611055270385775,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.4896103896103896,
+    "z": 5.030006443370472,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.013119533527696792,
+    "z": 1.850219145303174,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.29910714285714285,
+    "z": 3.70514493167866,
+    "n_pairs": 96
+   }
+  },
+  "J|Github|s1": {
+   "multihop": {
+    "rank": 0.2621359223300971,
+    "z": 3.836685398828636,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.14648295866569977,
+    "z": 2.8760104446747543,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.5,
+    "z": 5.072230815887451,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.014577259475218656,
+    "z": 1.8696253643352159,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.3080357142857143,
+    "z": 3.753457091632299,
+    "n_pairs": 96
+   }
+  },
+  "J|Github|s2": {
+   "multihop": {
+    "rank": 0.2579750346740638,
+    "z": 3.8195294026032234,
+    "n_pairs": 103
+   },
+   "multilingual": {
+    "rank": 0.145032632342277,
+    "z": 2.8609475317850785,
+    "n_pairs": 394
+   },
+   "order-ops": {
+    "rank": 0.4922077922077922,
+    "z": 5.031250395558097,
+    "n_pairs": 110
+   },
+   "poetry": {
+    "rank": 0.01020408163265306,
+    "z": 1.8547754169878912,
+    "n_pairs": 98
+   },
+   "typo": {
+    "rank": 0.30357142857142855,
+    "z": 3.727808784286026,
+    "n_pairs": 96
+   }
+  }
+ },
+ "logit_I_pooled_rank_flatmean7": 0.2190013977768011,
+ "provenance": {
+  "schema_version": 1,
+  "experiment": "CV7",
+  "utc": "2026-08-24T18:42:15+00:00",
+  "script": {
+   "path": "experiments/cv7_1b_rung.py",
+   "sha256": "17eed1b79d679e0044f836a0906667bbcfa1bcfe5dfe5399c99557321ea6f73d",
+   "n_lines": 316
+  },
+  "git": {
+   "commit": "180c1374d89790d6e60346e4d49581f5bba6b34f",
+   "commit_short": "180c1374d897",
+   "branch": "pythia",
+   "dirty": true,
+   "dirty_files": [
+    "experiments/cv7_1b_rung.py",
+    "figures/fA_effect_and_null.png",
+    "figures/fB_validity.png",
+    "figures/make_audit_figures.py",
+    "logs/",
+    "ocs/context/CONTEXT.md",
+    "paper/PAPER_HANDOFF.md",
+    "paper/audit_paper_5pp.tex",
+    "paper/audit_paper_7pp.tex",
+    "paper/build_audit.sh",
+    "paper/refs.bib",
+    "paper/repro-paper.tex",
+    "paper/tables_audit.tex",
+    "repro/30_repo_health.sh",
+    "repro/exp/cv7_1b.sh",
+    "repro/lib/banned_claims.py"
+   ],
+   "dirty_diff_sha256": "d380ffb38dde5c7efcbe498d9454e038c0e71c01a846d485b439e379c597db29",
+   "describe": "180c137-dirty"
+  },
+  "argv": [
+   "experiments/cv7_1b_rung.py",
+   "--device",
+   "cpu"
+  ],
+  "cwd": ".",
+  "inputs": [
+   {
+    "path": "results/ladder1b_b613/lens_Wikipedia_en_1b_n200_s0.pt",
+    "sha256": "f60c8ee8186e4d53e9a0195b5f0535506160d2d313b3bc174d3129b80d9426b2",
+    "bytes": 67112755,
+    "exists": true
+   },
+   {
+    "path": "results/ladder1b_b613/lens_Wikipedia_en_1b_n200_s1.pt",
+    "sha256": "25a07e77788297cc89a4603c0a1771d9cf2f8580a481c015bf77d039adcd0464",
+    "bytes": 67112755,
+    "exists": true
+   },
+   {
+    "path": "results/ladder1b_b613/lens_Wikipedia_en_1b_n200_s2.pt",
+    "sha256": "89838efaec6ce3b2f991ed142b320fd28c8f608e5205a8c934e0ac44675dd0fa",
+    "bytes": 67112755,
+    "exists": true
+   },
+   {
+    "path": "results/ladder1b_b613/lens_USPTO_Backgrounds_1b_n200_s0.pt",
+    "sha256": "4acda201f3fbc46f77e96d3f6f6840ab6cc53b8d80f694ec5080dab7321b054a",
+    "bytes": 67112825,
+    "exists": true
+   },
+   {
+    "path": "results/ladder1b_b613/lens_USPTO_Backgrounds_1b_n200_s1.pt",
+    "sha256": "4be9599eae66c8895479b2d0151fde7975929aedc420916b1b29d7415234eac1",
+    "bytes": 67112825,
+    "exists": true
+   },
+   {
+    "path": "results/ladder1b_b613/lens_USPTO_Backgrounds_1b_n200_s2.pt",
+    "sha256": "0f730745e88e5dae78b5f73b98a00ab55c28ed32dae8b7800ba8823445e090ff",
+    "bytes": 67112825,
+    "exists": true
+   },
+   {
+    "path": "results/ladder1b_b613/lens_Pile-CC_1b_n200_s0.pt",
+    "sha256": "3bf4ce26087e107fa74da45922f6ecc486d6eab6fb29b7fbbf9b356a0ca4eabd",
+    "bytes": 67112685,
+    "exists": true
+   },
+   {
+    "path": "results/ladder1b_b613/lens_Pile-CC_1b_n200_s1.pt",
+    "sha256": "f41713739a1f278fd0d1c5b5c23db6ecfe1df0991945aa6b83ad716200e9a8f0",
+    "bytes": 67112685,
+    "exists": true
+   },
+   {
+    "path": "results/ladder1b_b613/lens_Pile-CC_1b_n200_s2.pt",
+    "sha256": "fe7a6a5b4579d5f47f0119d943efc700eb70220fb16acdec55c1f724dcde0ac2",
+    "bytes": 67112685,
+    "exists": true
+   },
+   {
+    "path": "results/ladder1b_b613/lens_StackExchange_1b_n200_s0.pt",
+    "sha256": "5048cd087976e256a7ff9f30d8066dd95d442229f8c2b0ff5f4d273eef512c5b",
+    "bytes": 67112769,
+    "exists": true
+   },
+   {
+    "path": "results/ladder1b_b613/lens_StackExchange_1b_n200_s1.pt",
+    "sha256": "905a7e26d7af75d7e9a64a49ba5319156b0e220236c0bf8ec5c15d8989db2330",
+    "bytes": 67112769,
+    "exists": true
+   },
+   {
+    "path": "results/ladder1b_b613/lens_StackExchange_1b_n200_s2.pt",
+    "sha256": "78e10c625c615a13dc475fef35688ce8a90b8ab368ec84e1c8c7f88d70d3a040",
+    "bytes": 67112769,
+    "exists": true
+   },
+   {
+    "path": "results/ladder1b_b613/lens_Github_1b_n200_s0.pt",
+    "sha256": "459f997d6661ff07494e3ac0829864e02d71b15e0609875a705e032371a30158",
+    "bytes": 67112671,
+    "exists": true
+   },
+   {
+    "path": "results/ladder1b_b613/lens_Github_1b_n200_s1.pt",
+    "sha256": "187bd291fc66a6f3e5cf9a35c4640d9d963e1fd47434124dc3e7b7832f1a7950",
+    "bytes": 67112671,
+    "exists": true
+   },
+   {
+    "path": "results/ladder1b_b613/lens_Github_1b_n200_s2.pt",
+    "sha256": "e37d288145a6438be3c571ad3fac769b7fd314c00675573dc8e4e9b3812a2754",
+    "bytes": 67112671,
+    "exists": true
+   }
+  ],
+  "env": {
+   "python": "3.14.2",
+   "torch": "2.13.0",
+   "transformers": "5.14.1",
+   "numpy": "2.5.1",
+   "platform": "macOS-26.5.2-arm64-arm-64bit-Mach-O",
+   "machine": "arm64",
+   "hostname": "<redacted for double-blind review>",
+   "tf32": {
+    "allow_tf32_matmul": false,
+    "allow_tf32_cudnn": false,
+    "NVIDIA_TF32_OVERRIDE": "0"
+   }
+  },
+  "payload_sha256": "39a12fb962ca27e489684090b92af88ea55b4d2c0aab3e43061cd00eb5e627e0"
+ }
+}