| # ATTRIBUTION |
|
|
| This repository is **CC BY 4.0 without qualification**. It contains only what this work produced. |
|
|
| ## What is here |
|
|
| | what | size | licence | |
| |---|---|---| |
| | `results/lenses/**` — 382 fitted operators + sidecars | 16.2 GB | CC BY 4.0 | |
| | `results/**.json`, `results/**.npz` — experimental records | ~27 MB | CC BY 4.0 | |
| | `corpora/manifest.json`, `corpora/manifest_ood.json`, `corpora/dataset_revisions.json` | 12 KB | CC BY 4.0 | |
|
|
| Derived from **Pythia** (EleutherAI, Apache-2.0) via the **Jacobian-lens** reference |
| implementation (Apache-2.0, pinned `581d398`). Neither restricts these derivatives. |
|
|
| ## What is deliberately NOT here, and how to get it |
|
|
| The eleven corpus plaintext files are **not distributed**. They are third-party text under |
| upstream terms; `Wikipedia_en.jsonl` and `OOD_Wikipedia_2023.jsonl` are Wikipedia-derived and |
| carry **CC BY-SA**, which is share-alike and cannot be relicensed as CC BY 4.0. Shipping them |
| would have forced a mixed-licence repository and made the CC BY 4.0 claim above false. |
|
|
| Rebuild the pool from its public sources: |
|
|
| ```bash |
| bash repro/06_build_corpora.sh # builds, then hash-verifies against the manifests |
| bash repro/06_build_corpora.sh --verify # verify a pool you already have |
| ``` |
|
|
| `corpora/manifest.json` and `corpora/manifest_ood.json` ship here and carry, per corpus: the |
| SHA-256 of the exact file used, the document count, the source dataset id and config, and the |
| three disjoint seed-block index ranges. A rebuild that matches those hashes is the same pool. |
|
|
| **Measured 2026-08-31, rebuilding from scratch on a clean machine:** |
|
|
| | corpus | rebuild | |
| |---|---| |
| | Pile-CC, StackExchange, Wikipedia_en, Github, USPTO_Backgrounds | **5/5 bit-identical** | |
| | OOD_CommonPile, TRAP_FineWeb, CONTROL_PubMed_2023 | **3/3 bit-identical** | |
| | `corpora/manifest.json` regenerated | **byte-identical** | |
| | OOD_Wikipedia_2023, OOD_arXiv_2023 | not re-run (53 MB / 110 MB of streaming); both carry SHA-256 in the manifest and are checked by the script | |
|
|
| **One caveat, stated plainly.** `src/build_corpora.py` calls `load_dataset( |
| "monology/pile-uncopyrighted", ...)` with **no `revision=` pin**, so an upstream change to that |
| dataset would produce a different pool. The result above is evidence the upstream had not moved |
| as of that date — not a guarantee it never will. The manifest is the check: a mismatch means the |
| upstream moved, which is a finding to report, not a failure to work around. The OOD panel is more |
| stable by construction because its configs are date-stamped (`20231101.en`, `2024-06`). |
| |
