Tarush-AI commited on
Commit
c5c86dc
·
verified ·
1 Parent(s): 2fecd41

Card: name the author in LICENSE; correct the rebuild command and the revision-pinning caveat in ATTRIBUTION.md

Browse files
Files changed (2) hide show
  1. ATTRIBUTION.md +29 -19
  2. LICENSE +14 -15
ATTRIBUTION.md CHANGED
@@ -1,48 +1,58 @@
1
  # ATTRIBUTION
2
 
3
  This repository is **CC BY 4.0 without qualification**. It contains only what this work produced.
 
4
 
5
  ## What is here
6
 
7
  | what | size | licence |
8
  |---|---|---|
9
- | `results/lenses/**` — 382 fitted operators + sidecars | 16.2 GB | CC BY 4.0 |
10
- | `results/**.json`, `results/**.npz` — experimental records | ~27 MB | CC BY 4.0 |
11
- | `corpora/manifest.json`, `corpora/manifest_ood.json`, `corpora/dataset_revisions.json` | 12 KB | CC BY 4.0 |
12
 
13
  Derived from **Pythia** (EleutherAI, Apache-2.0) via the **Jacobian-lens** reference
14
  implementation (Apache-2.0, pinned `581d398`). Neither restricts these derivatives.
15
 
16
  ## What is deliberately NOT here, and how to get it
17
 
18
- The eleven corpus plaintext files are **not distributed**. They are third-party text under
19
- upstream terms; `Wikipedia_en.jsonl` and `OOD_Wikipedia_2023.jsonl` are Wikipedia-derived and
20
- carry **CC BY-SA**, which is share-alike and cannot be relicensed as CC BY 4.0. Shipping them
21
- would have forced a mixed-licence repository and made the CC BY 4.0 claim above false.
 
22
 
23
- Rebuild the pool from its public sources:
 
24
 
25
  ```bash
26
- bash repro/06_build_corpora.sh # builds, then hash-verifies against the manifests
27
- bash repro/06_build_corpora.sh --verify # verify a pool you already have
28
  ```
29
 
30
  `corpora/manifest.json` and `corpora/manifest_ood.json` ship here and carry, per corpus: the
31
  SHA-256 of the exact file used, the document count, the source dataset id and config, and the
32
  three disjoint seed-block index ranges. A rebuild that matches those hashes is the same pool.
 
 
33
 
34
- **Measured 2026-08-31, rebuilding from scratch on a clean machine:**
 
 
 
 
 
 
35
 
36
  | corpus | rebuild |
37
  |---|---|
38
  | Pile-CC, StackExchange, Wikipedia_en, Github, USPTO_Backgrounds | **5/5 bit-identical** |
39
  | OOD_CommonPile, TRAP_FineWeb, CONTROL_PubMed_2023 | **3/3 bit-identical** |
40
  | `corpora/manifest.json` regenerated | **byte-identical** |
41
- | OOD_Wikipedia_2023, OOD_arXiv_2023 | not re-run (53 MB / 110 MB of streaming); both carry SHA-256 in the manifest and are checked by the script |
42
-
43
- **One caveat, stated plainly.** `src/build_corpora.py` calls `load_dataset(
44
- "monology/pile-uncopyrighted", ...)` with **no `revision=` pin**, so an upstream change to that
45
- dataset would produce a different pool. The result above is evidence the upstream had not moved
46
- as of that date — not a guarantee it never will. The manifest is the check: a mismatch means the
47
- upstream moved, which is a finding to report, not a failure to work around. The OOD panel is more
48
- stable by construction because its configs are date-stamped (`20231101.en`, `2024-06`).
 
1
  # ATTRIBUTION
2
 
3
  This repository is **CC BY 4.0 without qualification**. It contains only what this work produced.
4
+ Copyright (c) 2026 Tarush Gupta.
5
 
6
  ## What is here
7
 
8
  | what | size | licence |
9
  |---|---|---|
10
+ | `results/**/*.pt`: 380 fitted Jacobian maps and 2 activation caches | 16.2 GB | CC BY 4.0 |
11
+ | `results/**/*.json`, `results/**/*.npz`: experimental records, plus 222 map sidecars | ~41 MB | CC BY 4.0 |
12
+ | `corpora/manifest.json`, `corpora/manifest_ood.json`, `corpora/dataset_revisions.json` | 13 KB | CC BY 4.0 |
13
 
14
  Derived from **Pythia** (EleutherAI, Apache-2.0) via the **Jacobian-lens** reference
15
  implementation (Apache-2.0, pinned `581d398`). Neither restricts these derivatives.
16
 
17
  ## What is deliberately NOT here, and how to get it
18
 
19
+ The corpus plaintext is **not distributed**: eleven files built from seven public datasets, plus
20
+ one derived from them. It is third-party text under upstream terms; `Wikipedia_en.jsonl` and
21
+ `OOD_Wikipedia_2023.jsonl` are Wikipedia-derived and carry **CC BY-SA**, which is share-alike and
22
+ cannot be relicensed as CC BY 4.0. Shipping them would have forced a mixed-licence repository and
23
+ made the CC BY 4.0 claim above false.
24
 
25
+ Rebuild the pool from its public sources with the code repository,
26
+ [github.com/tarush-ai/fittingcorpus](https://github.com/tarush-ai/fittingcorpus):
27
 
28
  ```bash
29
+ bash repro/06_data.sh --build # builds at the pinned revisions, then hash-verifies against the manifests
30
+ bash repro/06_data.sh --verify # verify a pool you already have
31
  ```
32
 
33
  `corpora/manifest.json` and `corpora/manifest_ood.json` ship here and carry, per corpus: the
34
  SHA-256 of the exact file used, the document count, the source dataset id and config, and the
35
  three disjoint seed-block index ranges. A rebuild that matches those hashes is the same pool.
36
+ `OOD_News_2024_dedup` has no upstream of its own: the manifest records the rule that derives it
37
+ from `OOD_News_2024`.
38
 
39
+ `corpora/dataset_revisions.json` pins the upstream dataset commit for all eight upstreams the code
40
+ reads: the seven corpus sources, and the Pythia training stream the containment scores are measured
41
+ against. Both builders (`src/build_corpora.py`, `src/build_ood_corpora.py`) read it and refuse to load a
42
+ dataset that has no pinned revision.
43
+
44
+ **Measured 2026-08-31, rebuilding from scratch on a clean machine** (the day before the revision
45
+ pins were added, so this is also evidence that the upstreams had not moved by then):
46
 
47
  | corpus | rebuild |
48
  |---|---|
49
  | Pile-CC, StackExchange, Wikipedia_en, Github, USPTO_Backgrounds | **5/5 bit-identical** |
50
  | OOD_CommonPile, TRAP_FineWeb, CONTROL_PubMed_2023 | **3/3 bit-identical** |
51
  | `corpora/manifest.json` regenerated | **byte-identical** |
52
+ | OOD_Wikipedia_2023, OOD_arXiv_2023 | not re-run (53 MB / 110 MB of streaming) |
53
+ | OOD_News_2024 and its derived `_dedup` | not in that record |
54
+
55
+ Every corpus in the last two rows carries its SHA-256 in the manifest, and `--build` checks it.
56
+ **A pinned revision protects against an upstream edit, but not against an upstream deletion.** If
57
+ a source dataset or its pinned commit is removed from the Hub, that corpus cannot be rebuilt; the
58
+ manifest hash still says which file the results were computed on.
 
LICENSE CHANGED
@@ -1,25 +1,24 @@
1
  Creative Commons Attribution 4.0 International (CC BY 4.0)
2
 
3
- Copyright (c) 2026 The Authors
4
 
5
  Everything in this repository is licensed under CC BY 4.0:
6
  https://creativecommons.org/licenses/by/4.0/
7
 
8
- That claim is unqualified, and it is unqualified BY CONSTRUCTION rather than by assertion: this
9
- repository contains only artifacts this work produced.
10
 
11
- results/lenses/** fitted Jacobian operators (.pt) and their provenance sidecars
12
- results/**.json, **.npz every experimental record and derived statistic
13
- corpora/manifest*.json the build record of the corpus pool: per-file SHA-256, document
14
- counts, source dataset id and config, disjoint seed-block ranges
15
 
16
- NO THIRD-PARTY CORPUS TEXT IS DISTRIBUTED HERE. The eleven corpus files this work used are
17
- third-party text under upstream terms, and two are Wikipedia-derived and carry CC BY-SA, a
18
- share-alike licence that cannot be relicensed as CC BY 4.0. Rather than qualify this licence,
19
- the pool is withheld in full and rebuilt from its public sources by the code repository's
20
- `repro/06_build_corpora.sh`. The manifests above are this work's own record of that pool and
21
- are what make the rebuild verifiable.
22
 
23
- The operators are derived from EleutherAI's Pythia suite (Apache-2.0) using the Jacobian-lens
24
  reference implementation (Apache-2.0, pinned at 581d398). Neither upstream licence restricts the
25
- licensing of the derived operators recorded here.
 
1
  Creative Commons Attribution 4.0 International (CC BY 4.0)
2
 
3
+ Copyright (c) 2026 Tarush Gupta
4
 
5
  Everything in this repository is licensed under CC BY 4.0:
6
  https://creativecommons.org/licenses/by/4.0/
7
 
8
+ The licence is unqualified because this repository contains only artifacts this work produced:
 
9
 
10
+ results/**/*.pt fitted Jacobian maps (and two activation caches)
11
+ results/**/*.json, *.npz every experimental record, derived statistic and map sidecar
12
+ corpora/*.json the build record of the corpus pool: per-file SHA-256, document
13
+ counts, source dataset id, config and pinned revision, seed-block ranges
14
 
15
+ NO THIRD-PARTY CORPUS TEXT IS DISTRIBUTED HERE. The corpus files this work used are third-party
16
+ text under upstream terms, and two are Wikipedia-derived and carry CC BY-SA, a share-alike
17
+ licence that cannot be relicensed as CC BY 4.0. Rather than qualify this licence, the pool is
18
+ withheld in full and rebuilt from its public sources by `bash repro/06_data.sh --build` in the
19
+ code repository, https://github.com/tarush-ai/fittingcorpus. The manifests above are this work's
20
+ own record of that pool and are what make the rebuild verifiable.
21
 
22
+ The maps are derived from EleutherAI's Pythia suite (Apache-2.0) using the Jacobian-lens
23
  reference implementation (Apache-2.0, pinned at 581d398). Neither upstream licence restricts the
24
+ licensing of the derived maps recorded here.