Card: name the author in LICENSE; correct the rebuild command and the revision-pinning caveat in ATTRIBUTION.md
Browse files- ATTRIBUTION.md +29 -19
- LICENSE +14 -15
ATTRIBUTION.md
CHANGED
|
@@ -1,48 +1,58 @@
|
|
| 1 |
# ATTRIBUTION
|
| 2 |
|
| 3 |
This repository is **CC BY 4.0 without qualification**. It contains only what this work produced.
|
|
|
|
| 4 |
|
| 5 |
## What is here
|
| 6 |
|
| 7 |
| what | size | licence |
|
| 8 |
|---|---|---|
|
| 9 |
-
| `results/
|
| 10 |
-
| `results/**.json`, `results/**.npz`
|
| 11 |
-
| `corpora/manifest.json`, `corpora/manifest_ood.json`, `corpora/dataset_revisions.json` |
|
| 12 |
|
| 13 |
Derived from **Pythia** (EleutherAI, Apache-2.0) via the **Jacobian-lens** reference
|
| 14 |
implementation (Apache-2.0, pinned `581d398`). Neither restricts these derivatives.
|
| 15 |
|
| 16 |
## What is deliberately NOT here, and how to get it
|
| 17 |
|
| 18 |
-
The
|
| 19 |
-
upstream terms; `Wikipedia_en.jsonl` and
|
| 20 |
-
carry **CC BY-SA**, which is share-alike and
|
| 21 |
-
would have forced a mixed-licence repository and
|
|
|
|
| 22 |
|
| 23 |
-
Rebuild the pool from its public sources
|
|
|
|
| 24 |
|
| 25 |
```bash
|
| 26 |
-
bash repro/
|
| 27 |
-
bash repro/
|
| 28 |
```
|
| 29 |
|
| 30 |
`corpora/manifest.json` and `corpora/manifest_ood.json` ship here and carry, per corpus: the
|
| 31 |
SHA-256 of the exact file used, the document count, the source dataset id and config, and the
|
| 32 |
three disjoint seed-block index ranges. A rebuild that matches those hashes is the same pool.
|
|
|
|
|
|
|
| 33 |
|
| 34 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
|
| 36 |
| corpus | rebuild |
|
| 37 |
|---|---|
|
| 38 |
| Pile-CC, StackExchange, Wikipedia_en, Github, USPTO_Backgrounds | **5/5 bit-identical** |
|
| 39 |
| OOD_CommonPile, TRAP_FineWeb, CONTROL_PubMed_2023 | **3/3 bit-identical** |
|
| 40 |
| `corpora/manifest.json` regenerated | **byte-identical** |
|
| 41 |
-
| OOD_Wikipedia_2023, OOD_arXiv_2023 | not re-run (53 MB / 110 MB of streaming)
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
stable by construction because its configs are date-stamped (`20231101.en`, `2024-06`).
|
|
|
|
| 1 |
# ATTRIBUTION
|
| 2 |
|
| 3 |
This repository is **CC BY 4.0 without qualification**. It contains only what this work produced.
|
| 4 |
+
Copyright (c) 2026 Tarush Gupta.
|
| 5 |
|
| 6 |
## What is here
|
| 7 |
|
| 8 |
| what | size | licence |
|
| 9 |
|---|---|---|
|
| 10 |
+
| `results/**/*.pt`: 380 fitted Jacobian maps and 2 activation caches | 16.2 GB | CC BY 4.0 |
|
| 11 |
+
| `results/**/*.json`, `results/**/*.npz`: experimental records, plus 222 map sidecars | ~41 MB | CC BY 4.0 |
|
| 12 |
+
| `corpora/manifest.json`, `corpora/manifest_ood.json`, `corpora/dataset_revisions.json` | 13 KB | CC BY 4.0 |
|
| 13 |
|
| 14 |
Derived from **Pythia** (EleutherAI, Apache-2.0) via the **Jacobian-lens** reference
|
| 15 |
implementation (Apache-2.0, pinned `581d398`). Neither restricts these derivatives.
|
| 16 |
|
| 17 |
## What is deliberately NOT here, and how to get it
|
| 18 |
|
| 19 |
+
The corpus plaintext is **not distributed**: eleven files built from seven public datasets, plus
|
| 20 |
+
one derived from them. It is third-party text under upstream terms; `Wikipedia_en.jsonl` and
|
| 21 |
+
`OOD_Wikipedia_2023.jsonl` are Wikipedia-derived and carry **CC BY-SA**, which is share-alike and
|
| 22 |
+
cannot be relicensed as CC BY 4.0. Shipping them would have forced a mixed-licence repository and
|
| 23 |
+
made the CC BY 4.0 claim above false.
|
| 24 |
|
| 25 |
+
Rebuild the pool from its public sources with the code repository,
|
| 26 |
+
[github.com/tarush-ai/fittingcorpus](https://github.com/tarush-ai/fittingcorpus):
|
| 27 |
|
| 28 |
```bash
|
| 29 |
+
bash repro/06_data.sh --build # builds at the pinned revisions, then hash-verifies against the manifests
|
| 30 |
+
bash repro/06_data.sh --verify # verify a pool you already have
|
| 31 |
```
|
| 32 |
|
| 33 |
`corpora/manifest.json` and `corpora/manifest_ood.json` ship here and carry, per corpus: the
|
| 34 |
SHA-256 of the exact file used, the document count, the source dataset id and config, and the
|
| 35 |
three disjoint seed-block index ranges. A rebuild that matches those hashes is the same pool.
|
| 36 |
+
`OOD_News_2024_dedup` has no upstream of its own: the manifest records the rule that derives it
|
| 37 |
+
from `OOD_News_2024`.
|
| 38 |
|
| 39 |
+
`corpora/dataset_revisions.json` pins the upstream dataset commit for all eight upstreams the code
|
| 40 |
+
reads: the seven corpus sources, and the Pythia training stream the containment scores are measured
|
| 41 |
+
against. Both builders (`src/build_corpora.py`, `src/build_ood_corpora.py`) read it and refuse to load a
|
| 42 |
+
dataset that has no pinned revision.
|
| 43 |
+
|
| 44 |
+
**Measured 2026-08-31, rebuilding from scratch on a clean machine** (the day before the revision
|
| 45 |
+
pins were added, so this is also evidence that the upstreams had not moved by then):
|
| 46 |
|
| 47 |
| corpus | rebuild |
|
| 48 |
|---|---|
|
| 49 |
| Pile-CC, StackExchange, Wikipedia_en, Github, USPTO_Backgrounds | **5/5 bit-identical** |
|
| 50 |
| OOD_CommonPile, TRAP_FineWeb, CONTROL_PubMed_2023 | **3/3 bit-identical** |
|
| 51 |
| `corpora/manifest.json` regenerated | **byte-identical** |
|
| 52 |
+
| OOD_Wikipedia_2023, OOD_arXiv_2023 | not re-run (53 MB / 110 MB of streaming) |
|
| 53 |
+
| OOD_News_2024 and its derived `_dedup` | not in that record |
|
| 54 |
+
|
| 55 |
+
Every corpus in the last two rows carries its SHA-256 in the manifest, and `--build` checks it.
|
| 56 |
+
**A pinned revision protects against an upstream edit, but not against an upstream deletion.** If
|
| 57 |
+
a source dataset or its pinned commit is removed from the Hub, that corpus cannot be rebuilt; the
|
| 58 |
+
manifest hash still says which file the results were computed on.
|
|
|
LICENSE
CHANGED
|
@@ -1,25 +1,24 @@
|
|
| 1 |
Creative Commons Attribution 4.0 International (CC BY 4.0)
|
| 2 |
|
| 3 |
-
Copyright (c) 2026
|
| 4 |
|
| 5 |
Everything in this repository is licensed under CC BY 4.0:
|
| 6 |
https://creativecommons.org/licenses/by/4.0/
|
| 7 |
|
| 8 |
-
|
| 9 |
-
repository contains only artifacts this work produced.
|
| 10 |
|
| 11 |
-
results/
|
| 12 |
-
results/**.json, *
|
| 13 |
-
corpora/
|
| 14 |
-
counts, source dataset id and
|
| 15 |
|
| 16 |
-
NO THIRD-PARTY CORPUS TEXT IS DISTRIBUTED HERE. The
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
are what make the rebuild verifiable.
|
| 22 |
|
| 23 |
-
The
|
| 24 |
reference implementation (Apache-2.0, pinned at 581d398). Neither upstream licence restricts the
|
| 25 |
-
licensing of the derived
|
|
|
|
| 1 |
Creative Commons Attribution 4.0 International (CC BY 4.0)
|
| 2 |
|
| 3 |
+
Copyright (c) 2026 Tarush Gupta
|
| 4 |
|
| 5 |
Everything in this repository is licensed under CC BY 4.0:
|
| 6 |
https://creativecommons.org/licenses/by/4.0/
|
| 7 |
|
| 8 |
+
The licence is unqualified because this repository contains only artifacts this work produced:
|
|
|
|
| 9 |
|
| 10 |
+
results/**/*.pt fitted Jacobian maps (and two activation caches)
|
| 11 |
+
results/**/*.json, *.npz every experimental record, derived statistic and map sidecar
|
| 12 |
+
corpora/*.json the build record of the corpus pool: per-file SHA-256, document
|
| 13 |
+
counts, source dataset id, config and pinned revision, seed-block ranges
|
| 14 |
|
| 15 |
+
NO THIRD-PARTY CORPUS TEXT IS DISTRIBUTED HERE. The corpus files this work used are third-party
|
| 16 |
+
text under upstream terms, and two are Wikipedia-derived and carry CC BY-SA, a share-alike
|
| 17 |
+
licence that cannot be relicensed as CC BY 4.0. Rather than qualify this licence, the pool is
|
| 18 |
+
withheld in full and rebuilt from its public sources by `bash repro/06_data.sh --build` in the
|
| 19 |
+
code repository, https://github.com/tarush-ai/fittingcorpus. The manifests above are this work's
|
| 20 |
+
own record of that pool and are what make the rebuild verifiable.
|
| 21 |
|
| 22 |
+
The maps are derived from EleutherAI's Pythia suite (Apache-2.0) using the Jacobian-lens
|
| 23 |
reference implementation (Apache-2.0, pinned at 581d398). Neither upstream licence restricts the
|
| 24 |
+
licensing of the derived maps recorded here.
|