docs/02-mix-plan.md

# 02 — Mix plan: sources, counts, rationale, and the build

Phase 1 deliverable. Target: **1.0 B trained tokens** (§2 band 900M–1.1B; frozen number confirmed at
Gate 3). Every repo id, size, date and licence below was read live on 2026-09-19 from the Hub card, the
`/api/datasets/<id>` metadata endpoint, or parquet footers over HTTP — not from memory. Where a card and
the stored data disagree, that is recorded rather than smoothed over.

## 1. Selection rules applied, operationally

- **§3.4 English only.** Every source is either English-config-scoped (`20250702.en`, `eng_Latn`,
  `sample/10BT`) or carries a per-row language field we filter on. Code and math are allowed and are
  deliberately included.
- **§3.5 No raw crawls.** Excluded as a class: `allenai/c4`, `HuggingFaceFW/fineweb`,
  `Skylion007/openwebtext`, `tiiuae/falcon-refinedweb`. What is admitted instead is
  **quality-scored** (`fineweb-edu` carries a per-row classifier `score`/`int_score`),
  **curated/rewritten** (`finewiki`, `cosmopedia`), **synthetic-textbook** (`finephrase`), or
  **openly-licensed domain corpora** (`common-pile/*`, `stack-v3-train`). Common-Crawl appears only as
  the *provenance* of a classifier-filtered subset, never as the selection criterion.
- **§3.3 No contamination.** Two mechanisms, not one assertion:
  1. **Prefer sources with published decontamination.** `HuggingFaceTB/finemath` documents 13-gram
     removal against GSM8K, MATH, MMLU and ARC test sets, with a public audit log
     (`HuggingFaceTB/finemath_contamination_report`).
  2. **Reject sources whose own cards validate on the eval suite.** That is why `openbmb/UltraData-Math`
     is **dropped entirely** despite being attractive on paper (apache-2.0, 170B+ tokens): its card
     lists MMLU/ARC-E/ARC-C/HellaSwag/PIQA/Winogrande/GSM8K/MATH500 as validation targets and its L3
     tiers are synthetic *exam-shaped* text. Its en+zh mixing would also need a language filter.
     Similarly excluded: `nvidia/Nemotron-Math-v2` (AoPS/SE solution traces, i.e. adjacent to GSM8K's
     own source pool, and solution-formatted in a way that quietly steers toward instruction data).
  3. The mechanical overlap audit of §5 runs against **validation/dev material only**; test splits stay
     untouched until Phase 6.

## 2. The mix: 1.31 B raw → ~1.05 B after dedup → ~1.0 B trained

| # | Source (config) | raw tokens | GB to pull | licence | rationale, from corpus properties |
|---|---|---|---|---|---|
| 1 | `HuggingFaceFW/fineweb-edu` `sample/10BT`, keep `int_score≥4` | 300M | 1.1 | ODC-BY | Classifier-scored general English; the only cheap source of broad register coverage. Score-gating is a *quality* filter, which §3.5 permits, and it is the source's own documented axis. |
| 2 | `HuggingFaceFW/finepdfs-edu` `data/eng_Latn` | 150M | 0.63 | ODC-BY + CC ToU | Globally-deduplicated long-form educational PDFs. Long document structure teaches coherence over >1k tokens, which chunked web prose does not. `eng_Latn` only (69 scripts exist). |
| 3 | `HuggingFaceTB/cosmopedia` stanford+openstax+khanacademy+auto_math_text | 130M | 0.33 | **Apache-2.0** | Synthetic textbooks: causal and explanatory chains ("because", "therefore", worked steps). This is the register that builds science-QA and multi-step reasoning ability. Per-config GB measured, not estimated. |
| 4 | `HuggingFaceTB/cosmopedia` `wikihow` | 30M | 0.05 | Apache-2.0 | Procedural how-to steps → ordered physical-world inference. Small slice, specific job. |
| 5 | `HuggingFaceFW/finewiki` `data/enwiki` | 80M | 0.43 | **CC-BY-SA-4.0** | Link-resolved, rewritten encyclopedic prose; definitional and comparative sentence forms that factual/recall-style prompts reward. Heaviest GB/token in the mix, hence capped. |
| 6 | `omarkamali/wikipedia-monthly` `20250702.en` | 60M | 0.03 | CC-BY-SA-4.0 | Cheapest encyclopedic tokens available (0.5 GB/Btok) and *current*; drop the `raw_mediawiki` column or it dominates the byte budget. |
| 7 | `HuggingFaceTB/finemath` `finemath-4plus` | 130M | 0.25 | ODC-BY | Best math density per GB (1.9) **and** the only candidate with a published eval-set decontamination report. |
| 8 | `open-web-math/open-web-math` | 70M | 0.15 | ODC-BY (card body) | Forum/webbook math in natural prose, incl. LaTeX. Complements #7's textbook skew with multi-turn human reasoning text. Oldest source here (2023-10) — cited as such. |
| 9 | `HuggingFaceFW/finephrase` `tutorial` + `faq` | 90M | 0.31 | ODC-BY | Rewritten into stepwise-tutorial and question/answer registers — forms largely absent from organic prose. |
| 10 | `HuggingFaceFW/finephrase` `table` | 30M | 0.10 | ODC-BY | Tabular→narrative realisation: reading quantities, units and row/column structure. |
| 11 | `HuggingFaceCode/stack-v3-train`, `license_type=="permissive"`, per-language cap | 130M | 0.52 | ODC-BY + per-file SPDX | Code for compositional syntax and deterministic multi-step logic. **Must** filter to permissive licences (constraint 4) and cap XML/HTML/JSON/JS, which the card notes are byte-dominant. |
| 12 | `SimpleStories/SimpleStories` | 40M | 0.07 | MIT | Syntactically simple long-range narrative. Disproportionately useful below ~200M params, where the model otherwise never sees a consistent referent across a full context window. |
| 13 | `common-pile/libretexts` + `common-pile/arxiv_abstracts` | 40M | 0.08 | CC0 / per-doc PD/CC-BY | Openly-licensed OER textbooks and scientific abstracts; cleanest licence hygiene in the pool (arXiv metadata is CC0). |
| | **total raw** | **1,310M** | **≈4.35 GB** | | |

**Headroom, and why it is sized like this.** §Phase 2 requires slack above the target for dedup loss
and a held-out split. Cross-source near-dedup is expected to cost **12–18 %** (FinePhrase's four configs
are rewrites of the *same* ~338.7M source documents, so they must be deduplicated against the shared
`id`/`url` or they buy topic echo rather than knowledge; #2/#5/#6 overlap on encyclopedic topics;
#1/#2 overlap on educational web content). 1,310M × 0.85 ≈ 1,114M, minus a 2 % validation holdout
(≈22M) → **~1.09 B available**, which brackets the 1.0 B target and leaves room to land at 0.9 B
without re-sourcing if Gate 3's throughput measurement forces the pre-registered fallback.

**Sits comfortably inside the instance.** ≈4.35 GB of shards + ~2 GB of Arrow/tokeniser working
overhead + 2 GB of final token store ≪ the measured **19.5 GB** `/kaggle/working`. Every source here is
pullable as individual 270 MB–3 GB shards, so the working directory is never the binding constraint —
provided the build pulls shard-by-shard and deletes after tokenising, which §4 requires it to do.

## 3. Excluded, with reasons (so nobody re-litigates from scratch)

| excluded | why |
|---|---|
| `allenai/c4`, `HuggingFaceFW/fineweb`, `Skylion007/openwebtext`, `tiiuae/falcon-refinedweb` | raw/unfiltered crawls → §3.5 |
| `openai/gsm8k`, `cais/mmlu`, `allenai/ai2_arc`, `hellaswag`, `piqa`, `winogrande`, `truthfulqa` | the eval tasks themselves → §3.3. Also: never glob a directory that could catch these. |
| `nvidia/Nemotron-CC*`, `-Math-v1`, `-Code-v1` | `gated: manual`; anonymous card and file reads return *"Access to dataset … is restricted"* → token counts unverifiable and republication status unclear |
| `openbmb/UltraData-Math` | card validates on the eval suite; exam-shaped L3 tiers; en+zh mixed |
| `nvidia/Nemotron-Math-v2` | solution-formatted AoPS/SE traces, GSM8K-adjacent; reads as instruction data |
| `allenai/peS2o` | ~11 GB per Btok by measurement — worst ratio in the pool for 42 B tokens we do not need |
| `HuggingFaceTB/smollm-corpus` `cosmopedia-v2` | duplicate role with #3; also a documented card-vs-data mismatch (card 17.8 B vs ~31.7 B implied by stored `token_length`) |
| `HuggingFaceFW/clean-wikipedia` | exists but README body is literally "Please see FineWiki instead" → deprecated |
| `HuggingFaceTB/smollm-corpus` `python-edu` | **not a corpus.** Verified schema is `blob_id, repo_name, path, length_bytes, score, int_score` — a filter list pointing at gated `bigcode/the-stack-v2`, with no `text` column. Do not budget code tokens from it. |
| `bigcode/the-stack-v2-dedup`, `codeparrot/github-code` | not verified this session; `the-stack-smol` is `gated: auto` |
| `TinyLlama` 32k tokenizer | Llama-2-derived; redistribution licence status unverified → excluded from both tokenizer and any data |
| `Cion-lab/Pretraining-10B-v1` (pre-existing in the account) | composition/mix unknown, token counts estimated with the **Qwen2.5** tokenizer, provenance not inspectable → cannot be cleared under §3.5 or §3.3 |

## 4. Card-vs-data warning

Two sources disagree with themselves: `cosmopedia-v2` (17.8 B card vs ~31.7 B stored) and `finephrase`
(486 B "completion tokens" card vs ~1.4 B implied by stored rows × inherited `token_count`). FinePhrase's
`token_count`/`score` columns describe the **source document, not the rewrite**. **Therefore: no token
number in this project is ever taken from a card.** Counts come from running our own tokenizer over the
bytes, which is also what the Gate 2 manifest will record.

## 5. Build pipeline (Phase 2 design, on Kaggle CPU)

1. **Pull shard-by-shard, delete after consuming.** 73–89 MB/s measured (35–89 across runs), so 4.35 GB
   arrives in minutes; the constraint is *peak residency*, not bandwidth. Never materialise the mix.
2. **Normalise + filter.** Language filter on the row-level field where present; `int_score≥4` for
   fineweb-edu; `license_type=="permissive"` + per-language cap for stack-v3; drop `raw_mediawiki`.
3. **Dedup, memory-bounded.** Exact: 64-bit hash of the normalised document. Near: MinHash + LSH over
   13-gram shingles (matching finemath's documented practice), **processed in bands against a
   disk-backed index**, because the box is a 30 GiB cgroup with **no swap** — a billion sketches in one
   dict is how a free-CPU job dies at hour six.
4. **Contamination audit (§3.3).** Mechanical 13-gram overlap of the surviving mix against benchmark
   **validation/dev** material only — including **MMLU's `dev` split specifically**, since those are the
   few-shot exemplars the harness will use, so overlap there is contamination by construction. A script
   computes and writes counts per source; **no item text is read, displayed or summarised by the agent.**
   Any source failing a stated threshold is dropped or cleaned; result written up in `docs/`. **Test
   splits are not opened until Phase 6.**
5. **Tokenise** with the frozen SmolLM2 BPE (49,152 — §3 of `01-plan`). Measured cost on this shape:
   660,943 tok/s across 4 cores → **1.31 B tokens in ~0.55 h**. Not a bottleneck; process-shard rather
   than thread-shard, since rayon scaling saturated at 2.7× on 4 cores.
6. **Shard for exact positional resumption (§Phase 2 requirement).** Concatenate token ids into fixed-size
   **`uint16`** shards (`vocab 49,152 < 65,536`, so uint16 is exact and halves the bytes): **40 M tokens =
   80 MB per shard**, ~28 shards, ~2 GB total. Then a resume position is literally
   `(shard_index, token_offset)` — one integer pair, memmap'd, so `skip_first_batches` on a replaying
   `Trainer` becomes an index advance rather than a decode. This is the property that makes §3.1's
   "same data, same position" checkable instead of hoped for.
7. **Hold out validation:** ~22 M tokens sampled **proportionally across sources**, never whole documents
   (whole-document holdout leaks topic statistics and makes PPL optimistic), written as its own shard so
   validation PPL is bit-reproducible at Gate 4.
8. **Publish** as public `ounce100m-mix-v1` with a manifest recording: per-source and total token counts,
   shard count and size, tokenizer identity (repo + revision + sha256 of `tokenizer.json`), per-shard
   sha256, the dedup parameters, the audit counts, and the cursor semantics of step 6.

## 6. Licensing the derived mix

#5 and #6 are **CC-BY-SA-4.0** (share-alike), and `wikimedia` additionally carries GFDL. A published
derivative containing them must be offered under SA with attribution and a change log. That is compatible
with §2's "repos are public", so the plan is: **license the mix CC-BY-SA-4.0** and ship an
`ATTRIBUTION.md` listing every source, its licence, and the transformations applied. ODC-BY sources
(#1, #2, #7–#10) require notices kept and changes documented; `stack-v3-train` requires the per-file
SPDX filter of §2 step 11 so we never republish a non-permissive file.

**If a fully permissive mix is later required**, drop #5 and #6 (140 M raw) and backfill from #1/#3. The
trade is a small loss of encyclopedic density for a licence with no reciprocal obligation — recorded
here so the choice is available and deliberate rather than discovered under time pressure.

## 7. Gate 2 checklist, restated as things that must be true

- [ ] dataset live on the Hub as a public `ounce100m-*` repo
- [ ] token count **verified against the manifest** by an independent recount, not by the builder's own number
- [ ] contamination audit passed **and written up** (per-source overlap counts, validation/dev only)
- [ ] validation split reserved, its token count recorded
- [ ] shard layout proven small enough that a training job reads a few shards at a time and never
      materialises the mix — demonstrated by a job that reports peak `/kaggle/working` usage while
      consuming shards, not asserted from the arithmetic above
- [ ] exact-position cursor `(shard_index, token_offset)` demonstrated to be reproducible

## 8. Risks

1. **Dedup is the only stage with real wall-clock cost** and the only one that can exhaust RAM. Design
   for bands + disk; measure on one source before all thirteen.
2. **Unauthenticated Hub rate limits.** Jobs currently read anonymously and the API warns about it; at
   ~50 shard downloads that is probably fine, but 429s mid-build are plausible. Mitigation is the
   mounted-credential path (§8 of `01-plan`), which is being verified now.
3. **Card token counts are unreliable** (§4) — every number in this file is either measured over HTTP or
   will be replaced by our own tokeniser's count at Gate 2.
4. **`finepdfs-edu` adds Common Crawl's Terms of Use** on top of ODC-BY. Included because the subset is
   classifier-selected and globally deduplicated rather than a dump, but it is the least clean licence
   admission in the mix and the easiest to cut if the attribution review gets nervous.
5. **Two sources are rewrites/synthetic** (#3, #4, #9, #10). That is permitted explicitly by §3.5, but a
   34 % synthetic share can produce fluent-but-homogeneous prose; the proportions above keep organic
   English (sources 1, 2, 5, 6, 8, 12, 13) in the majority, and that balance is a deliberate choice
   worth revisiting only with evidence.

---

## 9. Phase 2 step 1 ran — §2 corrected by measurement

`dodosoomro/ounce100m-p2-source-inventory` (CPU, 116 s, 1,500 rows per source, tokenised with the
**frozen SmolLM2 tokenizer**: vocab 49,152 confirmed, decode round-trips, 5.88 chars/token on a formal
probe sentence). **13 of 18 entries resolved; 5 failed on config names, and two sources turned out to be
unusable as described.** §2's estimates were right in aggregate (predicted ≈4.35 GB, measured 4.46 GB for
the 570 M tokens that resolved) but wrong in specifics, so the specifics are replaced rather than
averaged.

### 9.1 Config-name corrections (the datasets-server truth, from the error messages themselves)

| §2 said | correct value | note |
|---|---|---|
| `fineweb-edu` `sample/10BT` | **`sample-10BT`** | hyphen, not a path. Full set: `default`, `sample-10BT`, `sample-100BT`, `sample-350BT`, plus per-dump `CC-MAIN-2025-05/08/13/18/21/26…` |
| `finemath` `finemath4plus` | **`finemath-4plus`** | also `finemath-3plus`, `infiwebmath-3plus`, `infiwebmath-4plus` |
| `finewiki` `data/enwiki` | config **`en`** | configs are plain language codes (`ab`, `ace`, `af`, …), not directory paths |
| `wikipedia-monthly` `20250702.en` | **`20260101.en`** | the 2025-07 config is gone; **newer** dumps now exist (`20260101.*`), which is better for currency anyway |
| `stack-v3-train` `python` | **`default`** only | **there are no per-language configs.** Language is a row field, and rows are whole repositories with a nested `files` struct |

Every one of these would have been a mid-build failure. Recorded in `docs/00-platform-notes.md` style:
the card's own naming is not an API — `load_dataset`'s error is.

### 9.2 Measured yield per document — this is what the token arithmetic actually runs on

| source | tokens/doc | chars/doc | chars/token | non-English-flagged rows | verdict |
|---|---|---|---|---|---|
| `finepdfs-edu` `eng_Latn` | **1,321** | 14,352 | 3.98 | **128/1500 = 8.5 %** | keep, **must add our own language filter** |
| `cosmopedia` `stanford` | 932 | 4,917 | 5.30 | 15/1500 = 1.0 % | keep |
| `cosmopedia` `wikihow` | 907 | 4,541 | 4.97 | 0.7 % | keep |
| `cosmopedia` `openstax` | 765 | 3,820 | 4.93 | 0.8 % | keep |
| `cosmopedia` `auto_math_text` | 689 | 2,811 | 4.14 | 0.8 % | keep |
| `cosmopedia` `khanacademy` | 677 | 3,047 | 4.26 | 1.1 % | keep |
| `open-web-math` `default` | **1,182** | 6,795 | **3.31** | **112/1500 = 7.5 %** | keep, needs en filter; low chars/token = LaTeX density, which is the point |
| `finephrase` `tutorial` | 779 | 4,903 | 4.58 | 1.1 % | keep. Column is **`text`**, not `completion` |
| `finephrase` `faq` | 744 | 4,965 | 4.60 | 1.1 % | keep |
| `finephrase` `table` | 715 | 4,772 | 4.59 | 1.1 % | keep |
| `SimpleStories` | **285** | 1,260 | 4.33 | 0.5 % | keep, but it is 4× the *row* count for the same tokens |
| `common-pile/arxiv_abstracts` | 185 | 791 | 4.31 | 2.2 % | keep. Column is **`text`**, not `raw_content` |
| `common-pile/libretexts` | **75** | 964 | **2.42** | **1396/1500 = 93 %** | **DROP** |

**`libretexts` is disqualified, and the reason matters.** Its `text` field is not prose: 93 % of sampled
rows failed a naive English test, chars/token collapsed to 2.42, and the first example is a bare
`https://math.libretexts.org/Bookshelves/...` URL. The Common-Pile schema puts real content in
`metadata`/`raw_content` behind a per-source convention, so the column we assumed is an identifier field.
Rather than reverse-engineer four sub-corpora's schemas for 10–20M tokens of marginal value, it is cut.
`arxiv_abstracts` from the same publisher *did* work (its `text` is real abstract prose), so this is a
per-dataset fact and not a Common-Pile-wide one.

### 9.3 Two open problems this created, and how they are being handled

1. **Code no longer has a cheap path.** With no per-language config, getting 60–130M tokens of Python/JS
   from `stack-v3-train` means streaming whole-repo rows, opening a nested `files` struct, and filtering
   on a row field — for a source that concentrates in XML/HTML/JSON anyway. Code is not mandatory (§3.4
   makes proportions our call), and at 100M params / 1B tokens the marginal value is genuinely unclear.
   **Action: a dedicated mini-probe before committing** — measure rows-sampled-per-Python-repo and the
   nested schema — and hold code at **≤60M** in the plan until that returns. If it is awkward, drop it;
   that is a defensible §3.4 choice, not a failure.
2. **`eng_Latn` and `open-web-math` are not actually English-only** (8.5 % and 7.5 % of sampled rows fail
   a crude English test; the finepdfs sample included Cyrillic inline in a single "English" document).
   §3.4 is English-only, so **a language filter is now a required pipeline stage, not a courtesy** —
   applied per row on our side, on top of whatever the upstream config claims. `finepdfs` carries
   `full_doc_lid` / `per_page_languages` columns to do this cheaply; the general fallback is
   `language_score` where present and a fastText-style gate where not.

### 9.4 Revised totals

Confirmed-and-usable now: 570M target tokens across 13 resolved configs, **≈4.5 GB**, at measured yields
of roughly **909k documents** — plus `fineweb-edu`/`finewiki`/`wikipedia-monthly`/`finemath` recoverable
with the corrected config names (§9.1), which restores ~570M of the planned 1,310M. Net effect on the
plan: **the shape survives, the line items do not.** §2's per-source token targets stay as targets;
§2's config strings and two column names are superseded by this section.

**Directly measured consequence for the build:** 909k documents for 570M tokens ≈ 627 tokens/document
average, so the mix is ~1.6–2.1M documents for 1.0–1.3B tokens. That is the number the dedup design has
to be sized against — under 2.1M documents, a MinHash signature table comfortably fits the 30 GiB RAM,
which materially de-risks the one stage §8 called the real cost.

## 10. Builder implemented, and one blocking flaw found and fixed

`code/build/build_mix.py` (mirrored in `Cion-lab/ounce100m-code`) is the Phase 2 builder. It stages each
source separately (stream → normalise → gates → dedup → tokenise) and then **merges** the staged shards
into the final mixed shards by largest-deficit round-robin over documents.

**E-016, and why the smoke test was mandatory.** The first version consumed one source to its target and
only flushed a shard when the buffer filled. Since fineweb-edu's target alone is ~37 shards of 8M tokens,
the first third of the mix would have been *nothing but web text*, then textbooks, then math — a curriculum
drift across the run that no LR schedule can repair, and one the builder itself reported as success
(checksums, headers, token counts and cursor probes were all clean). The verifier caught it as
`distinct_sources_per_shard_min: 1`. The shuffle that was supposed to fix ordering only permuted documents
*inside* a single-source buffer. After the two-stage rewrite, the same smoke run reports
**`min = max = median = 4`** sources in every shard, 12,001,386 tokens recounted from the bytes,
0 out-of-range ids, 40/40 cursor probes and 40/40 document-boundary reconstructions correct.

**Shard format** (shared by staged and final, so document boundaries survive to the trainer):

```
header  : <IIII  = vocab, n_docs, n_tokens, flags        16 B
offsets : uint32 × (n_docs + 1), cumulative, [0] = 0
ids     : uint16 × n_tokens        (vocab 49,152 < 65,536, so uint16 is exact and halves the bytes)
doc i   = ids[offsets[i] : offsets[i+1]]
```

Target 8M tokens/shard (16 MB, ~150 shards, ~2.5 GB for the mix). The resume cursor is
`(shard_index, token_offset)` — one integer pair, which is what makes §3.1's "same data, same position"
checkable and what keeps `Trainer`'s replaying `skip_first_batches` cheap on a map-style read.

**Interruption safety, which was not optional.** `/kaggle/working` is destroyed when a session ends, and a
full staging pass is hours of streaming. So each source is published to `Cion-lab/ounce100m-mix-stage`
(the *stage* repo, distinct from the final `ounce100m-mix-v1`) the moment it completes, with a
`record.json` holding its shard list, token count and drop statistics. A new session rebuilds its stage
bookkeeping from the Hub and skips every source already published; only the source in flight at the moment
of the kill is re-staged. `code/build/smoke_hub_resume.py` rehearses this by wiping the local tree between
stage and merge, because "merge-only with an empty state and no restore path" is a failure mode that exits
with code 0 while emitting an empty mix.

**Revised raw total: 1.23 B tokens = 23 % headroom** over the 1.0 B trained target (raised fineweb-edu to
340M, finepdfs-edu to 180M, finemath to 160M, because §9's target edits had quietly pulled the plan from
1.31 B down to ~1.13 B — about 11 % headroom, which is not headroom once a 12–18 % dedup loss is applied).
16 sources, no code source yet (§9.3).

**Gates the build must satisfy before Gate 2 is claimed**, as assertions in `verify_mix.py` rather than
prose: every shard sha256 matches the manifest; no trailing bytes; offsets well-formed; zero out-of-range
ids; total tokens recounted from bytes equals the manifest; and **≥ half the distinct sources present in
every shard**. The last one exists because the first build passed everything else.

## 11. Interruption safety proven, and the build that this authorises

A CPU session's `/kaggle/working` is destroyed when it ends. Everything below was measured, not assumed
(`dodosoomro/ounce100m-p2-cold-resume-rehearsal` **v6**, 17:35Z, 55 s wall-clock, pinned REV `428c5234`).

| Step | Observed |
|---|---|
| 1. stage two sources, publish each | 2,400,867 tokens staged; `record.json` + `shard-0000.bin` per source on the Hub |
| 2. **wipe the local tree** (`root_exists: false`, `stage_dir_exists: false`) | faithful simulation of a session kill, not a soft restart |
| 3. `--merge-only` with the Hub as the only state | `hub: restored 2 staged source(s) (2,400,867 tokens)` → merged shard 2,352,747 tok / 5,815 docs, **2/2 sources in it**, val shard 48,120 tok |
| 4. `verify_mix.py` over the restored mix | `PASS: true`, `sha_ok 1/1`, `out_of_range_ids: 0`, `trailing_bytes: 0`, cursor probes **40/40 boundaries reconstructed, 0 mismatches**, `recount_matches_manifest: true` |
| 5. verdict + self-cleanup | `COLD_RESUME_PASSED=True`, `cleanup=deleted`, throwaway repo confirmed gone from the namespace |

v4 of the same rehearsal had reported `false` over an identically-working pipeline: a stray
`print("shards=", …)` inside the `BUILD_JSON` block made the parent's `json.loads` fail and a helper fold
that into `None` (memory/ERRORS.md **E-019**, and the report-shape recurrence **E-023**). The fix was to
the harness, plus a `(obj, why_null)` return and a `manifest.json` fallback so a verdict can never again
be silently null.

**The build now running** (`dodosoomro/ounce100m-p2-full-mix-build`, 17:38Z, CPU, `sessionTimeoutSeconds
43200`, pinned REV `98e49560`): `build_mix.py --root /kaggle/working/mixroot --hub-repo
Cion-lab/ounce100m-mix-stage` with defaults — stage cap 1.31 B tokens, per-source targets summing to
1.23 B, 8 M tokens/mixed shard, `VAL_STRIDE 50` holdout, shuffle seed 20260919. Each source publishes as
it completes, so an interruption costs at most one source's streaming time, and a re-launch restores the
finished ones from the Hub (step 3 above is that exact path).

**Audit semantics, hardened the same way.** `audit_contamination.py` compares 13-token windows against
`{train, validation, dev}` of the eight tasks — never `test` — and emits counts only. It now also reports
`shards_missing`, and its verdict is composite: `AUDIT_PASSED` requires every manifest shard present and
read, > 0 documents and grams measured, ≥ 6 readable reference sets, zero reference errors, and zero
overlap. Exit 6 means *overlap found* (a real finding); exit 7 means *the audit did not measure* (a bug).
A zero-overlap report over shards that were not on disk is no longer expressible.

**Publishing ships the tokenizer.** `publish_mix.py` copies `tokenizer.json` into the dataset root,
re-hashes it and aborts with exit 8 unless it matches the sha256 recorded in the manifest — so the
published ids and the published tokenizer are provably the pair that produced them (§3.12).

## 12. Gate 2 — the record, including where it failed and what that failure was worth

**Merge and verify: passed, twice, reproducibly.** From the 15 staged sources
(`Cion-lab/ounce100m-mix-stage`, 1,150,015,841 tokens staged):

| field | value |
|---|---|
| merged shards | **141** (each drawn from all 15 sources) |
| train tokens | **1,127,163,952** |
| held-out validation | **22,851,889** in 12 shards (every 50th merged document) |
| documents | 1,078,807 |
| per-source share | 75,144,263 tokens each, equal by construction |
| merge time | 16.4 s (restore from the Hub included: 123.8 s) |
| `verify_mix` | **`PASS: true`** — sha256 141/141, headers 141/141, 0 bad offsets, 0 out-of-range ids, 0 trailing bytes, recount matches manifest, 40/40 cursor probes, `distinct_sources_per_shard_min = 15` (floor was 7) |
| `finewiki__en` | **never staged.** Gate 2's own diagnostic showed the columns present and `first_row_ok: true` with a truncated `ImportError` in the loading script. 15 sources at 1.127 B is above the 1.0 B target with 12.7 % headroom, so this is a composition note, not a blocker |

**Then the audit refused to publish, and it was right to.** `audit_contamination` exited 7:

| task | 13-gram hits | reference size (verified against the dataset server) |
|---|---|---|
| mmlu | 904,288 | dev 285 + validation 1,531 items ≈ 149k grams |
| gsm8k | 185,600 | train 7,473 items = 360,656 grams |
| hellaswag | 1,852 | train 39,905 + validation 10,042 |
| arc_easy | 358 | train 2,251 + validation 570 |
| arc_challenge | 214 | train 1,119 + validation 299 |
| piqa | **not measured** | the loader yielded 0 of 16,113 train rows without raising |
| winogrande | (not in the visible excerpt) | train 40,398 + validation 1,267 |
| truthfulqa | (repo name was wrong) | validation 817, `multiple_choice` |

`expected_false_positives` for the whole scan was **≈0.0001**, so these are matches, not collisions. The
first reading I wrote in `STATE.md` — "GSM8K's whole train split is in the mix" — was an over-claim: hits
are counted per *mix* gram, so a small number of frequently repeated reference grams can produce a large
count, and 185,600 hits against 360,656 distinct GSM8K grams means "up to half of its gram vocabulary
occurs somewhere in 1.1 B tokens". The columns that settle severity are `mix_documents_majority_hit`
(documents whose grams are more than half matched → verbatim inclusion) and `source_hit_document_fraction`
(which source, and how much of it) — and both are only computable on the fixed audit, which is what Gate
2c/2d exist to produce.

**Why upstream did not catch it, verified rather than assumed:** FineMath's card states it removed 13-gram
overlap against the **test** sets of GSM8k, MATH, MMLU and ARC; Cosmopedia's describes a 10-gram +
`difflib` pass against **test benchmarks**; OpenWebMath documents only SimHash self-deduplication with no
benchmark claim. `validation`/`dev`/`train` overlap was therefore never in scope for any of them — and
`validation` is the split lm-eval scores HellaSwag, PIQA, Winogrande and TruthfulQA on
(`docs/05-eval-plan.md` §7).

**Gate 2 remained unsigned at the end of this stage.** `Cion-lab/ounce100m-mix-v1` did not exist yet. The
path to it: attribution →
drop the offending documents by mask (or a source if attribution says it is broadly soaked) → re-merge
with `--exclude-dir` → re-audit until `AUDIT_PASSED` with all 8 tasks covered → `publish_mix`, which now
refuses to ship unless the audit passed, covered all 8, scanned the val shards, and its `mix_bytes_sha256`
matches the exact manifest being published. `E-030` explains why the first filtered re-merge could not
work and what was changed.

### 12.4 Resolution — the filtered mix, measured (Gates 2b–2f, 2026-09-19)

**Attribution first, always.** Gate 2b/2c scanned the *staged* shards per source, which is the only way to
say where contamination lives rather than how much the merged mix contains. Of 1,955,832 staged documents,
**10,425 (0.533 %)** carried at least one matched 13-gram and **0 were majority-matched** — shared spans,
not wholesale inclusion. The count concentrated in five sources: finemath-4plus 2,688 documents, fineweb-edu
1,043, cosmopedia `auto_math_text` 568, finepdfs-edu 420, open-web-math 339, with the remaining ~5,367
spread over ten more. Per-source *hit counts* summed to ~50,700 while the merged scan reported 2,096,540 for
essentially the same gram volume; the discrepancy is unreconciled and recorded as **E-032c** rather than
smoothed over. It changes no decision: the exclusion is per-document and the post-filter overlap is zero
either way.

**The exclusion, and what it cost.** `audit_contamination --write-filter` emitted one `.u8` mask per source
(1 byte per staged document, ~700 KB total) marking documents that contain a matched window. Re-merging with
`--exclude-dir` dropped **5,878 documents = 17,366,967 tokens** (1.5 % of the mix) and the re-audit of the
result gave **`overlap_total = 0`** with 8/8 benchmark tasks covered, the held-out validation shards
scanned clean, and `MEASURED: true`. Final mix: **1,109,714,831 training tokens** in 141 shards plus
22,851,889 held-out validation tokens — still **11.0 %** above the 1.0 B target, so the pre-registered
0.9 B fallback (D-012) did not fire and the contamination fix cost nothing that the token budget had to
pay for.

**Reference coverage is now a precondition, not an assumption.** Gate 2b found that `piqa`'s train and
validation splits returned **0 rows** from the `datasets` library inside the Kaggle image while the server
reports 16,113 + 1,838 — the first audit had scored that as "clean". The audit now fetches each split's
server-side row count (`/size`), refuses any split it read to under 90 % of it, retries through the HTTP
`/rows` endpoint, and marks the whole report `MEASURED: false` if a loader still falls short. TruthfulQA's
answer text also sits under keys (`mc1_targets`, `mc2_targets`, `abstractions`) the first version never
read. All reference splits used are train/validation/dev; **no test split was opened** (§3.3).

**Why publication took three attempts (E-033).** Gates 2e and 2f both built and audited the mix correctly
and then died in the uploader: `publish_mix` sent one commit per file, and the Hub began rejecting every
request — including 30-byte files — after ~139 commits. That is a commit-rate ceiling, not a network fault,
and the "resumable" re-send was itself broken because `RepoFile.lfs` is a dict and `getattr(ent.lfs,
"sha256", None)` is therefore `None` for every shard, i.e. *nothing* looked published. The uploader now
reads `lfs["sha256"] or lfs["oid"]`, sends the remaining files as **one `upload_folder` commit**, and after
the commit re-lists the repo and re-hashes, exiting 11 if a path is absent and 12 if a remote digest
disagrees with the local bytes. `Cion-lab/ounce100m-mix-v1` held 142 of 173 files when Gate 2g re-ran the
same pipeline with it.

**Gate 2 is signed** (`dodosoomro/ounce100m-gate-2g-publish-mix-v1`, complete 2026-09-20T00:31Z, CPU, zero
GPU quota). The rebuilt mix reproduced Gate 2e's token total exactly — **1,109,714,831** — which is the
determinism claim in §11 tested a second time on a different session with the same inputs. `audit.json` on
the Hub reports `overlap_total = 0`, `mix_documents_with_any_hit = 0`, `majority_hit_docs = 0`,
`expected_false_positives = 0.0`, `tasks_covered` = all eight, `val_hits = 0` across 21,898 held-out
documents, and `AUDIT_PASSED`, `MEASURED`, `COVERED_ALL_SHARDS` all true, bound to these exact bytes by
`mix_bytes_sha256 = 732eec7f19e77f36…`. `verify_mix` exited 0. The batched uploader landed all 174 files
(139 train shards, 12 val shards, 16 `filter/` objects, `manifest.json`, `audit.json`, `tokenizer.json`,
`README.md`, `ATTRIBUTION.md`, `build_state.json`, `.gitattributes`), and four train plus two validation
shards were then read back from the anonymous `resolve` endpoint and matched the manifest's sha256 — so the
mix is public and serving, not merely listed (§3.13).

One editorial note on this section: §12.2's first draft treated `overlap_total` as a count of contaminated
documents. It is a count of matched *grams*, which is why a 41× difference between the per-source and
merged hit totals could coexist with a 1.3 % agreement on document counts (E-032c). The action taken —
drop the documents, re-audit to zero — was unaffected, but the number in the record was read wrongly for
two gates, and the fix was to write the density columns (`mix_documents_with_any_hit`,
`mix_documents_majority_hit`) and quote *those* when judging severity.