# Dataset The training corpus is real, licensed, public text. It is not templates and it is not model-generated filler. ## Why this exists Kernel v4 trained to a loss of **0.0168** on the old corpus. That number was not learning. The old corpus was eight fixed templates, so the model only had to memorise eight strings. Nothing about a 0.0168 loss on a template corpus says anything about the model, which is why the first thing that had to change was the data and not the hyperparameters. The old corpus also trained to 0.0168 after **12 hours**, and produced **zero checkpoints**. Both facts came from the same run, and both are what you get when a corpus is too small to hold a model's attention: memorise, then sit there re-reading the same eight strings until the session dies. ## The mix 50,000,000 tokens, every source filled to budget, no reweighting at train time. | Source | Language | Genre | Tokens | Share | Documents | License | | --- | --- | --- | --- | --- | --- | --- | | `idwiki` | id | encyclopedia | 23,500,000 | 47% | 14,622 | CC BY-SA 4.0 | | `enwiki` | en | encyclopedia | 23,500,000 | 47% | 41,000 | CC BY-SA 4.0 | | `openai/gsm8k` | en | math reasoning | 3,000,000 | 6% | 7,473 | MIT | Built corpus: **50,000,000 tokens**, 100,000,008 bytes, 63,095 documents. Held out, never trained on: `openai/openai_humaneval` test split, 164 problems. Every build runs an exact-text contamination check against it and **aborts** rather than emitting a cache if anything overlaps. `roneneldan/TinyStories` is pinned in the manifest but **disabled**. It is GPT-4 generated, not real text, which is the same defect that made the old corpus worthless, just at a larger scale. ## How bias is actually controlled **Shards are sampled with an even stride, never as a prefix.** This is the part that is easy to get wrong and invisible when you get it wrong. `wikimedia/wikipedia` is stored in page-id order, so its first row group is the articles whose titles start near "A". A naive "read the first N row groups" gives you a corpus sorted by title. It looks like plenty of text, and it is quietly a single-subject corpus. Measured, on the first build before this was fixed, the whole Indonesian half of the corpus would have been the "A" articles. `sample_row_groups` divides by `take - 1` so the **last** row group is sampled too, not stepped over: ```python return sorted({round(index * (total - 1) / (take - 1)) for index in range(take)}) ``` Sampling also spreads across **shards**, because each shard is a disjoint contiguous band of page ids. Evidence that the shipped sample is spread, taken from the pinned revision: | Source | Shards | Page-id range covered | First shard titles | Last shard titles | | --- | --- | --- | --- | --- | | `20231101.id` | 3 | 1 → 2,704,722 | `Asam deoksiribonukleat` … `Kapal selam` | `Mahkamah Konstitusi Azerbaijan` … `Baek Bo-ram` | | `20231101.en` | 41 | 12 → 2,989,158 | `Anarchism` … `Anal sex` | `Smolensk Governorate` … `Pictures of You (The Cure song)` | **Documents are shuffled per source with a seeded RNG**, and sources are interleaved round-robin while the cache is written, so the token stream does not arrive as 23.5M Indonesian followed by 23.5M English. **Every source is pinned twice**: an exact dataset revision (40-hex commit) and a content digest of the fetched text. The build recomputes the digest and **aborts** if it differs from the manifest, so upstream drift is a build failure rather than a silent corpus change. ## What this corpus does not cover Stated plainly, because the eval gate is calibrated against these gaps: - **No code.** There is no real code corpus in the mix. The coding domain is unserved, so `strict_gate`'s `coding >= 0.65` floor **cannot be reached from this corpus** no matter how long training runs. That is a property of the data, not a bug to be tuned away. Adding a permissively licensed code source is the single highest-value change to this mix. - **Encyclopedic only.** Both wikis are reference prose. There is no conversational, instructional, or narrative text, so the model will read as encyclopedic and write like it. - **GSM8K is small and English-only.** 3% of the corpus, all English, all grade-school word problems. It is the only reasoning-with-a-derivation signal in the mix. - **Byte-level tokenisation, no subword vocabulary.** 16,384 code points means every character is a token. That is the model that was specced; it is also why the checkpoint is 444 MB for 38.8M parameters. ## Reproducing it ```bash export HF_TOKEN=... # never committed; read from the environment python scripts/build_corpus.py ``` This reads pinned parquet shards **over HTTP range requests**, not whole-file downloads. The Indonesian shard is 267 MB and a 22.5M-token slice of it is about 9 MB of column chunks, so a full download would move 30x the bytes for the same result. The 41 English shards are likewise read one row group at a time. Output, gitignored: ``` data/cache/train.tokens 100,000,008 bytes flat uint16, magic ALGOTOK1 data/cache/train.tokens.json per-source counts, digests, budgets ``` Training memory-maps that file, so a resumed session reads an **absolute** token cursor: no re-tokenising, no re-shuffling, and no way for session N+1 to see a different corpus than session N. ## Pins as shipped | Source | Dataset revision | Content digest | | --- | --- | --- | | `idwiki` | `b04c8d1ceb2f5cd4588862100d08de323dccfbaa` | `430163e7eac84c38b07220ccf2eabb32bcfe217b0dd00f6bd8bcd711cc0eb405` | | `enwiki` | `b04c8d1ceb2f5cd4588862100d08de323dccfbaa` | `6ad6b951289c6e59f13fbf6ec8fe237a6c355b703720f4223223de6af36b3a43` | | `openai/gsm8k` | `740312add88f781978c0658806c59bc2815b9866` | `92e799c1a5104fbfd801f30ca61ad5a0d2523443a6d7eb1ef6b9e3c89c310737` | | `openai/openai_humaneval` (held out) | `7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544` | — | The two GSM8K and HumanEval revisions are the same commits the manifest pinned before any of this existed, which is a useful check that the pins were real.