algo-v1-codepoint / docs /DATASET.md
Miruamel's picture
Publish algo v1 (code-point tokenizer): 38.8M params, 16.4M tokens, loss 2.10
05f5056 verified
|
Raw History Blame Contribute Delete
6.09 kB

Dataset

The training corpus is real, licensed, public text. It is not templates and it is not model-generated filler.

Why this exists

Kernel v4 trained to a loss of 0.0168 on the old corpus. That number was not learning. The old corpus was eight fixed templates, so the model only had to memorise eight strings. Nothing about a 0.0168 loss on a template corpus says anything about the model, which is why the first thing that had to change was the data and not the hyperparameters.

The old corpus also trained to 0.0168 after 12 hours, and produced zero checkpoints. Both facts came from the same run, and both are what you get when a corpus is too small to hold a model's attention: memorise, then sit there re-reading the same eight strings until the session dies.

The mix

50,000,000 tokens, every source filled to budget, no reweighting at train time.

Source Language Genre Tokens Share Documents License
idwiki id encyclopedia 23,500,000 47% 14,622 CC BY-SA 4.0
enwiki en encyclopedia 23,500,000 47% 41,000 CC BY-SA 4.0
openai/gsm8k en math reasoning 3,000,000 6% 7,473 MIT

Built corpus: 50,000,000 tokens, 100,000,008 bytes, 63,095 documents.

Held out, never trained on: openai/openai_humaneval test split, 164 problems. Every build runs an exact-text contamination check against it and aborts rather than emitting a cache if anything overlaps.

roneneldan/TinyStories is pinned in the manifest but disabled. It is GPT-4 generated, not real text, which is the same defect that made the old corpus worthless, just at a larger scale.

How bias is actually controlled

Shards are sampled with an even stride, never as a prefix. This is the part that is easy to get wrong and invisible when you get it wrong.

wikimedia/wikipedia is stored in page-id order, so its first row group is the articles whose titles start near "A". A naive "read the first N row groups" gives you a corpus sorted by title. It looks like plenty of text, and it is quietly a single-subject corpus. Measured, on the first build before this was fixed, the whole Indonesian half of the corpus would have been the "A" articles.

sample_row_groups divides by take - 1 so the last row group is sampled too, not stepped over:

return sorted({round(index * (total - 1) / (take - 1)) for index in range(take)})

Sampling also spreads across shards, because each shard is a disjoint contiguous band of page ids. Evidence that the shipped sample is spread, taken from the pinned revision:

Source Shards Page-id range covered First shard titles Last shard titles
20231101.id 3 1 → 2,704,722 Asam deoksiribonukleat … Kapal selam Mahkamah Konstitusi Azerbaijan … Baek Bo-ram
20231101.en 41 12 → 2,989,158 Anarchism … Anal sex Smolensk Governorate … Pictures of You (The Cure song)

Documents are shuffled per source with a seeded RNG, and sources are interleaved round-robin while the cache is written, so the token stream does not arrive as 23.5M Indonesian followed by 23.5M English.

Every source is pinned twice: an exact dataset revision (40-hex commit) and a content digest of the fetched text. The build recomputes the digest and aborts if it differs from the manifest, so upstream drift is a build failure rather than a silent corpus change.

What this corpus does not cover

Stated plainly, because the eval gate is calibrated against these gaps:

  • No code. There is no real code corpus in the mix. The coding domain is unserved, so strict_gate's coding >= 0.65 floor cannot be reached from this corpus no matter how long training runs. That is a property of the data, not a bug to be tuned away. Adding a permissively licensed code source is the single highest-value change to this mix.
  • Encyclopedic only. Both wikis are reference prose. There is no conversational, instructional, or narrative text, so the model will read as encyclopedic and write like it.
  • GSM8K is small and English-only. 3% of the corpus, all English, all grade-school word problems. It is the only reasoning-with-a-derivation signal in the mix.
  • Byte-level tokenisation, no subword vocabulary. 16,384 code points means every character is a token. That is the model that was specced; it is also why the checkpoint is 444 MB for 38.8M parameters.

Reproducing it

export HF_TOKEN=...          # never committed; read from the environment
python scripts/build_corpus.py

This reads pinned parquet shards over HTTP range requests, not whole-file downloads. The Indonesian shard is 267 MB and a 22.5M-token slice of it is about 9 MB of column chunks, so a full download would move 30x the bytes for the same result. The 41 English shards are likewise read one row group at a time.

Output, gitignored:

data/cache/train.tokens        100,000,008 bytes   flat uint16, magic ALGOTOK1
data/cache/train.tokens.json   per-source counts, digests, budgets

Training memory-maps that file, so a resumed session reads an absolute token cursor: no re-tokenising, no re-shuffling, and no way for session N+1 to see a different corpus than session N.

Pins as shipped

Source Dataset revision Content digest
idwiki b04c8d1ceb2f5cd4588862100d08de323dccfbaa 430163e7eac84c38b07220ccf2eabb32bcfe217b0dd00f6bd8bcd711cc0eb405
enwiki b04c8d1ceb2f5cd4588862100d08de323dccfbaa 6ad6b951289c6e59f13fbf6ec8fe237a6c355b703720f4223223de6af36b3a43
openai/gsm8k 740312add88f781978c0658806c59bc2815b9866 92e799c1a5104fbfd801f30ca61ad5a0d2523443a6d7eb1ef6b9e3c89c310737
openai/openai_humaneval (held out) 7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544 —

The two GSM8K and HumanEval revisions are the same commits the manifest pinned before any of this existed, which is a useful check that the pins were real.