code/docs/data/corpus-v1.md

FarZenith corpus v1

Built 2026-10-30 with python -m zenith.data.pipeline {fetch,normalize,sample,tokenize}. Raw and processed data are git-ignored; the manifest (data/processed/manifest.json) records the tokenizer hash, per-source stats and shard hashes. Tokenizer: zenith/tokenizer/zenith-16k.json (+ .hf.json), 16384 BPE, 4.18 bytes/token on the 200 MB tokenizer sample, Python/HF parity check passed on 10 000 lines.

Sources and tokens

Source	Group	Licence	Docs kept	Tokens	Decontaminated (dropped)
Wikipedia (astronomy, astrophysics, spaceflight, physics categories; wikimedia/wikipedia 20231101.en)	astro	CC BY-SA 4.0	13,053	14.6M	1,807
arXiv metadata, title + abstract of astro-ph*, physics.*, gr-qc (Kaggle Cornell-University/arxiv)	arxiv	CC0 1.0 (metadata)	767,756	205.9M	3,572
OpenStax Astronomy 2e, University Physics vols 1–3	textbook	CC BY 4.0	485	1.2M	27
FineWeb-Edu sample/10BT, science-keyword filtered, capped at 2.2 GB of text	general	ODC-By 1.0	265,884	539.0M	660
Total			1,047,178	760.7M (train 756.8M, val 3.9M)	6,066

Per group (train / val tokens): arxiv 204.9M / 1.0M, astro 14.6M / 0.07M, general 536.2M / 2.8M, textbook 1.15M / 0.005M. Bytes per token: arxiv 4.53, astro 3.99, general 4.05, textbook 4.30.

Decontamination: 8-gram match against all 24,030 evaluation questions and showcase lines (98,429 distinct n-grams, eval hash 6bcb4ebae1a3…). Documents with any overlap are dropped.

Deviations from spec §4.1

• Total is 760.7M tokens, not ~1B. It meets the plan's ≥ 700M requirement. The small run (3815 steps × 262,144 tokens ≈ 1.0B tokens) therefore repeats data (see below).
• Wikipedia is 14.6M tokens, not 80–120M. The category crawl gave 22,622 titles; the dump scan found, and the 500-character minimum and dedupe kept, 14,860 articles (the other titles were absent from the snapshot or too short), and decontamination left 13,053. 1,807 articles (12% of the surviving Wikipedia docs) were dropped as contaminated: the evaluation questions are all astronomy, so the 8-gram check hits many encyclopedia articles. Fetching deeper categories would add documents but was not done for v1.
• arXiv is ~206M tokens, not ~100M, because gr-qc and all physics.* categories were included.
• OpenStax is 1.2M tokens, not ~2M (485 module documents after filtering).
• FineWeb-Edu was downloaded fresh (no HADES local copy), science-filtered, and capped at 2.2 GB.

Effective training mix for configs/small.yaml

max_epochs (astro 4, textbook 3; astro raised from 2 on 2026-10-30) caps the thin groups and gives their share to the others (zenith.train.data.effective_mix). With this corpus the small run sees:

Group	Target share	Effective share	Epochs
astro (Wikipedia)	30%	5.8%	4.0
arxiv	15%	21.7%	1.06
textbook	5%	0.3%	3.0
general (FineWeb-Edu)	50%	72.2%	1.35

The model will therefore be mostly general science web text plus arXiv abstracts, with little Wikipedia astronomy. This is acceptable for v1 because the model's job is mapping questions to engine specs (learned from the 300k SFT examples); the pretraining text mainly supplies language fluency. Options for a later version: raise max_epochs.astro, add more Wikipedia categories, or add more open astronomy text.

For the laptop tiny run (≈ 40M tokens) the target mix applies unchanged.

SFT data

python -m zenith.sft.generate --out data/sft --train 300000 --eval-per-op 2000 --seed 0: 300,000 training examples (225,272 simulation, 29,849 follow-up, 29,924 no-sim, 14,955 refusals) and 24,000 evaluation examples (20,000 simulation, 2,000 no-sim, 2,000 unsupported).