| # FarZenith corpus v1 |
|
|
| Built 2026-10-30 with `python -m zenith.data.pipeline {fetch,normalize,sample,tokenize}`. |
| Raw and processed data are git-ignored; the manifest (`data/processed/manifest.json`) records the |
| tokenizer hash, per-source stats and shard hashes. Tokenizer: `zenith/tokenizer/zenith-16k.json` |
| (+ `.hf.json`), 16384 BPE, **4.18 bytes/token** on the 200 MB tokenizer sample, Python/HF parity check |
| passed on 10 000 lines. |
|
|
| ## Sources and tokens |
|
|
| | Source | Group | Licence | Docs kept | Tokens | Decontaminated (dropped) | |
| |---|---|---|---|---|---| |
| | Wikipedia (astronomy, astrophysics, spaceflight, physics categories; `wikimedia/wikipedia` 20231101.en) | astro | CC BY-SA 4.0 | 13,053 | 14.6M | 1,807 | |
| | arXiv metadata, title + abstract of `astro-ph*`, `physics.*`, `gr-qc` (Kaggle `Cornell-University/arxiv`) | arxiv | CC0 1.0 (metadata) | 767,756 | 205.9M | 3,572 | |
| | OpenStax *Astronomy 2e*, *University Physics* vols 1–3 | textbook | CC BY 4.0 | 485 | 1.2M | 27 | |
| | FineWeb-Edu `sample/10BT`, science-keyword filtered, capped at 2.2 GB of text | general | ODC-By 1.0 | 265,884 | 539.0M | 660 | |
| | **Total** | | | **1,047,178** | **760.7M** (train 756.8M, val 3.9M) | 6,066 | |
|
|
| Per group (train / val tokens): arxiv 204.9M / 1.0M, astro 14.6M / 0.07M, general 536.2M / 2.8M, |
| textbook 1.15M / 0.005M. Bytes per token: arxiv 4.53, astro 3.99, general 4.05, textbook 4.30. |
|
|
| Decontamination: 8-gram match against all 24,030 evaluation questions and showcase lines |
| (98,429 distinct n-grams, eval hash `6bcb4ebae1a3…`). Documents with any overlap are dropped. |
|
|
| ## Deviations from spec §4.1 |
|
|
| - **Total is 760.7M tokens, not ~1B.** It meets the plan's ≥ 700M requirement. The `small` run |
| (3815 steps × 262,144 tokens ≈ 1.0B tokens) therefore repeats data (see below). |
| - **Wikipedia is 14.6M tokens, not 80–120M.** The category crawl gave 22,622 titles; the dump scan found, |
| and the 500-character minimum and dedupe kept, 14,860 articles (the other titles were absent from the |
| snapshot or too short), and decontamination left 13,053. 1,807 articles (12% of the surviving |
| Wikipedia docs) were dropped as contaminated: the evaluation questions are all astronomy, so the |
| 8-gram check hits many encyclopedia articles. Fetching deeper categories would add documents but |
| was not done for v1. |
| - **arXiv is ~206M tokens, not ~100M**, because `gr-qc` and all `physics.*` categories were included. |
| - **OpenStax is 1.2M tokens, not ~2M** (485 module documents after filtering). |
| - **FineWeb-Edu was downloaded fresh** (no HADES local copy), science-filtered, and capped at 2.2 GB. |
|
|
| ## Effective training mix for `configs/small.yaml` |
|
|
| `max_epochs` (astro 4, textbook 3; astro raised from 2 on 2026-10-30) caps the thin groups and gives their share to the others |
| (`zenith.train.data.effective_mix`). With this corpus the `small` run sees: |
|
|
| | Group | Target share | Effective share | Epochs | |
| |---|---|---|---| |
| | astro (Wikipedia) | 30% | **5.8%** | 4.0 | |
| | arxiv | 15% | 21.7% | 1.06 | |
| | textbook | 5% | **0.3%** | 3.0 | |
| | general (FineWeb-Edu) | 50% | 72.2% | 1.35 | |
|
|
| The model will therefore be mostly general science web text plus arXiv abstracts, with little |
| Wikipedia astronomy. This is acceptable for v1 because the model's job is mapping questions to |
| engine specs (learned from the 300k SFT examples); the pretraining text mainly supplies language |
| fluency. Options for a later version: raise `max_epochs.astro`, add more Wikipedia categories, or add |
| more open astronomy text. |
|
|
| For the laptop `tiny` run (≈ 40M tokens) the target mix applies unchanged. |
|
|
| ## SFT data |
|
|
| `python -m zenith.sft.generate --out data/sft --train 300000 --eval-per-op 2000 --seed 0`: |
| 300,000 training examples (225,272 simulation, 29,849 follow-up, 29,924 no-sim, 14,955 refusals) and |
| 24,000 evaluation examples (20,000 simulation, 2,000 no-sim, 2,000 unsupported). |
|
|
