# FarZenith corpus v1

Built 2026-10-30 with `python -m zenith.data.pipeline {fetch,normalize,sample,tokenize}`.
Raw and processed data are git-ignored; the manifest (`data/processed/manifest.json`) records the
tokenizer hash, per-source stats and shard hashes. Tokenizer: `zenith/tokenizer/zenith-16k.json`
(+ `.hf.json`), 16384 BPE, **4.18 bytes/token** on the 200 MB tokenizer sample, Python/HF parity check
passed on 10 000 lines.

## Sources and tokens

| Source | Group | Licence | Docs kept | Tokens | Decontaminated (dropped) |
|---|---|---|---|---|---|
| Wikipedia (astronomy, astrophysics, spaceflight, physics categories; `wikimedia/wikipedia` 20231101.en) | astro | CC BY-SA 4.0 | 13,053 | 14.6M | 1,807 |
| arXiv metadata, title + abstract of `astro-ph*`, `physics.*`, `gr-qc` (Kaggle `Cornell-University/arxiv`) | arxiv | CC0 1.0 (metadata) | 767,756 | 205.9M | 3,572 |
| OpenStax *Astronomy 2e*, *University Physics* vols 1–3 | textbook | CC BY 4.0 | 485 | 1.2M | 27 |
| FineWeb-Edu `sample/10BT`, science-keyword filtered, capped at 2.2 GB of text | general | ODC-By 1.0 | 265,884 | 539.0M | 660 |
| **Total** | | | **1,047,178** | **760.7M** (train 756.8M, val 3.9M) | 6,066 |

Per group (train / val tokens): arxiv 204.9M / 1.0M, astro 14.6M / 0.07M, general 536.2M / 2.8M,
textbook 1.15M / 0.005M. Bytes per token: arxiv 4.53, astro 3.99, general 4.05, textbook 4.30.

Decontamination: 8-gram match against all 24,030 evaluation questions and showcase lines
(98,429 distinct n-grams, eval hash `6bcb4ebae1a3…`). Documents with any overlap are dropped.

## Deviations from spec §4.1

- **Total is 760.7M tokens, not ~1B.** It meets the plan's ≥ 700M requirement. The `small` run
  (3815 steps × 262,144 tokens ≈ 1.0B tokens) therefore repeats data (see below).
- **Wikipedia is 14.6M tokens, not 80–120M.** The category crawl gave 22,622 titles; the dump scan found,
  and the 500-character minimum and dedupe kept, 14,860 articles (the other titles were absent from the
  snapshot or too short), and decontamination left 13,053. 1,807 articles (12% of the surviving
  Wikipedia docs) were dropped as contaminated: the evaluation questions are all astronomy, so the
  8-gram check hits many encyclopedia articles. Fetching deeper categories would add documents but
  was not done for v1.
- **arXiv is ~206M tokens, not ~100M**, because `gr-qc` and all `physics.*` categories were included.
- **OpenStax is 1.2M tokens, not ~2M** (485 module documents after filtering).
- **FineWeb-Edu was downloaded fresh** (no HADES local copy), science-filtered, and capped at 2.2 GB.

## Effective training mix for `configs/small.yaml`

`max_epochs` (astro 4, textbook 3; astro raised from 2 on 2026-10-30) caps the thin groups and gives their share to the others
(`zenith.train.data.effective_mix`). With this corpus the `small` run sees:

| Group | Target share | Effective share | Epochs |
|---|---|---|---|
| astro (Wikipedia) | 30% | **5.8%** | 4.0 |
| arxiv | 15% | 21.7% | 1.06 |
| textbook | 5% | **0.3%** | 3.0 |
| general (FineWeb-Edu) | 50% | 72.2% | 1.35 |

The model will therefore be mostly general science web text plus arXiv abstracts, with little
Wikipedia astronomy. This is acceptable for v1 because the model's job is mapping questions to
engine specs (learned from the 300k SFT examples); the pretraining text mainly supplies language
fluency. Options for a later version: raise `max_epochs.astro`, add more Wikipedia categories, or add
more open astronomy text.

For the laptop `tiny` run (≈ 40M tokens) the target mix applies unchanged.

## SFT data

`python -m zenith.sft.generate --out data/sft --train 300000 --eval-per-op 2000 --seed 0`:
300,000 training examples (225,272 simulation, 29,849 follow-up, 29,924 no-sim, 14,955 refusals) and
24,000 evaluation examples (20,000 simulation, 2,000 no-sim, 2,000 unsupported).