README: references point at in-repo docs/ and guide/, not one machine
Browse files
README.md
CHANGED
|
@@ -1,86 +1,87 @@
|
|
| 1 |
-
# ounce100m-code
|
| 2 |
-
|
| 3 |
-
Canonical mirror of the scripts for project `ounce100m`, so a Kaggle instance can fetch its own source
|
| 4 |
-
instead of having it pasted into a kernel. **Kept public deliberately: this is the training code, and it is
|
| 5 |
-
intended to be reused for a different model without reading the project's history.**
|
| 6 |
-
|
| 7 |
-
No credential ever appears in this repo. Anything that needs the Hugging Face token reads it at runtime
|
| 8 |
-
through `ounce100m_credentials.install()`, which fetches a private Kaggle dataset (D-006). The local
|
| 9 |
-
workspace's `MASTER_PROMPT.md` is the constitution and is **never** uploaded here.
|
| 10 |
-
|
| 11 |
-
## Layout
|
| 12 |
-
|
| 13 |
-
| path | what it is |
|
| 14 |
-
|---|---|
|
| 15 |
-
| `train/train_ounce100m.py` | the trainer: model build, fp16 autocast + fp32 master + GradScaler, DDP, cursor/checkpoint save, resume, `--dry-run` |
|
| 16 |
-
| `train/hubckpt.py` | the storage cycle: push → **verify bytes** → roll pointer → read pointer back → `prune_verified` (refuses to delete unless verification passed) |
|
| 17 |
-
| `train/shard_dataset.py` | **a library, not a script**: the map-style reader that makes resumption an index advance instead of a replay, plus `cursor_dict` / `save_cursor` / `load_cursor` / `perm_sha` / `mix_fingerprint`. The trainer imports it; tokenising and sharding is done by `build/build_mix.py` |
|
| 18 |
-
| `train/preflight.py` | Gate 3 stages, incl. the throughput cells that decide a config |
|
| 19 |
-
| `build/build_mix.py` | the corpus builder: sources, dedup, exclusion masks, held-out `val/` split |
|
| 20 |
-
| `build/audit_contamination.py` | the mechanical 13-gram overlap audit (counts only; **validation/dev only**, never test) |
|
| 21 |
-
| `build/publish_mix.py`, `build/verify_mix.py`, `build/hubsync.py` | dataset publication, anonymous re-hash verification, workspace↔Hub sync |
|
| 22 |
-
| `build/publish_model.py` | Phase 5: model + tokenizer + generated card, then `--verify-only` = the Gate 5 clean room |
|
| 23 |
-
| `eval/run_benchmarks.py` | Phase 6: lm-eval driver — smoke pass first, per-task incremental save, push after every task |
|
| 24 |
-
| `kernels/phase4_session.py` | the session planner: reads the pointer, tiles steps onto the session/quota grid, emits the `torchrun` line |
|
| 25 |
-
| `kernels/phase4_bootstrap.py` | 21-line GPU kernel body: fetch launcher at a pinned commit sha, sha256-assert, exec |
|
| 26 |
-
| `kernels/p4_stop_probe.py`, `kernels/phase4_prep.py`, `kernels/p4_plan_sweep.py` | the pre-launch evidence chain (GPU smoke/soak probe, free-CPU rehearsal, quota sweep) |
|
| 27 |
-
| `kernels/p6_evals_bootstrap.py` | benchmark launcher **with a heartbeat**: pushes its own log to the Hub every 120 s, so "queued / running / dead" is distinguishable without kernel-log access |
|
| 28 |
-
| `probes/` | Phase 0-2 platform measurements: GPU/DDP behaviour, filesystem, ingest rate, credential paths, push bandwidth, source inventory |
|
| 29 |
-
| `config/param_count.py` | parameter arithmetic used to prove the model is inside its allowed range |
|
| 30 |
-
|
| 31 |
-
## Training a new model
|
| 32 |
-
|
| 33 |
-
The trainer takes its geometry from flags, so a new model is normally a flag change and a new mix — not a new
|
| 34 |
-
script. Everything below ran on Kaggle 2×T4; nothing here is Kaggle-specific except `kernels/*`, which exist
|
| 35 |
-
because a kernel cannot be passed environment variables.
|
| 36 |
-
|
| 37 |
-
```bash
|
| 38 |
-
# 1. data. Assemble with headroom above the token target (dedup, masks and val/ all subtract),
|
| 39 |
-
# then audit and publish. Run each with --help for the full flag set.
|
| 40 |
-
python build/build_mix.py --root <mixroot> --budget-tokens 1310000000 # sources are the SOURCES table
|
| 41 |
-
python build/audit_contamination.py # 13-gram counts vs validation/dev only
|
| 42 |
-
python build/publish_mix.py --root <mixroot> --repo <you>/my-mix --dry-run
|
| 43 |
-
python build/verify_mix.py # anonymous re-hash against the manifest
|
| 44 |
-
|
| 45 |
-
# 2. architecture + schedule: the trainer's geometry flags, all defaulting to ounce100m's frozen values
|
| 46 |
-
torchrun --nproc_per_node=2 train/train_ounce100m.py \
|
| 47 |
-
--layers 22 --hidden 576 --seq-len 1024 --tokens 1000000000 \
|
| 48 |
-
--micro-batch 4 --accum 32 --lr 6e-4 --warmup-frac 0.02 --decay-frac 0.8 \
|
| 49 |
-
--attn eager --optim adamw_torch --no-grad-ckpt \
|
| 50 |
-
--root <mixroot> --hub-repo <you>/my-ckpt --push-every-steps 127 \
|
| 51 |
-
--max-steps 3814 --seed 20260919 --data-seed 20260919
|
| 52 |
-
```
|
| 53 |
-
|
| 54 |
-
Flags worth knowing about: `--dry-run` (build the model, print parameter counts, exit — no data, no GPU
|
| 55 |
-
needed), `--resume auto` (read the Hub pointer; never falls back to step 0 if the pointer exists and the
|
| 56 |
-
bytes do not), `--stop-after-steps N` (end a segment **on a Hub-verified checkpoint**, which is what makes a
|
| 57 |
-
capped session safe), `--prune` (delete local only after byte verification), `--grad-ckpt`, `--torch-compile`.
|
| 58 |
-
`train/hubckpt.py` and `train/preflight.py` are independently runnable.
|
| 59 |
-
|
| 60 |
-
### The four invariants that make a resumable run
|
| 61 |
-
|
| 62 |
-
Breaking any of them loses work silently, so they are asserted in code, not documented as convention:
|
| 63 |
-
|
| 64 |
-
1. **A sample index is a pure function.** `sample k` depends only on `(dataset fingerprint, shuffle_seed, k)`,
|
| 65 |
-
so the cursor is a position, not a state. Persisted with every checkpoint: `seq_len`, `samples_consumed`,
|
| 66 |
-
`shuffle_seed`, `shuffle_perm_sha`, `dataset_files_sha`, `step`, `tokens_consumed`. A resume **raises** if a
|
| 67 |
-
hash differs — it never re-shuffles quietly.
|
| 68 |
-
2. **Write → push → verify bytes → point → read pointer back → prune.** `prune_verified()` takes the
|
| 69 |
-
verification result and refuses to delete otherwise. Listing is not proof; an HTTP 200 on an LFS pointer is
|
| 70 |
-
not proof.
|
| 71 |
-
3. **Cadence divides the session length.** With a 127-step cadence and 762-step sessions, an interruption
|
| 72 |
-
costs ≤45 min. The trainer rejects a cadence that is not a whole multiple of its checkpoint grid.
|
| 73 |
-
4. **Cold resume is the only resume test that counts.** Delete the local run dir, start a fresh instance, and
|
| 74 |
-
recover from the Hub alone.
|
| 75 |
-
|
| 76 |
-
## Reproducing a run exactly
|
| 77 |
-
|
| 78 |
-
Every kernel fetches its script **at a commit sha and asserts the sha256 after download**, so a re-run names
|
| 79 |
-
the code it used. The `ounce100m` run is at REV `c237b478` (trainer `dcb0ac19…`, `hubckpt` `d4b50ed0…`,
|
| 80 |
-
launcher `5fb14bc3…`); Phase 5 published from REV `55f23ef5`. The trained artefacts are
|
| 81 |
-
`Cion-lab/ounce100m-v1` (model, tokenizer, card) and `Cion-lab/ounce100m-mix-v1` (mix, manifest, `audit.json`,
|
| 82 |
-
exclusion masks); `Cion-lab/ounce100m-ckpt` holds the 31 training checkpoints and `final/run_summary.json`.
|
| 83 |
-
|
| 84 |
-
The operational lessons — the evidence ladder, the checkpoint ordering, the contamination rules, the quota
|
| 85 |
-
arithmetic, and 54 numbered failures — are written up in `guide/`
|
| 86 |
-
`docs/final-report.md`
|
|
|
|
|
|
| 1 |
+
# ounce100m-code
|
| 2 |
+
|
| 3 |
+
Canonical mirror of the scripts for project `ounce100m`, so a Kaggle instance can fetch its own source
|
| 4 |
+
instead of having it pasted into a kernel. **Kept public deliberately: this is the training code, and it is
|
| 5 |
+
intended to be reused for a different model without reading the project's history.**
|
| 6 |
+
|
| 7 |
+
No credential ever appears in this repo. Anything that needs the Hugging Face token reads it at runtime
|
| 8 |
+
through `ounce100m_credentials.install()`, which fetches a private Kaggle dataset (D-006). The local
|
| 9 |
+
workspace's `MASTER_PROMPT.md` is the constitution and is **never** uploaded here.
|
| 10 |
+
|
| 11 |
+
## Layout
|
| 12 |
+
|
| 13 |
+
| path | what it is |
|
| 14 |
+
|---|---|
|
| 15 |
+
| `train/train_ounce100m.py` | the trainer: model build, fp16 autocast + fp32 master + GradScaler, DDP, cursor/checkpoint save, resume, `--dry-run` |
|
| 16 |
+
| `train/hubckpt.py` | the storage cycle: push → **verify bytes** → roll pointer → read pointer back → `prune_verified` (refuses to delete unless verification passed) |
|
| 17 |
+
| `train/shard_dataset.py` | **a library, not a script**: the map-style reader that makes resumption an index advance instead of a replay, plus `cursor_dict` / `save_cursor` / `load_cursor` / `perm_sha` / `mix_fingerprint`. The trainer imports it; tokenising and sharding is done by `build/build_mix.py` |
|
| 18 |
+
| `train/preflight.py` | Gate 3 stages, incl. the throughput cells that decide a config |
|
| 19 |
+
| `build/build_mix.py` | the corpus builder: sources, dedup, exclusion masks, held-out `val/` split |
|
| 20 |
+
| `build/audit_contamination.py` | the mechanical 13-gram overlap audit (counts only; **validation/dev only**, never test) |
|
| 21 |
+
| `build/publish_mix.py`, `build/verify_mix.py`, `build/hubsync.py` | dataset publication, anonymous re-hash verification, workspace↔Hub sync |
|
| 22 |
+
| `build/publish_model.py` | Phase 5: model + tokenizer + generated card, then `--verify-only` = the Gate 5 clean room |
|
| 23 |
+
| `eval/run_benchmarks.py` | Phase 6: lm-eval driver — smoke pass first, per-task incremental save, push after every task |
|
| 24 |
+
| `kernels/phase4_session.py` | the session planner: reads the pointer, tiles steps onto the session/quota grid, emits the `torchrun` line |
|
| 25 |
+
| `kernels/phase4_bootstrap.py` | 21-line GPU kernel body: fetch launcher at a pinned commit sha, sha256-assert, exec |
|
| 26 |
+
| `kernels/p4_stop_probe.py`, `kernels/phase4_prep.py`, `kernels/p4_plan_sweep.py` | the pre-launch evidence chain (GPU smoke/soak probe, free-CPU rehearsal, quota sweep) |
|
| 27 |
+
| `kernels/p6_evals_bootstrap.py` | benchmark launcher **with a heartbeat**: pushes its own log to the Hub every 120 s, so "queued / running / dead" is distinguishable without kernel-log access |
|
| 28 |
+
| `probes/` | Phase 0-2 platform measurements: GPU/DDP behaviour, filesystem, ingest rate, credential paths, push bandwidth, source inventory |
|
| 29 |
+
| `config/param_count.py` | parameter arithmetic used to prove the model is inside its allowed range |
|
| 30 |
+
|
| 31 |
+
## Training a new model
|
| 32 |
+
|
| 33 |
+
The trainer takes its geometry from flags, so a new model is normally a flag change and a new mix — not a new
|
| 34 |
+
script. Everything below ran on Kaggle 2×T4; nothing here is Kaggle-specific except `kernels/*`, which exist
|
| 35 |
+
because a kernel cannot be passed environment variables.
|
| 36 |
+
|
| 37 |
+
```bash
|
| 38 |
+
# 1. data. Assemble with headroom above the token target (dedup, masks and val/ all subtract),
|
| 39 |
+
# then audit and publish. Run each with --help for the full flag set.
|
| 40 |
+
python build/build_mix.py --root <mixroot> --budget-tokens 1310000000 # sources are the SOURCES table
|
| 41 |
+
python build/audit_contamination.py # 13-gram counts vs validation/dev only
|
| 42 |
+
python build/publish_mix.py --root <mixroot> --repo <you>/my-mix --dry-run
|
| 43 |
+
python build/verify_mix.py # anonymous re-hash against the manifest
|
| 44 |
+
|
| 45 |
+
# 2. architecture + schedule: the trainer's geometry flags, all defaulting to ounce100m's frozen values
|
| 46 |
+
torchrun --nproc_per_node=2 train/train_ounce100m.py \
|
| 47 |
+
--layers 22 --hidden 576 --seq-len 1024 --tokens 1000000000 \
|
| 48 |
+
--micro-batch 4 --accum 32 --lr 6e-4 --warmup-frac 0.02 --decay-frac 0.8 \
|
| 49 |
+
--attn eager --optim adamw_torch --no-grad-ckpt \
|
| 50 |
+
--root <mixroot> --hub-repo <you>/my-ckpt --push-every-steps 127 \
|
| 51 |
+
--max-steps 3814 --seed 20260919 --data-seed 20260919
|
| 52 |
+
```
|
| 53 |
+
|
| 54 |
+
Flags worth knowing about: `--dry-run` (build the model, print parameter counts, exit — no data, no GPU
|
| 55 |
+
needed), `--resume auto` (read the Hub pointer; never falls back to step 0 if the pointer exists and the
|
| 56 |
+
bytes do not), `--stop-after-steps N` (end a segment **on a Hub-verified checkpoint**, which is what makes a
|
| 57 |
+
capped session safe), `--prune` (delete local only after byte verification), `--grad-ckpt`, `--torch-compile`.
|
| 58 |
+
`train/hubckpt.py` and `train/preflight.py` are independently runnable.
|
| 59 |
+
|
| 60 |
+
### The four invariants that make a resumable run
|
| 61 |
+
|
| 62 |
+
Breaking any of them loses work silently, so they are asserted in code, not documented as convention:
|
| 63 |
+
|
| 64 |
+
1. **A sample index is a pure function.** `sample k` depends only on `(dataset fingerprint, shuffle_seed, k)`,
|
| 65 |
+
so the cursor is a position, not a state. Persisted with every checkpoint: `seq_len`, `samples_consumed`,
|
| 66 |
+
`shuffle_seed`, `shuffle_perm_sha`, `dataset_files_sha`, `step`, `tokens_consumed`. A resume **raises** if a
|
| 67 |
+
hash differs — it never re-shuffles quietly.
|
| 68 |
+
2. **Write → push → verify bytes → point → read pointer back → prune.** `prune_verified()` takes the
|
| 69 |
+
verification result and refuses to delete otherwise. Listing is not proof; an HTTP 200 on an LFS pointer is
|
| 70 |
+
not proof.
|
| 71 |
+
3. **Cadence divides the session length.** With a 127-step cadence and 762-step sessions, an interruption
|
| 72 |
+
costs ≤45 min. The trainer rejects a cadence that is not a whole multiple of its checkpoint grid.
|
| 73 |
+
4. **Cold resume is the only resume test that counts.** Delete the local run dir, start a fresh instance, and
|
| 74 |
+
recover from the Hub alone.
|
| 75 |
+
|
| 76 |
+
## Reproducing a run exactly
|
| 77 |
+
|
| 78 |
+
Every kernel fetches its script **at a commit sha and asserts the sha256 after download**, so a re-run names
|
| 79 |
+
the code it used. The `ounce100m` run is at REV `c237b478` (trainer `dcb0ac19…`, `hubckpt` `d4b50ed0…`,
|
| 80 |
+
launcher `5fb14bc3…`); Phase 5 published from REV `55f23ef5`. The trained artefacts are
|
| 81 |
+
`Cion-lab/ounce100m-v1` (model, tokenizer, card) and `Cion-lab/ounce100m-mix-v1` (mix, manifest, `audit.json`,
|
| 82 |
+
exclusion masks); `Cion-lab/ounce100m-ckpt` holds the 31 training checkpoints and `final/run_summary.json`.
|
| 83 |
+
|
| 84 |
+
The operational lessons — the evidence ladder, the checkpoint ordering, the contamination rules, the quota
|
| 85 |
+
arithmetic, and 54 numbered failures — are written up in `guide/` in this repo and summarised in
|
| 86 |
+
`docs/final-report.md`, also in this repo: https://huggingface.co/Cion-lab/ounce100m-code
|
| 87 |
+
Nothing in this repository depends on a file that lives only on one machine.
|