guide/03-preflight.md

03 — Preflight

The discipline of this phase is one sentence: spend the cheapest evidence first, and make each rung answer a different question. Every rung below caught, in the project this guide comes from, a bug that would have destroyed a multi-hour session.

1. The ladder

Rung	Budget	Question it answers
0. Static	seconds, local	Does it import? Does the launch command line parse? Do the hashes of the files I'm about to run match the files I reviewed?
1. Free-CPU rehearsal	minutes, no accelerator	Does the whole control flow work with the real modules — plan, resume decision, cursor math, checkpoint cycle — with training stubbed?
2. Cheap GPU smoke (20–30 steps)	budgeted ~10 min	Does the machinery run on the accelerator: install, device, one real optimiser step, one push, one verified resume?
3. Measurement probe	1–1.5 h billed	Does the config hold for hours: sustained rate, flat memory, cadence cost, cold resume from the Hub on an empty disk?
4. The run	everything	—

Two honest corrections about rung 2, because they are the argument for the ladder, not against it:

• It cost 1.25 h across eight attempts, not ten minutes. Every attempt but the last failed on a real defect: torchrun handed the interpreter instead of the script (E-037), a stale-stop guard refusing correctly on a dirty repo (E-038), an argument-order slip in delete_folder (E-039), a harness that could not read its own tee'd output (E-041), and finally a rank-only assertion that killed every successful leg (E-044). A smoke rung's budget is ten minutes; its cost is the number of bugs in the code it is the first to execute. That is still cheaper than finding them in a 4.5-hour session — but plan the budget as "10 min, or 1.5 h if the code is new", not as "10 min".
• The CPU rehearsal caught the plumbing defects (E-036, E-037). It did not catch the expensive one: the rank-only-on_save crash needed GPU legs to reproduce and took seven attempts to diagnose. Don't oversell the free rung — it buys you command lines and control flow, not distributed semantics.

Order by price-per-question, and re-run rungs 0–1 on every edit to run code. They are free; the only cost of a rehearsal is the nine minutes you didn't spend on something else.

2. Make the rehearsal rehearse the real thing

• Import the real modules — a rehearsal of a reimplementation proves nothing about the original.
• Print the command line the launcher would execute and parse it in the rehearsal, so a flag typo fails there.
• Point it at real artefacts where it can: hash the actually-published shards, read the actually-verified checkpoint. The Phase 4 rehearsal re-hashed all 151 mix files, which is why the run started knowing its data.
• Assert the same invariants the run will assert, so a rehearsal pass means the run's guards are armed, not merely present.

3. Model size: prove size-dependent things at the real size

"Preflight on tiny models" is the instruction, and taken literally it is wrong. Memory, throughput, checkpoint byte-size, DDP rank behaviour and optimizer-state resume are all size-dependent — a 15M model proves the code path and nothing about the run's numbers. This project therefore ran every Gate 3 proof at the real 106M geometry on 2×T4, and used CPU/tiny cells only for properties that genuinely don't scale (reader semantics, order determinism, the storage cycle's logic). Choose per proof: if the quantity you're measuring is a function of size, the size has to be right; if it isn't, spend nothing.

4. What preflight must demonstrate, with logs

The eight proofs. Each is a line in the preflight report, with the log excerpt that shows it.

1. Upload a checkpoint to the Hub; restart and pull it back.
2. Exact data position on resume — recorded index/hash and a token-offset check, not "loss looked continuous".
3. Optimizer and LR state restored: print the scheduler value on the first resumed step and compare it to what the interrupted run would have produced.
4. Loss continuity across the restart within noise, with numbers.
5. Measured throughput on the target hardware, and the ETA it implies.
6. Cold resume: interrupt, kill the instance, start fresh with an empty disk, recover solely from the Hub. This is the only resume test that counts, because every real interruption looks like this. A resume that reads a local leftover file is testing something that will not exist.
7. The storage cycle: upload → verify by fetching bytes back → delete locally → confirm free space → keep training, at sizes representative of the real model plus optimizer state (07: ~12 bytes/param, not the weight file alone).
8. Two sequential resumes — position must never drift backwards or re-read consumed data.

Anything unproven here is a bug you have not found yet.

5. Then measure the config, and let the measurement change it

Preflight is the last cheap moment to choose. Throughput is not a spec sheet; it's a distribution on your hardware with your batch shape, and small-N numbers lie:

• A 20-step average is not a rate. The 20-step figure here was 6 % optimistic — the difference between fitting the week and not. Take the sustained number over a long leg (two legs of 120 and 60 steps measured 12,312 and 12,221 tok/s; the planning number adopted was 12,300).
• Measure memory as a time series, not a peak. max_memory_reserved is process-lifetime, so a post-training validation pass "raises the peak" without any leak. Gate on the leg that does not evaluate, and compare step 10 against step 120 in the same process.
• Report the cross-rank max, obtained by an explicit all_reduce(MAX). Rank-0-only peak statistics under- report, and that is the number you will otherwise make an OOM decision on.
• Cost the checkpoint cycle in seconds per push at real conditions: whole-object read-back measured 797 s off a cold CDN; bounded ranged verification runs inside a 7–10 s push→point gap on the live run. That single measurement decides how often you may push, hence how much work an interruption costs.
• If the measurement shows headroom, ask what your current setting is still paying for. 5.99 GB peak on a 15.4 GB card meant gradient checkpointing was buying memory nobody needed; since it computes identical gradients, dropping it was plumbing, not a loss-landscape change. It bought ~30 % throughput and turned an infeasible 29.7-hour run into a 22.6-hour one.

Rule for reopening a frozen choice: only before results exist, only when the change provably leaves gradients identical, and shrink the blast radius first (finer cadence so a failure costs less). Never after a number is on the board — that is tuning, and it invalidates the report.

6. Prove the failure paths, and prefer a discriminating job to an argument

Rungs 0–3 are where you inject faults, because the alternative is discovering them mid-run:

• Serve checkpoint bytes back wrong (flipped byte inside the ranged window, wrong span, a server that ignores Range) and confirm verification fails. A verifier that has never rejected anything is decoration.
• Point prune at an unverified result and confirm it refuses to delete.
• Kill a run mid-leg: confirm the stop guard fires for the right reason and the resume lands where it should.
• Test the absence case: an empty repo must read as "nothing here", not as a timeout or a 403.
• Test rank-blind guards on every rank. on_save ran on rank 0 only, so a "did you push?" assertion was unsatisfiable on rank 1 and killed every correct leg (E-044). Two rules follow: assert only predicates the rank can actually satisfy, and print diagnostics on all ranks — a rank-0-only logger turns a crash into silence, which is how E-044 survived seven attempts.
• When an error fits a plausible story, spend five minutes disproving the story directly (E-017). A cheap, targeted job settled in one run what a morning of reasoning about token scopes could not. The plausible explanation is the expensive one to trust.

7. Sign the report

Write the phase's report with each of the eight proofs (§4) and its evidence, plus measured tokens/sec, the main-run ETA, the GPU-hour estimate, and the arithmetic showing it fits the remaining quota. Named files, not conversation — a reader without this project's history must be able to check every claim. If a proof cannot state a number, it is missing, and saying so is cheaper than discovering it later.