guide/06-checklists.md

06 — Checklists

The working page. Tick nothing from memory. Each gate names the error/decision records it came from, so a claim here is checkable against the ledger rather than taken on trust.

Gate 0 — Orientation · E-001..E-019

• Credentials verified by a call returning identity, not by a file existing
• A secret store's privacy proved by two different calls (authenticated listing and anonymous read) — 403/404 alone cannot distinguish "private" from "never created" (E-011)
• Remote quota read live; hours remaining + reset instant written in the ledger (look up the weekday)
• One throwaway CPU job and one throwaway GPU job completed end to end; mechanics recorded
• Any async job from a previous session found and accounted for (running / finished / dead)
• Memory files match reality; every discrepancy logged, not silently corrected

Gate 1 — Plan · E-020..E-026

• Architecture, hyperparameters, token total, step count written and dated
• Throughput estimate with its source, and the GPU-hour arithmetic it implies
• Budget has a line for every remaining phase, including publish and benchmarks (D-019)
• Per-benchmark target bands pre-registered, each with a citation
• Eval split + metric named per task before any eval code exists
• What you may change mid-run vs what is frozen, stated as one sentence
• Every choice traceable to a source you actually looked at

Gate 2 — Data · E-027..E-035

• Assembled above target; validation split reserved before sharding
• Deduplicated; mechanical overlap audit at overlap_total 0
• Audit touched validation/dev only; no test split opened
• Audit denominators (rows per task) asserted nonzero and printed beside the counts
• No benchmark item text in any published audit artefact, including exception strings
• Shard layout proven small enough that a job reads a few shards, not the mix
• Packing boundary choice measured, not assumed (seam rate)
• Published public; token count re-verified from the Hub copy against the manifest
• Manifest self-consistent (sums, counts, no empties); per-file + set fingerprints recorded
• Cursor design supports exact positional resume; fingerprint keys chosen and persisted

Gate 3 — Preflight · E-036..E-044 · D-012..D-018

• Rehearsal re-run after every edit to run code — it's free
• Rehearsal imports the real modules and parses the real launch command line
• Each proof run at the size the property depends on (03 §3), not at a token tiny model
• Fault injection: wrong bytes, ignored Range, unverified prune, absent repo — all rejected
• Guards tested on every rank; diagnostics printed on every rank, not just rank 0
• Upload → restart → pull-back, on the Hub
• Exact data position proven by index/hash and token offset
• Optimizer/LR state restored, values printed; loss continuous across restart
• Cold resume on a fresh instance, empty disk, recovering solely from the Hub
• Storage cycle end to end at representative size (weights + optimizer state), free space logged
• Two sequential resumes, no backward drift, no re-read data
• Sustained tokens/sec over a long leg (not 20 steps); memory as a time series, cross-rank max
• Checkpoint-cycle cost measured in seconds per push
• Report: the eight proofs (03 §4) each with evidence + ETA + GPU-hour arithmetic that fits the quota

Gate 4 — Run · E-042, D-018

• Config frozen; hash recorded and asserted at every session start
• Cadence divides session length; session tiling ends exactly on the horizon
• Per checkpoint: write → push → verify bytes → point → read pointer back → prune
• Prune gated by the verification result inside the prune function
• Free space logged beside loss, every cycle
• Ledger row booked before each GPU job; hours remaining re-read at each launch
• tokens == steps × tokens_per_step and cursor sample math checked at each checkpoint
• Loss anomalies documented with grad-norm and LR context, config unchanged
• Results written before trailing cross-checks; verification cannot destroy a completed report (E-012)
• Multi-week plan written if the estimate exceeds remaining hours

Gate 5 — Publish · E-045, E-046

• Card fields generated from artefacts; required-field guard demonstrably fails on an empty run
• Parameter/vocab/geometry assertions recomputed from the pushed files
• Clean-room from_pretrained + generate: env set before import, token=False, and every token var popped from the environment
• Anonymous re-hash of every published file
• 404 vs timeout vs 403 distinguished in the "nothing there" path

Gate 6 — Benchmarks · E-045..E-049

• Harness version pinned; task names validated against the harness's own registry, by executing it
• Smoke pass with --limit green: metric keys present, splits correct, installs complete with deps
• Per task: effective shot count + provenance, metric name verbatim, split, n_input
• Failed / missing-metric / no-denominator cells carry a status and fail the "all ok" check
• Incremental saving; completed tasks skipped on re-run only if config matches
• Run against the published repo, not local weights
• Task order pre-declared; a partial table never published as a result
• Table + exact reproduction config pushed into the model repo

Gate 7 — Report

• Measured vs. targeted per task
• What went wrong, with steps and dates
• Contamination statement: method, splits used (never test), denominators, exclusions, residual risk
• Limitations, incl. any mid-run finding that touched the loss landscape and was not fixed
• Which week each number came from, if the schedule spanned a reset
• Status file set to COMPLETE; re-verify and stop

The five recurring failure patterns

Nearly every ledger entry is one of these (E-001..E-049). Check for them before checking for anything exotic.

1. A claim written from memory instead of from the artefact. A doc says a function does X; it doesn't. A log line says a gate passed; the gate read a default. Fix: assert against the file, the bytes, or the executed call — never against the note.
2. A guard that cannot fail. Empty-list denominators, dict.get(k, 0), .get("ok", True), a required-field check over defaults, a verifier that has never rejected anything. Fix: every guard gets one negative test that makes it fire.
3. Verification by listing instead of by bytes. Commit returned, file in the tree, HTTP 200 on a pointer — none of them authorise deleting the local copy.
4. A check that is blind to its own context. Guards and logs written rank-0-only in a 2-rank job (E-039, E-044); clean rooms that inherit an installed token (E-045); "which environment runs this step" never decided, so a job depends on packages the other image doesn't have (E-007, E-010, E-028, E-047). Fix: state the executor — rank, container, interpreter — for every assertion and every install.
5. Assuming the platform matches its docs. A CLI subcommand that isn't; a field that moved; a split that isn't there; billing on one clock and reporting on another. Fix: execute before you rely.

Meta-pattern underneath all five: the check itself needs a test. Twice here a fix for a verified bug was itself wrong (E-046: four of thirteen fixes), and it passed because the check was written by the same reasoning that wrote the bug. Independent review of the test, not just the code, is the cheap way out.

Runbook: a session died at 04:00Z and nobody is awake

1. Read the remote pointer and the commit list. latest -> N tells you exactly what survives. Do not read the job status as the authority.
2. Compute what is lost: at most one cadence interval (04 §3). Nothing before N needs re-running.
3. Check remaining quota against remaining steps from N. Both numbers come from live reads.
4. If quota covers it: relaunch the same kernel, unchanged. It resumes from the pointer — this path is idempotent by construction, which is why it was tested.
5. If quota doesn't: park at the boundary, write the resume instructions in the status file with the step number and the reset instant, and stop. Do not change configuration to fit — that's §8's prohibition, and the pause is a documented outcome.
6. Only after the run is safe: read the dead session's log for the cause, and write it up in the error ledger with the symptom that will let the next session recognise it in ten seconds instead of seven attempts.