docs/ARCHITECTURE.md

# cascade architecture

## The thesis

A time-series foundation model is only as good as the data it was trained on.
cascade makes **synthetic training data** the competitive resource: miners
write data generators, the subnet owner trains **a Toto2-4M backbone from random
initialisation** on each, and the generator whose data yields the best forecaster
wins. By holding the model architecture and the entire training process constant,
the subnet turns a noisy question ("is this model good?") into a controlled one
("is this *data* good?").

Training **from scratch** (not fine-tuning a released checkpoint) is what makes
that attribution clean: a fine-tune confounds data quality with what the
pretrained weights already encode, whereas from random init the corpus is the
*only* source of learned signal. This mirrors Toto 2.0 itself, whose pretraining
mix is 57.5% synthetic and 0% public time series yet still tops GIFT-Eval — the
synthetic prior is the lever, and cascade competes it.

## Roles and data flow

### 1. Miner — submits a generator

A miner writes `generator.py` exposing `Generator(DataGenerator)`, pushes the
repo to the **Hippius Hub registry** (OCI) with `cascade deploy`, and commits a
single on-chain pointer:

```
metro-v1:gen:hippius:<repo>@<digest>
```

The Hub `repo@digest` content-addresses the generator code, `config.json`, and
`requirements.txt` together — it both locates and pins the submission (the OCI
digest *is* the content hash, so there is no separate git SHA). A generator is
code-only (purely algorithmic): no shipped weights of any kind, so a miner cannot
distill a pretrained forecaster into the "generator". The distinction from horizon
is *what is scored*: cascade scores the **data** (via a fixed model trained on it),
horizon scores the submitted model directly. See `docs/INTERFACE.md`.

### 2. Trainer — owner-operated, the GPU boundary

A round is one ~24h epoch (`[round] epoch_blocks`); the trainer runs one round
per epoch (so the king is trained once per day). Each round:

1. Resolves on-chain commitments to `(hotkey, uid, repo, revision)`, keeping only
   those revealed **before the epoch boundary** (`resolve_commitments(...,
   cutoff_block=epoch_start)`) — that boundary is the submission deadline.
2. Identifies the reigning **king** (highest-incentive UID on the metagraph) and
   the eligible **challenger field**.
3. Derives one `RoundSeeds` from the round's base seed (the block hash **at the
   epoch boundary**): a shared `generation_seed` and a shared `training_seed`,
   used by every training in the round — heat and final, all sizes — so the whole
   day shares one random init.
4. **Heat (screen).** Trains every eligible challenger cheaply
   (`[round] heat_train_hours`, ~30min, on the primary/smallest size), scores each
   on the held-out pool (geomean of CRPS/MASE), and keeps the top
   `[round] finalists` (default 1). A challenger that fails to train or score just
   doesn't qualify.
5. **Final.** For the king and each surviving finalist, at **every configured
   size** (the `[training]` primary plus each `[[training.sizes]]`, e.g. 4M + 22M),
   **under that one shared seed pair**:
   - opens the round's corpus stream (`cascade.trainer.stream.open_round_stream`,
     selected by `[training] corpus_mode`): `stream_cpu` streams *fresh* `(C, L)`
     series from a sandboxed generator with no reuse (rolling byte-exact digest);
     `cache_reuse` draws a fixed corpus once (also sandboxed) and cycles it. Either
     way the trainer gets one budget-capped iterator (univariate `C = 1` today;
     the channel axis is carried so multivariate priors need no schema change),
   - trains a **fresh Toto2 model from random init** at that size via the owner's
     `BaseTrainer` (`cascade.trainer.contract`; reference:
     `cascade.trainer.toto2_trainer`) — it pulls series until the stream ends, for
     the per-size budget (~3h on the reference GPU, enforced as a fixed
     `train_tokens` count so king and challenger get identical compute), streaming
     per-step metrics (loss, lr, throughput) to **Hippius S3** (and, when
     `[wandb] enabled`, mirroring the *same* records into a live wandb run — one
     per round/competitor/size, tagged with the miner hotkey — so miners can watch
     their generator train as it occurs; observability only, never fed to scoring),
   - pushes the checkpoint to the **Hippius Hub registry** (OCI) and records its
     size-tagged ref.
6. Signs a `TrainingManifest` (trainer hotkey) listing every trained-model ref
   (one king + finalist pair per size, each tagged with its `size`) and the
   corpus/contract digests, and publishes it to the **Hippius S3** manifest
   bucket (`round-<id>.json` + `latest.json`).

`BaseTrainer` is a `Protocol` — the single GPU-dependent seam. Everything else
in the trainer is numpy/CPU and unit-tested. A reference implementation (a
Toto2-4M backbone trained from random init under the `chain.toml [training]`
recipe — `head_dim 64`, `patch_size 32`, a 9-quantile pinball head, u-μP, the
NorMuon+AdamW split) is the operator's to provide; it must be **stateless across
the king and challenger calls** so no information leaks between the two training
runs (shared `training_seed` ⇒ identical random init for both).

#### Two-device (remote) training

By default the king and challenger train sequentially on the trainer's own GPU.
For faster rounds the trainer can dispatch them **in parallel to separate
SSH-reachable GPU pods** (e.g. rented Lium/Targon boxes) via `--remote-hosts`
(`cascade.trainer.remote`). The remote unit is a **round-worker**
(`cascade.trainer.worker`), not a remote `BaseTrainer`: each pod pulls its
generator from the registry by ref, builds the corpus in its own sandbox, trains,
uploads the checkpoint, and returns a `TrainedEntry` receipt over SSH. The
orchestrator collects the receipts and signs + publishes the manifest, so **the
trainer hotkey never lands on a rented box**; pods need registry/S3 access, not
the wallet. The host list is a trainer-local file (`scripts/remote_hosts.example.toml`),
never `chain.toml`.

This preserves the controlled experiment: the budget is a fixed `train_tokens`
count, so king and challenger get **identical compute** regardless of which (or
how fast a) device runs them. King failure aborts the round; a challenger failure
just drops that challenger.

**Byte-exact audit (pinned GPU).** The reference trainer runs deterministically
(deterministic cuBLAS/cuDNN, the math attention kernel, all RNGs seeded from
`training_seed`), so on a **fixed GPU SKU** a re-derived run reproduces the exact
checkpoint. Each run records its `torch.cuda.get_device_name(...)` into the
manifest entry's `gpu_name`, and the validator's gate enforces matched hardware:
with `[training] expected_gpu` set, every entry must report that SKU; otherwise
king and challenger must at least match each other. So pin one SKU on both pods
(e.g. both an H100) and the round is byte-reproducible end-to-end; leave
`expected_gpu` empty and you only lose the cross-round SKU pin, not the
king-vs-challenger guarantee.

### 3. Validator — reads the manifest, decides the throne

The validator never trains. Each round it:

1. Reads the current manifest, verifies its signature and that king and
   challenger share the **contract digest** and **base-arch digest** (the
   controlled-experiment gate — `ValidatorRunner.check_manifest`). The contract
   digest covers every size at once (`[[training.sizes]]` is folded into it).
2. For each trained **size**, pulls the king's and finalist's checkpoints and
   scores them on the **same** held-out real-world eval windows
   (`cascade.validator.evaluator`), then **pools** the per-window scores across
   sizes (king-vs-finalist), preserving pairing because each size shares the
   window `abs_target`.
3. Runs ONE paired-bootstrap KOTH verdict on the pooled scores
   (`cascade.eval.koth.evaluate_round`) — a single throne decided on the combined
   4M+22M skill — and folds it into the champion state.
4. Sets weights: an equal share across the current king plus up to
   `[scoring] reward_prior_kings` registered prior kings (`reward_prior_kings = 0`
   ⇒ winner-take-all on the king; burns to `burn_uid` if none are registered).

### Cascade — king-reign promotion

On top of the daily KOTH sits **Cascade** (`cascade.validator.cascade`), a
wall-clock ratchet that periodically raises the floor the whole field trains up
from. A **reign clock** counts days since the current king last took the throne;
every dethrone re-crowns and resets it (Cascade reuses the KOTH dethrone signal —
it never re-implements dethroning). During a reign every checkpoint the king
produces is scored on the three public suites — **GIFT-Eval, BOOM, and TIME** —
`score = geomean(gifteval_crps, gifteval_mase, boom_crps, boom_mase, time_crps,
time_mase)`, lower better — and kept in a per-reign log. All three suites report
CRPS/MASE the same way — the shifted geometric mean, across tasks, of each metric
**normalized by the Seasonal-Naive baseline** (≈1.0 = baseline parity) — so the
six numbers are the same kind of quantity before they enter the geomean. When a king holds the
throne `[scoring] cascade_reign_days` (default 7) consecutive days undethroned —
counted in blocks (7200/day), anchored to the manifest's epoch-start block so
every validator fires on the same round — a
**Cascade** fires: the reign's lowest-score checkpoint (a lookup, not a re-eval)
is installed **as-is** as the warm-start init for all subsequent rounds; the king
**persists** on the throne with a fresh reign clock (DEC-CA-0004 — both roles
train from the shared init, so promotion confers no advantage worth vacating
over, and the throne only changes hands via a genuine dethrone). The reign clock
and checkpoint log persist next to the champion state, so Cascade survives
validator restarts.

Those six numbers are **authoritative from the trainer, not recomputed per
validator**. The trainer (owner-operated, already the manifest trust anchor) runs
the benchmark sidecar once on the king's checkpoint and stamps the numbers onto
that entry in the *signed* manifest (`manifest.BenchScores`), so every validator
records the identical values — Cascade selection is deterministic across
validators rather than each re-running a non-bit-reproducible GPU sweep. A
validator falls back to scoring the checkpoint itself only when the manifest
carries no scores (e.g. a trainer predating the hook). The eval is the **full**
GIFT-Eval + BOOM + TIME battery each round (`[eval] cascade_bench_max_series = 0`;
BOOM full ≈ 26 min on an RTX 5090, run with `--bench-device cuda`), and TIME's
Seasonal-Naive baseline — checkpoint-independent — is cached so only the model
forward is paid per round. The dethrone verdict itself stays entirely on the
private eval pool; these public-benchmark numbers drive only Cascade's warm-start
promotion. Cascade is opt-in — `[scoring] cascade_enabled`
(off by default) — and when off the trainer skips the eval and validators run pure
KOTH.

## The controlled-experiment invariant

For a round to be a fair measurement of data quality, at **each size** the king's
model and the challenger's model must differ in **exactly one** thing: the corpus.
cascade enforces this on three sides:

* **Trainer:** one `RoundSeeds` instance is reused for every run in the round —
  heat and final, king and challenger, all sizes — so weight initialisation
  (`training_seed`, the from-scratch init) and the generation seed are identical;
  only the per-size width/depth changes between sizes, never between king and
  challenger of the same size.
* **Manifest:** `contract_digest` (sha256 of the `TrainingContractConfig`,
  including every `[[training.sizes]]`) and `base_arch_digest` are recorded once;
  each size's frozen-arch digest is folded into the contract digest.
* **Validator:** rejects any manifest whose digests don't match its own
  `chain.toml`, so a tampered or mismatched training run can't score.

Auditability: because both seeds derive deterministically from the chain block
hash and every corpus carries a `corpus_digest`, a second honest trainer (or a
suspicious validator) can re-draw the corpus and re-train to confirm the run.

## Scoring

Per window, per channel, per model: MASE (Hyndman seasonal-naive denominator) and
the gluonts `MeanWeightedSumQuantileLoss` components `(qloss_per_q, abs_target)`
over the 9-level grid `0.1…0.9`. That grid is *exactly* Toto 2.0's training
objective — its quantile head predicts those nine levels under pinball loss — so
cascade's **score objective equals the model's train objective**, which
collapses the metric-layer gap between what's trained and what's measured.
Univariate windows produce one score each (`channel = 0`); a multivariate window
contributes one row per channel.

The KOTH decision is a **paired bootstrap LCB** on the relative improvement of
`geomean(MWSQL, mean MASE)`, challenger vs king, resampling window indices once
per bag and aggregating MWSQL numerator/denominator before dividing (robust to
near-zero-mean windows). The challenger wins a round iff that LCB clears the
win margin on at least `min_windows` common windows. The windows
are a **rotating private slice** (`cascade.validator.windows`): seeded by the
round's block hash so every validator scores the identical set and the king/
challenger comparison is paired, but rotated each round so no fixed eval set can
be distribution-matched.

Dethroning is configurable. The shipped `chain.toml` sets `dethrone_cp = 1` with
a flat margin (`win_margin_start == win_margin_end`, `margin_warmup_rounds = 0`),
so a single round that clears the margin takes the throne and every king is
equally challengeable regardless of tenure. The sticky, tenure-weighted variant
is still available: set `dethrone_cp > 1` (a challenger must then win that many
**consecutive** rounds; a single loss or inconclusive round resets the streak)
and let `win_margin_end > win_margin_start` ramp over `margin_warmup_rounds` of
tenure so an entrenched king must be beaten more decisively.

**Public-benchmark no-regression gate (optional, off by default).** With
`[scoring] gift_gate_mode = "enforce"`, a dethrone additionally requires that the
challenger has not *statistically meaningfully regressed* on broad public data
(GIFT-Eval). On a private-pool win, both models are scored via the isolated
`benchmarks/` sidecar and a **paired no-regression bootstrap**
(`cascade.eval.gift_gate`) checks `lcb >= -gift_gate_tolerance` on the shared
configs. The gate is **not winnable** — it can only block a dethrone the private
LCB already granted — and an uncomputable gate (sidecar down, too few configs,
or king/challenger on different pinned data revisions) makes the round
inconclusive rather than silently passing or failing. `gift_gate_mode = "shadow"`
computes and logs the verdict without enforcing it, to calibrate the tolerance
against real noise first.

## Trust model (v1) and the path to decentralisation

v1 centralises training in the owner's trainer and trust in `[manifest]
trainer_hotkey`. This is the pragmatic bootstrap: it makes the controlled
experiment trivially enforceable. The corpus/contract digests already make every
run *reproducible*, which is the hook for decentralising training later (have
validators or a trainer quorum re-derive and challenge a manifest).

## What's implemented vs. a boundary

Implemented and tested (numpy/CPU): the generator contract + output checks (with
the MV-ready `(C, L)` channel axis), the static guard, commit/pointer parsing
(Hippius Hub `repo@digest` scheme), config (the full from-scratch Toto2 contract, digest-pinned),
the manifest schema + digests + **signing/verification**, the full scoring + KOTH
math, the champion state machine, corpus building from a generator, the trainer's
pairing logic, the **Hippius storage layer** (Hub ref grammar + S3
manifest/log/pool-snapshot layout), the rotating private window selection *and* the **eval-pool
loader** (`cascade.validator.pool`), and the trainer-round assembly + both
**live service loops** (`trainer/main.py`, `validator/main.py`).

The **Toto2-4M from-scratch `BaseTrainer`** ships as a runnable reference
(`cascade.trainer.toto2_trainer`) behind the `[train]` extra — a causal patch
transformer with a 9-quantile pinball head, u-μP-style init, a Muon+AdamW
optimiser split, and a token-budget LR schedule. It is the one piece that needs a
**GPU to validate end-to-end** (no GPU in CI); run a real round on your reference
box, then pin `base_arch_digest` / `ref_throughput_tokens_per_s`. Other operator
inputs before launch: the Hippius `[storage]` credentials/endpoints and the
held-out eval-pool ref (`[eval] window_pool`). The corpus sandbox subprocess
caveats are unchanged.