Validator ↔ Trainer Communication Map
Every channel between the subnet's roles, in one place: who writes, who reads,
what carries integrity, and what happens when several validators run at once.
docs/ARCHITECTURE.md explains why the round works this way; this page is
the wire-level how.
Terminology worth pinning down first: the trainer publishes the round's manifest (signed training receipts); each validator publishes a round receipt (signed scoring/weights record). Validators never post manifests — they gate, score, set weights, and publish receipts.
The picture
Bittensor chain
commitments ▲ │ commitments, ▲ weights
(gen refs) │ │ incentive, blocks │ (per validator,
│ ▼ │ independent)
┌────────┐ ┌─────────────────────┐ ┌──────────────┐
│ miners │ │ trainer │ │ validators │ × N
└────────┘ │ (owner GPU boundary)│ └──────────────┘
│ └─────────────────────┘ ▲ ▲ ▲ │
│ generator │ checkpoints │ manifest │ │ │ │ receipts (signed,
│ repos ▼ ▼ │ │ │ ▼ per-validator prefix)
▼ ┌───────────┐ ┌─────────────────────────────────┐
┌───────────┐ │ Hippius │ │ manifest bucket (Hippius S3) │
│ Hippius │ │ Hub (OCI) │ │ manifests/… logs/… │
│ Hub (OCI) │ └───────────┘ │ receipts/<hotkey>/… index.html│
└───────────┘ └─────────────────────────────────┘
▲
┌───────────┐ pool bucket │ snapshots (tar+sha256,
TSBench-Forge ─► │ owner pool│ ───────────────┘ selected by
(private raw │ builder │ effective_block)
data bucket) └───────────┘
Channels
| # | Channel | Writer → Reader | Medium / key | Integrity | Multi-validator behaviour |
|---|---|---|---|---|---|
| 1 | Generator commitments | miner → trainer, validators | chain reveal commitments, metro-v1:gen:hippius:<repo>@<digest> |
on-chain, digest-pinned | read-only; identical view per block |
| 2 | Round seed | chain → everyone | epoch-start block hash → seed_from_block_hash |
consensus | deterministic, shared |
| 3 | Training manifest | trainer → validators | manifests/round-<id>.json + manifests/latest.json (manifest bucket) |
signed by [manifest] trainer_hotkey; gated by check_manifest |
read-only; single writer (the trainer) |
| 4 | Checkpoints / generators | trainer, miners → validators | Hippius Hub OCI, repo@sha256:… |
content-addressed digest self-verifies | read-only |
| 5 | Eval pool | owner pool builder → validators | pool/snapshots/block-<N>.tar + pool/index.json (pool bucket) |
sha256 pinned in the signed manifest (eval_pool_key/eval_pool_sha256, gated by check_pool_pin); index sha256 as legacy fallback for unpinned manifests |
read-only; all validators resolve the same snapshot for a round and verify it against the pin |
| 6 | Weights | each validator → chain | set_weights |
on-chain, per-hotkey | independent by construction — no shared state |
| 7 | Round receipts | each validator → auditors, dashboard | receipts/<hotkey>/round-<id>.json + receipts/<hotkey>/latest.json, shared receipts/latest.json pointer |
signed per validator hotkey; public-read | single-writer prefixes — validators cannot clobber each other; only the convenience pointer is last-writer-wins |
| 8 | Receipts index | each validator → dashboard | receipts/index.json, entries keyed (round_id, validator_hotkey) |
unsigned, presentational only | merge-keyed read-modify-write; a simultaneous write can drop one entry until that validator's next round — never audit- or weight-bearing |
| 8b | Live chain status | validator (or scripts/publish_chain_status.py) → dashboard |
status/chain.json: block anchor, epoch grid, stage windows, revealed submissions (see cascade.shared.chain_status) |
unsigned, presentational only | last-writer-wins whole-object overwrite — honest publishers write near-identical chain state; never audit- or weight-bearing |
| 9 | Training logs | trainer → observers | logs/round-<id>/<role>.jsonl (+ optional wandb) |
none (observability) | read-only |
| 10 | Trainer ↔ GPU pods | trainer orchestrator ↔ rented pods | SSH, receipt-sentinel stdout | wallet never leaves the orchestrator | n/a |
| 11 | Validator ↔ eval pod | validator ↔ its own GPU pod | SSH+scp (--eval-hosts) |
only public checkpoint + report cross | per-validator, private |
The channels a verdict depends on (1–6) are all either on-chain, signed, or content-addressed and deterministic per round — N validators reach the same verdict independently, which is why weights need no coordination. Channels 7–9 are evidence and observability, never inputs to consensus.
Because storage keys aren't prefix-scoped, anyone holding a bucket's write credentials can overwrite any object in it. The design assumes that and anchors integrity above the storage layer: manifests fail the trainer-hotkey signature gate if touched, checkpoints are digest-pinned, and the eval pool is pinned inside the signed manifest. What write access does buy an attacker is vandalism — deleted receipts, a defaced dashboard, stalled rounds — which is detectable (signatures, the R2 mirror) but not preventable; the mitigation is key hygiene: one pair per bucket, pool keys separate from manifest keys, forge keys never shared beyond the owner.
What a validator needs access to — and only this
| Resource | Access | Credential |
|---|---|---|
| Bittensor chain | read + set_weights |
hotkey wallet |
| Manifest bucket | read manifests/…, write receipts/<own hotkey>/… + index |
HIPPIUS_S3_* |
| Pool bucket | read-only snapshots | POOL_S3_* — issue a pair separate from the manifest-bucket keys (S3 keys aren't prefix-scoped; a shared pair could write the pool) |
| Hippius Hub | read (pull checkpoints/generators) | HIPPIUS_HUB_TOKEN |
| HF benchmark datasets + bench sidecar (opt-in only, see below) | public read | HF_TOKEN if gated |
The benchmark sidecar (benchmarks/, or an SSH eval pod via --eval-hosts) is
not part of a stock validator. All three of its call sites default off:
[eval] run_benchmarks (log-only numbers on a dethrone), [scoring] gift_gate_mode (the gift-eval gate — the only mode in which the sidecar is
load-bearing, since enforce turns a sidecar failure into an inconclusive
round), and [scoring] cascade_enabled (warm-start promotion, which prefers
the trainer-signed bench_scores on the manifest and touches the sidecar only
as a non-consensus-safe fallback for manifests that carry none).
Deliberately not on the list: the TSBench-Forge raw-data bucket. Validators
consume forge data only through the built pool snapshots (channel 5). The raw
catalog + parquet relay (tsbench-forge-sources on Hippius — see
docs/EVAL_POOL.md) is an owner-orchestrator-side secret; keeping it private
is the anti-gaming lever that stops miners from fitting generators to the
held-out pool.
TSBench-Forge → validator data path
forge repo cron (scrape-data.yml)
└─ sync_storage.py ─► s3://tsbench-forge-sources [private, owner-only]
owner orchestrator
└─ aws s3 sync … ./tsforge
cascade-pool publish --sources tsbench_forge ─► pool bucket snapshots
validators
└─ select snapshot by effective_block, verify sha256, rotate windows by round seed
One bucket contract end to end: sources.yaml + data/<source_id>/<date>.parquet
at the relay, deterministic tar + pool/index.json at the pool. The forge cron
and the pool-publish cron are decoupled; the staleness guard
(max_stale_days, fail-loud when every feed is stale) lives in the pool
builder so a dead scraper can't silently erode freshness.