Affine SN120 mining harness

Two-stage pre-screen for Affine (netuid 120) so no hotkey is spent on a checkpoint that was never going to crown.

Contract facts this is built on (verified against the live chain, 2026-08-08): one submission per hotkey ever; crown needs paired margin > 3·se and > 0.02; 4 of 271 submissions have ever been accepted.

Why two stages

The ranking term is S = Λ2 + w·clip(L1lift, ±0.1).

• Λ2 = lpC(y_C|z_A) − lpC(y_C|∅) needs the teacher's logprobs.
• L1lift = lpA(y_C|z_A) − lpA(y_C|∅) is entirely challenger-side.

The validator publishes complete duel dumps, which contain the teacher's sampled (z_C, y_C) per turn and let the reigning king's per-turn scores be recovered. So ΔL1lift can be measured by serving one model on 2 GPUs — and it predicts the paired duel margin at R² = 0.90.

Backtested leave-one-out on 79 published duels, a screen gate of 0.020 keeps both known crowns while removing 92% of the field.

Runbook

Phase 0 — offline, free (do this first)

python3 harness/replay.py          # scorer must reproduce all published verdicts
python3 harness/analyze.py         # what winners do differently
python3 harness/levers.py          # which quantity actually predicts margin
python3 harness/calibrate.py       # fit ΔL1lift -> margin
python3 harness/backtest.py        # leave-one-out + screen gate sweep
python3 harness/selftest.py        # dry-run the live code path, no GPU

replay.py must print 95/95 exact. If it doesn't, stop — nothing downstream is trustworthy.

Refresh the inputs

python3 harness/sync.py            # re-pull eval dumps + corpus (sha-verified)

The king changes every ~24 h and the corpus is refreshed as an anti-memorisation measure (it went from 9,871 to 18,384 turns on 2026-08-08). Re-sync before every screening session and point --ref at the current king's crowning duel.

Stage 1 — screen a candidate (2 GPUs)

vllm serve <repo> --revision <sha> --served-model-name <repo> \
    --tensor-parallel-size 2 --max-model-len 32768 \
    --gpu-memory-utilization 0.80 --max-num-batched-tokens 8192 --port 8002

python3 harness/screen.py --repo <repo> --revision <sha> --out result.json

--gpu-memory-utilization 0.80 is not a typo. Teacher-forced echo+logprobs materialises an fp32 log_softmax over (tokens × vocab) outside vLLM's budgeted pool — the validator's own config records that 0.85 OOM'd on 268 GiB B300s.

Verdict is PROMOTE or REJECT. Promote only survivors to the full duel.

Phase 1 — train a candidate

# 1. dataset. Default samples a served teacher over turns NO dump has used
#    (16,645 of 18,384 = 90.5% of the corpus), keeping train and eval disjoint.
python3 harness/distill_data.py --from-teacher --n-turns 4000 --out data/sft.jsonl
#    free shakedown alternative (RT-6 risk — read the docstring):
python3 harness/distill_data.py --from-dumps --out data/sft_dumps.jsonl

# 2. LoRA on the reigning king
python3 harness/train_lora.py --base <king repo> --revision <sha> \
    --data data/sft.jsonl --out runs/v1

# 3. merge a checkpoint into full weights
python3 harness/merge.py --base <king repo> --revision <sha> \
    --adapter runs/v1/checkpoint-300 --out merged/v1-ckpt300

# 4. screen it, and only then upload

Training examples are rendered with chat.gen_prompt(...) plus the canonical </think>\nTHOUGHT: {z}\n\n{y} span, byte-identical to chat.force_text() — what the validator teacher-forces. Loss is masked to that span, with the boundary located by offset mapping exactly as score_action locates it.

Screen several checkpoints. The crown went to ckpt300; the same operator's ckpt1000 measured z=1.96 and ckpt1500 never crowned. Default saves every 150 steps.

What the LoRA actually touches

Validated against the owner's published adapter and the king's index: 160 of the checkpoint's 1,026 tensors — mlp.shared_expert.{gate,up,down}_proj on all 40 layers, plus self_attn.{q,k,v,o}_proj on the 10 full_attention layers (the architecture interleaves 3 linear_attention layers per full one). The routed MoE experts (mlp.experts.*) and the entire vision tower are untouched.

The merge trap

merge_and_unload().save_pretrained() drops the vision tower on this multimodal checkpoint, and vLLM then refuses to load it — the unservable verdict that burned 7 of 93 duels.

The cause is a key-namespace mismatch: the checkpoint stores text weights under model.language_model.layers.N..., but peft records base_model.model.model.layers.N.... merge.py streams the base shards and applies W += (alpha/r)·B@A only to resolved targets, copying the other 866 tensors (including all 333 model.visual.*) through byte-for-byte, into shards with the base's own names and index. Peak memory is one tensor, not one 70 GB model. It refuses to write if any adapter target fails to resolve, and its pre-flight asserts the output differs from the base (model_copy defence), the size caps, no *.py, and no auto_map.

Phase 2 — the whole pipeline on one pod

python3 harness/run.py --out runs/v1 --plan     # cost estimate, spends nothing

# calibrate the rates first, for a few dollars
python3 harness/run.py --out runs/smoke --n-turns 40 --max-steps 20 \
    --save-steps 10 --keep-checkpoints 2

# real run
python3 harness/run.py --out runs/v1 --n-turns 4000 --max-steps 900

run.py chains sync → data → train → merge → screen → rank, reshaping the GPU layout per stage on an 8-GPU pod:

Each stage is resumable — artifacts already on disk are skipped, so a crash or --phase screen picks up where it stopped. run.py never submits anything on-chain; it ends at a ranked table and the hotkey decision stays manual.

The king is re-read from index.jsonl on every invocation and pinned into runs/<name>/king.json, so a crown change mid-run is visible rather than silent.

The --plan rates are assumptions, not measurements. --gen-tok-per-sec, --step-secs, --merge-mins and --screen-mins are guesses; a wrong throughput guess moves the bill by hundreds of dollars. Run the smoke config, then re-plan with real numbers.

Gates

Do not train on the screening turns

The duel slice is drawn from the full corpus by blake2b(reveal_block_hash ‖ hotkey). Training on the turn ids used here is exactly the RT-6 memorisation channel the validator patched on 2026-08-06 — it makes the screen look great and the real duel disappointing.

Files

score.py, chat.py and vllm_client.py are copies of the subnet's own code. Re-copy them whenever the contract changes — a local edit silently decalibrates everything.

Two contract eras

23 of the dumps predate the 2026-08-05/06 changes and record only 6 gate keys. They ran under l1_clip=0.1, r_lo=1.0, r_hi=4.0, no baseline_band, δ=0.05. Replaying them with today's defaults makes 6 look mis-scored — including reigns 1–2, which really were crowned retroactively when r_lo fell to 0.3. replay.py detects the era per dump; keep that behaviour.

Pod setup

pip install -r requirements.txt