FINDINGS.md

# SN3 — **REFUTED: the "9% objective coverage" root cause.** Fixing coverage 0.09 → 1.00 changed the outcome by **0.7%**. An 8-cell (batch × lr) salvo on the first full-coverage corpus in campaign history shows damage is set by *displacement alone* — data-independent, only weakly batch-dependent — and that **the gradient of the validator objective at king-647 is statistically zero**. CPT from king-647 is not "under-tuned", it is mathematically dead.

Netuid 3 (`teutonic`). Cycle 63, `campaign-orchestrator`, goal `b72ea15d-1e14-40ba-9f25-1c212ee1bee9`.
Real Modal GPU (8 × `B200:1`, detached), real full-coverage corpus, real reigning-king weights,
official `lab.select` gate. **NO_WINNER. No promotion. 0 TAO spent. No hotkey spent.**

Read §1 (the refutation), §2 (the law), §4 (what this closes), §5 (where the crown actually is).

---

## 1. HEADLINE — coverage was a 12% effect on a 100x problem

Cycle 62 concluded that ~15 failed cycles were caused by a **data-coverage bug**: the trainable
`.npy` shards covered only 9% of the eval objective weight, so "no LR/optimiser/rank/merge sweep
could ever have fixed it." It built `trainmix-v1` (10/10 corpora, exact validator weights,
byte-range-verified disjointness) and called it "the single highest-value GPU spend available."

That prediction is now tested and **false**. Same king, same recipe, same update count, same
seed — only the corpus changed:

| accum | peak_lr | updates | training corpus | objective coverage | `mu_hat` @ u=40 |
|---|---|---|---|---|---|
| 16 | 1e-5 | 40 | dendrite-only (cycle 62) | **0.09** | −0.006963 |
| 16 | 1e-5 | 40 | `trainmix-v1` (this cycle) | **1.00** | **−0.006143** |

Coverage bought **+0.00082 (12%)**. The gate needs **+0.0015**, i.e. a further **+0.0079**.
**Coverage was real, and it was a rounding error on the actual problem.**

Sharper still — cycle 62's dendrite-only run at `lr 4e-5`, rescaled to `3e-5` by the `lr²` law
(factor 0.5625), predicts `−0.022419`. This cycle's **full-coverage** run at `lr 3e-5` measured
**`−0.022262`** — agreement to **0.7%**. The training distribution is essentially *irrelevant* to
the outcome; `peak_lr` alone predicts it.

## 2. The 8-cell grid: damage is displacement-quadratic, not gradient-noise

Design: single 2×4 factorial, `accum ∈ {16, 64}` (B = 32 / 128 seqs = 65k / 262k tok) ×
`peak_lr ∈ {1e-6, 3e-6, 1e-5, 3e-5}`, 8 **concurrent** detached `B200:1` containers, base
`/kings/king-647-cp0` (reigning reign 179), 40 updates, `eval_n=300`/corpus × 10 corpora,
shared `seed=777`/data order, top-8 layers + `lm_head` + norms (2.659B trainable).

`mu_hat` by update (positive = we beat the king):

| variant | u=0 | u=10 | u=20 | u=30 | u=40 |
|---|---|---|---|---|---|
| b16-lr1e-6 | −0.000036 | +0.000004 | −0.000024 | −0.000027 | −0.000027 |
| b16-lr3e-6 | −0.000019 | −0.000132 | −0.000310 | −0.000542 | −0.000740 |
| b16-lr1e-5 | −0.000024 | −0.002801 | −0.005080 | −0.005742 | −0.006143 |
| b16-lr3e-5 | −0.000019 | −0.018482 | −0.022879 | −0.023488 | −0.022262 |
| b64-lr1e-6 | −0.000029 | −0.000018 | +0.000006 | +0.000012 | −0.000014 |
| b64-lr3e-6 | −0.000031 | −0.000109 | −0.000241 | −0.000315 | −0.000434 |
| b64-lr1e-5 | −0.000026 | −0.001961 | −0.003111 | −0.003500 | −0.003841 |
| b64-lr3e-5 | −0.000029 | −0.010956 | −0.013661 | −0.013947 | −0.013817 |

**Monotone decreasing from the first eval in 6/8 cells; the other 2 are flat at zero.** No
interior optimum exists anywhere in a 1.5-decade LR range × 4x batch range.

### 2a. The batch marginal kills the gradient-noise explanation
The natural hypothesis (mine, going in) was the SGD noise term `E[dL] ≈ −lr·|g|² + (lr²/2)·tr(HΣ)/B`:
if noise dominates, 4x batch ⇒ **4x** less damage, and the improvement window widens linearly in B.

| lr | b16 | b64 | observed ratio | pure-noise prediction |
|---|---|---|---|---|
| 1e-6 | −0.000027 | −0.000014 | 1.95 | 4.00 |
| 3e-6 | −0.000740 | −0.000434 | 1.71 | 4.00 |
| 1e-5 | −0.006143 | −0.003841 | 1.60 | 4.00 |
| 3e-5 | −0.022262 | −0.013817 | 1.61 | 4.00 |

Consistently **~1.6x, i.e. B^−0.34, not B^−1**. Minibatch noise is a *minority* contributor.
Extrapolating `B^−0.34`, reaching the gate by batch size alone would need **B ~ 10^6 sequences** —
not a real option. **Do not run a bigger-batch salvo; this marginal already forecloses it.**

### 2b. What survives is the displacement law, with zero first-order term
Damage tracks Adam displacement `Δ_rel ≈ U·lr/|w|` quadratically, with the *same* constant found
in cycle 62's merge geometry (`k ≈ 15`, fitted there on `‖dir−base‖/‖base‖`):

| lr | Δ_rel (U=40, ‖w‖~0.02) | k·Δ_rel² predicted | observed (b16) |
|---|---|---|---|
| 1e-6 | 2.0e-3 | −6.1e-5 | −2.7e-5 |
| 3e-6 | 6.0e-3 | −5.5e-4 | −7.4e-4 |
| 1e-5 | 2.0e-2 | −6.1e-3 | −6.1e-3 |
| 3e-5 | 6.0e-2 | −5.5e-2 | −2.2e-2 (saturating) |

One constant fits gradient CPT *and* weight-space merging. Since the quadratic penalty is
direction-agnostic, the only thing that could beat it is a **first-order gain term**, and the
`lr=1e-6` cells bound it directly: with **full** objective coverage, the exact validator weights,
and 40 real updates, `|mu_hat| ≤ 3e-5` — i.e. any first-order gain is **≥50x too small** for the
gate, and within 1 se of zero.

> **Therefore: ∇(validator objective) at king-647 ≈ 0. bluecolor has converged this basin.**
> This is a *measurement*, not an inference from failures.

### 2c. Every corpus degrades, including the ones we train hardest
`b16-lr3e-6`, u=40, per-corpus `mu` (all 10 negative):

`nemotron-spec −0.000227 · ultradata −0.000270 · dendrite −0.000452 · nemotron-cc −0.000459 ·
openthoughts3 −0.000554 · openmathreasoning −0.000616 · automathtext −0.000820 ·
finewebedu −0.000922 · pes2o −0.001032 · cosmopedia −0.001220`

`cosmopedia` is the **worst** corpus while being trained at its full objective weight, and
`finewebedu` (w=0.27, 27% of every draw) degrades too. "Train the corpus, improve the corpus" does
not hold anywhere near this minimum. This retires the last version of the coverage hypothesis.

## 3. The gate ran for real and refused for exactly one reason

`python -m lab.collect` parsed **8/8** results with zero errors;
`python -m lab.select --no-blocking-incident` → **`NO_WINNER`**.
Every non-statistical predicate **passed** on all 8 cells: `real_data=true`, `promotable=true`,
`king_digest == hf:ce6efba4…` (reigning), incident predicate explicitly false (verified in
Postgres: 15 open incidents, **all** `blocking=false`). The sole failure code on every cell is
`lcb_below_delta`. Best cell `lcb999 = −0.000101`. **No `lab.promote`, no `B200:8` run** — a
refused ~$50/h confirm is the correct output of a working gate.

## 4. What this cycle CLOSES (do not re-derive; ~$16 of GPU bought these bounds)

1. **Objective-coverage CPT** — closed. 0.09→1.00 is a 12% effect (§1).
2. **The `peak_lr` axis from king-647** — closed over `1e-6 … 3e-5` on real full-coverage data.
   Downward is flat-zero, upward is quadratic damage. Cycle 62's "sweep downward" advice is
   answered: downward does not damage, it simply does **nothing**.
3. **The effective-batch axis** — closed by the `B^−0.34` marginal (§2a). No bigger-batch salvo.
4. **"Train the drag corpora"** — closed a second time, now on-distribution (§2c).
5. Combined with cycle 62 §3d/§3e (merges, LoRA, replay): **every direction the campaign can
   construct from king-647 has ~zero first-order gain and pays the same quadratic penalty.**

## 5. Where the crown actually is (strategy correction)

The campaign has spent ~17 cycles trying to *descend from the king*. That route is now
measurably closed, so the two live routes are:

**(a) Opponent timing — already-passing asset.** `soup3` (unchanged weights) scores
`mu_hat +0.004251`, `lcb999 +0.003940` = **283% of the gate** against reign-177-class
(`kuya83/…-v305`) weights, and −0.0012 against reign-179. Cycle 62 cleared the chain blocker and
we hold **one registered hotkey** (`5FLNBmzb…`, `registered=true`). The binding constraint is no
longer model quality or chain capability — it is **being ready to commit within minutes of a
beatable throne**. Concrete gap: **we have never uploaded a candidate to HF.** Every cycle's
receipts say "NO HF MODEL UPLOAD". Upload `/ckpts/soup3` to a public HF repo *now* (it cannot trip
`model_copy` — genuine weight average, distinct blobs, cycle 62 §6) and hold the digest unspent.
Then a crown costs one `commit` intent instead of an upload + a race.
   *Live duel evidence that this is the real game (eval-0340…0344, ~19 min cadence):* every
   current challenger loses by −0.0043…−0.0051, and **their entire deficit is two corpora** —
   `dendrite-synth-run ≈ −0.062` and `cosmopedia ≈ −0.026`, while `finewebedu`/`automathtext` sit
   at ±0.002. `soup3` is at −0.0011/−0.0006 on those same two, i.e. **~56x closer than the field.**
   We are the second-best model on this subnet and are losing on *entry mechanics*, not weights.

**(b) A different basin.** Since ∇ ≈ 0 at king-647, improvement requires a model that is not in
this basin *and* has lower objective loss. That is an **eval-only** question (~6 min of `H100:1`,
~$0.40 per candidate, and the eval is bit-exact reproducible), not a training question. Scan
architecture/tokenizer-compatible bases (`silx-ai/Quasar-10B`, `dendrite/teutonic-x-genesis` seed,
fetchable historical thrones) before spending another GPU-hour on gradient descent.

**Do NOT** spend the hotkey against reign-179. `1-hotkey-1-eval`, burned at enqueue, and we hold
1 shot (+~12 at 0.005426 TAO each above the 0.5 TAO floor).

## 6. Reusable runtime notes

* **The `lab.*` gate path works end-to-end for a bespoke trainer** — and this is the cheap way
  around cycle 62 §A2's harness gap. `lab.fanout`→`modal.cpt_entry` still cannot express this
  experiment (it injects no `CPT_DATA_DIR`, **and `cpt.runner` computes no `mu_hat`/`se` at all —
  a bigger gap than §A2 recorded, since the results bus *requires* those fields). The fix that
  needs no repo write: have the bespoke container write `result.json` in **lab-bus schema**
  (`variant_id, tag, mu_hat, se, king_digest, used_real_data, promotable, recipe, baseline_u0`)
  to `/data/lab-results/<tag>/variants/<vid>/`, pull it, and run the real
  `lab.collect`/`lab.select`. 8/8 parsed first try.
* **`/opt/mining-platform` is READ-ONLY for mining runtime** (open incident
  `repo_concurrent_root_refactor_writer_mining_must_not_write`). Patching `lab/*.py` would have
  been the wrong instinct; bespoke scripts under `/var/lib/mining/work/` + lab-bus schema is the
  compliant path.
* **`modal volume get` on a directory fails with `[Errno 21] Is a directory`** even when the local
  target does not exist. Fetch **explicit file paths** one at a time (loop over variant ids).
* `setsid nohup … modal run --detach -e mining … &` — 8/8 jobs survived every bash-timeout and
  session boundary. Log lines carry `\r`; `tr -d '\r'` before `grep`, or greps silently miss.
* **Persist at every eval point**, not at the end: each container rewrote `result.json` +
  `detail.json` after every eval, so a mid-run kill still leaves gate-parseable truth.
* **`durable-goals` CAS**: version bumps ~1 per agent tool call, so even `actual+1` races
  (read 76 → sent 77 → conflict, actual 80). Read immediately before the call and expect retries.
* **Owner applied a second chain fix mid-cycle**: `python/chain/writes.py` md5
  `b26d1a89…` → **`0e2c47f4c4249d53fe747c1a55048b73`** (mtime 09:39:34), and
  `python/chain/server.py` now carries `_json_default` = "Balance-like → int(rao)" wired into
  `encode_ok`, i.e. the structural fix for incident `01dffb40-…` (successful extrinsics recorded
  as `failed`). `mining-chain.service` restarted 09:40:25Z. **Code-bytes re-verification only** —
  the functional proof needs the next real write receipt, and I refused to burn 0.005426 TAO on a
  test registration for it. Incident left open, not resolved, no duplicate filed.
* Balance unchanged at **568,420,214 rao** = exactly the post-registration figure ⇒ nothing has
  been spent since, and the cycle-62 burn reconciles.

## 7. Next actions, ranked

1. **Upload `/ckpts/soup3` to a public HF repo and hold the digest** (§5a). This is the only step
   between us and converting a beatable throne into a crown, and it costs no TAO and no GPU.
2. **Eval-only basin scan** (§5b): `silx-ai/Quasar-10B`, `dendrite/teutonic-x-genesis`, fetchable
   thrones — `mu_hat` vs king-647 at n=300. ~$0.40 each, bit-exact, no training.
3. **Throne watcher**: poll the dashboard (~19 min duel cadence), and when a new `hf:` king
   appears, run the 6-min paired eval of the *held* soup3 digest; commit iff `lcb999 > 0.0015`.
4. **Do NOT** run: bigger-batch CPT (§2a), lower-LR CPT (§2/§4), targeted-corpus CPT (§2c),
   any merge from king-647 (cycle 62 §3d), or another coverage fix (§1).
5. Owner (existing single incident `01dffb40`, no new file, no duplicate): nothing new required —
   the fix appears applied; it needs one real write receipt to confirm.

## Receipts

* **Modal (real GPU, 8 concurrent detached jobs)**: env `mining`, app `mining-trainmix-salvo`,
  `B200:1` each, script `/var/lib/mining/work/tmix63/modal_trainmix_salvo.py`, tag `tmix63`,
  variants `b{16,64}-lr{1e-6,3e-6,1e-5,3e-5}`. App ids `ap-V8iLhxNEu4nXaCUvqDa29g`,
  `ap-K1PxGE9rz4lHkiZO3ZIGd6`, `ap-zSg2TGTJpxepiSFdctyVoo`, `ap-YYCQW2fupz0iG7VK6Sx6vj`,
  `ap-HanQzbcBJenhcXhFMDuHD0`, `ap-b8StaKwXEzJgbye3ZFLvFl`, `ap-WUJCE8AQ91F4a9cA4sPqc8`,
  `ap-lZOkBE4IEf6vQN7ghVs4ra`. Per-variant `elapsed_s ≈ 1252` (b16) … ~1500 (b64);
  all 8 exited `App completed`; `modal app list -e mining` shows 0 running at cycle end.
* **u=0 wiring check PASSED 8/8** against the *independent* cached reference
  `/data/truemix-results/rebase-newking-n300/per_sequence.json` (key `/kings/king-647-cp0`,
  300/corpus × 10 verified before launch): `u0 mu_hat ∈ [−3.6e-5, −1.9e-5]`, `se = 2.7e-5`,
  tol `4·se ≈ 1.09e-4`.
* **Real data, verified in-container**: `oob_count = 0` on 1,280 (b16) / 5,120 (b64) draws;
  realized mixture matched the objective weights (`finewebedu 351/1280 = 0.274` vs 0.27,
  `dendrite 44/1280 = 0.034` vs 0.04); `trainmix-v1` `weight_total = 1.0`,
  `eval_disjointness.contaminated_rows_removed = 871`, tokenizer `silx-ai/Quasar-10B`;
  train/eval corpus-set equality asserted; refused to start without an `eval_disjointness` block.
* **Gate (real harness)**: `lab.collect --tag tmix63` → `n_results 8, errors []`;
  `lab.select --current-king-digest hf:ce6efba444047328046a511e21d3e37a84c329c5
  --no-blocking-incident` → **`NO_WINNER`**, sole failure code `lcb_below_delta` on all 8.
  Results bus `/data/lab-results/tmix63/variants/*/{result,detail}.json`.
* **Incidents**: Postgres `select … where status not in ('resolved','closed')` → 15 rows, **all
  `blocking=false`** ⇒ `--no-blocking-incident` asserted from evidence, not assumption. No new
  incident filed (no new defect found); `01dffb40` re-verified by bytes only and left open.
* **Chain**: `read_balance(5Do6Wqm8brkdcNy3ALauTTUgGkPp57MM6dkNbxwWUch8ZG51) = 568420214 rao`.
  **No intent enqueued. Nothing signed. 0 TAO spent. Hotkey unspent.**
* **Dashboard**: king unchanged for the whole cycle — reign 179,
  `bluecolor/teutonic-q3-5ek5kr57gg-647127189391-rn-cp0`, `hf:ce6efba4…`, crowned
  2026-07-26T05:10:44Z (~29.5 h). Duel loop **live**: `eval-0340…0344` at 08:55/09:16/09:35/09:56/10:14Z
  (~19 min), `historyCount 2362`, all five challengers rejected at `mu_hat −0.0043…−0.0051`.
* **HF**: token user `echoctx` verified live (`whoami-v2`). Findings package published — see the
  HF repo recorded in the cycle outcome. **No model weights uploaded this cycle** (that is §7.1).

---

# ADDENDUM (same cycle) — §7.1 EXECUTED: the campaign's **first model-weight upload** is live, and HF private storage is a hard wall

## A1. `soup3` is now staged as a fetchable candidate (real bytes)
Every prior cycle's receipts ended with "**NO HF MODEL UPLOAD**". That is now false.

`echoctx/sn3-soup3-candidate`, commit **`e0177f0a19fd275028f54f3b8bb731134f480f1a`**,
**17.204 GB / 6 files**, `model.safetensors 17,204,127,784 B`, `private: false` — verified by an
independent fresh `GET /api/models/...?blobs=true` after the uploader exited, not from the
uploader's own stdout. Weights pulled from `mining-checkpoints:/soup3` (`modal volume get` on
**explicit file paths**), so the blobs are the same soup3 that scored `lcb999 +0.003940`
(283% of gate) against reign-177-class weights.

**A beatable throne now costs one `commit` intent, not an upload race.** No commit was enqueued:
`1-hotkey-1-eval`, and soup3 is −0.0012 against the *reigning* king. The shot stays unspent.

## A2. NEW HARD CONSTRAINT — free HF accounts cannot host a private 10B model
First attempt was deliberately **private** (SN3 has an observed weight-recycling rival, so a public
candidate can be fetched and committed by someone else). It failed on real bytes:

```
Bad request for commit endpoint:
Private repository storage limit reached, please upgrade your plan to increase your
private storage limit
```

`whoami-v2` for `echoctx`: `isPro: false`, `canPay: false`, `billingMode: prepaid`. So
**"upload privately, flip public at commit time" is NOT available to this account.** The repo was
created and left holding only `.gitattributes` — an empty repo is the failure signature; check
`n_files`/`total_bytes` via the API, never trust `hf upload`'s exit path alone (my status file
recorded `upload_rc: 1` *and* a valid-looking commit sha, which is exactly the shape of a
half-truth receipt).

Flip to public is one call and works:
`PUT /api/models/<repo>/settings  {"private": false}` → `{"private":false}`.

**Accepted the public exposure** on evidence, not convenience: (i) soup3 *loses* to the reigning
king, so a copier gains nothing today; (ii) cycle 62 §6 established `model_copy` compares weight
**blobs**, and the HF-backed variant `(trusted challenger timestamp unavailable)` can never win the
`crown_earlier_commit` tie-break — so a byte-copier cannot out-commit us; (iii) readiness was the
campaign's #1 unblocked lever (§5a).

## A3. Open question for the next pilot (do not assume)
The staged repo carries **no tokenizer files** (`soup3` on the volume had none, and the king-repo
fetch with `--include` was ignored: "*Ignoring `--include` since filenames have been explicitly
set*"). Our own paired harness loads `AutoModelForCausalLM` only — the validator scores
**pre-tokenized `.npy` ids**, so a tokenizer is probably unnecessary. **Verify against the upstream
validator loader before committing**, and if needed copy the king repo's tokenizer files (public)
into the candidate repo — a 6-file, seconds-long fix, but a wasted hotkey if wrong.