log: v7-v12 stability campaign, 3 bugs fixed, no strength gain

docs/RESEARCH_LOG.md

@@ -2665,3 +2665,127 @@ slower** per update, and 96 threads is 5.8x slower — the update step is small
 enough that thread-synchronisation overhead dominates past 16. More cores do not
 help this workload; the 192-core machine is best used for parallel self-play
 (16 workers x 2 threads) rather than for wider intra-op parallelism.
+
+## `main_v6` is INVALID: `--resume` silently fell back to random init (MEASURED, FIXED)
+
+**Status: MEASURED. `main_v6` must not be cited as a result.**
+
+`main_v6` was launched with `--resume main_baseline/latest.pt`, a path inside the
+**hub model repo**, not a local file. The container-rebuild recovery block only
+knew how to pull `<basename(--out)>/latest.pt` — i.e. `main_v6/latest.pt`, which
+did not exist yet on a first launch. Both lookups failed, the `if args.resume and
+os.path.exists(...)` guard was simply false, and training **started from random
+init with no warning**. The metrics file confirms it: an `init` event is present
+and the first iteration is `iter 0`, not `iter 270`.
+
+The damage was not merely a wasted run. `--eval-every` evaluates the current model
+against `<out>/init.pt`, and `init.pt` was now a **random network**. The loop
+therefore reported, at iter 9:
+
+    EVAL vs init (adj=material): score=0.850 W42/D18/L0 elo+301
+
+which reads as a large improvement and is really "a 10-iteration model beats a
+random one". Direct head-to-head against the real baseline says the opposite:
+
+| comparison | opening plies | T | score | W/D/L | Wilson CI |
+|---|---:|---:|---:|---|---|
+| v6 vs iter269 | 6 | 0.0 | 0.500 | 0/120/0 | [0.412,0.588] |
+| v6 vs iter269 | 6 | 0.3 | **0.317** | 5/66/49 | [0.240,0.404] |
+| v6 vs iter269 | 10 | 0.0 | 0.500 | 0/120/0 | [0.412,0.588] |
+| v6 vs iter269 | 10 | 0.3 | **0.325** | 3/72/45 | [0.248,0.413] |
+| exact in-loop repro (seed 99, n=60) | 2 | 0.3 | 0.325 | 1/37/22 | — |
+
+Reproducing the in-loop call exactly but with iter269 as the opponent yields
+0.325, not 0.850 — confirming the opponent, not the protocol, was the difference.
+v6 (10 iterations from scratch) is **clearly weaker** than iter269, as expected.
+
+Two lessons, both now enforced in code:
+1. **An in-loop "EVAL vs init" number is only meaningful if `init.pt` is the
+   intended reference.** It silently redefines itself on a fresh run.
+2. **A missing `--resume` target must be fatal.** `train_selfplay.py` now resolves
+   a hub-style path by downloading it, and raises
+   `FATAL: --resume ... Refusing to start from random init silently` otherwise.
+   Verified both branches: bad path -> exit 1; `main_baseline/latest.pt` ->
+   `[recover] pulled ckpt` then `[resume] iter=270`.
+
+Note the temp-0 rows: 120/120 identical draws. At temperature 0 with only 2-10
+random opening plies, two deterministic policies replay **the same game**, so a
+temp-0 arena of any size carries almost no information. Comparisons must use
+T>0 or heavily randomised openings.
+
+### `main_v7` (relaunch, correct resume)
+Genuinely continues from iter269 (`[resume] iter=270`), 1024 games/192 updates,
+`--threads 16`. Learn phase **197-220 s vs v6's 359-381 s** — the thread
+benchmark transferred to the real loop (~1.75x faster per iteration).
+Early iterations: plies ~144, decisive 0.58-0.67, entropy 1.90 -> 1.76.
+**No strength claim yet** — pending an eval against iter269 at iter 280.
+
+### `main_v7` @ iter 292 — independent verification (MEASURED, NOT SIGNIFICANT)
+
+In-loop eval vs iter269 (n=60): iter 280 **0.542** (+29 Elo), iter 290 **0.558**
+(+41 Elo). Independent arena, larger n, several conditions:
+
+| condition | score | W/D/L | Wilson CI |
+|---|---:|---|---|
+| T=0.3, open 6  | 0.506 | 13/136/11 | [0.430,0.583] |
+| T=0.3, open 10 | 0.491 | 10/137/13 | [0.414,0.567] |
+| T=0.5, open 6  | 0.591 | 41/107/12 | [0.513,0.664] |
+| **POOLED n=480** | **0.529** | — | **[0.484,0.573]** |
+
+**The pooled CI contains 0.5: `main_v7` is NOT yet measurably stronger than
+iter269.** Only the T=0.5 condition is nominally above 0.5, and selecting it
+alone would be cherry-picking across three conditions. The in-loop n=60 numbers
+(+29/+41 Elo) are inside their own noise and must not be quoted as progress.
+
+`runs/mainval269/latest.pt` and hub `main_baseline/latest.pt` remain canonical.
+Self-play trend so far is favourable (decisive 0.61 -> 0.77, entropy 1.90 -> 1.46
+over iters 270-292) but self-play statistics are not strength.
+
+## Optimisation-stability campaign v7-v12: three real bugs fixed, NO strength gain (MEASURED)
+
+All runs resume from iter269 (`main_baseline/latest.pt`), 1024 games/iter.
+
+### Bugs found and fixed (all verified in isolation)
+1. **`--resume` silently fell back to random init** for hub-style paths. `main_v6`
+   is invalid; its "+301 Elo" was against a random `init.pt`. Now FATAL. Verified.
+2. **Repetition feature destroyed by the FEN round-trip.** Fired at play time
+   (0.667), always 0.0 at train time. Now recorded per position. Verified: 83/1343
+   samples restored.
+3. **Last-move feature destroyed by the same round-trip.** Play time encodes with
+   the last move, training encoded with `None`. Now recorded. Verified: mean
+   |logp_old - logp_recomputed| on an UNCHANGED model drops **0.0222 -> 0.0080
+   (2.8x)** — that residual was pure spurious PPO ratio error.
+4. **Entropy-controller windup.** `ent_coef *= 1.5` every iteration while the
+   target was unreachable (0.01 -> 0.17) as approx_kl went 1.3 -> 59.8 and
+   grad_norm 6 -> 1918. Added `--ent-kl-ceiling` (back off when KL diverges).
+5. **`--updates 192` was the dominant divergence driver.** `main_v4` (48 updates)
+   held approx_kl at 0.01 for 75 iterations; v7/v8 (192 updates) reached 70.
+
+### Config search over the KL controls (MEASURED)
+
+| run | updates | KL controls | outcome |
+|---|---:|---|---|
+| v7  | 192 | none | KL 1.9 -> 70, grad 1918, decisive 0.77 -> 0.53 |
+| v8  | 192 | ent-kl-ceiling 6 | ent_coef pinned to min, KL still 63 -> entropy was NOT the cause |
+| v9  | 48 | ent-kl-ceiling 6 | learn 47 s/iter (8x faster), KL ~6-8, then drift to 33 |
+| v10 | 48 | + target-kl 1.0, min-updates 8 | KL **0.03-0.13, loss 0.4** for ~30 iters, then collapse (plies 144->78, ent 1.60->0.55) |
+| v11 | 48 | + kl-hard 3.0, min-updates 4 | **nup=0 on most iterations** — silent no-ops, the known failure mode |
+| v12 | 48 | kl-hard 3.0, min-updates 8 | no no-ops, but the 8 forced updates keep KL at 18-25; entropy 1.14 -> 0.94 |
+
+### Strength: flat throughout
+In-loop eval vs iter269 (n=60) across v7, v9, v10, v11, v12:
++29, +41, +17, -41, +12, -0, -12, +17, -23, +6, -35, -6, +12, -12, +17, -0 Elo.
+**Every value is inside n=60 noise and the sequence has no trend.**
+
+Independent arena, `main_v7` @292 vs iter269, n=480 pooled over three conditions:
+**0.529, Wilson CI [0.484, 0.573]** — contains 0.5.
+
+**Conclusion (MEASURED, negative): the three encoding/optimisation bugs were real
+and are fixed, but fixing them did not make the model stronger.** `main_v10`
+demonstrates that the loop *can* be held in a well-conditioned regime
+(KL ~0.05, loss ~0.4) — and in that regime the model simply does not improve; it
+holds until the entropy controller lets it collapse again. This reproduces, with
+clean optimisation, the standing result that ~100K params plateau at this level.
+
+`runs/mainval269/latest.pt` (iter 269) REMAINS THE CANONICAL BEST MODEL.
+No checkpoint from v6-v12 beats it. `main_baseline/latest.pt` was not overwritten.