| # Progress report β 2026-07-10 / 2026-07-11 |
|
|
| Session goal: push the latest **200M compact-vocab** ChessTransformer toward high Elo |
| **without MCTS** (pure next-move / policy argmax), while growing better soft-target data. |
|
|
| ## Summary |
|
|
| | Item | Result | |
| |------|--------| |
| | Base model | `avewright/chess-transformer-200m-compact-soft` | |
| | Finetune run | **exp189** β soft MultiPV + hard deep labels + h-flip | |
| | Deep data harvest | **exp190** β phase-balanced SF18 MultiPV (ongoing) | |
| | Checkpoint pushed | `latest` step **2851**, `best` by holdout metric at step **500** | |
| | Pure-policy Elo (vs SF18 limited) | **~1700** (bracket 1500β1800) | |
| | Training status | Stopped on request after plateau | |
|
|
| ## What we shipped (code) |
|
|
| | Path | Role | |
| |------|------| |
| | `experiments/exp189_200m_maxelo_policy.py` | Max-Elo policy finetune; mixes shallow soft + deep soft + hard HF | |
| | `experiments/exp190_phase_deep_harvest.py` | Phase-quota MultiPV harvest (SF18, Syzygy, multi-source FENs) | |
| | `scripts/run_exp189_200m_maxelo.sh` | Initial exp189 launch | |
| | `scripts/run_exp189_deep_mix.sh` | Resume with `--deep-soft-cache` mix | |
| | `scripts/run_exp190_phase_deep.sh` | Harvest launcher (40 workers, SF18 vnni512) | |
| | `play_factory_gui.py` | Fix: Chess960 castling UCI (`e8a8`) β standard (`e8c8`) for chess.js | |
|
|
| ## Training recipe (exp189) |
|
|
| Resumed from compact-soft, then continued with deep-mix: |
|
|
| - Soft frac **0.72**, soft Ξ± **0.40**, deep-mix frac **0.40** among soft steps |
| - Batch **448 Γ 2** accum, LR **4e-6** cosine, warmup 200 |
| - H-flip aug **p=0.5** on soft batches |
| - Hard ballast: HF lichess-sf stream, `min_depthβ₯15` |
| - Eval: shallow holdout + deep holdout; `best.pt` tracks blended top-1 |
|
|
| ### Observed learning |
|
|
| Deep-mix run plateaued around: |
|
|
| - Shallow soft holdout top-1 β **38%** |
| - Deep soft holdout top-1 β **41%** |
| - Blended best metric **0.3966** recorded at step **500** of the deep-mix resume |
| - Shutdown save: `latest.pt` at step **2851** |
|
|
| Losses stayed flat (~soft 2.38, hard CE ~1.79) β model had largely extracted the |
| signal from the then-available mix (~2M shallow + ~96β175k deep). |
|
|
| ## Soft targets β why they matter |
|
|
| Hard labels = one Stockfish best move. Soft MultiPV labels = a **distribution** over |
| plausible moves (score β softmax). That teaches: |
|
|
| 1. Near-equal alternatives (not overconfident on one line) |
| 2. Better calibration under legal-mask argmax (no search) |
| 3. Deeper MultiPV (exp190) = sharper teacher in middlegame/endgame |
|
|
| exp186 already provided ~2M shallow soft rows (`avewright/exp186-sf-multipv-2m`). |
| exp190 adds **phase-balanced deeper** labels so training does not drown in openings. |
|
|
| ## exp190 harvest design |
|
|
| Targets: **22% opening / 48% middlegame / 30% endgame** |
|
|
| | Lever | Design | |
| |-------|--------| |
| | Depth-by-phase | opening 10β14, mid 12β16, endgame 14β18 | |
| | Teacher | Stockfish **18** full strength (`stockfish/stockfish-latest` vnni512) | |
| | Sources | HF streams + book playouts + endgame templates + random walks | |
| | Quotas | Deficit-first producer + hard gate at writer | |
| | Tags | `phase`, `label_depth` in `soft_cache.pt` for stratified training | |
| | Syzygy | Workers probe tablebases when present | |
|
|
| At upload time harvest was ~**187k** written positions (target 1M, still running), |
| phase mix roughly on quota. See `outputs/exp190_phase_deep/ARCHITECTURE.md`. |
|
|
| ## Elo evaluation |
|
|
| Script: `elo_eval_latest.py` |
| Checkpoint: `outputs/exp189_200m_maxelo_policy/latest.pt` |
| Opponent: SF18 `UCI_LimitStrength`, 50ms/move, 8 openings Γ both colors, Syzygy + book. |
|
|
| | SF Elo | Score | WβDβL | |
| |--------|-------|-------| |
| | 1500 | 0.625 | 7β6β3 | |
| | 1800 | 0.438 | 4β6β6 | |
|
|
| **Estimated Elo β 1700** (noisy; Β±~150 with this sample). |
| Weakness: **Black** side (more losses / short mates). No MCTS at inference. |
|
|
| ## Hugging Face artifacts |
|
|
| | Repo | Type | Contents | |
| |------|------|----------| |
| | `avewright/chess-transformer-200m-maxelo` | model | `best_model.pt`, `latest_model.pt`, config, Elo + progress docs | |
| | `avewright/exp190-phase-deep-soft` | dataset | `soft_cache.pt`, jsonl shards, status, architecture notes | |
|
|
| Base still: `avewright/chess-transformer-200m-compact-soft` |
| Shallow soft still: `avewright/exp186-sf-multipv-2m` |
|
|
| ## How to load for inference |
|
|
| ```bash |
| export MOVE_VOCAB_VERSION=compact |
| python play_factory_gui.py --checkpoint path/to/latest_model.pt |
| # or |
| python elo_eval_latest.py path/to/latest_model.pt |
| ``` |
|
|
| Policy-only: legal-mask argmax on `policy_logits` (see `elo_eval_latest.get_model_move_generic`). |
|
|
| ## Next levers (not done this session) |
|
|
| 1. Grow exp190 to 1M+ and retrain with stratified phase sampling + depth weights |
| 2. Longer Elo gauntlet (more games / level) and evaluate `best` vs `latest` |
| 3. AlphaZero-style expert iteration (`exp183_selfplay.py` / `rl_selfplay/`) after supervised plateau |
| 4. Fix Black-side weakness (phase-balanced Black STM oversampling, or STM-normalized training audit) |
|
|
| ## Ops notes |
|
|
| - Training stopped cleanly: `Saved on shutdown at step 2851` |
| - Harvest left running on CPU (40 workers) so it does not starve a future GPU train |
| - `.env` / tokens are gitignored β never commit HF or GitHub secrets |
|
|
