chorcat commited on
Commit
2ea8567
·
verified ·
1 Parent(s): cc5ccda

docs(card): regenerated from the current table and course

Browse files
Files changed (1) hide show
  1. README.md +17 -48
README.md CHANGED
@@ -27,7 +27,7 @@ Play against it in the browser: [https://rukh.borjaglez.com/?stage=tiny-int8](ht
27
 
28
  ## Results
29
 
30
- Measured with `rukh eval --suite full` on 2026-09-20.
31
 
32
  | Metric | Value |
33
  |---|---|
@@ -36,7 +36,7 @@ Measured with `rukh eval --suite full` on 2026-09-20.
36
  | Top-1 next move | 40.3 % |
37
  | Top-3 next move | 67.1 % |
38
  | Puzzles solved | 8.9 % |
39
- | Estimated Elo | 921 (95 % CI 713-1040) |
40
 
41
  Puzzles by difficulty band:
42
 
@@ -66,7 +66,7 @@ flattering or not.
66
  | Bar | Target | Measured | Verdict |
67
  |---|---|---|---|
68
  | Legality without the mask, argmax | at least 99 % | 94.5 % | **not met** |
69
- | Estimated Elo | at least 1200 | 921 (95 % CI 713-1040) | **not met** |
70
 
71
  **The legality bar is not met**: without the legality mask the weights propose an illegal move
72
  more often than one time in a hundred. The demo masks before sampling and therefore never plays
@@ -98,23 +98,11 @@ Every stage was then re-evaluated on the measured ladder, and the numbers on thi
98
  those runs. The correction moves all stages by a similar amount, so comparisons between them are
99
  unchanged.
100
 
101
- ### An Elo belongs to the pair model+sampling
102
-
103
- The same weights (`4591cf8cca0b`) measure **921 Elo** at
104
- temperature 0.05, top-k 1 and **64 Elo** at temperature 0.6, top-k 20.
105
- Nothing about the tensors changed between the two runs; only the way a move is drawn from them
106
- did. A rating is therefore a property of the pair model+sampling and not of the file you
107
- download, and any table that compares stages has to fix the sampling first.
108
-
109
- The results above are the temperature 0.05, top-k 1 point. The demo samples for variety
110
- rather than always playing its best move, so what a player meets over the board is the weaker of
111
- the two numbers.
112
-
113
  How to read these numbers:
114
 
115
  - legality_argmax is the share of validation positions where the single most likely token is a legal move (no temperature, no top-k, no mask): this is the >= 99 % bar of GOAL.md. legality_sampled draws the token the way the demo does (temperature 0.05, top-k 1) and is always the lower of the two.
116
  - the Elo interval covers sampling noise only: the four ``skill-*`` rungs are nominal ``Skill Level`` anchors rather than measured ratings, and Stockfish plays at 0.1 s per move, far below any setting ``UCI_Elo`` is calibrated for
117
- - 3 of 160 games hit the context limit and were adjudicated (3 of them) instead of being scored as draws
118
 
119
 
120
  ## Input and output
@@ -174,38 +162,10 @@ weights are the same, the arithmetic is not.
174
  ## Training recipe
175
 
176
 
177
- | Parameter | Value |
178
- |---|---|
179
- | `batch_size` | `128` |
180
- | `betas` | `[0.9, 0.95]` |
181
- | `block` | `200` |
182
- | `ckpt_every` | `1000` |
183
- | `compile` | `True` |
184
- | `data_manifest_sha` | `2274906c04342b8fc1632d3e1c3bff82e33f535936af742c7a7f9e4b052afa31` |
185
- | `device` | `cuda` |
186
- | `eval_batches` | `50` |
187
- | `eval_every` | `250` |
188
- | `grad_accum` | `2` |
189
- | `grad_clip` | `1.0` |
190
- | `log_every` | `10` |
191
- | `lr` | `0.001` |
192
- | `max_steps` | `6000` |
193
- | `min_lr_ratio` | `0.1` |
194
- | `model` | `None` |
195
- | `num_params` | `5258752` |
196
- | `out_dir` | `checkpoints` |
197
- | `precision` | `bf16` |
198
- | `preset` | `tiny` |
199
- | `run_name` | `tiny` |
200
- | `seed` | `42` |
201
- | `tokens_dir` | `data/tokens/uci` |
202
- | `unique_run_name` | `True` |
203
- | `vocab_hash` | `527c5dda224cab570cb84da8f7dcda0f43fe53edaf86822f899568f5dd824b46` |
204
- | `warmup` | `500` |
205
- | `weight_decay` | `0.1` |
206
- | `workers` | `4` |
207
-
208
- MLflow run: `0a25dd056bfe4ca9b89204a6d75686f3`.
209
 
210
  ```json
211
  {
@@ -257,6 +217,15 @@ converted from SAN to legal UCI. Validation uses a month the model never saw.
257
  anchors rather than measured ratings, and games that run out of context are adjudicated on the
258
  final position instead of being scored as draws.
259
 
 
 
 
 
 
 
 
 
 
260
  ## License
261
 
262
  APACHE-2.0. The code and the weights are released under the Apache License 2.0;
 
27
 
28
  ## Results
29
 
30
+ Measured with `rukh eval --suite full` on 2026-09-21.
31
 
32
  | Metric | Value |
33
  |---|---|
 
36
  | Top-1 next move | 40.3 % |
37
  | Top-3 next move | 67.1 % |
38
  | Puzzles solved | 8.9 % |
39
+ | Estimated Elo | 778 (95 % CI 479-904) |
40
 
41
  Puzzles by difficulty band:
42
 
 
66
  | Bar | Target | Measured | Verdict |
67
  |---|---|---|---|
68
  | Legality without the mask, argmax | at least 99 % | 94.5 % | **not met** |
69
+ | Estimated Elo | at least 1200 | 778 (95 % CI 479-904) | **not met** |
70
 
71
  **The legality bar is not met**: without the legality mask the weights propose an illegal move
72
  more often than one time in a hundred. The demo masks before sampling and therefore never plays
 
98
  those runs. The correction moves all stages by a similar amount, so comparisons between them are
99
  unchanged.
100
 
 
 
 
 
 
 
 
 
 
 
 
 
101
  How to read these numbers:
102
 
103
  - legality_argmax is the share of validation positions where the single most likely token is a legal move (no temperature, no top-k, no mask): this is the >= 99 % bar of GOAL.md. legality_sampled draws the token the way the demo does (temperature 0.05, top-k 1) and is always the lower of the two.
104
  - the Elo interval covers sampling noise only: the four ``skill-*`` rungs are nominal ``Skill Level`` anchors rather than measured ratings, and Stockfish plays at 0.1 s per move, far below any setting ``UCI_Elo`` is calibrated for
105
+ - 1 of 160 games hit the context limit and were adjudicated (1 of them) instead of being scored as draws
106
 
107
 
108
  ## Input and output
 
162
  ## Training recipe
163
 
164
 
165
+ The MLflow run for this checkpoint was not available when the card was generated; the shape of
166
+ the model is in `config.json`.
167
+
168
+
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
169
 
170
  ```json
171
  {
 
217
  anchors rather than measured ratings, and games that run out of context are adjudicated on the
218
  final position instead of being scored as draws.
219
 
220
+ ## In the course
221
+
222
+ - Built in [M2 · El decoder: entrenarlo y medirlo](https://lab.rukh.borjaglez.com/curso/m2/02-entrenar-y-medir/), the lesson that runs every command behind this repository.
223
+ - Measured in the [single results table](https://lab.rukh.borjaglez.com/proyecto/) as the stage `tiny-greedy`: every stage of the course, the same suite, the same day.
224
+ - Play it in the browser: [https://rukh.borjaglez.com/?stage=tiny-int8](https://rukh.borjaglez.com/?stage=tiny-int8).
225
+ - Bring it to the paths the configs read: `uv run rukh pull tiny`.
226
+
227
+
228
+
229
  ## License
230
 
231
  APACHE-2.0. The code and the weights are released under the Apache License 2.0;