chorcat commited on
Commit
68b2691
·
verified ·
1 Parent(s): 3c8021b

Publish tiny-greedy

Browse files
Files changed (2) hide show
  1. README.md +41 -15
  2. config.json +1 -1
README.md CHANGED
@@ -17,26 +17,34 @@ tags:
17
  # chorcat/rukh-tiny
18
 
19
  A GPT decoder written from scratch that plays chess by predicting the next move of a game
20
- written in UCI. This is the `tiny` stage of [Rukh](https://github.com/borja-glez/rukh), a course
21
  that builds a chess language model end to end: 5,309,952 parameters, a
22
  vocabulary of 2030 fixed tokens and a context of
23
  200 moves.
24
 
25
- Play against it in the browser: [https://rukh.borjaglez.com/?stage=tiny](https://rukh.borjaglez.com/?stage=tiny) · read how it was built:
26
  [https://lab.rukh.borjaglez.com](https://lab.rukh.borjaglez.com)
27
 
28
  ## Results
29
 
30
- Measured with `rukh eval --suite quick` on 2026-09-19.
31
 
32
  | Metric | Value |
33
  |---|---|
34
  | Legality without the mask, argmax | 94.5 % |
35
- | Legality without the mask, sampled (T=0.6, top-k 20) | 93.8 % |
36
  | Top-1 next move | 40.3 % |
37
  | Top-3 next move | 67.1 % |
38
- | Puzzles solved | n/a |
39
- | Estimated Elo | 64 (95 % CI -200-292) |
 
 
 
 
 
 
 
 
40
 
41
  Legality is measured **without** the legality mask, twice, because the two numbers answer
42
  different questions:
@@ -58,23 +66,41 @@ flattering or not.
58
  | Bar | Target | Measured | Verdict |
59
  |---|---|---|---|
60
  | Legality without the mask, argmax | at least 99 % | 94.5 % | **not met** |
61
- | Estimated Elo | at least 1200 | 64 (95 % CI -200-292) | **not met** |
62
 
63
  **The legality bar is not met**: without the legality mask the weights propose an illegal move
64
  more often than one time in a hundred. The demo masks before sampling and therefore never plays
65
  one, but the bar is about the weights and the weights do not clear it.
66
 
67
- **The Elo bar is not met**, and the reason is the corpus rather than the recipe: this model was
68
- trained on 5.9 M games, against the 16 M of the reference work it is measured against, and the
69
- `medium` stage with three times the parameters and the same data bought only 84 more Elo, with
70
- overlapping intervals. The constraint is data, not capacity, so the honest fix is more months of
71
- Lichess and not more layers.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
72
 
73
  How to read these numbers:
74
 
75
- - legality_argmax is the share of validation positions where the single most likely token is a legal move (no temperature, no top-k, no mask): this is the >= 99 % bar of GOAL.md. legality_sampled draws the token the way the demo does (temperature 0.6, top-k 20) and is always the lower of the two.
76
- - puzzles not found at E:\work\ai\chess-lm\rukh\data\puzzles\puzzles.parquet: puzzle suite skipped
77
  - the Elo interval covers sampling noise only: the four ``skill-*`` rungs are nominal ``Skill Level`` anchors rather than measured ratings, and Stockfish plays at 0.1 s per move, far below any setting ``UCI_Elo`` is calibrated for
 
78
 
79
 
80
  ## Input and output
@@ -175,7 +201,7 @@ MLflow run: `0a25dd056bfe4ca9b89204a6d75686f3`.
175
  "model_type": "rukh-move-decoder",
176
  "library_name": "rukh",
177
  "rukh_version": "0.0.1",
178
- "stage": "tiny",
179
  "step": 6000,
180
  "params": 5309952,
181
  "tokenizer": "uci",
 
17
  # chorcat/rukh-tiny
18
 
19
  A GPT decoder written from scratch that plays chess by predicting the next move of a game
20
+ written in UCI. This is the `tiny-greedy` stage of [Rukh](https://github.com/borja-glez/rukh), a course
21
  that builds a chess language model end to end: 5,309,952 parameters, a
22
  vocabulary of 2030 fixed tokens and a context of
23
  200 moves.
24
 
25
+ Play against it in the browser: [https://rukh.borjaglez.com/?stage=tiny-int8](https://rukh.borjaglez.com/?stage=tiny-int8) · read how it was built:
26
  [https://lab.rukh.borjaglez.com](https://lab.rukh.borjaglez.com)
27
 
28
  ## Results
29
 
30
+ Measured with `rukh eval --suite full` on 2026-09-20.
31
 
32
  | Metric | Value |
33
  |---|---|
34
  | Legality without the mask, argmax | 94.5 % |
35
+ | Legality without the mask, sampled (T=0.05, top-k 1) | 94.5 % |
36
  | Top-1 next move | 40.3 % |
37
  | Top-3 next move | 67.1 % |
38
+ | Puzzles solved | 8.9 % |
39
+ | Estimated Elo | 921 (95 % CI 713-1040) |
40
+
41
+ Puzzles by difficulty band:
42
+
43
+ | Band | Solved |
44
+ |---|---|
45
+ | 1000-1500 | 14.1 % |
46
+ | 1500-2000 | 8.6 % |
47
+ | 2000+ | 3.9 % |
48
 
49
  Legality is measured **without** the legality mask, twice, because the two numbers answer
50
  different questions:
 
66
  | Bar | Target | Measured | Verdict |
67
  |---|---|---|---|
68
  | Legality without the mask, argmax | at least 99 % | 94.5 % | **not met** |
69
+ | Estimated Elo | at least 1200 | 921 (95 % CI 713-1040) | **not met** |
70
 
71
  **The legality bar is not met**: without the legality mask the weights propose an illegal move
72
  more often than one time in a hundred. The demo masks before sampling and therefore never plays
73
  one, but the bar is about the weights and the weights do not clear it.
74
 
75
+ **The Elo bar is not met.** This stage exists as a baseline, not as a release: it is the smallest
76
+ model in the project and it is published so the larger stages have something to be compared
77
+ against. The stages that carry the project's answer to this bar are `rukh-medium` (1504) and
78
+ `rukh-medium-dpo` (1529), trained on 19.0 M games where this one saw 2.9 M.
79
+
80
+ A note on the number itself: an earlier release of these cards reported far lower ratings —
81
+ 1007 for `small`, 64 for this stage — and gave the bar as missed across the board. Those figures
82
+ were wrong. Four of the eight Stockfish rungs the rating is fitted against carried Elo labels
83
+ written by hand, and playing the ladder against itself showed they were 428 to 581 points too
84
+ low. The ladder is now measured, every stage was re-evaluated on it, and the numbers above are
85
+ from those runs.
86
+
87
+ ### An Elo belongs to the pair model+sampling
88
+
89
+ The same weights (`4591cf8cca0b`) measure **921 Elo** at
90
+ temperature 0.05, top-k 1 and **64 Elo** at temperature 0.6, top-k 20.
91
+ Nothing about the tensors changed between the two runs; only the way a move is drawn from them
92
+ did. A rating is therefore a property of the pair model+sampling and not of the file you
93
+ download, and any table that compares stages has to fix the sampling first.
94
+
95
+ The results above are the temperature 0.05, top-k 1 point. The demo samples for variety
96
+ rather than always playing its best move, so what a player meets over the board is the weaker of
97
+ the two numbers.
98
 
99
  How to read these numbers:
100
 
101
+ - legality_argmax is the share of validation positions where the single most likely token is a legal move (no temperature, no top-k, no mask): this is the >= 99 % bar of GOAL.md. legality_sampled draws the token the way the demo does (temperature 0.05, top-k 1) and is always the lower of the two.
 
102
  - the Elo interval covers sampling noise only: the four ``skill-*`` rungs are nominal ``Skill Level`` anchors rather than measured ratings, and Stockfish plays at 0.1 s per move, far below any setting ``UCI_Elo`` is calibrated for
103
+ - 3 of 160 games hit the context limit and were adjudicated (3 of them) instead of being scored as draws
104
 
105
 
106
  ## Input and output
 
201
  "model_type": "rukh-move-decoder",
202
  "library_name": "rukh",
203
  "rukh_version": "0.0.1",
204
+ "stage": "tiny-greedy",
205
  "step": 6000,
206
  "params": 5309952,
207
  "tokenizer": "uci",
config.json CHANGED
@@ -5,7 +5,7 @@
5
  "model_type": "rukh-move-decoder",
6
  "library_name": "rukh",
7
  "rukh_version": "0.0.1",
8
- "stage": "tiny",
9
  "step": 6000,
10
  "params": 5309952,
11
  "tokenizer": "uci",
 
5
  "model_type": "rukh-move-decoder",
6
  "library_name": "rukh",
7
  "rukh_version": "0.0.1",
8
+ "stage": "tiny-greedy",
9
  "step": 6000,
10
  "params": 5309952,
11
  "tokenizer": "uci",