| --- |
| license: apache-2.0 |
| library_name: rukh |
| base_model: chorcat/rukh-medium |
| tags: |
| - chess |
| - rukh |
| - lora |
| - adapter |
| --- |
| |
| # chorcat/rukh-lora-e4 |
|
|
| A LoRA adapter for [`chorcat/rukh-medium`](https://huggingface.co/chorcat/rukh-medium). On its own it does |
| nothing: it is 393,216 numbers (1.6 MB) that |
| correct that model's weights, and without them there is no model to correct. |
|
|
| Written by hand rather than imported, as part of [Rukh](https://github.com/borja-glez/rukh), a course that builds |
| a chess language model end to end. How it works and why: [https://lab.rukh.borjaglez.com](https://lab.rukh.borjaglez.com). |
|
|
| ## What it is |
|
|
| | | | |
| |---|---| |
| | Rank `r` | 8 | |
| | Scaling `alpha` | 16 (16/8 applied to `B·A`) | |
| | Adapted matrices | `q, v`, in every block | |
| | Trainable parameters | 393,216 | |
| | File size | 1.6 MB | |
|
|
| The base weights never move. Everything outside `A` and `B` is frozen during training, which is |
| what makes the file this small and what limits what it can learn: the correction has to pass |
| through 8 dimensions per matrix. |
|
|
| ## What it changed |
|
|
| The model's own probability for `e2e4` from the opening position, read off the |
| softmax over the twenty legal first moves with no sampling -- so the number is a property of the |
| weights and two readings agree to the last decimal: |
|
|
| | | P(`e2e4`) | first-move entropy | |
| |---|---:|---:| |
| | `chorcat/rukh-medium` | 59.64 % | 1.7695 bits | |
| | with this adapter | 99.85 % | 0.0209 bits | |
|
|
| And what the style cost, on the same suite every other stage of the project is measured with -- |
| same validation positions, same puzzles, same seed: |
|
|
| | | `chorcat/rukh-medium` | with this adapter | |
| |---|---:|---:| |
| | Legality without the mask, argmax | 99.8 % | 99.8 % | |
| | Top-1 next move | 54.4 % | 54.7 % | |
| | Puzzles solved | 37.5 % | 37.8 % | |
|
|
|
|
| ## How it was trained |
|
|
| 200,000 games selected with `split_part(uci, ' ', 1) = 'e2e4' AND white_elo >= 1800 AND black_elo >= 1800`, from the |
| 1800+ Lichess corpus of the project. |
|
|
| ## Using it |
|
|
| ```python |
| from rukh.models import MoveDecoder |
| from rukh.models.lora import load_adapter |
| from rukh.train import load_model |
| |
| model, _ = load_model("<the base checkpoint>") |
| load_adapter(model, "adapter.safetensors") # the config travels next to it |
| ``` |
|
|
| `rukh.models.lora.merge_lora(model)` folds it into the weights, after which the model is an |
| ordinary `MoveDecoder`: same module names, same state dict, exportable to ONNX like any other. |
|
|
| ### In the browser |
|
|
| `web/adapter.bin` is the same factors again as one flat little-endian `float32` buffer |
| (1.6 MB): `A` first, then `B`, both stacked over the blocks in the |
| order they run, with the `alpha/r` scaling already applied. `web/adapter.json` carries the shapes, |
| because a buffer of floats says nothing about itself. |
|
|
| It is meant for the ONNX export of `chorcat/rukh-medium` that takes its LoRA factors as **inputs of the |
| graph** (`lora_a` and `lora_b`) rather than baked into the weights. Fed an adapter of zeros that |
| file is the base model exactly; fed this one it is this style. So the demo changes style by |
| downloading 1.6 MB instead of a second copy of the model, which is |
| the only reason a low-rank correction is worth keeping low-rank once it leaves the training loop. |
|
|
| ## In the course |
|
|
| - Built in [M4 · Fine-tuning: los labs](https://lab.rukh.borjaglez.com/curso/m4/03-labs-de-afinado/), the lesson that runs every command behind this repository. |
| - Measured in the [single results table](https://lab.rukh.borjaglez.com/proyecto/) as the stage `lora-e4`: every stage of the course, the same suite, the same day. |
| - Bring it to the paths the configs read: `uv run rukh pull lora-e4`. |
|
|
|
|
|
|
| ## Licence |
|
|
| apache-2.0, the same as the base model. |
|
|
