plainer wording, Korean suite table

README.md

@@ -9,29 +9,36 @@ tags: [decision-model, classification, calibration, kev, system-one, lora, point
 
 # Malkuth-4B
 
-A small decision model for multilingual and Korean classification. Given text, a question and a set of criteria, it
-returns a probability distribution over the options instead of generating an answer. Post-trained from
-[Kev](https://github.com/jaredpalmer/kev) and compatible with TypeSafe's System One API.
+A small multilingual decision model for classification, with a focus on Korean.
 
-Code, training mixes and evaluation: [newfull5/malkuth](https://github.com/newfull5/malkuth).
-Companion model: [dhtocks/malkuth-2b](https://huggingface.co/dhtocks/malkuth-2b) (2B).
+Given a text, a question, and a set of criteria, Malkuth returns probabilities over the possible choices instead of
+generating text.
 
-## Model
+Malkuth-4B is a LoRA adapter trained on [Kev](https://github.com/jaredpalmer/kev) and built on a frozen
+`Qwen/Qwen3.5-4B-Base`. It has 33.8M trainable parameters and a 124 MB adapter.
 
-- Rank-16 LoRA adapter and a pointer head on a frozen `Qwen/Qwen3.5-4B-Base`; 33.8M trainable parameters, 124 MB adapter
-- `choice` (1-255 options), `noul` (yes/no) and `score` (ordered levels), several questions per request
-- Questions share the input text but cannot read each other, so one document can be classified, checked and scored at once
-- Probabilities are temperature-calibrated on a validation split; the temperature never changes which option wins
+It supports three question types:
 
-## Usage
+- `choice`: select one of up to 255 options
+- `noul`: yes or no
+- `score`: select an ordered level
+
+Multiple questions can be evaluated against the same input. Questions are evaluated independently, so they cannot see
+each other's answers. Probabilities are temperature-calibrated on a held-out validation split.
 
-Serve with Kev; the official `typesafe-sdk` works against it with a `base_url` change.
+Malkuth is served through Kev and works with the TypeSafe System One API. The companion model is
+[dhtocks/malkuth-2b](https://huggingface.co/dhtocks/malkuth-2b) (2B). Code, training mixes and evaluation live in
+[newfull5/malkuth](https://github.com/newfull5/malkuth).
+
+## Usage
 
 ```bash
 git clone https://github.com/jaredpalmer/kev.git kev && cd kev && uv sync --extra serve
 uv run --extra serve python -m kev.serve --run dhtocks/malkuth-4b --port 8009
 ```
 
+The official `typesafe-sdk` works against the local server with a `base_url` change.
+
 ```python
 from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
 
@@ -51,36 +58,54 @@ print(r.choices["team"].choice, r.choices["team"].probabilities)
 
 ## Evaluation
 
-29 suites, about 129,000 questions, the same items for every model; Jev measured through its API.
-**held-out** is the nine datasets absent from the training mix — the number to read. **all** includes ten we trained
-on. Brier scores the probabilities, not just the chosen answer; lower is better.
+29 suites and about 129,000 questions. Every model answered the same items. Jev was measured through its API.
 
-| Model | Accuracy (held-out) ↑ | Accuracy (all) ↑ | Brier (held-out) ↓ |
+- **held-out**: the nine datasets we did not train on
+- **all**: all 29 suites, ten of which we trained on
+- **Brier**: quality of the probabilities, lower is better
+
+| Model | Accuracy (held-out) | Accuracy (all) | Brier (held-out) |
 |---|---:|---:|---:|
 | Jev API | **0.754** ±0.0042 | 0.790 | **0.355** |
-| **Malkuth-4B** | 0.724 ±0.0044 | **0.794** | 0.385 |
-| **Malkuth-2B** | 0.703 ±0.0044 | 0.765 | 0.412 |
+| Malkuth-4B | 0.724 ±0.0044 | **0.794** | 0.385 |
+| Malkuth-2B | 0.703 ±0.0044 | 0.765 | 0.412 |
 | Kev-9B | 0.700 ±0.0046 | 0.737 | 0.407 |
 | Kev-4B | 0.686 ±0.0046 | 0.720 | 0.407 |
 | Kev-0.8B | 0.589 ±0.0050 | 0.584 | 0.539 |
 | Laya (best of four) | 0.544 ±0.0044 | 0.507 | 0.654 |
 
-Model time per request on one A100 80GB in bf16, median of 20, new state / prefix-cached state: 48 / 32 ms for six questions, 145 / 33 ms on a 2,200-token state.
+On the five Korean suites:
+
+| Suite | Jev | Malkuth-4B | Malkuth-2B | Kev-9B | Kev-4B |
+|---|---:|---:|---:|---:|---:|
+| `klue_nli` | **0.936** | 0.912 | 0.832 | 0.882 | 0.866 |
+| `klue_ynat` | 0.724 | **0.852** | 0.824 | 0.758 | 0.728 |
+| `nsmc` | 0.850 | **0.878** | 0.862 | 0.856 | 0.806 |
+| `kobest` | 0.903 | **0.904** | 0.819 | 0.850 | 0.809 |
+| `kold` | 0.806 | **0.820** | 0.768 | 0.602 | 0.654 |
+| mean | 0.844 | **0.873** | 0.821 | 0.790 | 0.773 |
+
+Four of these five contributed their train splits to our mix, so this table is in-distribution for Malkuth.
 
-Per-dataset scores, calibration analysis and third-party test sets:
+Latency on one A100 80GB in bf16, median of 20 requests. The second number is a repeated input served from the prefix
+cache. Six questions take 48 ms on a new input and 32 ms on a repeated one.
+
+Per-dataset scores and calibration analysis are in
 [docs/evaluation.md](https://github.com/newfull5/malkuth/blob/main/docs/evaluation.md).
 
 ## Training data
 
-89,791 labelled requests from 20 public sources — 62,400 `choice`, 24,790 `noul` and 7,401 `score` questions, across
-ten core languages with five Korean-only sources. Built by
-[`tools/build_train_mix.py`](https://github.com/newfull5/malkuth/blob/main/tools/build_train_mix.py); per-source
-licences in [docs/benchmarks.md](https://github.com/newfull5/malkuth/blob/main/docs/benchmarks.md).
+89,791 labelled requests from 20 public sources. 62,400 `choice`, 24,790 `noul` and 7,401 `score` questions, across ten
+core languages with five Korean-only sources.
+
+Built by [`tools/build_train_mix.py`](https://github.com/newfull5/malkuth/blob/main/tools/build_train_mix.py).
+Per-source licences are in
+[docs/benchmarks.md](https://github.com/newfull5/malkuth/blob/main/docs/benchmarks.md).
 
 ## License
 
-- **Weights** - the training mixes include CC-BY-NC datasets.
-- **Code** - Apache-2.0, same as Kev and the Qwen bases.
+- Weights: the training mixes include CC-BY-NC datasets.
+- Code: Apache-2.0, same as Kev and the Qwen bases.
 
 ## Citation