Model card: chance baselines and DI skill per Knowledge & Reasoning benchmark; HLE stays at chance; ChessBench truncation

README.md

@@ -92,22 +92,31 @@ the always-think gain while thinking on 48 % of the questions. Thinking length:
 The area's benchmarks run through the server (`POST /v1/decide`, `thinking: "auto"`, threshold 0.8, budget 8,192 thinking
 tokens), scored with the Decision Index kit. Eight of the ten benchmarks were run to completion:
 
-| benchmark | metric | System 1 | **adaptive** | questions that thought |
-|---|---|---:|---:|---:|
-| GPQA Diamond | accuracy | 42.9 | **78.6** | 87 % |
-| CRUXEval | accuracy | 67.5 | **90.7** | 42 % |
-| CLadder | accuracy | 71.0 | **86.6** | 51 % |
-| HLE | accuracy | 8.4 | **17.8** | 88 % |
-| GSM8K | accuracy | 97.6 | **99.1** | 5 % |
-| SATA-Bench | case exact accuracy | 34.2 | **35.5** | 21 % |
-| MuSR | accuracy | 67.3 | **67.6** | 51 % |
-| ChessBench | accuracy | 23.7 | 23.6 | 88 % |
-| MMLU-Pro (first 1,611 of 12,032 questions) | accuracy | 59.2 | **80.1** | 68 % |
-| BBH | accuracy | 75.2 | not run | — |
-
-Scores in %. The Knowledge & Reasoning area skill rises from 0.430 to **0.551** with eight of ten benchmarks adaptive
-(MMLU-Pro and BBH counted with System 1). Thinking helps most on science, code and causal questions; chess does not
-benefit.
+| benchmark | chance | System 1 | **adaptive** | DI skill, System 1 → adaptive | questions that thought |
+|---|---:|---:|---:|---:|---:|
+| GPQA Diamond | 25.0 | 42.9 | **78.6** | 0.238 → **0.714** | 87 % |
+| CRUXEval | 37.0 | 67.5 | **90.7** | 0.485 → **0.853** | 42 % |
+| CLadder | 50.0 | 71.0 | **86.6** | 0.420 → **0.732** | 51 % |
+| GSM8K | 25.0 | 97.6 | **99.1** | 0.969 → 0.989 | 5 % |
+| SATA-Bench (case exact) | 1.3 | 34.2 | 35.5 | 0.334 → 0.346 | 21 % |
+| MuSR | 37.1 | 67.3 | 67.6 | 0.480 → 0.484 | 51 % |
+| ChessBench | 8.2 | 23.7 | 23.6 | 0.169 → 0.168 | 88 % |
+| HLE | 16.4 | 8.4 | 17.8 | 0.000 → 0.016 | 88 % |
+| MMLU-Pro (first 1,611 of 12,032 questions) | 11.1 | 59.2 | **80.1** | counted with System 1 | 68 % |
+| BBH | 31.0 | 75.2 | not run | counted with System 1 | — |
+
+Accuracy in %; "chance" is the kit's random baseline, and the Decision Index skill rescales accuracy so that chance is
+0 (below chance counts as 0). The Knowledge & Reasoning area skill rises from 0.430 to **0.551** with eight of ten
+benchmarks adaptive (MMLU-Pro and BBH counted with System 1).
+
+* **Real gains** on science, code and causal reasoning: GPQA Diamond, CRUXEval and CLadder.
+* **HLE only returns to chance.** System 1 is below chance (8.4 % against 16.4 %), and the questions where it is confident
+  enough not to think are almost all wrong (1.6 %). Thinking brings HLE to 17.8 %, the level of guessing, so its skill
+  stays near 0.
+* **Chess does not benefit.** Two thirds of the ChessBench thoughts hit the 8,192-token budget; the answer read from a
+  truncated thought is no better than System 1 (17.0 % against 17.7 % on those questions), while finished thoughts gain
+  a little (19.8 % → 22.7 %). Answers read from truncated thoughts are also over-confident, so the equal mix does not
+  damp them.
 
 ### Latency
 
@@ -321,8 +330,9 @@ were chosen on data outside the Decision Index suite.
 
 ## Limitations
 
-* Where the teacher is wrong, System 1 often is too; adaptive thinking helps most on math, logic and multi-step
-  questions, and can slightly lower accuracy on commonsense questions (CommonsenseQA 86.7 → 85.0).
+* Where the teacher is wrong, System 1 often is too, and it can be confidently wrong: on HLE the questions it answers
+  without thinking are 1.6 % correct. Adaptive thinking helps most on science, code, math and logic, does not help on
+  chess or expert-level HLE questions, and can slightly lower accuracy on commonsense questions (CommonsenseQA 86.7 → 85.0).
 * Thinking is slow on hard inputs: on expert-level and chess questions it often uses the full 8,192-token budget, and
   the adaptive mode does not meet the Decision Index latency limit on the Knowledge & Reasoning benchmarks.
 * The Decision Index score above combines adaptive runs of eight Knowledge & Reasoning benchmarks with System 1 runs of