decider-4b v2.1

decider-4b v1 + merged LoRA (rank 64, attention and MLP) with replay trained toward v1's distribution; temperature 1.099, temperature_by_type choice 1.110 / noul 1.560 / score 1.287 (decider-ai 1.4.0). v2 is under the tag v2, v1 under v1.

README.md

@@ -17,93 +17,126 @@ Base model: [Qwen/Qwen3.5-4B-Base](https://huggingface.co/Qwen/Qwen3.5-4B-Base):
 attention and 24 with gated delta-net linear attention, hidden size 2,560. Two supervised stages. Stage 1 (decider-4b v1): one
 pass of cross-entropy on the slot readout over mixture v2, the public decision mixture of
 [decider-2b](https://huggingface.co/Mapika/decider-2b) plus 26 further public decision datasets and ten programmatically
-generated families with verifiable gold (742M tokens). Stage 2 (v2): a LoRA of rank 64 on the attention and MLP weights,
-trained for 2 epochs on 29,356 rows of harder decisions and replay, then merged into the weights. There is no
-reinforcement-learning stage. **This repository holds v2**, the bf16 weights (8.4 GB). v1 stays available under the Hub tag
-`v1` (see [Changes from v1](#changes-from-v1) for who should keep using it). The other sizes are listed under The decider
-family. `decider/` in this repository is the inference subset of the GitHub package.
-
-Against decider-2b v10 on the same rows: accuracy is higher on 66 of the 95 regression tasks (in-task 0.824 against 0.805,
-held-out 0.779 against 0.755), +1.8 points on the 847 validation rows, +3.5 on OpenJev, +4.8 on Mind2Web, +5.9 on the TypeSafe
-workflow rows (interval includes zero), JevBench hard tier 0.676 against 0.459, Bespoke's public suite 0.773 against 0.704 macro.
-Against decider-35b-a3b it is 3.1 to 3.2 points lower on the regression set, 1.6 to 5.0 points lower on three fixtures, level on
-TypeSafe and level on the JevBench hard tier (0.676 both), at 8.4 GB instead of 65 GB. On live browser tasks it plays at the level
-of the 2B in greedy mode (92.6% against 90.9%) and below it on the six held-out tasks (81.2% against 91.7%), because it has no RL
-stage. Details under Evaluation.
-
-**Contents:** [Changes from v1](#changes-from-v1) · [The decider family](#the-decider-family) · [Usage](#usage) · [How it works](#how-it-works) · [Training](#training) · [Evaluation](#evaluation) · [Calibration](#calibration) · [Speed](#speed) · [Limitations](#limitations) · [Changelog](#changelog) · [Reproduction](#reproduction)
-
-## Changes from v1
-
-v2 is v1 plus the stage-2 LoRA described under Training. It is better than v1 on hard and document-based judgments and on
-TypeSafe, OpenJev and Bespoke's suite, and worse on sampled game and browser play, on some text games and by about 1 point on
-everyday tasks. All rows below are on identical inputs, v1 at its stored temperature 1.05 and v2 at 1.935, through decider-ai
-1.2.1; intervals are 95% paired bootstrap intervals of v2 minus v1 (tasks, rows, items, boards or task-seed pairs).
-
-| set | v1 | v2 | v2 minus v1 |
-|---|---|---|---|
-| regression set, 67 in-task tasks, accuracy / NLL / ECE | 0.834 / 0.404 / 0.027 | 0.824 / 0.441 / 0.041 | accuracy −1.0 (−1.2 to −0.7), NLL +0.037 (+0.029 to +0.046), ECE +0.014 (+0.008 to +0.021) |
-| regression set, 28 held-out tasks | 0.788 / 0.558 / 0.071 | 0.779 / 0.566 / 0.080 | accuracy −0.9 (−2.0 to +0.1), NLL +0.008 (−0.024 to +0.035), ECE +0.009 (−0.003 to +0.021) |
-| 847 in-task validation rows, accuracy / NLL | 86.1% / 0.417 | 85.0% / 0.419 | −1.1 (−2.6 to +0.5); NLL +0.002 (−0.019 to +0.021) |
-| OpenJev, 5,252 rows | 63.9% / 0.893 | 66.7% / 0.789 | +2.8 (+1.8 to +3.9); NLL −0.104 (−0.121 to −0.087) |
-| Mind2Web, 1,770 rows | 88.4% / 0.366 | 87.4% / 0.393 | −1.0 (−2.0 to +0.1); NLL +0.027 (+0.011 to +0.044) |
-| TypeSafe workflow decisions, 102 rows | 81.4% / 0.611 | 86.3% / 0.410 | +4.9 (−1.0 to +10.8); NLL −0.201 (−0.348 to −0.072) |
-| Bespoke's public suite, macro / micro | 0.757 / 0.765 | 0.773 / 0.781 | macro +1.6 |
-| JevBench public items, easy / standard / hard | 1.000 / 0.958 / 0.550 | 1.000 / 0.986 / 0.676 | hard +12.6 (+4.5 to +20.7) |
-| JevBench hard tier, top-label ECE (v2: recomputed at T 1.935 from stored probabilities, the 6 Score items at T 1.719) | 0.288 | 0.071 | |
-| Decision Index 4,000-request sample, calibration error (site definition; v2 recomputed at T 1.935 from stored probabilities) | 0.086 | 0.074 | |
-| live MiniWoB++, 22 tasks x 8 seeds, greedy (all / 6 held-out tasks) | 91.5% / 75.0% | 92.6% / 81.2% | +1.1 (−1.7 to +4.0) / +6.2 (−2.1 to +16.7) |
-| live MiniWoB++, sampled (all / 16 rewarded / 6 held-out) | 90.9% / 96.1% / 77.1% | 88.1% / 93.0% / 75.0% | −2.8 (−7.4 to +1.7) / −3.1 (−7.8 to +0.8) / −2.1 (−14.6 to +10.4) |
-| zero-shot games, win rate, sampled (234 boards) | 27.8% | 22.4% | −5.3 (−7.6 to −3.1) |
-| zero-shot games, greedy | 29.1% | 27.8% | −1.3 (−5.6 to +2.6) |
-| bag-draw games alone, sampled / greedy | 56.6% / 62.5% | 37.9% / 48.4% | −18.8 (−24.6 to −12.9) / −14.1 (−23.4 to −6.2) |
-| ten text games, greedy: CliffWalking / BabyAI-GoTo / Breakout | −13 / 0.54 / 14 | −60 / 0.35 / 12 | |
-| behaviour probes: model-router tier / needs-live-data / touches-outside-project | 0.968 / 0.871 / 0.956 | 0.935 / 0.839 / 0.933 | |
-| behaviour probes: abstention battery / catch-all / command risk / browser element and action | 7 of 8 / 0.90 / 0.889 / 0.875 and 0.875 | 8 of 8 / 0.95 / 0.911 / 0.938 and 0.938 | |
-
-The v1 column is v1 measured again on 2026-09-24 through decider-ai 1.2.1, so that both versions are read by the same code on
-the same day. It differs from the numbers first published for v1 by at most 1.0 point on the regression set, the fixtures and
-JevBench, by up to 2.1 points on the live browser and game rows (sampled held-out browser tasks 77.1% against 79.2%, bag-draw
-greedy 62.5% against 60.9%), and on Breakout (14 against 18). The first-published values are in the v1 card under the tag `v1`.
+generated families with verifiable gold (742M tokens). Stage 2 (v2.1): a LoRA of rank 64 on the attention and MLP weights,
+trained for 2 epochs on 29,325 rows of harder decisions and replay, with the replay rows trained toward v1's own answer
+distribution, then merged into the weights. There is no reinforcement-learning stage. **This repository holds v2.1**, the bf16
+weights (8.4 GB), with one temperature per answer type. v2 stays available under the Hub tag `v2` and v1 under the tag `v1`
+(see [Changes from v2](#changes-from-v2) for who should keep using them). The other sizes are listed under The decider family.
+`decider/` in this repository is the inference subset of the GitHub package.
+
+v2.1 keeps most of v2's gain on hard decisions and gets back most of what v2 lost in sampled play. Against v2 on the same rows:
+bag-draw games in sampled play 52.0% against 37.9% wins (v1 56.6%), zero-shot games sampled 26.9% against 22.4% (v1 27.8%), live
+browser tasks sampled 93.2% against 88.1% (v1 90.9%), CliffWalking −13 against −60, the regression set 0.831 / 0.784 against 0.824
+/ 0.779 (v1 0.834 / 0.788); held-out generated decision families 0.556 against 0.560 (v1 0.469) and the JevBench public hard tier
+0.649 against 0.676 (v1 0.550). It is less well calibrated than v2 on hard items (details under
+[Calibration](#calibration)), still answers one of the two form-filling cases of issue #9 wrongly, and is worse than v1 on
+BabyAI-GoTo and on greedy bag-draw play. Against decider-35b-a3b it is 2.4 and 2.6 points lower on the regression set.
+
+**Contents:** [Changes from v2](#changes-from-v2) · [The decider family](#the-decider-family) · [Usage](#usage) · [How it works](#how-it-works) · [Training](#training) · [Evaluation](#evaluation) · [Calibration](#calibration) · [Speed](#speed) · [Limitations](#limitations) · [Changelog](#changelog) · [Reproduction](#reproduction)
+
+## Changes from v2
+
+v2.1 differs from v2 in two things:
+
+* **Replay rows are trained toward v1's own distribution.** Stage 2 of v2 trained every row, including 6,700 replay rows from v1's
+  training mixture, on its hard label. v2.1 trains the replay rows toward v1's answer distribution instead, with the loss
+  KL(p_v1 ‖ p_model) over the options of each answer, and keeps hard labels on every other row. Everything else is v2's recipe:
+  the same rows (29,325 after removing 31 rows that shared a context with an evaluation file; v2 had 29,356), LoRA rank 64, learning
+  rate 1e-4, 2 epochs. Training the replay rows on hard labels had sharpened the logits everywhere; the fitted temperature then
+  rose to 1.935 and flattened every served answer, which is what cost v2 its sampled play. With the replay rows trained toward
+  v1, the fitted temperature is 1.099 (v1 1.05). On 1,000 held-out replay rows the mean KL(p_v1 ‖ p_model) at temperature 1 is 0.021
+  nats (v2: 0.112), and 96.3% of the argmax answers equal v1's (v2: 94.1%).
+* **One temperature per answer type.** `decider_config.json` has `temperature` 1.099 and `temperature_by_type`
+  `{"choice": 1.110, "noul": 1.560, "score": 1.287}`, fitted by NLL with `decider.calibrate` (decider-ai 1.4.0). decider-ai 1.4.0
+  and later use the map. **decider-ai 1.3.0 and earlier ignore the map and serve every answer at 1.099**; a temperature does not
+  change which option is most probable, so the answers are the same either way (except exact ties between the level rows of an
+  isolated Score answer, where rounding can break the tie differently: 1 of 5,000 held-out generated-family rows and 2 of 447
+  document-question validation rows changed); the
+  probabilities differ, mostly on yes/no and Score answers. The map changes little in practice: sampled
+  play, the fixtures and the probes are the same within noise with and without it, and it lowers the calibration error on
+  held-out yes/no answers (0.146 to 0.101) and Score answers (0.140 to 0.108). It does not fix the overconfidence on hard
+  Choice answers (see Calibration).
+
+All rows below are on identical inputs and seeds: v2.1 through decider-ai 1.4.0 with its map, v1 and v2 through decider-ai 1.3.0
+at their stored temperatures (1.05 and 1.935), measured in the same session on 2026-09-24. Intervals are 95% paired bootstrap
+intervals (rows for the fixtures, boards for the games, task-seed pairs for the browser). The regression set and the two held-out
+sets were read from stored temperature-1 logits at each model's served temperatures. The JevBench files of v1 and v2 were read
+earlier through decider-ai 1.2.1, v2's at the candidate temperature 1.719; accuracy does not depend on the temperature.
+
+| set | v1 (T 1.05) | v2 (T 1.935) | v2.1 (map) | v2.1 minus v1 | v2.1 minus v2 |
+|---|---|---|---|---|---|
+| regression set, 67 in-task tasks, accuracy / NLL / ECE | 0.834 / 0.404 / 0.027 | 0.824 / 0.441 / 0.041 | 0.831 / 0.414 / 0.031 | −0.3 | +0.7 |
+| regression set, 28 held-out tasks | 0.788 / 0.558 / 0.071 | 0.779 / 0.566 / 0.080 | 0.784 / 0.569 / 0.077 | −0.4 | +0.5 |
+| 847 in-task validation rows, accuracy / NLL | 86.1% / 0.417 | 85.0% / 0.419 | 85.0% / 0.432 | −1.1 (−2.1 to +0.0); NLL +0.015 (+0.003 to +0.028) | 0.0 (−1.5 to +1.5); NLL +0.014 (−0.006 to +0.035) |
+| OpenJev, 5,252 rows, accuracy / NLL | 63.9% / 0.893 | 66.7% / 0.789 | 66.0% / 0.846 | +2.1 (+1.2 to +2.9); NLL −0.047 (−0.060 to −0.035) | −0.7 (−1.6 to +0.2); NLL +0.057 (+0.044 to +0.070) |
+| Mind2Web, 1,770 rows, accuracy / NLL | 88.4% / 0.366 | 87.4% / 0.393 | 87.5% / 0.375 | −0.8 (−1.7 to +0.0); NLL +0.009 (−0.004 to +0.023) | +0.1 (−0.8 to +1.1); NLL −0.018 (−0.032 to −0.005) |
+| TypeSafe workflow decisions, 102 rows, accuracy / NLL | 81.4% / 0.611 | 86.3% / 0.410 | 84.3% / 0.441 | +2.9 (−2.9 to +8.8); NLL −0.170 (−0.324 to −0.036) | −2.0 (−7.8 to +3.9); NLL +0.031 (−0.069 to +0.134) |
+| held-out generated families (heldout_jb), 5,000 rows, accuracy / ECE | 0.469 / 0.260 | 0.560 / 0.046 | 0.556 / 0.147 | +8.7 | −0.4 |
+| held-out document questions (test_teacher2), 449 rows, accuracy / ECE | 0.726 / 0.110 | 0.826 / 0.053 | 0.820 / 0.043 | +9.4 | −0.7 |
+| JevBench public items, easy / standard / hard accuracy | 1.000 / 0.958 / 0.550 | 1.000 / 0.986 / 0.676 | 1.000 / 0.986 / 0.649 | hard +11 items | hard −3 items |
+| JevBench hard tier, top-label ECE (v1, v2: files read through 1.2.1, v2 at T 1.719) | 0.288 | 0.104 | 0.184 | | |
+| Bespoke's public suite, macro / micro | 0.757 / 0.765 | 0.773 / 0.781 | 0.756 / 0.765 | −0.1 macro | −1.6 macro |
+| live MiniWoB++, sampled, all 22 tasks | 90.9% | 88.1% | 93.2% | +2.3 (−1.7 to +6.2) | +5.1 (+0.6 to +9.7) |
+| live MiniWoB++, sampled, 16 rewarded tasks | 96.1% | 93.0% | 95.3% | −0.8 (−3.9 to +2.3) | +2.3 (−2.3 to +7.0) |
+| live MiniWoB++, sampled, 6 held-out tasks | 77.1% | 75.0% | 87.5% | +10.4 (0.0 to +20.8) | +12.5 (0.0 to +25.0) |
+| live MiniWoB++, greedy, all 22 tasks | 91.5% | 92.6% | 93.8% | +2.3 (−0.6 to +5.7) | +1.1 (−1.7 to +4.5) |
+| live MiniWoB++, greedy, 16 rewarded tasks | 97.7% | 96.9% | 96.1% | −1.6 (−3.9 to +0.0) | −0.8 (−2.3 to +0.0) |
+| live MiniWoB++, greedy, 6 held-out tasks | 75.0% | 81.2% | 87.5% | +12.5 (+4.2 to +22.9) | +6.2 (−4.2 to +16.7) |
+| zero-shot games, 234 boards, sampled, win rate | 27.8% | 22.4% | 26.9% | −0.9 (−2.6 to +1.0) | +4.5 (+2.5 to +6.6) |
+| bag-draw games, 64 boards, sampled, win rate | 56.6% | 37.9% | 52.0% | −4.7 (−9.0 to −0.4) | +14.1 (+8.2 to +19.9) |
+| slippery-grid games, 64 boards, sampled, win rate | 16.4% | 16.0% | 16.8% | +0.4 (−2.7 to +3.5) | +0.8 (−2.3 to +3.9) |
+| zero-shot games, 234 boards, greedy, win rate | 29.1% | 27.8% | 25.2% | −3.8 (−7.3 to −0.4) | −2.6 (−6.4 to +1.3) |
+| bag-draw games, 64 boards, greedy, win rate | 62.5% | 48.4% | 53.1% | −9.4 (−17.2 to −3.1) | +4.7 (−3.1 to +12.5) |
+| slippery-grid games, 64 boards, greedy, win rate | 12.5% | 18.8% | 10.9% | −1.6 (−7.9 to +4.7) | −7.8 (−15.6 to +0.0) |
+| ten text games, greedy: Pong / Breakout / CliffWalking / BabyAI-GoTo / Freeway / Blackjack | −21 / 14 / −13 / 0.54 / 0 / −0.6 | −21 / 12 / −60 / 0.35 / 1 / −0.6 | −5 / 35 / −13 / 0.19 / 0 / −0.6 | | |
+| behaviour probes: model-router tier / needs-live-data (31 items) | 0.968 / 0.871 | 0.935 / 0.839 | 0.968 / 0.839 | 0 / −1 item | +1 / 0 items |
+| behaviour probes: command risk / touches-outside-project (45 items) | 0.889 / 0.956 | 0.911 / 0.933 | 0.889 / 0.933 | 0 / −1 item | −1 / 0 items |
+| behaviour probes: generic bucket / catch-all / abstention battery / browser element and action | 1.00 / 0.90 / 7 of 8 / 0.875 and 0.875 | 0.95 / 0.95 / 8 of 8 / 0.938 and 0.938 | 1.00 / 0.95 / 8 of 8 / 0.938 and 0.750 | | |
+| issue #9 form cases c_1 / c_2 (probability of the gold option) | right (0.79) / right (0.98) | wrong (0.16) / right (0.28) | wrong (0.20) / right (0.41) | | |
 
 **Regressions, stated plainly.**
-* Sampled play in games: the bag-draw games (choose the bag with the highest expected value) fall from 56.6% to 37.9% wins in
-  sampled play, and zero-shot games overall from 27.8% to 22.4%. An earlier candidate (the Qwen3.5-4B instruct model plus a LoRA on
-  15,768 of these 29,356 rows) showed a drop of 12.5 points on the same bag-draw games; this points to the new data, but it was not tested
-  directly.
-* Sampled browser play: −2.8 points over all 22 tasks and −3.1 on the 16 rewarded tasks (intervals include zero); the losses
-  are on click-dialog-2, click-tab, click-checkboxes-large, click-checkboxes-soft, click-checkboxes-transfer and click-option,
-  the gains on click-tab-2, click-tab-2-hard and focus-text-2.
-* Text games (greedy): CliffWalking −60 against −13, BabyAI-GoTo 0.35 against 0.54, Breakout 12 against 14.
-* Behaviour probes: model-router tier 0.935 against 0.968, needs-live-data 0.839 against 0.871, touches-outside-project 0.933
-  against 0.956, generic bucket choice 0.95 against 1.00.
-* Form filling: on the two form-filling cases reported in issue #9 (choose the document entity for a form field, or skip /
-  click / check), v1 answers both correctly (0.79 and 0.98 on the gold option). v2 answers one wrongly ("skip" at 0.68, gold
-  0.16) and the other correctly by 0.02 (0.28 against "click" at 0.26). Two cases do not measure a rate, but for form filling
-  we suggest v1.
-* Everyday tasks: regression in-task accuracy −1.0 point and held-out −0.9 (the held-out interval includes zero), lower on 75
-  of 95 tasks, mostly by under 3 points; the largest losses are CommitmentBank −8.9, PubMedQA −6.2, hate-speech tweets −4.7,
-  emotion −4.3, LIAR −3.9, QuALITY (full) −3.3, MedQA −3.0. Regression-set calibration is worse (in-task ECE 0.041 against
-  0.027), and rating (Score) tasks are the least calibrated in the probe (see Calibration).
-
-If you rely on sampled play (games, browser agents that sample actions) or on the probes above, pin revision `v1`. The
-package loads a local folder, so download the revision first:
+* BabyAI-GoTo (greedy text game): 0.19 against v1's 0.54 and v2's 0.35.
+* Greedy bag-draw play: 53.1% against v1's 62.5% wins (−9.4 points, interval −17.2 to −3.1); greedy zero-shot games overall 25.2%
+  against 29.1% (−3.8, interval −7.3 to −0.4). Sampled bag-draw play is 4.7 points under v1 (interval −9.0 to −0.4).
+* Behaviour probes: needs-live-data 0.839 against v1's 0.871 and touches-outside-project 0.933 against 0.956, one item each (as
+  v2); the browser-agent action choice is 12 of 16 items (0.750), 2 fewer than v1 (0.875) and 3 fewer than v2
+  (0.938).
+* Form filling: issue #9 case c_1 is still answered wrongly. The field is "Degree earned" and the document entity is "Studied:
+  Associate of Arts"; v2.1 chooses "skip" at 0.76 and gives the gold entity 0.20 (v1: gold 0.79; v2: gold 0.16). Case c_2 is
+  answered correctly at 0.41. For form filling we suggest v1.
+* Calibration on hard items: on the held-out generated families the calibration error is 0.147 against v2's 0.046 (see
+  Calibration), and on the JevBench public hard tier 0.184 against v2's 0.104.
+* JevBench public hard tier: 72 of 111 items (0.649) against v2's 75 (0.676).
+* Against v2: TypeSafe −2.0 points and OpenJev −0.7 (intervals include zero), NLL higher on both; Bespoke's suite 0.756 against
+  0.773 macro (v1 0.757); greedy slippery-grid play 10.9% against 18.8%.
+
+**How v2.1 was chosen, and why it is released although it did not pass.** The run had pre-registered release rules. v2.1 passed
+the numeric items (held-out document questions 0.820 against the required 0.816, held-out generated families 0.556 against 0.550,
+regression accuracy and calibration not worse than v2's, sampled bag-draw above the midpoint of v1 and v2, sampled zero-shot
+games and browser play not shown to be below v1's (upper end of the 95% interval at least 0), each probe at most one item below v1) and failed the last item, which required both
+issue #9 form cases to be right: c_1 is wrong. By the rule it was not recommended. A second pre-registered rule for the
+temperature map required the calibration error on the held-out generated families to be at most 0.08; the map reaches 0.147
+(0.163 without it), so the map did not pass either. v2.1 is released with the map on a decision made after reading the full comparison above: it is better than v2 on sampled play, CliffWalking, the model-router probe and the regression set, at v2's
+level on our held-out hard sets, and its failures are listed in this section. The JevBench public items were read once for the
+candidate at its global temperature and once with the map, after the rule decisions; they were not used for training,
+selection or the temperatures.
+
+**Which version to use.**
+* v2.1 (this revision): the default. Sampled play (games, browser agents that sample actions), hard decisions.
+* v2 (`revision="v2"`): if you rely on confidence values on hard multi-step items, where v2 is better calibrated (held-out
+  generated families 0.046, JevBench hard tier 0.104), or on its slightly higher JevBench hard tier and TypeSafe accuracy.
+* v1 (`revision="v1"`): form filling (issue #9), BabyAI-GoTo-like grid navigation, greedy bag-draw play.
+
+The package loads a local folder, so download the revision first:
 
 ```python
 from huggingface_hub import snapshot_download
 from decider.infer import Decider
-d = Decider(snapshot_download("Mapika/decider-4b", revision="v1"))
+d = Decider(snapshot_download("Mapika/decider-4b", revision="v2"))      # or revision="v1"
 ```
 
 For the HTTP server, set `DECIDER_MODEL` to the same downloaded folder.
 
-**How v2 was chosen.** The training run had a pre-registered rule for replacing v1: the candidate had to beat an earlier
-candidate (the Qwen3.5-4B instruct model plus a LoRA on 15,768 of the same hard-decision rows) on our own held-out hard sets. v2 did not meet it:
-it was 0.8 points short on the held-out teacher-written set and 0.5 points short on the held-out generated families, that is, it
-tied the earlier candidate there. The earlier candidate had lost v1's everyday skills and v2 keeps them within about 1 point, so
-v2 was then measured on every set of this card, and the release was decided on that full comparison with v1. The JevBench public
-items were read once for v2, after selection; they were not used for training, selection or the temperature.
-
 ## The decider family
 
 All six repositories share one interface (`decider.infer.Decider`, `POST /v1/systemone` in TypeSafe's format) and one
@@ -111,8 +144,8 @@ readout: the letter logits at an answer slot, softmaxed over the options. Pick b
 
 | model | base | weights | use it for | numbers |
 |---|---|---|---|---|
-| [decider-2b](https://huggingface.co/Mapika/decider-2b) v10 | Qwen3.5-2B-Base | 3.5 GB bf16 | the default: routing, classification, judgments, browser agents; 4 ms per request with CUDA graphs on one GPU | regression set 0.805 in-task / 0.755 held-out; live browser 93%; Bespoke suite 0.704 |
-| [decider-4b](https://huggingface.co/Mapika/decider-4b) v2 | Qwen3.5-4B-Base | 8.4 GB bf16 | the middle point: knowledge questions and hard judgments above the 2B in a dense 8.4 GB model; no RL stage; v1 under the tag `v1` for sampled play | 0.824 / 0.779, above the 2B on 66 of 95 tasks; JevBench hard 0.676; Bespoke 0.773 |
+| [decider-2b](https://huggingface.co/Mapika/decider-2b) v11 | Qwen3.5-2B-Base | 3.8 GB bf16 | the default: routing, classification, judgments, browser agents; 4 ms per request with CUDA graphs on one GPU; v10 under the tag `v10` | regression set 0.802 in-task / 0.752 held-out; JevBench hard 0.577; live browser 90% sampled; Bespoke suite 0.706 |
+| [decider-4b](https://huggingface.co/Mapika/decider-4b) v2.1 | Qwen3.5-4B-Base | 8.4 GB bf16 | the middle point: knowledge questions and hard judgments above the 2B in a dense 8.4 GB model; no RL stage; v2 and v1 under the tags `v2` and `v1` | 0.831 / 0.784; JevBench hard 0.649; live browser 93% sampled; Bespoke 0.756 |
 | [decider-35b-a3b](https://huggingface.co/Mapika/decider-35b-a3b) v1 | Qwen3.5-35B-A3B-Base (3B active) | 65 GB bf16 | when accuracy is worth 3 to 4 times the cost per decision: knowledge and multi-step questions, long policies | 0.855 / 0.810, above the 2B on 93 of 95 tasks; JevBench hard 0.676; Bespoke 0.774; no RL stage |
 | [decider-35b-a3b-nvfp4](https://huggingface.co/Mapika/decider-35b-a3b-nvfp4) | the 35B in NVFP4 | 19.6 GB | the 35B on Blackwell through vLLM or TensorRT-LLM | 1.0 to 1.5 points under bf16 on the measured fixtures |
 | [decider-0.8b](https://huggingface.co/Mapika/decider-0.8b) | Qwen3.5-0.8B-Base | 1.4 GB bf16 | the smallest: routing, yes/no and short-state lookups within 1 to 4 points of the 2B, 1.5x faster | 0.776 / 0.707 on the single-run protocol (2B: 0.809 / 0.739) |
@@ -133,19 +166,22 @@ d.decide("My card was charged twice for the same purchase.",
 
 The API is the same as decider-2b's: `decide_batch` scores many states with many questions in one call, `abstain_below=t`
 returns `None` under a confidence threshold, a question can have 2 to 255 options, and `system_one` / `decider.serve` accept
-TypeSafe's `POST /v1/systemone` request shape (the official `typesafe-sdk` works with `TYPESAFE_BASE_URL` pointing at the
-server; checked with decider-2b, not again with this model). Every question and every Score level is scored in its own row. The state may be a string, object or array of up to
-32k tokens. See the decider-2b card for the full description of the request shape, field types and the schema cache.
+TypeSafe's `POST /v1/systemone` request shape. Every question and every Score level is scored in its own row. The state may be a
+string, object or array of up to 32k tokens. See the decider-2b card for the full description of the request shape, field types
+and the schema cache.
 
 Requirements: `torch`, `transformers>=5`, and `flash-linear-attention` (Triton kernels for the Qwen3.5 linear-attention layers;
-the model runs without it but several times slower). The weights take 8.4 GB in bf16. v2 uses the plain prompt layout, as v1
-does, so it needs no new package version. decider-ai 1.0.2, 1.1.4, 1.2.1 and 1.2.2 and the `decider/` subset in this repository
-(taken from 1.2.2) were checked on CUDA: each loads v2 with the name `decider-4b-v2` and temperature 1.935 and gives the same
-probabilities to four decimals on the usage example above, on the eager path and on the CUDA-graph path (the two paths differ in
-the third decimal, as for every model). The 1.2.2 HTTP server was checked with one `/v1/systemone` request. Other versions,
-devices and the FP8 path were not checked on v2. On Blackwell GPUs use 1.0.2 or later (1.0.0 and 1.0.1 have the cuDNN attention fault fixed
-in 1.0.2). The model is dense, so the CUDA-graph engine, `torch.compile` and the FP8 path of the helper package apply to it as
-to decider-2b (`use_graphs=False` selects eager PyTorch). The measurements below were taken with the eager path unless stated.
+the model runs without it but several times slower). The weights take 8.4 GB in bf16. v2.1 uses the plain prompt layout, as v1 and
+v2 do. The per-type temperatures need decider-ai 1.4.0 or later (or the `decider/` subset in this repository, taken from 1.4.0).
+Checked on CUDA, eager path: decider-ai 1.3.0 loads v2.1 with the name `decider-4b-v2.1` and temperature 1.099 and gives exactly
+the probabilities that 1.4.0 gives with the map switched off; 1.4.0 and the `decider/` subset in this repository load the map
+and give the same probabilities as each other. For Choice and yes/no answers the 1.4.0 output with the map equals
+softmax(log(p) · 1.099 / T_type) of the 1.3.0 output p (to 1e-7 on `decide`, to 8e-5 on the four-decimal `system_one` values);
+an isolated-level Score answer is rescaled per level row (each level's yes/no pair at 1.287 instead of 1.099) before the
+levels are normalised, so it cannot be recomputed from the combined Score probabilities. The 1.4.0 HTTP server with CUDA graphs was checked with `/v1/systemone` and `/decide` requests. Other package
+versions, MPS and the FP8 path were not checked on v2.1. On Blackwell GPUs use 1.0.2 or later (1.0.0 and 1.0.1 have the cuDNN
+attention fault fixed in 1.0.2). The model is dense, so the CUDA-graph engine, `torch.compile` and the FP8 path of the helper
+package apply to it as to decider-2b (`use_graphs=False` selects eager PyTorch).
 
 Without the helper package, the same computation in plain `transformers`:
 
@@ -159,7 +195,7 @@ ids = tok(prompt, return_tensors="pt").to("cuda")
 with torch.no_grad():
     logits = m(**ids).logits[0, -1]
 letters = [tok.encode(L, add_special_tokens=False)[0] for L in "ABC"]
-probs = torch.softmax(logits[letters].float() / 1.935, -1)     # 1.935 is the stored temperature (v1: 1.05)
+probs = torch.softmax(logits[letters].float() / 1.110, -1)     # 1.110 is the stored choice temperature (yes/no 1.560, Score 1.287)
 ```
 
 ## How it works
@@ -167,8 +203,13 @@ probs = torch.softmax(logits[letters].float() / 1.935, -1)     # 1.935 is the st
 The prompt is `Context: ...` followed by, for each question, the question text, the lettered options `(A) ... (B) ...` and an
 answer slot `Answer k: (`. The hidden state at each slot is projected with the option-letter rows of the LM head and softmaxed
 over the valid letters, divided by the temperature in `decider_config.json`. Letters are never generated, so all slots are read
-from one pass. Large label sets were sub-sampled to at most 10 options per training example (gold always kept, order shuffled),
-so the model conditions on the supplied candidates rather than on a fixed head.
+from one pass. From decider-ai 1.4.0 the config may also hold `temperature_by_type`, one temperature per answer type
+(`choice`, `noul`, `score`; a missing type uses `temperature`). This release's config has such a map: Choice answers use 1.110,
+yes/no (noul) answers 1.560, Score answers 1.287 on each of their level rows. Package versions before 1.4.0 use `temperature`
+(1.099) for every answer. `DECIDER_TEMPERATURE=T` or `Decider(path, temperature=T)` replaces `temperature` with T and switches the
+map off, so every answer then uses T. Large label sets were sub-sampled to at most 10 options per
+training example (gold always kept, order shuffled), so the model conditions on the supplied candidates rather than on a fixed
+head.
 
 ## Training
 
@@ -201,250 +242,171 @@ schedule, only the optimizer changed), AdamW on the bf16 parameters beat AdamW w
 through and moves the weights further from the base model; without it, updates below the bf16 resolution round away and more of
 the base model's knowledge is kept.
 
-**Stage 2 (v2).** A LoRA of rank 64 (alpha 128) on the attention and MLP weights of v1, trained with cross-entropy on the slot
-readout for 2 epochs over 29,356 rows in v1's plain state-first layout with isolated Score levels, then merged into the bf16
-weights:
+**Stage 2 (v2.1).** A LoRA of rank 64 (alpha 128) on the attention and MLP weights of v1, trained for 2 epochs over 29,325 rows
+in v1's plain state-first layout with isolated Score levels, then merged into the bf16 weights:
 
-| source | rows | content |
-|---|---|---|
-| generated decision families | 8,000 | ten families (temporal and numeric decisions, subtle answer judgment, long policies, multi-hop lookup, abstention, probability, constrained trade-offs, safety judgment, paraphrase sensitivity, adversarial traps); the answers are computed by the generating code |
-| questions over business documents, written by Qwen3.6-27B with thinking on | 11,356 | in two rounds, one realistic business document (up to 34 domains and 22 document kinds) plus three or four typed questions per writer call; each question was answered twice more by the same model in fresh contexts, with shuffled options and without the writer's answer, and kept only when both answers agreed with the writer's (89% and 91% kept) |
-| human-labelled public sets (training halves) | 3,300 | MMLU, ARC, CommonsenseQA, BoolQ, MNLI, SNLI, Banking77, RACE, OpenBookQA, LogiQA 2, MedQA, Winogrande |
-| replay of mixture v2 | 6,700 | 100 rows from the training half of each of the 67 in-task regression tasks |
+| source | rows | content | target |
+|---|---|---|---|
+| generated decision families | 8,000 | ten families (temporal and numeric decisions, subtle answer judgment, long policies, multi-hop lookup, abstention, probability, constrained trade-offs, safety judgment, paraphrase sensitivity, adversarial traps); the answers are computed by the generating code | the label |
+| questions over business documents, written by Qwen3.6-27B with thinking on | 11,356 | in two rounds, one realistic business document (up to 34 domains and 22 document kinds) plus three or four typed questions per writer call; each question was answered twice more by the same model in fresh contexts, with shuffled options and without the writer's answer, and kept only when both answers agreed with the writer's (89% and 91% kept) | the label |
+| human-labelled public sets (training halves) | 3,293 | MMLU, ARC, CommonsenseQA, BoolQ, MNLI, SNLI, Banking77, RACE, OpenBookQA, LogiQA 2, MedQA, Winogrande | the label |
+| replay of mixture v2 | 6,676 | 100 rows from the training half of each of the 67 in-task regression tasks | v1's answer distribution: loss KL(p_v1 ‖ p_model) at temperature 1 |
+
+These are v2's rows without 31 that the v2.1 checks found to share a context with an evaluation file (7 human-labelled rows and
+24 replay rows).
 
 | stage 2 | |
 |---|---|
 | trainable parameters | LoRA rank 64, alpha 128, on the attention and MLP projections; merged after training |
-| schedule | learning rate 1e-4, 5% warm-up then cosine, 1,517 steps of 65,536 tokens (2 epochs), seed 0 |
-| hardware | one NVIDIA B300, 87 minutes |
+| loss | cross-entropy on the slot readout for labelled rows; KL(p_v1 ‖ p_model) over the options for replay rows, with p_v1 from the frozen v1 weights; the step loss is the sum over answers divided by the number of answers |
+| schedule | learning rate 1e-4, 5% warm-up then cosine, 1,518 steps of 65,536 tokens (2 epochs), seed 0 |
+| hardware | one NVIDIA B300, 94 minutes |
 
 No JevBench item and no Decision Index item was used for training, for writing the generators or the document questions, for
-selecting the checkpoint, or for the temperature. The ten generated families and the skill list of the document questions were
+selecting the checkpoint, or for the temperatures. The ten generated families and the skill list of the document questions were
 written from the family names that JevBench publishes for its sealed set, not from its items. Every training row was checked
-against every evaluation file used for selection and against the evaluation half of all 95 regression tasks: there is no exact
-state-and-question overlap. The held-out sets used for selection are generated families from held-out templates and document
-questions from business domains that are not in the training data.
-
-**Temperature.** 1.935, fitted by NLL (all rows pooled) on the in-task half of the public regression set without Banking77,
-CLINC-OOS, MMLU, ARC, Winogrande and HellaSwag: 61 tasks, 102,804 rows. Fitted on all 67 in-task tasks the value is 1.942. v1's
-temperature (1.05) was fitted on the 67 in-task tasks, the same set and method as decider-2b and decider-35b-a3b. No
-reinforcement-learning stage was run on this model; the RL recipe of decider-2b v10 is documented in `docs/RL.md` of the GitHub
-repository.
+against every evaluation file used for selection, the canonical examples of the evaluation half of all 153 mixture-v2 evaluation
+tasks, the four request fixtures and the issue #9 cases: there is no exact state-and-question overlap (a few contexts under 40
+characters, such as a short utterance, occur under another task and question). The held-out sets used for selection are
+generated families from held-out templates (heldout_jb) and document questions from business domains that are not in the
+training data (test_teacher2). Two arms were trained to the end (B4 and C4, which differ in the size of the replay; two further arms were
+stopped or cancelled at the user's request); the rule selected B4 after epoch 2.
+
+**Temperatures.** `temperature` 1.099, fitted by NLL (all rows pooled) on the in-task half of the public regression set without
+Banking77, CLINC-OOS, MMLU, ARC, Winogrande and HellaSwag: 61 tasks, 102,804 rows. `temperature_by_type` fitted by NLL per answer
+type with `decider.calibrate.fit_by_type` on a pool of those regression rows and our own validation rows (the validation halves of
+the generated families and document questions, human-labelled validation rows, held-out replay rows): 108,927 Choice answers,
+1,677 yes/no answers and 535 Score answers. The Choice pool is 94% regression rows, so the Choice temperature (1.110) stays close
+to the global one. v2's temperature (1.935) and v1's (1.05) were one value each.
 
 ## Evaluation
 
-All v2 numbers are at the stored temperature 1.935 through decider-ai 1.2.1, except where a paragraph says otherwise. Greedy
-play and accuracy do not depend on the temperature; sampled play and calibration do, and those were measured again at 1.935.
+All v2.1 numbers are with the per-type map through decider-ai 1.4.0, except where a paragraph says otherwise. Greedy play and
+accuracy do not depend on the temperature; sampled play and calibration do.
 
 **Public regression set**, rebuilt on this machine (95 tasks: 67 in-task, 28 held-out; large label sets sub-sampled to 10
-options; one temperature per model fitted on in-task data). All four rows are the same rows. ECE is the expected calibration
-error with 15 bins; intervals are 95% bootstrap intervals over tasks, paired.
+options; the temperatures fitted on in-task data). The rows are the same for every model. ECE is the expected calibration
+error with 15 bins, per-task mean. The regression rows are all Choice answers, so v2.1 reads them at the Choice temperature 1.110.
 
 | model | in-task acc / NLL / ECE (67 tasks) | held-out acc / NLL / ECE (28 tasks) |
 |---|---|---|
-| decider-2b v10, T=1.30 | 0.805 / 0.474 / 0.037 | 0.755 / 0.622 / 0.084 |
+| decider-2b v10, T=1.30 | 0.806 / 0.474 / 0.038 | 0.755 / 0.622 / 0.084 |
+| decider-2b v11, per-type map | 0.802 / 0.481 / 0.038 | 0.752 / 0.626 / 0.083 |
 | decider-4b v1, T=1.05 | 0.834 / 0.404 / 0.027 | 0.788 / 0.558 / 0.071 |
-| **decider-4b v2 (this repository), T=1.935** | **0.824 / 0.441 / 0.041** | **0.779 / 0.566 / 0.080** |
+| decider-4b v2, T=1.935 | 0.824 / 0.441 / 0.041 | 0.779 / 0.566 / 0.080 |
+| **decider-4b v2.1 (this repository), per-type map** | **0.831 / 0.414 / 0.031** | **0.784 / 0.569 / 0.077** |
 | decider-35b-a3b v1, T=1.08 | 0.855 / 0.357 / 0.026 | 0.810 / 0.497 / 0.069 |
 
-Against v10: +1.9 in-task points (+1.0 to +2.8) and +2.4 held-out points (+1.1 to +3.7); NLL −0.032 (−0.054 to −0.011) and
-−0.056 (−0.094 to −0.024); accuracy higher on 66 of the 95 tasks, equal on two, lower on 27 (hate-speech tweets −4.9 points,
-emotion −3.6, ADE −2.9, abstention probe −2.3, SST-5 −2.1, HelpSteer3 preference −2.1, LIAR −2.1, the others under 2). The
-largest gains are on knowledge and reasoning tasks: MedQA +13.4, MedMCQA +12.3, Winogrande +11.7, TruthfulQA +11.6, MMLU +11.3,
-OpenBookQA +8.8, StrategyQA +8.0. Mean accuracy over nine knowledge tasks (MMLU, ARC, HellaSwag, MedQA, MedMCQA, Social IQa,
-COPA, TruthfulQA, TREC): 0.797 (v1 0.801) against v10's 0.707 and the 35B's 0.862. Against the 35B: −3.1 in-task (−4.0 to −2.3)
-and −3.2 held-out points (−4.5 to −1.8), lower on 85 of 95 tasks and higher on 7; the largest gaps are MedQA −17.7, HelpSteer3
-preference −16.6, MedMCQA −11.9, StrategyQA −10.8, TruthfulQA −10.0.
+Per-task paired intervals were not recomputed for v2.1; the v2 card (tag `v2`) has them for v2 against v10 and the 35B, and v2.1
+is within 0.7 points of v2 on both halves.
 
-<details>
-<summary><b>Per-task accuracy / ECE on the 28 held-out datasets, decider-2b v10, this model, decider-35b-a3b</b></summary>
+**The same rows as v1 and v2** are in the table under [Changes from v2](#changes-from-v2): the four fixtures (847 in-task
+validation rows, OpenJev, Mind2Web, TypeSafe), our held-out hard sets, JevBench, Bespoke, live browser tasks, zero-shot games,
+text games, behaviour probes and the issue #9 cases.
 
-| task | decider-2b v10 | decider-4b v2 | decider-35b-a3b |
-|---|---|---|---|
-| abstain_probe | 0.606 / 0.134 | 0.582 / 0.133 | 0.622 / 0.085 |
-| ade | 0.817 / 0.038 | 0.789 / 0.107 | 0.837 / 0.035 |
-| arena_pref | 0.483 / 0.189 | 0.503 / 0.218 | 0.521 / 0.121 |
-| bbc_news | 0.927 / 0.013 | 0.951 / 0.068 | 0.944 / 0.027 |
-| cb | 0.857 / 0.093 | 0.839 / 0.118 | 0.893 / 0.084 |
-| cr_reviews | 0.903 / 0.031 | 0.898 / 0.046 | 0.914 / 0.033 |
-| dbpedia_l2 | 0.950 / 0.018 | 0.951 / 0.017 | 0.961 / 0.010 |
-| dbpedia_l3 | 0.987 / 0.005 | 0.991 / 0.012 | 0.992 / 0.004 |
-| dolly_category | 0.299 / 0.203 | 0.374 / 0.142 | 0.354 / 0.098 |
-| fin_phrasebank | 0.694 / 0.042 | 0.699 / 0.077 | 0.759 / 0.110 |
-| fin_sentiment | 0.793 / 0.058 | 0.826 / 0.051 | 0.839 / 0.136 |
-| hermes_tools | 0.723 / 0.208 | 0.737 / 0.124 | 0.799 / 0.085 |
-| hwu64 | 0.961 / 0.030 | 0.963 / 0.037 | 0.975 / 0.022 |
-| massive_scenario | 0.756 / 0.041 | 0.793 / 0.052 | 0.799 / 0.027 |
-| offtopic_probe | 0.841 / 0.027 | 0.823 / 0.043 | 0.870 / 0.038 |
-| paws | 0.724 / 0.145 | 0.794 / 0.124 | 0.729 / 0.169 |
-| pubmedqa | 0.756 / 0.085 | 0.758 / 0.055 | 0.820 / 0.078 |
-| quality | 0.494 / 0.233 | 0.561 / 0.179 | 0.632 / 0.096 |
-| quality_full | 0.508 / 0.198 | 0.505 / 0.188 | 0.565 / 0.112 |
-| reward_bench | 0.819 / 0.045 | 0.853 / 0.032 | 0.919 / 0.024 |
-| sciq | 0.982 / 0.024 | 0.991 / 0.021 | 0.993 / 0.011 |
-| social_iqa | 0.708 / 0.077 | 0.759 / 0.062 | 0.823 / 0.025 |
-| strategyqa | 0.552 / 0.138 | 0.632 / 0.157 | 0.739 / 0.036 |
-| student_questions | 0.925 / 0.045 | 0.940 / 0.052 | 0.954 / 0.090 |
-| trec | 0.784 / 0.066 | 0.842 / 0.027 | 0.832 / 0.160 |
-| truthfulqa | 0.537 / 0.090 | 0.654 / 0.063 | 0.754 / 0.068 |
-| tweet_irony | 0.795 / 0.052 | 0.827 / 0.027 | 0.861 / 0.129 |
-| xstory_cloze | 0.962 / 0.017 | 0.971 / 0.017 | 0.995 / 0.016 |
-
-</details>
-
-**On the same rows as decider-2b v10 and decider-35b-a3b.** Every row below is scored by all three models on identical inputs
-and seeds. Intervals are 95% paired bootstrap intervals (rows for the fixtures, boards for the games, task-seed pairs for the
-browser). The table under [Changes from v1](#changes-from-v1) has v1 on the same rows.
-
-| | decider-2b v10 | **decider-4b v2** | decider-35b-a3b | 4B minus 2B | 4B minus 35B |
-|---|---|---|---|---|---|
-| 847 in-task validation rows, accuracy / NLL | 83.2% / 0.444 | 85.0% / 0.419 | 90.0% / 0.329 | +1.8 (−0.5 to +3.9) | −5.0 (−7.1 to −2.8) |
-| OpenJev, 5,252 rows, accuracy / NLL | 63.3% / 0.916 | 66.7% / 0.789 | 68.3% / 0.752 | +3.5 (+2.2 to +4.7) | −1.6 (−2.8 to −0.3) |
-| Mind2Web element and action choice, 1,770 rows | 82.7% / 0.543 | 87.4% / 0.393 | 89.6% / 0.316 | +4.8 (+3.1 to +6.4) | −2.2 (−3.7 to −0.6) |
-| TypeSafe workflow decisions, 102 rows, accuracy / NLL | 80.4% / 0.585 | 86.3% / 0.410 | 86.3% / 0.342 | +5.9 (−2.9 to +14.7) | 0.0 (−5.9 to +4.9) |
-| Bespoke's public suite, 13 subsets, macro / micro | 0.704 / 0.711 | 0.773 / 0.781 | 0.774 / 0.787 | | |
-| JevBench public items, easy / standard / hard accuracy | 1.000 / 0.889 / 0.459 | 1.000 / 0.986 / 0.676 | 1.000 / 0.972 / 0.676 | | |
-| live MiniWoB++ click tasks, 22 tasks x 8 seeds, greedy play | 90.9% | 92.6% | 97.2% | +1.7 (−4.0 to +7.4) | −4.5 (−8.0 to −1.7) |
-| the same, 6 tasks v10 never used for reward, greedy | 91.7% | 81.2% | 97.9% | −10.4 (−22.9 to +2.1) | −16.7 (−27.1 to −6.2) |
-| live MiniWoB++ click tasks, sampled play | 93.2% | 88.1% | 86.4% | −5.1 (−10.8 to +0.6) | +1.7 (−3.4 to +6.8) |
-| the same, 6 held-out tasks, sampled | 91.7% | 75.0% | 79.2% | −16.7 (−31.2 to −4.2) | −4.2 (−16.7 to +8.3) |
-| zero-shot games, win rate, sampled play (234 boards) | 23.7% | 22.4% | 24.1% | −1.3 (−4.2 to +1.5) | −1.7 (−3.7 to +0.2) |
-| zero-shot games, greedy play | 26.5% | 27.8% | 37.2% | +1.3 (−4.3 to +6.8) | −9.4 (−15.0 to −3.8) |
-| bag-draw games alone, sampled | 41.4% | 37.9% | 41.8% | −3.5 (−9.0 to +2.0) | −3.9 (−8.6 to +0.8) |
-
-On the two fixtures whose rows are in neither model's training data (TypeSafe, OpenJev) v2 is above the 2B (OpenJev +3.5, interval
-excludes zero; TypeSafe +5.9, interval includes zero) and on TypeSafe level with the 35B. Stage 2 contains teacher-written and
-human-labelled judgment rows, not TypeSafe or OpenJev rows. The browser rows show the missing RL stage: greedy play is level
-with v10 over all tasks and 10 points below it on the six tasks that v10's RL never rewarded (interval includes zero), where the
-4B's weakest tasks are focus-text-2 (0.375) and click-collapsible-2 (0.5). On the bag-draw games v1 won about 15 points more
-often than either other model in sampled play; v2 does not (37.9% against 41.4% and 41.8%). On tic-tac-toe, the slippery grid
-and minesweeper every model is near the random floor.
-
-**Ten text games, zero-shot** (`decider.games.play`, five episodes per game, greedy, eager; the 2B was trained on the first
-four, the 4B and the 35B on none): Pong −21 (teacher 8, 2B 8, 35B −21), Breakout 12 (22, 22, 6), CliffWalking −60 (teacher −13,
-2B −13, 35B −1,248), MiniGrid-Empty 0 (teacher 0.96, both others 0), Freeway 1 (teacher 5, 2B 0, 35B 1), FrozenLake 0 (teacher 1,
-others 0), Blackjack −0.6 (teacher −0.6, 2B −1, 35B −0.6), MiniGrid-LavaGap 0, MiniGrid-DoorKey 0 (teacher 0), BabyAI-GoTo 0.35
-(teacher 0.34, 2B 0, 35B 0.19). v1 reached the teacher on CliffWalking (−13) and read 0.54 on BabyAI-GoTo; v2 keeps neither.
-Neither version learns Pong from the text state. Greedy play takes the most probable option, so these results do not depend on
-the temperature.
+**Ten text games, zero-shot** (`decider.games.play`, five episodes per game, greedy, eager; the 2B was trained on the first four,
+the 4B on none): Pong −5 (v1 −21, v2 −21; the 2B 8), Breakout 35 (v1 14, v2 12; the 2B 22), CliffWalking −13 (v1 −13, v2 −60;
+teacher −13), BabyAI-GoTo 0.19 (v1 0.54, v2 0.35), Freeway 0 (v2 1), Blackjack −0.6, FrozenLake, MiniGrid-Empty, LavaGap and DoorKey
+0 as for v1 and v2. Greedy play takes the most probable option, so these results do not depend on the temperature.
 
 **JevBench public items** (231 items of [Benchmark Heaven](https://benchmarkheaven.com/jev-models); argmax over the exact
-label set with the request the harness's TypeSafe adapter builds). The items were read once for v2, after selection, at the
-candidate temperature 1.719; the accuracies do not depend on the temperature, and the calibration numbers below are recomputed
-at 1.935 from the stored probabilities. Jev 1.13.0 is at 1.000 / 0.986 / 0.730, SemIf (also a Qwen3.5-4B) at 1.000 / 0.986 /
-0.613, decider-35b-a3b at 1.000 / 0.972 / 0.676 on the same items. The one standard-tier family with misses is adequacy (0.92).
-Hard-tier families (v1 as first published in brackets): trap 1.00 (1.00), hard routing 1.00 (1.00), adversarial 1.00 (0.67), probability 0.90 (0.60),
-ambiguous 0.86 (0.29), multi-hop 0.72 (0.78), trade-off 0.67 (0.33), long policy 0.63 (0.47), judge-hard 0.47 (0.47),
-temporal-numeric 0.27 (0.13). Top-label ECE is 0.008 / 0.085 / 0.071 by tier (v1, read again on 2026-09-24: 0.001 / 0.047 / 0.288;
-the 35B's hard tier 0.151). On the hard tier the mean confidence is 0.74 at accuracy 0.68; 31% of hard items are answered at confidence 0.9 or more,
-with accuracy 0.94 on them (v1: 50% at 0.73). The 18 Score items were answered with isolated levels, whose per-level values are
-not stored, so they keep their T 1.719 probabilities inside these ECE values. The public hard tier is 111 items (95% interval
-about ±9 points), and the gain over v1 there (+12.6) is larger than the gain on our own held-out hard sets.
+label set with the request the harness's TypeSafe adapter builds). Read once for v2.1 at its global temperature (1.3.0) and once
+with the map (1.4.0), both after the rule decisions: easy 1.000, standard 0.986, hard 0.649 (72 of 111) in both reads. Top-label
+ECE on the hard tier is 0.184 with the map (0.210 at the global temperature); by answer type with the map, Choice 0.156 (67 items),
+yes/no 0.259 (38), Score 0.354 (6). Mean confidence on the hard tier is 0.799 at accuracy 0.649. For comparison on the same items:
+v2 0.676 (ECE 0.104 as read at T 1.719; 0.071 recomputed at its release temperature 1.935 with 6 Score items kept at 1.719), v1
+0.550 (ECE 0.288), decider-35b-a3b 0.676, Jev 1.13.0 0.730. The public hard tier is 111 items (95% interval about ±9 points).
 
 **Bespoke's public suite** (13 human-labelled subsets, 3,880 records in Jev's wire format, answered through `system_one` as
-shipped). Nimble-9B and Jev 1.13.0 numbers are copied from Bespoke's report.
+shipped). v1 and v2 through decider-ai 1.3.0, v2.1 through 1.4.0 with the map, same session. Nimble-9B (0.748 / 0.759 macro /
+micro) and Jev 1.13.0 (0.760 / 0.773) are in Bespoke's report.
 
-| subset (type) | decider-2b v10 | decider-4b v2 | decider-35b-a3b | Nimble-9B | Jev 1.13.0 |
-|---|---|---|---|---|---|
-| vitaminc-dev (choice) | 0.639 | 0.778 | 0.795 | 0.766 | 0.801 |
-| massive-en-US (choice; trained) | 0.823 | 0.869 | 0.880 | 0.869 | 0.874 |
-| massive-de-DE (choice, German) | 0.797 | 0.837 | 0.869 | 0.834 | 0.869 |
-| boolq (noul; trained) | 0.803 | 0.860 | 0.887 | 0.860 | 0.897 |
-| squad2 (noul) | 0.776 | 0.793 | 0.749 | 0.806 | 0.829 |
-| paws (noul; trained) | 0.720 | 0.832 | 0.768 | 0.828 | 0.892 |
-| multinli (choice; trained) | 0.856 | 0.926 | 0.910 | 0.853 | 0.829 |
-| civil_comments (noul; trained) | 0.840 | 0.857 | 0.907 | 0.703 | 0.810 |
-| aegis2 (noul) | 0.728 | 0.812 | 0.808 | 0.812 | 0.804 |
-| helpsteer2 (score; trained) | 0.426 | 0.466 | 0.478 | 0.390 | 0.341 |
-| summeval-relevance (score) | 0.354 | 0.463 | 0.483 | 0.492 | 0.350 |
-| summeval-consistency (score) | 0.660 | 0.826 | 0.757 | 0.757 | 0.812 |
-| pubmedqa (choice; trained) | 0.724 | 0.724 | 0.768 | 0.756 | 0.772 |
-| **macro / micro** | 0.704 / 0.711 | **0.773 / 0.781** | 0.774 / 0.787 | 0.748 / 0.759 | 0.760 / 0.773 |
-
-On the six subsets whose training split is not in the mixture the macro accuracy is 0.751 (v1 0.731, 2B 0.659, 35B 0.744). v2 is
-1.3 points above Jev 1.13.0 and 2.5 above Nimble-9B on the average (v1: 0.757). It is still behind Jev where a claim has to be
-checked against evidence that nearly matches it (PAWS 0.832 against 0.892, VitaminC 0.778 against 0.801) and on SQuAD2
-answerability (0.793 against 0.829), where it is now 1.7 points above the 2B (v1 was 7 points under it).
-
-**Behaviour probes** (teacher-labelled, same probes as the other releases; v1 in brackets): generic-versus-specific bucket choice
-0.95 / 1.00 (1.00 / 1.00), catch-all when nothing fits 0.95 (0.90; 35B 0.95), abstention battery 8 of 8 (7 of 8); model-router
-tier 0.935 (0.968) and needs-live-data 0.839 (0.871); command-risk classification 0.911 (0.889) with no destructive command
-called safe (35B 0.933), touches-outside-project 0.933 (0.956); browser-agent element and action choice 0.938 / 0.938 (0.875 /
-0.875; 35B 0.938). Scoring a Score level alone against scoring it with its neighbours changes accuracy by at most 2.0 points on
-five rating datasets, and the per-level fits sum to between 0.92 and 1.00 (this probe reads the raw logits, T=1).
+| subset (type) | decider-4b v1 | decider-4b v2 | decider-4b v2.1 |
+|---|---|---|---|
+| vitaminc-dev (choice) | 0.756 | 0.778 | 0.743 |
+| massive-en-US (choice; trained) | 0.860 | 0.869 | 0.863 |
+| massive-de-DE (choice) | 0.843 | 0.837 | 0.840 |
+| boolq (noul; trained) | 0.873 | 0.860 | 0.867 |
+| squad2 (noul) | 0.706 | 0.793 | 0.789 |
+| paws (noul; trained) | 0.716 | 0.832 | 0.764 |
+| multinli (choice; trained) | 0.933 | 0.926 | 0.933 |
+| civil_comments (noul; trained) | 0.880 | 0.857 | 0.873 |
+| aegis2 (noul) | 0.820 | 0.812 | 0.804 |
+| helpsteer2 (score; trained) | 0.466 | 0.466 | 0.450 |
+| summeval-relevance (score) | 0.425 | 0.463 | 0.396 |
+| summeval-consistency (score) | 0.833 | 0.826 | 0.799 |
+| pubmedqa (choice; trained) | 0.728 | 0.724 | 0.708 |
+| **macro / micro** | 0.757 / 0.765 | 0.773 / 0.781 | 0.756 / 0.765 |
+| macro over the subsets not trained on | 0.731 | 0.751 | 0.728 |
+
+v2.1 is at v1's level on this suite and 1.6 points under v2 (macro). The largest differences to v2 are PAWS (0.764 against 0.832),
+SummEval relevance (0.396 against 0.463) and VitaminC (0.743 against 0.778).
+
+**Behaviour probes** (teacher-labelled, same probes as the other releases; v1 / v2 in brackets): generic-versus-specific bucket
+choice 1.00 / 1.00 (1.00 / 1.00; 0.95 / 1.00), catch-all when nothing fits 0.95 (0.90; 0.95), abstention battery 8 of 8 (7 of 8;
+8 of 8); model-router tier 0.968 (0.968; 0.935) and needs-live-data 0.839 (0.871; 0.839); command-risk classification 0.889
+(0.889; 0.911) with no destructive command called safe, touches-outside-project 0.933 (0.956; 0.933); browser-agent element and
+action choice 0.938 / 0.750 (0.875 / 0.875; 0.938 / 0.938). The yes/no Brier score on the probe batteries is 0.010 with the map
+and 0.006 at the global temperature: the yes/no temperature 1.560 flattens easy yes/no answers there.
 
 ## Calibration
 
-The stored temperature 1.935 was fitted on 61 of the 67 in-task regression tasks (see Training). On the regression set v2 is less
-well calibrated than v1 and the 35B (ECE 0.041 / 0.080 against v1's 0.027 / 0.071 and the 35B's 0.026 / 0.069), with per-task
-exceptions: hate-speech tweets 0.349 (in-task), Arena preferences 0.218, QuALITY (full) 0.188, QuALITY 0.179, StrategyQA 0.157,
-Dolly categories 0.142, abstention probe 0.133; the in-task preference tasks HH-RLHF (0.113) and SHP (0.111) are also above 0.1.
-On the five rating datasets of the Score-level probe, which reads the raw logits (T=1) and scores all levels together, the ECE
-is 0.13 to 0.27 (v1 0.05 to 0.09 in the same probe); these sets were not measured at the served temperatures. One temperature
-does not fit every task type. Outside the regression set v2 is better calibrated than v1 on most sets:
+The per-type map was fitted on everyday regression rows and our own validation rows. It helps where it can: yes/no and Score
+answers on the held-out sets. It does not change Choice answers much, because the Choice temperature fitted on a pool that is
+94% everyday rows stays at 1.110. On the held-out generated families, whose answers are two-thirds Choice, v2.1 is
+overconfident: calibration error 0.147 with the map, where our own release limit is 0.08. v2 is at 0.046 there, only because its
+one temperature of 1.935 flattens every answer, which is also what cost it sampled play. ECE below is 10-bin top-label ECE on
+stored temperature-1 logits read at the served temperatures.
 
-| set | measure | decider-2b v10 | decider-4b v1 (read again 2026-09-24) | decider-4b v2 | decider-35b-a3b |
-|---|---|---|---|---|---|
-| JevBench hard tier, 111 items | top-label ECE | 0.304 | 0.288 | 0.071 | 0.151 |
-| TypeSafe, 102 rows | 10-bin ECE / mean total variation to the frontier reference | 0.091 / 0.242 | 0.124 / 0.232 | 0.072 / 0.196 | 0.065 / 0.165 |
-| OpenJev, 5,252 rows | 10-bin ECE | 0.150 | 0.158 | 0.131 | 0.049 |
-| 847 in-task validation rows | 10-bin ECE | | 0.037 | 0.031 | |
-| Mind2Web, 1,770 rows | 10-bin ECE | | 0.016 | 0.051 | |
-| Decision Index 4,000-request sample, 33 benchmarks | benchmark-weighted ECE (the site's definition) | 0.093 | 0.086 | 0.074 | 0.027 |
-| the same | share of all answers given at 95% or more confidence and wrong | 0.010 | 0.022 | 0.012 | 0.005 |
-| the same | mean confidence against accuracy | 0.658 / 0.566 | 0.722 / 0.637 | 0.690 / 0.637 | 0.684 / 0.690 |
-| the same | sample index (about 4 points under a full run) | 42.3 | 48.5 | 48.3 | 50.4 |
-
-The Decision Index rows are a readout of the 4,000-request sample (about 88 cases per benchmark) through the public server,
-computed with the site's definition (benchmark-weighted pooled bins, ten bins; the same computation gives 0.027 on the 35B
-against the site's published 0.031). v2 was read once at the candidate temperature 1.719 (ECE 0.090, sample index 48.3); the
-chosen answers do not change with the temperature, and the calibration rows above are recomputed at 1.935 from the stored
-probabilities. The JevBench hard-tier value is recomputed the same way, except its 6 Score items, which keep their T 1.719
-probabilities. The sample index was not recomputed. Nothing was fitted on index rows. v2's confidence exceeds its accuracy by
-0.053 on that sample (v1 0.085, the 2B 0.092; the 35B has no gap). The gap is per-benchmark heterogeneity, not a global scale: a
-single temperature that removes it on the knowledge benchmarks would make the wide label sets underconfident. If you route on
-confidence, calibrate on your own labels.
+| set | rows | accuracy | ECE at the global T / with the map | choice / noul / score ECE with the map |
+|---|---|---|---|---|
+| heldout_jb | 5,000 | 0.556 | 0.163 / 0.147 | 0.170 / 0.101 / 0.108 |
+| test_teacher2 | 449 | 0.820 | 0.050 / 0.043 | 0.057 / 0.048 / 0.086 |
+| guard | 2,994 | 0.819 | 0.019 / 0.017 | 0.017 / − / − |
+| cal_human | 1,595 | 0.856 | 0.019 / 0.021 | 0.021 / − / − |
+
+On the held-out generated families: v1 (T 1.05) 0.260, v2 (T 1.935) 0.046, v2.1 at T = 1 0.182. A map fitted without the
+regression rows (Choice 1.259) would reach 0.127 there and would make the regression set less calibrated (in-task ECE 0.0375
+against 0.0309); it was not used. On the fixtures (10-bin ECE, map): 847 validation rows 0.035 (v1 0.037, v2 0.031), TypeSafe
+0.063 (v1 0.124, v2 0.072), OpenJev 0.154 (v1 0.158, v2 0.131), Mind2Web 0.017 (v1 0.016, v2 0.051). The Decision Index
+sample was not read for v2.1. If you route on confidence, calibrate on your own labels; `python -m decider.calibrate` fits a
+per-type map from your own answers read at temperature 1.
 
 ## Speed
 
-v2 has the same architecture and size as v1, so it runs at v1's speed. These timings were taken with the same weights in the
-candidate readout session of 2026-09-24 (the server was configured with the candidate temperature 1.719; the temperature does
-not change the amount of computation). In one session on one unshared NVIDIA B300 (bf16), both models measured one after the
-other: 34.6 ms (v1 32.4 ms) median per decision over 200 game-state decisions of 156 tokens median,
-batch of one, eager PyTorch without CUDA graphs or `torch.compile`, timed around the forward pass with `torch.cuda.synchronize()`.
-The eager path is launch-bound, so host load changes it: v1's first measurement was 24.7 ms (10th to 90th percentile 24.6 to
-43.5 ms), with decider-2b at 17.9 ms and decider-35b-a3b at 41.4 ms on the same decisions and method. With the helper's CUDA
-graphs and `torch.compile`, one support-ticket request (228 tokens, 3 questions) takes 5.2 ms (FP8 5.0 ms); a batch of 32 such
-states takes 81.5 ms, 1,178 decisions per second (FP8 71.2 ms, 1,349 per second). The HTTP server (`/decide`, bf16) answers 72.8
-requests per second at a median of 13.3 ms with one client and 190 requests per second with 64 clients.
+v2.1 has the same architecture and size as v1 and v2, and the temperature map is a division per answer; the speed was not
+measured again. The measurements of v1 and v2 (the same weights layout): in one session on one unshared NVIDIA B300 (bf16), 34.6
+ms (v2) and 32.4 ms (v1) median per decision over 200 game-state decisions of 156 tokens median, batch of one, eager PyTorch
+without CUDA graphs or `torch.compile`, timed around the forward pass with `torch.cuda.synchronize()`. The eager path is
+launch-bound, so host load changes it: v1's first measurement was 24.7 ms (10th to 90th percentile 24.6 to 43.5 ms), with
+decider-2b at 17.9 ms and decider-35b-a3b at 41.4 ms on the same decisions and method. With the helper's CUDA graphs and
+`torch.compile`, one support-ticket request (228 tokens, 3 questions) takes 5.2 ms (FP8 5.0 ms); a batch of 32 such states takes
+81.5 ms, 1,178 decisions per second (FP8 71.2 ms, 1,349 per second). The HTTP server (`/decide`, bf16) answers 72.8 requests per
+second at a median of 13.3 ms with one client and 190 requests per second with 64 clients.
 
 ## Limitations
 
-* No reinforcement-learning stage: stated beliefs about action outcomes were not trained against exact laws, and on live
-  browser tasks the model is 10 points below decider-2b v10 on the six held-out tasks in greedy play (81% against 92%, interval
-  includes zero) and 17 points below it in sampled play (75% against 92%).
-* Sampled play is worse than v1's: bag-draw games 37.9% against 56.6% wins, zero-shot games 22.4% against 27.8%, sampled browser
-  play −2.8 points; CliffWalking −60 against −13. Pin revision `v1` if you depend on these (see Changes from v1).
-* Less calibrated than v1 on the regression set (in-task ECE 0.041 against 0.027), on Mind2Web (0.051 against 0.016) and on
-  rating tasks in the Score-level probe (ECE 0.13 to 0.27 on the five rating sets at T=1); still overconfident outside the regression set: OpenJev ECE 0.13,
-  Decision Index sample ECE 0.074 where the 35B reads 0.027. On the Decision Index sample 1.2% of all answers are given at 95% or
-  more confidence and are wrong, against 0.5% for the 35B. Calibration is measured on public datasets, not on your traffic.
-* About 1 point under v1 on everyday tasks (regression set, validation rows, Mind2Web) and lower than v1 on the model-router and
-  command-scope probes.
-* Below the 35B on 85 of 95 regression tasks by 3.1 / 3.2 points and on three fixtures by 1.6 to 5.0 points; level on TypeSafe
-  and on the JevBench hard tier. The largest gaps are on MedQA, MedMCQA, StrategyQA, TruthfulQA and preference judgments.
-* The abstention probe is 2.4 points under the 2B (0.582 against 0.606), with ECE 0.133.
-* The JevBench public hard tier is 111 items; v2's gain there (+12.6 points over v1) is larger than its gain on our own held-out
-  hard sets, where it tied an earlier candidate. Do not read it as a gain of that size on hard items in general.
+* No reinforcement-learning stage: stated beliefs about action outcomes were not trained against exact laws. On live browser
+  tasks v2.1 is at 93.2% sampled and 87.5% on the six held-out tasks; decider-2b v10, which has the RL stage, is at 93.2% and 91.7%.
+* Overconfident on hard multi-step items: calibration error 0.147 on our held-out generated families and 0.184 on the JevBench
+  public hard tier, against v2's 0.046 and 0.104. On those items, do not read a confidence of 0.8 as an 80% chance of being right.
+* Issue #9 form case c_1 is answered wrongly (see Changes from v2); BabyAI-GoTo 0.19 against v1's 0.54; greedy bag-draw play 9.4
+  points under v1.
+* About 0.3 points under v1 on the regression set and 1.1 points under it on the 847 validation rows; needs-live-data and
+  touches-outside-project probes one item under v1 each.
+* The per-type map needs decider-ai 1.4.0 or later. With 1.3.0 or earlier, or with `DECIDER_TEMPERATURE` set, every answer uses
+  1.099: the answers are the same, yes/no answers are sharper (on the held-out generated families their ECE is 0.146 instead of
+  0.101) and Score answers are sharper.
+* Below the 35B by 2.4 / 2.6 points on the regression set and 2.7 on the JevBench public hard tier (3 items).
+* The JevBench public hard tier is 111 items; differences of a few items between versions are within its noise.
 * The stage-2 LoRA was trained only in the plain state-first layout. The schema-first layout (the server's opt-in schema cache)
-  was not measured on v2, and `decider_config.json` does not mark v2 as trained for it (`schema_first_trained: false`), so
-  `DECIDER_SCHEMA_CACHE=1` does not turn the schema cache on for this model.
+  was not measured on v2.1, and `decider_config.json` does not mark it as trained for that layout (`schema_first_trained:
+  false`), so `DECIDER_SCHEMA_CACHE=1` does not turn the schema cache on for this model.
 * Mixture v2's 26 additional public datasets and ten programmatic families, and stage 2's generators and document questions, are
   described above but their builders are not in the public package; `scripts/train.sh full` reproduces the 60% of stage 1's data
-  that is the public mixture. On the held-out variants of mixture v2's programmatic families v1 was at 0.635 mean accuracy (logs
-  0.28, plans 0.44, probability 0.43, code 0.88, policy 0.99); this was not measured on v2.
+  that is the public mixture.
 * English is the main language; the multilingual rows (XNLI, PAWS-X, MASSIVE, Belebele, XCOPA) are a small share of the data
-  and were not measured beyond the mixture-v2 evaluation set.
+  and were not measured beyond the mixture-v2 evaluation set and Bespoke's German MASSIVE subset.
 * Everything else in the decider-2b card's limitations (packed questions see each other, long JSON arrays by position, full
   label sets against sampled options, abstention wording, rules in the question) applies; those shapes were not re-measured at
   this size.
@@ -453,7 +415,8 @@ requests per second at a median of 13.3 ms with one client and 190 requests per
 
 | version | what changed |
 |---|---|
-| **v2** (2026-09-24, these weights) | v1 + a merged LoRA (rank 64, attention and MLP, 2 epochs, 29,356 rows: generated decision families, document questions written by Qwen3.6-27B and kept when two independent answers agreed, human-labelled public sets, replay of mixture v2); temperature 1.935; plain layout, checked with decider-ai 1.0.2, 1.1.4, 1.2.1 and 1.2.2. Better on hard judgments, TypeSafe, OpenJev and Bespoke's suite; worse on sampled play, some text games and by about 1 point on everyday tasks |
+| **v2.1** (2026-09-24, these weights) | v1 + a merged LoRA on v2's rows (29,325 after removing 31 that overlapped evaluation files), with the replay rows trained toward v1's own answer distribution; temperature 1.099 and `temperature_by_type` {choice 1.110, noul 1.560, score 1.287} (decider-ai 1.4.0; older versions use 1.099). Against v2: sampled play recovered (bag-draw 52.0% against 37.9%, browser 93.2% against 88.1%), CliffWalking −13 against −60, regression set +0.7 / +0.5; hard sets at v2's level; less calibrated on hard items (0.147 against 0.046); issue #9 c_1 still wrong. Did not pass its pre-registered rules; released on the full comparison |
+| v2 (2026-09-24, Hub tag `v2`) | v1 + a merged LoRA (rank 64, attention and MLP, 2 epochs, 29,356 rows: generated decision families, document questions written by Qwen3.6-27B and kept when two independent answers agreed, human-labelled public sets, replay of mixture v2); temperature 1.935. Better on hard judgments, TypeSafe, OpenJev and Bespoke's suite; worse on sampled play, some text games and by about 1 point on everyday tasks |
 | v1 (2026-09-22, Hub tag `v1`) | first release: one pass over mixture v2 on Qwen3.5-4B-Base with AdamW on bf16 parameters, no RL stage; temperature 1.05 |
 
 The GitHub repository's [docs/CHANGELOG.md](https://github.com/Mapika/decider/blob/main/docs/CHANGELOG.md) lists every
@@ -464,10 +427,10 @@ decider release.
 Code, data registry, training and evaluation scripts and the per-version history: https://github.com/Mapika/decider
 (`docs/HISTORY.md`, section "decider-4b"). Stage 1 was trained with the data-parallel trainer of the architecture A/B study
 (`arch_ab/train_dp_optvar.py` in the research repository, optimizer variant `bf16`); stage 2 with a LoRA trainer in the research
-repository. Both were evaluated with the public `decider.evaluate` and the head-to-head tools, and uploaded with
-`scripts/upload_hf.py`. `eval_results.json` in this repository has the per-task regression metrics at T 1.935, the fixtures, JevBench, Bespoke's
-suite, the games, the browser, the text games, the Decision Index calibration, the behaviour probes and speed, and the paired
-comparisons with v1 (regression set, fixtures, JevBench, games, browser).
+repository. Both were evaluated with the public `decider.evaluate` and the head-to-head tools. `eval_results.json` in this
+repository has the regression metrics with the map and at the global temperature, our held-out sets by answer type, the
+fixtures, games, browser and Bespoke results with the paired comparisons against v1 and v2, the text games, the behaviour
+probes, the issue #9 cases and the JevBench public items.
 
 **Independence.** This is an independent project. It is not affiliated with or endorsed by TypeSafe AI. It is an open
 reproduction of the "System One" model class (TypeSafe AI's Jev); nothing was distilled from Jev. The training data is public

decider/calibrate.py

@@ -0,0 +1,222 @@
+"""Fit decider_config.json "temperature_by_type" by NLL, one temperature per answer type (decider.temperature).
+
+    python -m decider.calibrate records.jsonl [--min-rows 50]
+    -> {"temperature_by_type": {"choice": 1.48, "noul": 2.22, "score": 1.38}, "temperature": 1.6, "rows": {...}, "nll": {...}}
+
+A record is one answer with its gold, read at temperature 1:
+    {"type": "choice" | "noul" | "score", "logits": [one value per option], "gold": option index}
+    {"type": "score", "level_logits": [[no, yes], one pair per level], "gold": level index}      Score with isolated levels
+"probs" / "level_probs" (probabilities at temperature 1) may stand in for "logits" / "level_logits": log p is the logit up to a
+constant, which the softmax ignores.  A noul gold is 1 for true, 0 for false.  Every record is checked first (check_record):
+a malformed one (gold out of range or not an integer, wrong shape, NaN, a row without a finite logit, an isolated Score whose
+levels all have P(yes) = 0) raises ValueError naming its position.
+
+An isolated-levels Score answer is fitted through the readout the server uses: every level row is softmax(row / T), and the
+answer is P(yes) of each level divided by their sum (decider.systemone.combine_isolated).  So the "score" temperature is fitted on
+Score answers whichever readout the model serves them with; fit it on records collected with the same isolated_levels setting.
+
+`collect(decider, examples)` produces records from a decider.infer.Decider and labelled /v1/systemone-shaped examples.
+"temperature" in the output is one temperature fitted on all records together, for comparison; a type with fewer than
+--min-rows records is left out of the map (it then uses "temperature" of the config).
+"""
+import json, math, sys
+import numpy as np
+
+from decider.temperature import TYPES
+
+GRID = np.exp(np.linspace(math.log(0.05), math.log(20.0), 801))       # 0.05 .. 20, about 0.75 % apart
+
+
+def _log(x):
+    x = np.asarray(x, dtype=np.float64)
+    with np.errstate(divide="ignore"):
+        return np.where(x > 0, np.log(np.clip(x, 1e-300, None)), -np.inf)
+
+
+def _logits(rec, key):
+    if key in rec:
+        return np.asarray(rec[key], dtype=np.float64)
+    alt = {"logits": "probs", "level_logits": "level_probs"}[key]
+    return _log(rec[alt])
+
+
+def _lse(z, axis=-1):
+    m = np.max(z, axis=axis, keepdims=True)
+    m = np.where(np.isfinite(m), m, 0.0)
+    with np.errstate(divide="ignore"):                              # an all -inf row gives -inf, handled by the callers
+        return (m + np.log(np.sum(np.exp(z - m), axis=axis, keepdims=True))).squeeze(axis)
+
+
+def nll(rec, T):
+    """Negative log-likelihood of one record's gold at temperature T."""
+    return float(_Batch([rec]).nll(T))
+
+
+def _isolated(rec):
+    return "level_logits" in rec or "level_probs" in rec
+
+
+def check_record(rec, i=None):
+    """Raise ValueError unless `rec` is a well-formed record (module docstring): an integral gold within the options or levels,
+    a 1-D row of at least two options or an [levels, 2] array of at least two levels, logits that are not NaN or +inf,
+    probabilities in [0, 1] with positive mass, and isolated levels only on a Score record."""
+    where = f"record {i}" if i is not None else "record"
+    if not isinstance(rec, dict):
+        raise ValueError(f"{where}: not a JSON object")
+    if rec.get("type") not in TYPES:
+        raise ValueError(f"{where}: type {rec.get('type')!r}, expected one of {', '.join(TYPES)}")
+    iso = _isolated(rec)
+    keys = ("level_logits", "level_probs") if iso else ("logits", "probs")
+    if sum(k in rec for k in keys) != 1 or (iso and ("logits" in rec or "probs" in rec)):
+        raise ValueError(f"{where}: give exactly one of logits, probs, level_logits, level_probs")
+    if iso and rec["type"] != "score":
+        raise ValueError(f"{where}: isolated levels (level_logits / level_probs) belong to a score record, not {rec['type']!r}")
+    key = keys[0] if keys[0] in rec else keys[1]
+    try:
+        a = np.asarray(rec[key], dtype=np.float64)
+    except (TypeError, ValueError):
+        raise ValueError(f"{where}: {key} is not a numeric array") from None
+    if iso and (a.ndim != 2 or a.shape[1] != 2 or a.shape[0] < 2):
+        raise ValueError(f"{where}: {key} must be one [no, yes] pair per level, at least two levels; got shape {a.shape}")
+    if not iso and (a.ndim != 1 or a.shape[0] < 2):
+        raise ValueError(f"{where}: {key} must be one value per option, at least two options; got shape {a.shape}")
+    if key.endswith("probs"):
+        if not np.all(np.isfinite(a)) or np.any(a < 0) or np.any(a > 1) or np.any(a.sum(-1) <= 0):
+            raise ValueError(f"{where}: {key} must be probabilities in [0, 1] with positive mass per row")
+    elif np.any(np.isnan(a)) or np.any(a == np.inf) or not np.all(np.any(np.isfinite(a), axis=-1)):
+        raise ValueError(f"{where}: {key} contains NaN or +inf, or a row without a finite value")
+    if rec["type"] == "noul" and a.shape[0] != 2:
+        raise ValueError(f"{where}: a noul record has exactly two options (false, true); got {a.shape[0]}")
+    if iso and not np.any(a[:, 1] > 0 if key == "level_probs" else np.isfinite(a[:, 1])):
+        raise ValueError(f"{where}: no level has a yes probability above 0, so the served Score answer has no distribution")
+    g = rec.get("gold")
+    if isinstance(g, bool) or not isinstance(g, (int, np.integer)) or not 0 <= g < a.shape[0]:
+        raise ValueError(f"{where}: gold must be an integer index in 0..{a.shape[0] - 1}, got {g!r}")
+    return rec
+
+
+class _Batch:
+    """Records packed into padded arrays, so one temperature is scored over all of them at once."""
+    def __init__(self, records):
+        records = [check_record(r, i) for i, r in enumerate(records)]
+        lists = [r for r in records if not _isolated(r)]
+        isos = [r for r in records if _isolated(r)]
+        self.n = len(lists) + len(isos)
+        self.L = self.Lg = self.I = self.Im = self.Ig = None
+        if lists:
+            zs = [_logits(r, "logits") for r in lists]
+            K = max(len(z) for z in zs)
+            self.L = np.full((len(zs), K), -np.inf)
+            for i, z in enumerate(zs):
+                self.L[i, :len(z)] = z
+            self.Lg = np.array([int(r["gold"]) for r in lists])
+        if isos:
+            zs = [_logits(r, "level_logits").reshape(-1, 2) for r in isos]
+            n = max(len(z) for z in zs)
+            self.I = np.zeros((len(zs), n, 2)); self.Im = np.zeros((len(zs), n), dtype=bool)
+            for i, z in enumerate(zs):
+                self.I[i, :len(z)] = z; self.Im[i, :len(z)] = True
+            self.Ig = np.array([int(r["gold"]) for r in isos])
+
+    def nll(self, T):
+        """Sum of the records' NLL at temperature T."""
+        tot = 0.0
+        if self.L is not None:
+            z = self.L / T
+            zg = z[np.arange(len(z)), self.Lg]
+            tot += float(np.sum(np.where(np.isfinite(zg), _lse(z) - zg, 690.0)))
+        if self.I is not None:                          # in log space: P(yes) of very confident rows underflows otherwise
+            z = self.I / T
+            with np.errstate(invalid="ignore"):
+                lp = np.where(self.Im, z[..., 1] - _lse(z), -np.inf)          # log P(yes) per level, -inf on padding
+            norm = _lse(lp)                                                  # log of the summed P(yes)
+            lg = lp[np.arange(len(z)), self.Ig]
+            with np.errstate(invalid="ignore"):
+                per = np.where(np.isfinite(lg), norm - lg, 690.0)               # check_record guarantees a finite norm
+            tot += float(np.sum(per))
+        return tot
+
+
+def mean_nll(records, T):
+    b = records if isinstance(records, _Batch) else _Batch(records)
+    return b.nll(T) / b.n
+
+
+def fit(records, grid=GRID):
+    """The temperature with the lowest mean NLL on the grid, refined by a golden-section search between its neighbours."""
+    records = records if isinstance(records, _Batch) else _Batch(list(records))
+    if not records.n:
+        raise ValueError("no records to fit")
+    vals = [mean_nll(records, T) for T in grid]
+    i = int(np.argmin(vals))
+    lo, hi = math.log(grid[max(i - 1, 0)]), math.log(grid[min(i + 1, len(grid) - 1)])
+    r = (math.sqrt(5) - 1) / 2
+    a, b = hi - r * (hi - lo), lo + r * (hi - lo)
+    fa, fb = mean_nll(records, math.exp(a)), mean_nll(records, math.exp(b))
+    for _ in range(40):
+        if fa < fb:
+            hi, b, fb = b, a, fa; a = hi - r * (hi - lo); fa = mean_nll(records, math.exp(a))
+        else:
+            lo, a, fa = a, b, fb; b = lo + r * (hi - lo); fb = mean_nll(records, math.exp(b))
+    T = math.exp((lo + hi) / 2)
+    return T if mean_nll(records, T) <= vals[i] else float(grid[i])
+
+
+def fit_by_type(records, min_rows=50):
+    """-> {"temperature_by_type": {type: T}, "temperature": pooled T, "rows": {type: n}, "nll": {type: {"T=1": .., "fitted": ..}}}"""
+    records = [check_record(r, i) for i, r in enumerate(records)]
+    groups = {t: [] for t in TYPES}
+    for r in records:
+        groups[r["type"]].append(r)
+    out = {"temperature_by_type": {}, "temperature": round(fit(records), 3) if records else None,
+           "rows": {t: len(g) for t, g in groups.items()}, "nll": {}}
+    for t, g in groups.items():
+        if len(g) < max(1, min_rows):
+            continue
+        b = _Batch(g); T = fit(b)
+        out["temperature_by_type"][t] = round(T, 3)
+        out["nll"][t] = {"T=1": round(mean_nll(b, 1.0), 4), "fitted": round(mean_nll(b, T), 4)}
+    return out
+
+
+def collect(decider, examples, independent=True):
+    """Records for fit_by_type from a decider.infer.Decider, read at temperature 1 on the uncached state-first path.
+    examples: iterable of (state, questions, golds) with questions as /v1/systemone takes them and golds {question id: gold},
+    the gold being a Choice option name, a Noul true/false, or a Score level index.  Questions without a gold are skipped.
+    Score questions are read with the Decider's isolated_levels setting, as system_one serves them."""
+    recs = []
+    for state, questions, golds in examples:
+        rqs, index, items = decider._system_one_items(state, questions, independent, layout="state_first")
+        rows = decider._system_one_probs(items, "state_first", temperature=1.0)
+        for k, kind, s, n in index:
+            if k not in golds:
+                continue
+            rq, g = rqs[k], golds[k]
+            if rq["type"] == "noul" and not (isinstance(g, bool) or g in (0, 1)):
+                raise ValueError(f"gold of noul question {k!r} must be true or false, got {g!r}")
+            if rq["type"] == "score" and (isinstance(g, bool) or not isinstance(g, int) or not 0 <= g < len(rq["names"])):
+                raise ValueError(f"gold of score question {k!r} must be a level index in 0..{len(rq['names']) - 1}, got {g!r}")
+            key = bool(g) if rq["type"] == "noul" else g
+            if key not in rq["names"]:
+                raise ValueError(f"gold of choice question {k!r} is not one of its options: {g!r}")
+            gold = rq["names"].index(key)
+            if kind == "iso":
+                recs.append({"type": rq["type"], "level_probs": [rows[s + j][:2] for j in range(n)], "gold": gold})
+            else:
+                recs.append({"type": rq["type"], "probs": rows[s][:len(rq["options"])], "gold": gold})
+    return recs
+
+
+def main(argv=None):
+    import argparse
+    ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
+    ap.add_argument("records", help="JSON lines, one record per line (see the module docstring)")
+    ap.add_argument("--min-rows", type=int, default=50, help="fit a type only when it has at least this many records")
+    a = ap.parse_args(argv)
+    with open(a.records) as f:
+        recs = [json.loads(line) for line in f if line.strip()]
+    print(json.dumps(fit_by_type(recs, a.min_rows), indent=1))
+
+
+if __name__ == "__main__":
+    main(sys.argv[1:])

decider/engine.py

@@ -7,6 +7,7 @@ option-letter logits for all positions [B, T, K]; slots are gathered outside.
 import time, torch, torch._dynamo, torch.nn.functional as F
 from decider.model import DecisionModel, collate
 from decider.prompt import build, MAX_OPTIONS
+from decider.temperature import scaled_softmax, slot_temperatures
 
 T_BUCKETS = [64, 128, 192, 256, 320, 384, 512, 640, 768, 1024, 1280, 1536, 2048]
 B_BUCKETS = [1, 2, 4, 8, 16, 32, 64]
@@ -46,11 +47,12 @@ def patch_conv():
 
 def read_slots(out, rows, slots, nopts, temperature, n_per_item):
     """One gather + one softmax + one device-to-host copy for the whole batch (was: three small kernels and a sync per item).
-    out [B, T, K] logits; rows/slots/nopts: flat python lists, one entry per question; n_per_item: questions per item."""
+    out [B, T, K] logits; rows/slots/nopts: flat python lists, one entry per question; n_per_item: questions per item.
+    temperature: a number for every question, or a flat list with one temperature per question (decider.temperature)."""
     dev = out.device; idx = torch.tensor([rows, slots, nopts], dtype=torch.long).to(dev, non_blocking=True)
     lg = out[idx[0], idx[1]]                                                                  # [N, K]
     lg = lg.masked_fill(torch.arange(lg.shape[1], device=dev)[None, :] >= idx[2][:, None], float("-inf"))
-    p = torch.softmax(lg / temperature, -1).cpu()
+    p = scaled_softmax(lg, temperature).cpu()
     return list(torch.split(p, n_per_item))
 
 
@@ -140,14 +142,15 @@ class Engine:
 
     @torch.no_grad()
     def score_items(self, items, temperature=1.0):
-        """items: list of dicts from prompt.build. Returns list of [n_q, MAX_OPTIONS] prob tensors (cpu)."""
+        """items: list of dicts from prompt.build. Returns list of [n_q, MAX_OPTIONS] prob tensors (cpu).
+        temperature: a number, or one entry per item (a number or one number per slot; decider.temperature.for_items)."""
         Tmax = max(len(it["ids"]) for it in items)
         T = _bucket(Tmax, T_BUCKETS) or -(-Tmax // LONG_STEP) * LONG_STEP
         B = (_bucket(len(items), B_BUCKETS) or len(items)) if T <= GRAPH_MAX_T else len(items)
         ids = fill_ids([it["ids"] for it in items], B, T, self.tok.pad_token_id)
         out = self.logits_all(ids.to(self.dev, non_blocking=True))
         return read_slots(out, [b for b, it in enumerate(items) for _ in it["slots"]], [s for it in items for s in it["slots"]],
-                          [n for it in items for n in it["nopts"]], temperature, [len(it["slots"]) for it in items])
+                          [n for it in items for n in it["nopts"]], slot_temperatures(temperature, items), [len(it["slots"]) for it in items])
 
     @torch.no_grad()
     def score_shared(self, items, temperature=1.0, min_prefix=192):

decider/engine_v2.py

@@ -19,6 +19,7 @@ import time, torch, torch.nn.functional as F
 from decider import shared_prefix
 from decider.engine import read_slots, fill_ids, patch_conv, set_attention_backend_policy
 from decider.model import DecisionModel
+from decider.temperature import item_slice, slot_temperatures
 
 T_BUCKETS = [64, 128, 192, 256, 320, 384, 512, 640, 768, 1024, 1280, 1536, 2048, 3072, 4096, 6144, 8192]
 B_BUCKETS = [1, 2, 4, 8, 16, 32]
@@ -150,9 +151,11 @@ class EngineV2:
     # ---- scoring -----------------------------------------------------------
     @torch.no_grad()
     def score_items(self, items, temperature=1.0):
-        """items: dicts from prompt.build / build_rows.  -> one [n_q, MAX_OPTIONS] cpu probability tensor per item."""
+        """items: dicts from prompt.build / build_rows.  -> one [n_q, MAX_OPTIONS] cpu probability tensor per item.
+        temperature: a number, or one entry per item (a number or one number per slot; decider.temperature.for_items)."""
         if not items:
             return []
+        slot_temperatures(temperature, items)                      # a length mismatch fails before any forward
         Tmax = max(len(it["ids"]) for it in items)
         T = self.t_bucket(Tmax)
         if T is None:
@@ -168,7 +171,7 @@ class EngineV2:
             lg = self.logits_all(ids.to(self.dev, non_blocking=True))
             out += read_slots(lg, [b for b, it in enumerate(chunk) for _ in it["slots"]],
                               [s for it in chunk for s in it["slots"]], [n for it in chunk for n in it["nopts"]],
-                              temperature, [len(it["slots"]) for it in chunk])
+                              slot_temperatures(item_slice(temperature, i - len(chunk), i), chunk), [len(it["slots"]) for it in chunk])
         return out
 
     @torch.no_grad()

decider/infer.py

@@ -12,6 +12,7 @@ import logging
 import torch
 from decider.model import DecisionModel, collate
 from decider.prompt import build, MAX_OPTIONS, resolve_layout, chat_template
+from decider import temperature as TT
 from dataclasses import dataclass
 
 
@@ -49,11 +50,15 @@ class _NoShuffle:                              # keep option order as given
 
 
 class CompiledSchema:
-    def __init__(self, d, rqs, h, index): self.d, self.rqs, self.h, self.index = d, rqs, h, index
+    def __init__(self, d, rqs, h, index):
+        from decider.systemone import row_types
+        self.d, self.rqs, self.h, self.index = d, rqs, h, index
+        self.types = row_types(rqs, index)                         # answer type of every schema row (decider.temperature)
 
     def batch(self, states, max_state_tokens=32768):
         from decider.systemone import render_state, assemble
-        probs = self.d._se.score(self.h, [render_state(s) for s in states], temperature=self.d.T_schema, max_ctx_tokens=max_state_tokens)
+        T = TT.for_types(self.d.T_schema, self.d.T_schema_by_type, self.types)
+        probs = self.d._se.score(self.h, [render_state(s) for s in states], temperature=T, max_ctx_tokens=max_state_tokens)
         return [{"model": self.d.name, "answers": assemble(self.rqs, self.index, [p.tolist() for p in pr])} for pr in probs]
 
     def __call__(self, state, max_state_tokens=32768):
@@ -67,10 +72,15 @@ class Decider:
     the optional MPS patch; CPU defaults to bfloat16. Set ``use_graphs=False``
     for eager execution or debugging.
     """
-    def __init__(self, path, device=None, dtype=None, temperature=None, abstain_below=0.0, use_graphs=None):
+    def __init__(self, path, device=None, dtype=None, temperature=None, abstain_below=0.0, use_graphs=None, temperature_by_type=None):
         """The prompt layout comes from decider_config.json: "layout": "chat" (chat-trained checkpoints) wraps every prompt in the
         tokenizer's chat template (decider.prompt.build_chat); no "layout" key is the plain layout of every earlier model.
-        An unknown layout raises ValueError before the weights are loaded."""
+        An unknown layout raises ValueError before the weights are loaded.
+
+        Temperatures (decider.temperature): decider_config.json "temperature" and the optional "temperature_by_type"
+        {"choice": T, "noul": T, "score": T}.  temperature= overrides "temperature" and switches the config's by-type map off;
+        temperature_by_type= sets the map explicitly.  An invalid temperature or map key raises ValueError before the weights
+        are loaded."""
         if device is None:
             device = "cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu")
         if dtype is None:
@@ -85,8 +95,7 @@ class Decider:
         except Exception:
             pass
         self.layout = resolve_layout(cfg)
-        if temperature is None:
-            temperature = float(cfg.get("temperature", 1.0))
+        (temperature, self.T_by_type), (T_schema, self.T_schema_by_type) = TT.from_config(cfg, temperature, temperature_by_type)
         self.neutralize_none = bool(cfg.get("neutralize_none", True))   # v4 and earlier learned the literal string as an abstain signal
         if use_graphs is None:
             use_graphs = str(device).startswith("cuda")
@@ -102,7 +111,7 @@ class Decider:
         self.dev = device; self.T = temperature; self.abstain_below = abstain_below
         self.name = "decider-" + str(cfg.get("version", "dev"))
         self.schema_first = bool(cfg.get("schema_first", False)) and self.eng is not None      # default layout. Questions-first (the cacheable one) costs accuracy
-        self.T_schema = float(cfg.get("temperature_schema_first", temperature))                # (about 1.5 points on fixed label sets, more elsewhere): opt in with schema()
+        self.T_schema = T_schema                                              # (about 1.5 points on fixed label sets, more elsewhere): opt in with schema()
         self.isolated_levels = bool(cfg.get("isolated_levels", False))      # Score levels judged one per row (v8+)
         self._se = None; self._schemas = {}
 
@@ -116,19 +125,22 @@ class Decider:
                 assert 2 <= len(q["options"]) <= MAX_OPTIONS, f"2..{MAX_OPTIONS} options required"
             exs.append(Example(context, [Q(q["question"], list(q["options"]), 0) for q in qs], "infer"))
         items = [build(e, self.m.tok, _NoShuffle(), max_options=MAX_OPTIONS, max_ctx_tokens=max_ctx_tokens, chat=self.chat) for e in exs]
+        for it, (_, qs) in zip(items, requests):                     # answer type per slot: a plain question is "choice"
+            it["types"] = [q.get("_type", "choice") for q in qs]
         return requests, items
 
     @torch.no_grad()
     def decide_batch(self, requests, max_ctx_tokens=1536):
         """requests: list of (context:str, questions:list[dict(question, options)]). One forward pass for everything."""
         requests, items = self._decide_items(requests, max_ctx_tokens)
+        T = TT.for_items(self.T, self.T_by_type, items)                  # the scalar self.T when there is no by-type map
         if self.eng is not None:
-            probs = torch.cat(self.eng.score_items(items, temperature=self.T))
+            probs = torch.cat(self.eng.score_items(items, temperature=T))
         else:
             b = collate(items, self.m.tok.pad_token_id)
             logits = self.m.slot_logits(b["input_ids"].to(self.dev), b["attention_mask"].to(self.dev), b["slot_idx"].to(self.dev),
                                         b["slot_batch"].to(self.dev), b["nopts"].to(self.dev))
-            probs = torch.softmax(logits / self.T, -1).cpu()
+            probs = TT.scaled_softmax(logits, TT.slot_temperatures(T, items)).cpu()
         out, k = [], 0
         for context, qs in requests:
             res = []
@@ -176,27 +188,33 @@ class Decider:
         (later questions can then see earlier question texts)."""
         from decider.systemone import unique_tokens, assemble
         rqs, index, items = self._system_one_items(state, questions, independent, max_state_tokens, layout, isolated)
-        with torch.no_grad():
-            if self.eng is not None and len(items) > 1 and layout == "state_first":
-                probs = self.eng.score_shared(items, temperature=self.T)
-            else:
-                probs = []; per = max(1, max_fwd_tokens // max(len(it["ids"]) for it in items))
-                for i in range(0, len(items), per):
-                    if self.eng is not None:
-                        probs += self.eng.score_items(items[i:i + per], temperature=self.T)
-                    else:
-                        bt = collate(items[i:i + per], self.m.tok.pad_token_id)
-                        lg = self.m.slot_logits(*[bt[k].to(self.dev) for k in ("input_ids", "attention_mask", "slot_idx", "slot_batch", "nopts")])
-                        pr = torch.softmax(lg / self.T, -1).cpu(); c = 0
-                        for it in items[i:i + per]:
-                            probs.append(pr[c:c + len(it["slots"])]); c += len(it["slots"])
-        flatp = [p.tolist() for ps in probs for p in ps]
+        flatp = self._system_one_probs(items, layout, max_fwd_tokens, TT.for_items(self.T, self.T_by_type, items))
         return {"model": self.name, "answers": assemble(rqs, index, flatp),
                 "usage": {"input_tokens": unique_tokens(items), "output_tokens": 0}}
 
+    @torch.no_grad()
+    def _system_one_probs(self, items, layout, max_fwd_tokens=65536, temperature=None):
+        """Score system_one's uncached rows.  temperature: as Engine.score_items takes it (a number, or one entry per item).
+        -> one probability list per question row, in row order."""
+        if self.eng is not None and len(items) > 1 and layout == "state_first":
+            probs = self.eng.score_shared(items, temperature=temperature)
+        else:
+            probs = []; per = max(1, max_fwd_tokens // max(len(it["ids"]) for it in items))
+            for i in range(0, len(items), per):
+                T = TT.item_slice(temperature, i, i + per)
+                if self.eng is not None:
+                    probs += self.eng.score_items(items[i:i + per], temperature=T)
+                else:
+                    bt = collate(items[i:i + per], self.m.tok.pad_token_id)
+                    lg = self.m.slot_logits(*[bt[k].to(self.dev) for k in ("input_ids", "attention_mask", "slot_idx", "slot_batch", "nopts")])
+                    pr = TT.scaled_softmax(lg, TT.slot_temperatures(T, items[i:i + per])).cpu(); c = 0
+                    for it in items[i:i + per]:
+                        probs.append(pr[c:c + len(it["slots"])]); c += len(it["slots"])
+        return [p.tolist() for ps in probs for p in ps]
+
     def _system_one_items(self, state, questions, independent=True, max_state_tokens=32768, layout=None, isolated=None):
         """The prompt rows of system_one's uncached path: -> (rendered questions, answer index, items)."""
-        from decider.systemone import render_state, render_question, plan_rows
+        from decider.systemone import render_state, render_question, plan_rows, row_types
         layout = layout or ("schema_first" if self.schema_first else "state_first")
         isolated = (self.isolated_levels if isolated is None else isolated) and independent
         ctx = render_state(state); rqs = {k: render_question(v) for k, v in questions.items()}
@@ -205,6 +223,9 @@ class Decider:
         rows = [[r] for r in flat] if independent else [flat]
         items = [build(Example(ctx, [Q(r["question"], opts(r), 0) for r in row]), self.m.tok, _NoShuffle(), max_options=MAX_OPTIONS,
                        max_ctx_tokens=max_state_tokens, layout=layout, chat=self.chat) for row in rows]
+        types = row_types(rqs, index)                                 # answer type per row (decider.temperature)
+        for it, ts in zip(items, [[t] for t in types] if independent else [types]):
+            it["types"] = ts
         return rqs, index, items
 
     # ---- typed schema interface: {question: {"type": "bool"} | {"type": "choice", "options": [...]}
@@ -257,14 +278,14 @@ class Decider:
         for qtext, spec in schema.items():
             t = spec.get("type", "choice")
             if t == "bool":
-                qs.append(dict(question=qtext, options=["no", "yes"]))
+                qs.append(dict(question=qtext, options=["no", "yes"], _type="noul"))
             elif t == "choice":
-                qs.append(dict(question=qtext, options=list(spec["options"])))
+                qs.append(dict(question=qtext, options=list(spec["options"]), _type="choice"))
             elif t == "scale":
                 leg = spec["legend"]
                 keys = sorted(leg, key=lambda k: float(k)) if isinstance(leg, dict) else list(range(len(leg)))
                 labels = [f"{k}: {leg[k]}" if isinstance(leg, dict) else f"{i}: {leg[i]}" for i, k in enumerate(keys)]
-                qs.append(dict(question=qtext, options=labels, _keys=keys, _legend=leg))
+                qs.append(dict(question=qtext, options=labels, _keys=keys, _legend=leg, _type="score"))
             else:
                 raise ValueError(f"unknown field type {t}")
         return qs

decider/schema_engine.py

@@ -124,12 +124,15 @@ class SchemaEngine:
         return next((t for t in TS_BUCKETS if t >= n_tokens), -(-n_tokens // 256) * 256)
 
     def score(self, h, contexts, temperature=1.0, max_ctx_tokens=1536):
-        """-> one [n_questions, MAX_OPTIONS] probability tensor per context."""
+        """-> one [n_questions, MAX_OPTIONS] probability tensor per context.  temperature: as in score_rows."""
         return self.score_rows(h, [self.tokenize(h, c, max_ctx_tokens) for c in contexts], temperature)
 
     @torch.no_grad()
     def score_rows(self, h, rows, temperature=1.0):
-        """rows: [(suffix ids, slots)] from tokenize()."""
+        """rows: [(suffix ids, slots)] from tokenize().  temperature: a number, or a list with one temperature per schema row
+        (h.nq values, in the order of prepare's questions), applied to every request."""
+        if isinstance(temperature, (list, tuple)) and len(temperature) != h.nq:
+            raise ValueError(f"temperature: {len(temperature)} values for a schema with {h.nq} rows")
         Tmax = max(len(r[0]) for r in rows); Ts = next((t for t in TS_BUCKETS if t >= Tmax), None); n = len(rows)
         R = next((b for b in B_BUCKETS if b >= n), n) if Ts else n; Ts = Ts or -(-Tmax // 256) * 256
         ids = fill_ids([x for x, _ in rows for _ in range(h.P)], R * h.P, Ts, self.tok.pad_token_id).to(self.dev, non_blocking=True)
@@ -143,4 +146,5 @@ class SchemaEngine:
             rws = [r for r in range(n) for _ in range(h.nq)]; sls = [x for _, sl in rows for x in sl]
         else:                                                  # independent: one slot in each of the request's P rows
             rws = [r * h.P + p for r in range(n) for p in range(h.P)]; sls = [sl[0] for _, sl in rows for _ in range(h.P)]
-        return read_slots(out, rws, sls, h.nopts * n, temperature, [h.nq] * n)
+        temps = list(temperature) * n if isinstance(temperature, (list, tuple)) else temperature
+        return read_slots(out, rws, sls, h.nopts * n, temps, [h.nq] * n)

decider/serve.py

@@ -40,6 +40,12 @@ Variables (default):  DECIDER_DEVICE (auto: cuda, else mps, else cpu)  DECIDER_C
   DECIDER_T_BUCKETS  DECIDER_B_BUCKETS  DECIDER_GRAPH_TOKEN_BUDGET (32768)  DECIDER_WARMUP (1)  DECIDER_TOKENIZE_THREADS (8)
   DECIDER_MAX_ROWS (1024)  DECIDER_MAX_ROW_TOKENS (DECIDER_MAX_STATE_TOKENS + 4096)  DECIDER_MAX_REQUEST_TOKENS (1048576)
   DECIDER_MAX_QUEUE_ROWS (4096)  DECIDER_TEMPERATURE  DECIDER_SCHEMA_CACHE (0)  DECIDER_SCHEMA_MIN_SEEN (2)  DECIDER_SCHEMAS
+
+Temperatures (1.4.0, decider.temperature): decider_config.json "temperature", and optionally "temperature_by_type"
+{"choice": T, "noul": T, "score": T} (a /decide "bool" field is "noul", a "scale" field is "score"; a missing type uses
+"temperature").  Every row carries the answer type of each of its slots, so a batch that mixes requests and types still
+applies each answer's own temperature.  DECIDER_TEMPERATURE replaces "temperature" and switches the by-type map off.
+/health and the ready line report the temperature every answer type gets.
 """
 import asyncio, json, os, time
 from concurrent.futures import ThreadPoolExecutor
@@ -47,6 +53,7 @@ from contextlib import asynccontextmanager
 from fastapi import FastAPI, HTTPException
 from pydantic import BaseModel
 from decider import systemone as S1
+from decider import temperature as TT
 from decider.batching import DEFAULT_MERGE_OVERHEAD_TOKENS, plan_batches
 from decider.prompt import build, MAX_OPTIONS, resolve_layout, chat_template
 from decider.prompt_fast import build_rows, unique_tokens
@@ -78,7 +85,7 @@ MAX_REQUEST_TOKENS = _env_int("DECIDER_MAX_REQUEST_TOKENS", 1 << 20)       # sum
 MAX_QUEUE_ROWS = _env_int("DECIDER_MAX_QUEUE_ROWS", 4096)                   # rows admitted and not yet scored, over all requests
 DECIDE_MAX_CTX_TOKENS = 1536                                                # /decide context cap, unchanged from 1.0.x
 
-MODEL_NAME = "decider"; TEMP = 1.0; TEMP_SCHEMA = 1.0; RELEASE_DATE = "2026-09-17"; ISOLATED = False; NEUTRALIZE_NONE = True
+MODEL_NAME = "decider"; TEMP = 1.0; TEMP_SCHEMA = 1.0; TEMP_BY_TYPE = {}; TEMP_SCHEMA_BY_TYPE = {}; RELEASE_DATE = "2026-09-17"; ISOLATED = False; NEUTRALIZE_NONE = True
 SCHEMA_FIRST = False; LAYOUT = "plain"; CHAT = None; se = None; squeue = None; schemas = {}; seen = {}
 eng = None; queue = None; gpu = None; cpu = None; batcher_task = None; schema_task = None
 outstanding = 0; REQ_SEQ = 0
@@ -118,6 +125,9 @@ def prepare(tok, state, questions, independent, isolated=False, max_state_tokens
     pairs = [(r["question"], list(r["options"])) for r in flat]
     rows = [[p] for p in pairs] if independent else [pairs]
     items, ctx_len = build_rows(tok, ctx, rows, max_ctx_tokens=max_state_tokens, chat=chat)
+    types = S1.row_types(rqs, index)                                 # answer type per slot (decider.temperature)
+    for it, ts in zip(items, [[t] for t in types] if independent else [types]):
+        it["types"] = ts
     return rqs, index, items, ctx_len
 
 
@@ -133,6 +143,7 @@ def _prepare_decide(context, schema):
             q["options"], q["_back"] = neutralize_options(q["options"])
     ex = Example(context, [Q(q["question"], list(q["options"]), 0) for q in qs])
     it = build(ex, eng.tok, _NoShuffle(), max_options=MAX_OPTIONS, max_ctx_tokens=DECIDE_MAX_CTX_TOKENS, chat=CHAT)
+    it["types"] = [q["_type"] for q in qs]                            # bool -> noul, scale -> score (decider.temperature)
     return qs, it
 
 
@@ -188,11 +199,11 @@ def _release(n):
 
 # ---- GPU work ----------------------------------------------------------------
 def _score_items(items):
-    return eng.score_items(items, temperature=TEMP)
+    return eng.score_items(items, temperature=TT.for_items(TEMP, TEMP_BY_TYPE, items))      # the scalar TEMP without a by-type map
 
 
 def _score_shared(items):
-    return eng.score_shared(items, temperature=TEMP)
+    return eng.score_shared(items, temperature=TT.for_items(TEMP, TEMP_BY_TYPE, items))
 
 
 async def _collect(q, wait_ms=None, adaptive_ms=0.0, idle_reset_ms=None):
@@ -294,6 +305,7 @@ def _schema_handle(questions, independent, compile=False):
             old = next(iter(schemas)); hid = schemas.pop(old)[1].id
             for k in [k for k in se.graphs if k[0] == hid]: del se.graphs[k]
         h = se.prepare(rows, independent=independent, compile=compile)
+        h.types = S1.row_types(rqs, index)                             # answer type per schema row (decider.temperature)
         schemas[key] = (rqs, h, index)
     return schemas[key]
 
@@ -308,7 +320,7 @@ def _worth_caching(questions, independent):
 
 
 def _score_schema(h, rows):
-    return se.score_rows(h, rows, temperature=TEMP_SCHEMA)
+    return se.score_rows(h, rows, temperature=TT.for_types(TEMP_SCHEMA, TEMP_SCHEMA_BY_TYPE, h.types))
 
 
 async def schema_batcher():
@@ -333,12 +345,13 @@ async def schema_batcher():
 
 # ---- start-up / shutdown -------------------------------------------------------
 def apply_config(cfg):
-    global MODEL_NAME, TEMP, TEMP_SCHEMA, RELEASE_DATE, ISOLATED, NEUTRALIZE_NONE, SCHEMA_FIRST, LAYOUT
+    global MODEL_NAME, TEMP, TEMP_SCHEMA, TEMP_BY_TYPE, TEMP_SCHEMA_BY_TYPE, RELEASE_DATE, ISOLATED, NEUTRALIZE_NONE, SCHEMA_FIRST, LAYOUT
     LAYOUT = resolve_layout(cfg)                    # ValueError for an unknown layout, before the engine is built
     NEUTRALIZE_NONE = bool(cfg.get("neutralize_none", True))
     MODEL_NAME = "decider-" + str(cfg.get("version", "dev"))
-    TEMP = float(os.environ.get("DECIDER_TEMPERATURE", cfg.get("temperature", 1.0)))
-    TEMP_SCHEMA = float(cfg.get("temperature_schema_first", TEMP))
+    # ValueError for a temperature that is not finite and > 0 or an unknown by-type key, before the engine is built.
+    # DECIDER_TEMPERATURE replaces "temperature" and switches "temperature_by_type" off (decider.temperature).
+    (TEMP, TEMP_BY_TYPE), (TEMP_SCHEMA, TEMP_SCHEMA_BY_TYPE) = TT.from_config(cfg, os.environ.get("DECIDER_TEMPERATURE"))
     RELEASE_DATE = str(cfg.get("release_date", RELEASE_DATE))
     ISOLATED = bool(cfg.get("isolated_levels", False))
     trained = bool(cfg.get("schema_first", False) or cfg.get("schema_first_trained", False))
@@ -403,7 +416,9 @@ async def _start():
         t = await loop.run_in_executor(gpu, lambda: eng.warmup(log=lambda s: print(s, flush=True)))
         print(f"[serve] captured {len(eng.graphs)} graphs in {t:.0f}s", flush=True)
     eng.seal()
-    print("[serve] ready", json.dumps(dict(model=MODEL_NAME, layout=LAYOUT, temperature=TEMP, isolated_levels=ISOLATED, schema_first=SCHEMA_FIRST,
+    print("[serve] ready", json.dumps(dict(model=MODEL_NAME, layout=LAYOUT, temperature=TEMP, temperature_by_type=TT.effective(TEMP, TEMP_BY_TYPE),
+                                          **({"temperature_schema_first_by_type": TT.effective(TEMP_SCHEMA, TEMP_SCHEMA_BY_TYPE)} if SCHEMA_FIRST else {}),
+                                          isolated_levels=ISOLATED, schema_first=SCHEMA_FIRST,
                                           shared=SHARED, graphs=len(eng.graphs), limits=dict(
                                               max_rows=MAX_ROWS, max_row_tokens=MAX_ROW_TOKENS, max_request_tokens=MAX_REQUEST_TOKENS,
                                               max_queue_rows=MAX_QUEUE_ROWS))), flush=True)
@@ -535,7 +550,9 @@ async def models():
 @app.get("/health")
 async def health():
     return {"ok": eng is not None and bool(getattr(eng, "sealed", True)) and _alive(), "model": MODEL,
-            "device": str(getattr(eng, "dev", "")) if eng is not None else None, "layout": LAYOUT}
+            "device": str(getattr(eng, "dev", "")) if eng is not None else None, "layout": LAYOUT,
+            "temperature": TEMP, "temperature_by_type": TT.effective(TEMP, TEMP_BY_TYPE),
+            **({"temperature_schema_first_by_type": TT.effective(TEMP_SCHEMA, TEMP_SCHEMA_BY_TYPE)} if SCHEMA_FIRST else {})}
 
 
 @app.get("/stats")

decider/shared_prefix.py

@@ -33,6 +33,7 @@ import torch
 import torch.nn.functional as F
 
 from decider.engine import fill_ids, read_slots
+from decider.temperature import item_slice, slot_temperatures
 
 DEFAULT_FORK_GB = 8.0
 STATE_NAMES = ("keys", "values", "indexer_keys", "conv_states", "recurrent_states")
@@ -159,6 +160,7 @@ def score_shared(engine, items, temperature=1.0, min_prefix=192, budget_bytes=No
     request does not qualify (fewer than two rows, or a common prefix below `min_prefix`) and the caller should use
     `score_items`.
 
+    `temperature`: a number, or one entry per item (decider.temperature.slot_temperatures).
     `rows_per_fork` forces the chunk size; it exists for the tests that compare chunked against unchunked answers."""
     ids = [it["ids"] for it in items]
     n = len(ids)
@@ -167,6 +169,7 @@ def score_shared(engine, items, temperature=1.0, min_prefix=192, budget_bytes=No
     lcp = common_prefix_len(ids)
     if lcp < min_prefix:
         return None
+    slot_temperatures(temperature, items)                  # a length mismatch fails before any forward
     core, W, dev, pad = engine.core, engine.W, engine.dev, engine.tok.pad_token_id
     pre = torch.tensor(ids[0][:lcp], device=dev)[None]
     cache = core(input_ids=pre, use_cache=True).past_key_values
@@ -191,6 +194,7 @@ def score_shared(engine, items, temperature=1.0, min_prefix=192, budget_bytes=No
         sl = [s - lcp for it in part for s in it["slots"]]
         idx = torch.tensor([rows, sl], device=dev)
         out += read_slots(F.linear(h[idx[0], idx[1]], W).float()[:, None, :], list(range(len(rows))), [0] * len(rows),
-                          [k for it in part for k in it["nopts"]], temperature, [len(it["slots"]) for it in part])
+                          [k for it in part for k in it["nopts"]],
+                          slot_temperatures(item_slice(temperature, i, i + b), part), [len(it["slots"]) for it in part])
         del fork, h, suf                                  # drop this chunk's fork before the next one is built
     return out

decider/systemone.py

@@ -3,7 +3,8 @@
     state      str | dict | list            JSON state is serialised compactly; questions may name a part by path (`ticket.messages[0].text`)
     questions  {id: {"type": "choice", "instructions": ..., "criteria": {name: description | {...} | [...] | None}}      up to 255 options
                      {"type": "score",  "instructions": ..., "criteria": [level 0 description, level 1 description, ...]}  2..10 levels
-                     {"type": "noul",   "instructions": ..., "criteria": {"true": ..., "false": ...} (optional)}}
+                     {"type": "noul",   "instructions": ... (optional), "criteria": {"true": ..., "false": ...} (optional)}}
+    a noul question needs instructions or at least one true/false description.
     ids are never shown to the model.  `instructions` and every description may be a string or any JSON value.
 """
 import json, math
@@ -38,7 +39,15 @@ def render_state(state, index_arrays=True):
 
 def render_question(spec):
     """-> dict(question=str, options=[str], type=..., names=[...])  (names: what the answer reports for each option)"""
-    t = spec.get("type", "choice"); ins = _txt(spec.get("instructions", spec.get("question", ""))); crit = spec.get("criteria", spec.get("options"))
+    t = spec.get("type", "choice"); crit = spec.get("criteria", spec.get("options"))
+    raw = spec.get("instructions", spec.get("question", ""))
+    if t in ("noul", "bool") and raw in (None, ""):          # the criteria carry the question (NOUL_WITHOUT_INSTRUCTIONS)
+        ins = NOUL_WITHOUT_INSTRUCTIONS
+        described = isinstance(crit, dict) and any(d not in (None, "") for d in (crit.get("true", crit.get(True)), crit.get("false", crit.get(False))))
+        if not described and (crit is None or isinstance(crit, dict)):     # a non-map is rejected below with the criteria message
+            raise ValueError("noul question without instructions: criteria must describe true or false")
+    else:
+        ins = _txt(raw)
     if not ins:
         raise ValueError("question without instructions")
     if t == "choice":
@@ -65,6 +74,12 @@ def render_question(spec):
                 isolated=bool(spec.get("isolated", True)))
 
 
+# A noul question may omit `instructions` (TypeSafe's OpenAPI file marks it optional).  The question id is never shown to the
+# model, so the question text is this fixed sentence and the true/false descriptions, rendered as the options "no: ..." and
+# "yes: ...", say what is being asked.  A request that gives instructions is rendered exactly as before.
+NOUL_WITHOUT_INSTRUCTIONS = "Which answer fits the context?"
+
+
 # ---- isolated levels: every Score level is judged in its own row, without its number or its neighbours
 ISOLATED = "{q}\nProposed answer: {level}\nDoes the proposed answer fit?"
 _NUM = None
@@ -100,6 +115,15 @@ def plan_rows(rqs, isolated=True):
     return rows, index
 
 
+def row_types(rqs, index):
+    """The answer type ("choice", "noul" or "score") of every plan_rows row, in row order.  An isolated Score question's
+    yes/no level rows carry "score": together they are one Score answer (decider.temperature)."""
+    types = [None] * sum(n for _, _, _, n in index)
+    for k, _, s, n in index:
+        types[s:s + n] = [rqs[k]["type"]] * n
+    return types
+
+
 def assemble(rqs, index, probs):
     """probs: one probability list per row (plan_rows order) -> {id: answer}."""
     out = {}
@@ -118,16 +142,50 @@ def certainty(p):
     return max(0.0, 1.0 - h / math.log(len(p))) if len(p) > 1 else 1.0
 
 
+def _clip01(x):
+    return min(1.0, max(0.0, x))
+
+
+def _normalised(p):
+    """As the adapter's _normalize: a distribution with zero total counts as uniform."""
+    tot = sum(p)
+    return [1.0 / len(p)] * len(p) if tot == 0 else [x / tot for x in p]
+
+
+def choice_confidence(p):
+    """TypeSafe's Choice confidence: the largest probability rescaled so that a uniform distribution gives 0 and all mass on one
+    option gives 1, (n * p_max - 1) / (n - 1); 1 for a single option."""
+    n = len(p); p = _normalised(p)
+    return 1.0 if n <= 1 else _clip01((n * max(p) - 1) / (n - 1))
+
+
+def score_confidence(p):
+    """TypeSafe's Score confidence (system-one-adapter-python, confidence_metrics.score_confidence): 1 minus the expected distance
+    from the most likely level, divided by D = mean over levels i of |i - (n - 1)/2| (the mean distance of the levels from the
+    middle of the scale), floored at 0; 1 for a single level.
+    With two levels it equals choice_confidence."""
+    n = len(p); p = _normalised(p)
+    if n <= 1:
+        return 1.0
+    k = max(range(n), key=p.__getitem__)
+    spread = sum(x * abs(i - k) for i, x in enumerate(p))
+    uniform = sum(abs(i - (n - 1) / 2) for i in range(n)) / n
+    return _clip01(1.0 - spread / uniform)
+
+
 def format_answer(rq, p, nd=4):
-    """rq: render_question output; p: probabilities in option order."""
+    """rq: render_question output; p: probabilities in option order.
+    `confidence` is TypeSafe's (choice_confidence / score_confidence); `x_p_max` is the largest probability, which was
+    `confidence` before 1.3.0."""
     p = [float(x) for x in p[:len(rq["options"])]]; s = sum(p) or 1.0; p = [x / s for x in p]
     j = max(range(len(p)), key=p.__getitem__)
     if rq["type"] == "noul":
         return {"type": "noul", "noul": round(p[1], nd)}
     if rq["type"] == "choice":
-        return {"type": "choice", "choice": rq["names"][j], "confidence": round(p[j], nd), "certainty": round(certainty(p), nd),
-                "probabilities": {n: round(x, nd) for n, x in zip(rq["names"], p)}}
-    return {"type": "score", "score": round(sum(i * x for i, x in enumerate(p)), 2), "confidence": round(p[j], nd), "certainty": round(certainty(p), nd),
+        return {"type": "choice", "choice": rq["names"][j], "confidence": round(choice_confidence(p), nd), "x_p_max": round(p[j], nd),
+                "certainty": round(certainty(p), nd), "probabilities": {n: round(x, nd) for n, x in zip(rq["names"], p)}}
+    return {"type": "score", "score": round(sum(i * x for i, x in enumerate(p)), 2), "confidence": round(score_confidence(p), nd), "x_p_max": round(p[j], nd),
+            "certainty": round(certainty(p), nd),
             "legend": {str(i): d for i, d in enumerate(rq["legend"])}, "probabilities": {str(i): round(x, nd) for i, x in enumerate(p)}}
 
 

decider/temperature.py

@@ -0,0 +1,144 @@
+"""Answer temperatures from decider_config.json (1.4.0).
+
+Every answer is softmax(logits / T) over its option letters.  decider_config.json sets T:
+
+    "temperature": 1.3                                   one value for every answer (the only form before 1.4.0)
+    "temperature_by_type": {"choice": 1.48, "noul": 2.22, "score": 1.38}
+                                                         optional; one value per answer type, a missing type uses "temperature"
+    "temperature_schema_first": 1.18                     optional; the schema cache (questions-first layout), as before
+    "temperature_schema_first_by_type": {...}            optional; per answer type on the schema cache
+
+The answer types are the /v1/systemone question types.  A /decide field maps onto them: "choice" -> "choice", "bool" -> "noul",
+"scale" -> "score".  Plain `Decider.decide` questions (a question and its options, no type) are "choice".  A Score question
+read with isolated levels (one yes/no row per level) uses the "score" temperature on every one of its level rows: the rows form
+one Score answer, and a temperature fitted on Score answers is fitted through that same readout (decider.calibrate).
+
+On the state-first layout the temperature of an answer of type t is temperature_by_type[t], else temperature.  On the schema
+cache it is temperature_schema_first_by_type[t], else temperature_schema_first, else (no schema-first value at all) the
+state-first temperature of t.  An explicit override (Decider(temperature=...), DECIDER_TEMPERATURE) replaces the state-first
+"temperature" and switches temperature_by_type off, so the override is the one temperature of every state-first answer, as it
+was before 1.4.0.
+
+A config without a by-type map gives every path the single number it gave in 1.3.0, and that number reaches the engines as the
+same Python float, so the probabilities are bit-identical to 1.3.0.
+"""
+import math
+
+TYPES = ("choice", "noul", "score")
+FIELD_TYPES = {"choice": "choice", "bool": "noul", "scale": "score"}        # /decide field type -> answer type
+_KEYS_TEXT = ('the keys are "choice", "noul" and "score" (a /decide "bool" field is "noul", a "scale" field is "score"; '
+              'a /v1/systemone "bool" question is "noul")')
+
+
+def positive(value, where):
+    """A temperature: a number (or a numeric string, as float() reads it, which 1.3.0 accepted for "temperature") that is finite
+    and > 0.  Raises ValueError naming `where`."""
+    if isinstance(value, bool):
+        raise ValueError(f"{where} must be a finite number > 0, got {value!r}")
+    try:
+        v = float(value)
+    except (TypeError, ValueError):
+        raise ValueError(f"{where} must be a finite number > 0, got {value!r}") from None
+    if not math.isfinite(v) or v <= 0:
+        raise ValueError(f"{where} must be a finite number > 0, got {value!r}")
+    return v
+
+
+def by_type(m, where):
+    """Validate a {answer type: temperature} map.  None -> {}.  Unknown keys, non-numbers and values that are not finite and > 0
+    raise ValueError."""
+    if m is None:
+        return {}
+    if not isinstance(m, dict):
+        raise ValueError(f"{where} must be a map {{answer type: temperature}}, got {type(m).__name__}; " + _KEYS_TEXT)
+    out = {}
+    for k, v in m.items():
+        if k not in TYPES:
+            raise ValueError(f"{where} has the unknown key {k!r}; " + _KEYS_TEXT)
+        if isinstance(v, bool) or not isinstance(v, (int, float)):
+            raise ValueError(f"{where}[{k!r}] must be a finite number > 0, got {v!r}")
+        out[k] = positive(v, f"{where}[{k!r}]")
+    return out
+
+
+def from_config(cfg, temperature=None, temperature_by_type=None):
+    """-> ((T, by_type) for the state-first layout, (T, by_type) for the schema cache).
+
+    temperature / temperature_by_type: explicit overrides (Decider arguments, DECIDER_TEMPERATURE).  An explicit temperature
+    without an explicit map switches the config's map off (see the module docstring)."""
+    cfg = cfg or {}
+    where = "decider_config.json"
+    if temperature is not None:
+        T = positive(temperature, "temperature")
+        m = by_type(temperature_by_type, "temperature_by_type") if temperature_by_type is not None else {}
+    else:
+        T = positive(cfg.get("temperature", 1.0), f'{where} "temperature"')
+        m = (by_type(temperature_by_type, "temperature_by_type") if temperature_by_type is not None
+             else by_type(cfg.get("temperature_by_type"), f'{where} "temperature_by_type"'))
+    ms = by_type(cfg.get("temperature_schema_first_by_type"), f'{where} "temperature_schema_first_by_type"')
+    if "temperature_schema_first" in cfg:
+        schema = (positive(cfg["temperature_schema_first"], f'{where} "temperature_schema_first"'), ms)
+    else:
+        schema = (T, {**m, **ms})
+    return (T, m), schema
+
+
+def effective(T, m):
+    """{answer type: the temperature it gets} for reporting (/health, the ready line)."""
+    return {t: m.get(t, T) for t in TYPES}
+
+
+def for_types(T, m, types):
+    """One temperature per slot, or the scalar T itself when there is no map (the 1.3.0 call)."""
+    if not m:
+        return T
+    return [m.get(t, T) for t in types]
+
+
+def item_types(it):
+    """The answer type of every slot of a prompt item ("types", set where the item is built); None for an item without it."""
+    ts = it.get("types")
+    return list(ts) if ts is not None else [None] * len(it["slots"])
+
+
+def for_items(T, m, items):
+    """The `temperature` argument of Engine.score_items / score_shared: the scalar T when there is no map (the 1.3.0 call),
+    else one list of per-slot temperatures per item."""
+    if not m:
+        return T
+    return [[m.get(t, T) for t in item_types(it)] for it in items]
+
+
+def slot_temperatures(temperature, items):
+    """Engine side.  temperature: a scalar (returned as it is), or one entry per item, each a scalar or one value per slot of
+    that item.  -> the scalar, or a flat list with one temperature per slot in item order."""
+    if not isinstance(temperature, (list, tuple)):
+        return temperature
+    if len(temperature) != len(items):
+        raise ValueError(f"temperature: {len(temperature)} entries for {len(items)} items")
+    flat = []
+    for t, it in zip(temperature, items):
+        n = len(it["slots"])
+        if isinstance(t, (list, tuple)):
+            if len(t) != n:
+                raise ValueError(f"temperature: {len(t)} values for an item with {n} slots")
+            flat += list(t)
+        else:
+            flat += [t] * n
+    return flat
+
+
+def item_slice(temperature, lo, hi):
+    """The per-item temperature entries of items[lo:hi] (a scalar is shared by every item)."""
+    return temperature[lo:hi] if isinstance(temperature, (list, tuple)) else temperature
+
+
+def scaled_softmax(lg, temperature):
+    """softmax(lg / T) over the last axis.  A scalar T is the 1.3.0 expression unchanged; a list gives one T per row of lg."""
+    import torch
+    if isinstance(temperature, (list, tuple)):
+        if len(temperature) != lg.shape[0]:
+            raise ValueError(f"temperature: {len(temperature)} values for {lg.shape[0]} slots")
+        t = torch.tensor(temperature, dtype=lg.dtype).to(lg.device, non_blocking=True)[:, None]
+        return torch.softmax(lg / t, -1)
+    return torch.softmax(lg / temperature, -1)

decider_config.json

@@ -1,7 +1,12 @@
 {
- "temperature": 1.935,
+ "temperature": 1.099,
+ "temperature_by_type": {
+  "choice": 1.11,
+  "noul": 1.56,
+  "score": 1.287
+ },
  "neutralize_none": false,
- "version": "4b-v2",
+ "version": "4b-v2.1",
  "base": "Mapika/decider-4b v1 + LoRA (merged); v1 is Qwen/Qwen3.5-4B-Base + one supervised pass over mixture v2",
  "layout": "plain",
  "max_options": 255,
@@ -10,5 +15,6 @@
  "schema_first_trained": false,
  "isolated_levels": true,
  "release_date": "2026-09-24",
- "stage": "decider-4b v1 + LoRA rank 64 (alpha 128) on attention and MLP, LR 1e-4, 2 epochs (1,517 steps of 65,536 tokens) over 29,356 rows in the plain state-first layout (generated decision families with code-computed answers, Qwen3.6-27B-written document questions kept when two independent answers agreed, human-labelled public sets, replay of v1's mixture v2), merged into the bf16 weights; no RL stage; temperature fitted by NLL on 61 in-task regression tasks (the 67 in-task tasks without banking77, clinc_oos, mmlu, arc, winogrande, hellaswag)"
+ "requires": "decider-ai>=1.4.0 for temperature_by_type; older versions serve every answer at temperature",
+ "stage": "decider-4b v1 + LoRA rank 64 (alpha 128) on attention and MLP, LR 1e-4, 2 epochs (1,518 steps of 65,536 tokens) over v2's 29,325-row mix in the plain state-first layout (generated decision families with code-computed answers, Qwen3.6-27B-written document questions kept when two independent answers agreed, human-labelled public sets, replay of v1's mixture v2), with the replay rows trained toward v1's own answer distribution (KL to v1) instead of their labels, merged into the bf16 weights; no RL stage; temperature fitted by NLL on 61 in-task regression tasks (the 67 in-task tasks without banking77, clinc_oos, mmlu, arc, winogrande, hellaswag); temperature_by_type fitted with decider.calibrate.fit_by_type on the same regression rows plus our own validation rows (choice, noul and score answers)"
 }

eval_results.json

@@ -1,1796 +1,1549 @@
 {
- "model": "decider-4b-v2",
- "temperature": 1.935,
- "temperature_fit": {
-  "method": "one scalar, pooled row NLL, golden-section search on log T in [0.3, 5]; stored rounded to 3 decimals",
-  "tasks": 61,
-  "rows": 102804,
-  "excluded_in_task_tasks": [
-   "banking77",
-   "clinc_oos",
-   "mmlu",
-   "arc",
-   "winogrande",
-   "hellaswag"
-  ],
-  "fitted": 1.9349278889417365,
-  "grid_fit_temperature": 1.9420684917099538,
-  "stored": 1.935,
-  "note": "fitted on the in-task half of the public regression set without banking77, clinc_oos, mmlu, arc, winogrande and hellaswag (61 tasks); no JevBench or Decision Index item was used for training, selection or temperature"
+ "model": "decider-4b-v2.1",
+ "temperature": 1.099,
+ "temperature_by_type": {
+  "choice": 1.11,
+  "noul": 1.56,
+  "score": 1.287
  },
- "protocol": "decider-4b v1 + merged LoRA (rank 64, attention and MLP, 2 epochs, 29,356 rows). Public regression set rebuilt on this machine: eval half of the mixture, 95 tasks (67 in-task / 28 held-out), state-first plain layout, max_ctx 1536, large label sets sub-sampled to 10 options; metrics at the stored temperature, ECE with 15 bins. Other sets through decider-ai 1.2.1 at T 1.935: fixtures through h2h/score_public.py (eager, 32k context), Bespoke suite through decider.bench.public_suite, games and browser through the h2h/serve_public.py shim, behaviour probes; JevBench public items read once at T 1.719 and rescaled to T 1.935 from the stored probabilities (18 isolated-level Score items kept at T 1.719); Decision Index sample read at T 1.719 and rescaled likewise; ten text games greedy (argmax, independent of T). Readout only.",
- "aggregate": {
-  "in_task": {
-   "acc": 0.8241469454813016,
-   "nll": 0.441108525216357,
-   "ece": 0.04134314671333522,
-   "tasks": 67
-  },
-  "heldout": {
-   "acc": 0.7787589834608305,
-   "nll": 0.5663163406508309,
-   "ece": 0.08033093216064069,
-   "tasks": 28
-  }
+ "temperature_fit": {
+  "global": "one scalar, pooled row NLL on the in-task half of the public regression set without banking77, clinc_oos, mmlu, arc, winogrande and hellaswag (61 tasks, 102,804 rows)",
+  "by_type": "decider.calibrate.fit_by_type (decider-ai 1.4.0), min_rows 50, on the same regression rows plus our own validation rows (val_jb, val_teacher, val_teacher2, cal_human without mmlu, arc and banking77, val_replay without six excluded families)",
+  "rows": {
+   "choice": 108927,
+   "noul": 1677,
+   "score": 535
+  },
+  "note": "no JevBench or Decision Index item was used for training, selection or temperature"
  },
- "results": {
-  "abstain_probe": {
-   "heldout": true,
-   "acc": 0.5820987654320988,
-   "nll": 1.1091256141662598,
-   "ece": 0.13279259307884878
-  },
-  "ade": {
-   "heldout": true,
-   "acc": 0.7886666666666666,
-   "nll": 0.5347910523414612,
-   "ece": 0.10660141297181445
-  },
-  "ag_news": {
-   "heldout": false,
-   "acc": 0.9033333333333333,
-   "nll": 0.294931024312973,
-   "ece": 0.04695633757114412
-  },
-  "agenttraj": {
-   "heldout": false,
-   "acc": 0.8973333333333333,
-   "nll": 0.2499454766511917,
-   "ece": 0.03513214365641278
-  },
-  "amazon_stars": {
-   "heldout": false,
-   "acc": 0.602,
-   "nll": 0.9151002764701843,
-   "ece": 0.035494098405043265
-  },
-  "arc": {
-   "heldout": false,
-   "acc": 0.939419795221843,
-   "nll": 0.1868872344493866,
-   "ece": 0.0222670579577062
-  },
-  "arena_pref": {
-   "heldout": true,
-   "acc": 0.5033333333333333,
-   "nll": 1.2348569631576538,
-   "ece": 0.21849741025765737
-  },
-  "banking77": {
-   "heldout": false,
-   "acc": 0.972,
-   "nll": 0.10399508476257324,
-   "ece": 0.0268547142942746
-  },
-  "bbc_news": {
-   "heldout": true,
-   "acc": 0.951,
-   "nll": 0.18327508866786957,
-   "ece": 0.06801495689153671
-  },
-  "bias_in_bios": {
-   "heldout": false,
-   "acc": 0.9486666666666667,
-   "nll": 0.15249282121658325,
-   "ece": 0.013438965007662774
-  },
-  "bitext_support": {
-   "heldout": false,
-   "acc": 0.998,
-   "nll": 0.016967715695500374,
-   "ece": 0.01042288645108536
-  },
-  "boolq": {
-   "heldout": false,
-   "acc": 0.9033333333333333,
-   "nll": 0.24676379561424255,
-   "ece": 0.029159295320510863
-  },
-  "cb": {
-   "heldout": true,
-   "acc": 0.8392857142857143,
-   "nll": 0.4322512745857239,
-   "ece": 0.11792815795966557
-  },
-  "civil_comments": {
-   "heldout": false,
-   "acc": 0.9328,
-   "nll": 0.1662478744983673,
-   "ece": 0.010289057048161809
-  },
-  "clinc_oos": {
-   "heldout": false,
-   "acc": 0.9826666666666667,
-   "nll": 0.08267337828874588,
-   "ece": 0.024905774275461832
-  },
-  "cola": {
-   "heldout": false,
-   "acc": 0.8331735378715245,
-   "nll": 0.3790046274662018,
-   "ece": 0.02507039284546103
-  },
-  "commonsense_qa": {
-   "heldout": false,
-   "acc": 0.823914823914824,
-   "nll": 0.4847744107246399,
-   "ece": 0.030845976944930447
-  },
-  "copa": {
-   "heldout": false,
-   "acc": 1.0,
-   "nll": 0.030766163021326065,
-   "ece": 0.028578150868415832
-  },
-  "counterfactual": {
-   "heldout": false,
-   "acc": 0.9573333333333334,
-   "nll": 0.13998979330062866,
-   "ece": 0.012234268069267251
-  },
-  "cr_reviews": {
-   "heldout": true,
-   "acc": 0.897742363877822,
-   "nll": 0.24005697667598724,
-   "ece": 0.045616757347289316
-  },
-  "dbpedia": {
-   "heldout": false,
-   "acc": 0.9933333333333333,
-   "nll": 0.027618318796157837,
-   "ece": 0.013420193096001943
-  },
-  "dbpedia_l2": {
-   "heldout": true,
-   "acc": 0.9506666666666667,
-   "nll": 0.1401711255311966,
-   "ece": 0.017278132518132504
-  },
-  "dbpedia_l3": {
-   "heldout": true,
-   "acc": 0.9913333333333333,
-   "nll": 0.03723839297890663,
-   "ece": 0.011898971815903936
-  },
-  "dolly_category": {
-   "heldout": true,
-   "acc": 0.374,
-   "nll": 1.7009543180465698,
-   "ece": 0.1419213018019994
-  },
-  "emotion": {
-   "heldout": false,
-   "acc": 0.8333333333333334,
-   "nll": 0.4790882170200348,
-   "ece": 0.025491972724596655
-  },
-  "enron_spam": {
-   "heldout": false,
-   "acc": 0.988,
-   "nll": 0.03346330299973488,
-   "ece": 0.006207752108573955
-  },
-  "fever": {
-   "heldout": false,
-   "acc": 0.8806666666666667,
-   "nll": 0.32262682914733887,
-   "ece": 0.01808753836154938
-  },
-  "fin_phrasebank": {
-   "heldout": true,
-   "acc": 0.6989690721649484,
-   "nll": 0.6234146952629089,
-   "ece": 0.07664692620026696
-  },
-  "fin_sentiment": {
-   "heldout": true,
-   "acc": 0.826,
-   "nll": 0.42305630445480347,
-   "ece": 0.051281696399052924
-  },
-  "glaive_tools": {
-   "heldout": false,
-   "acc": 0.9369436201780416,
-   "nll": 0.2514188289642334,
-   "ece": 0.10150587457047022
-  },
-  "go_emotions": {
-   "heldout": false,
-   "acc": 0.776,
-   "nll": 0.6853362917900085,
-   "ece": 0.03606782659888266
-  },
-  "hate_offensive": {
-   "heldout": false,
-   "acc": 0.9133333333333333,
-   "nll": 0.26428458094596863,
-   "ece": 0.014751521805922176
-  },
-  "hate_speech_scales": {
-   "heldout": false,
-   "acc": 0.5582666666666667,
-   "nll": 0.9945353269577026,
-   "ece": 0.03258252822558084
-  },
-  "hellaswag": {
-   "heldout": false,
-   "acc": 0.9346666666666666,
-   "nll": 0.1996963769197464,
-   "ece": 0.015657478451728855
-  },
-  "helpsteer2": {
-   "heldout": false,
-   "acc": 0.5980732177263969,
-   "nll": 0.9590670466423035,
-   "ece": 0.02256552288183126
-  },
-  "helpsteer3_pref": {
-   "heldout": false,
-   "acc": 0.38605442176870747,
-   "nll": 1.5417133569717407,
-   "ece": 0.061995073291314706
-  },
-  "hermes_tools": {
-   "heldout": true,
-   "acc": 0.7373333333333333,
-   "nll": 0.5897114872932434,
-   "ece": 0.1240579075018565
-  },
-  "hh_rlhf": {
-   "heldout": false,
-   "acc": 0.6451612903225806,
-   "nll": 0.680734395980835,
-   "ece": 0.11346792193349971
-  },
-  "hwu64": {
-   "heldout": true,
-   "acc": 0.9628252788104089,
-   "nll": 0.12855161726474762,
-   "ece": 0.03735405791093864
-  },
-  "imdb": {
-   "heldout": false,
-   "acc": 0.968,
-   "nll": 0.09611714631319046,
-   "ece": 0.01006534445285793
-  },
-  "insincere_questions": {
-   "heldout": false,
-   "acc": 0.9553333333333334,
-   "nll": 0.12364762276411057,
-   "ece": 0.01763951281706497
-  },
-  "liar2": {
-   "heldout": false,
-   "acc": 0.35,
-   "nll": 1.513845682144165,
-   "ece": 0.06293645719687144
-  },
-  "massive_intent": {
-   "heldout": false,
-   "acc": 0.9606666666666667,
-   "nll": 0.13908076286315918,
-   "ece": 0.02024710043271384
-  },
-  "massive_scenario": {
-   "heldout": true,
-   "acc": 0.7933333333333333,
-   "nll": 0.5642920732498169,
-   "ece": 0.05245566362142563
-  },
-  "medmcqa": {
-   "heldout": false,
-   "acc": 0.6233333333333333,
-   "nll": 0.9161791801452637,
-   "ece": 0.07326398479938506
-  },
-  "medqa": {
-   "heldout": false,
-   "acc": 0.6779261586802828,
-   "nll": 0.8010168075561523,
-   "ece": 0.0796375350744258
-  },
-  "mind2web": {
-   "heldout": false,
-   "acc": 0.822,
-   "nll": 0.48613396286964417,
-   "ece": 0.04646803086996078
-  },
-  "mmlu": {
-   "heldout": false,
-   "acc": 0.742,
-   "nll": 0.6732416749000549,
-   "ece": 0.04000420778989793
-  },
-  "mnli": {
-   "heldout": false,
-   "acc": 0.9026666666666666,
-   "nll": 0.28677940368652344,
-   "ece": 0.019106337686379753
-  },
-  "mrpc": {
-   "heldout": false,
-   "acc": 0.8431372549019608,
-   "nll": 0.35424306988716125,
-   "ece": 0.031691053334404445
-  },
-  "msmarco_rel": {
-   "heldout": false,
-   "acc": 0.6874135546334716,
-   "nll": 0.586560845375061,
-   "ece": 0.035539872212693564
-  },
-  "multirc": {
-   "heldout": false,
-   "acc": 0.9053333333333333,
-   "nll": 0.2743209898471832,
-   "ece": 0.025939866781234736
-  },
-  "newsgroups": {
-   "heldout": false,
-   "acc": 0.820109439124487,
-   "nll": 0.5166534781455994,
-   "ece": 0.0274887290539885
-  },
-  "offtopic_probe": {
-   "heldout": true,
-   "acc": 0.8234567901234567,
-   "nll": 0.5037797093391418,
-   "ece": 0.043301358119941055
-  },
-  "openbookqa": {
-   "heldout": false,
-   "acc": 0.904,
-   "nll": 0.2716519832611084,
-   "ece": 0.033541144907474535
-  },
-  "paws": {
-   "heldout": true,
-   "acc": 0.794,
-   "nll": 0.5527914762496948,
-   "ece": 0.12406627035140992
-  },
-  "piqa": {
-   "heldout": false,
-   "acc": 0.89,
-   "nll": 0.2665766477584839,
-   "ece": 0.026692868391672748
-  },
-  "prosocial_safety": {
-   "heldout": false,
-   "acc": 0.512,
-   "nll": 1.1915110349655151,
-   "ece": 0.030761619885762537
-  },
-  "pubmedqa": {
-   "heldout": true,
-   "acc": 0.758,
-   "nll": 0.6072925329208374,
-   "ece": 0.05524836438894271
-  },
-  "qasc": {
-   "heldout": false,
-   "acc": 0.8596112311015118,
-   "nll": 0.47038528323173523,
-   "ece": 0.06783940297094844
-  },
-  "qnli": {
-   "heldout": false,
-   "acc": 0.946,
-   "nll": 0.14609812200069427,
-   "ece": 0.01228535489241284
-  },
-  "qqp": {
-   "heldout": false,
-   "acc": 0.848,
-   "nll": 0.3371011018753052,
-   "ece": 0.03070641124248504
-  },
-  "quality": {
-   "heldout": true,
-   "acc": 0.5606666666666666,
-   "nll": 1.1722065210342407,
-   "ece": 0.17895789907375972
-  },
-  "quality_full": {
-   "heldout": true,
-   "acc": 0.505,
-   "nll": 1.2171955108642578,
-   "ece": 0.1881948218246301
-  },
-  "race": {
-   "heldout": false,
-   "acc": 0.8953333333333333,
-   "nll": 0.33984363079071045,
-   "ece": 0.02000871032476428
-  },
-  "reward_bench": {
-   "heldout": true,
-   "acc": 0.8526666666666667,
-   "nll": 0.34169474244117737,
-   "ece": 0.03198828514417014
-  },
-  "rte": {
-   "heldout": false,
-   "acc": 0.9205776173285198,
-   "nll": 0.1940610110759735,
-   "ece": 0.04438359793342838
-  },
-  "sciq": {
-   "heldout": true,
-   "acc": 0.991,
-   "nll": 0.044908951967954636,
-   "ece": 0.02056253519654279
-  },
-  "shp": {
-   "heldout": false,
-   "acc": 0.7353333333333333,
-   "nll": 0.594969630241394,
-   "ece": 0.1105338017543157
-  },
-  "sms_spam": {
-   "heldout": false,
-   "acc": 0.987,
-   "nll": 0.055126290768384933,
-   "ece": 0.006505105614662178
-  },
-  "snli": {
-   "heldout": false,
-   "acc": 0.9227642276422764,
-   "nll": 0.24105508625507355,
-   "ece": 0.03949038551105716
-  },
-  "social_iqa": {
-   "heldout": true,
-   "acc": 0.7593333333333333,
-   "nll": 0.5866184830665588,
-   "ece": 0.06180198764801027
-  },
-  "sst2": {
-   "heldout": false,
-   "acc": 0.948394495412844,
-   "nll": 0.14050182700157166,
-   "ece": 0.01752297238472409
-  },
-  "sst5": {
-   "heldout": false,
-   "acc": 0.5826666666666667,
-   "nll": 0.9461587071418762,
-   "ece": 0.0451480305393537
-  },
-  "strategyqa": {
-   "heldout": true,
-   "acc": 0.6317321688500728,
-   "nll": 0.6970252990722656,
-   "ece": 0.1567338213129335
-  },
-  "student_questions": {
-   "heldout": true,
-   "acc": 0.94,
-   "nll": 0.20459797978401184,
-   "ece": 0.05185317244132365
-  },
-  "subjectivity": {
-   "heldout": false,
-   "acc": 0.9566666666666667,
-   "nll": 0.11102409660816193,
-   "ece": 0.015209745287895248
-  },
-  "support_tickets": {
-   "heldout": false,
-   "acc": 0.5586666666666666,
-   "nll": 1.0297108888626099,
-   "ece": 0.024460139827595817
-  },
-  "synth": {
-   "heldout": false,
-   "acc": 0.8066666666666666,
-   "nll": 0.4917465150356293,
-   "ece": 0.04993985931078593
-  },
-  "toolace": {
-   "heldout": false,
-   "acc": 0.931,
-   "nll": 0.2593076229095459,
-   "ece": 0.06617412987351419
-  },
-  "toxic_chat": {
-   "heldout": false,
-   "acc": 0.977,
-   "nll": 0.06245163083076477,
-   "ece": 0.00981356755892436
-  },
-  "trec": {
-   "heldout": true,
-   "acc": 0.842,
-   "nll": 0.5022116899490356,
-   "ece": 0.02651941215991974
-  },
-  "truthfulqa": {
-   "heldout": true,
-   "acc": 0.653610771113831,
-   "nll": 1.0122824907302856,
-   "ece": 0.06321512999849775
-  },
-  "tweet_emotion": {
-   "heldout": false,
-   "acc": 0.8578465869106263,
-   "nll": 0.41329097747802734,
-   "ece": 0.039397706623063994
-  },
-  "tweet_hate": {
-   "heldout": false,
-   "acc": 0.5126666666666667,
-   "nll": 1.136211633682251,
-   "ece": 0.34860085622469583
-  },
-  "tweet_irony": {
-   "heldout": true,
-   "acc": 0.826530612244898,
-   "nll": 0.39445212483406067,
-   "ece": 0.027230415189144572
-  },
-  "tweet_offensive": {
-   "heldout": false,
-   "acc": 0.8523255813953489,
-   "nll": 0.3407946228981018,
-   "ece": 0.021613172181817006
-  },
-  "tweet_sentiment": {
-   "heldout": false,
-   "acc": 0.7206666666666667,
-   "nll": 0.610216498374939,
-   "ece": 0.028348857482274364
-  },
-  "ultrafeedback_pref": {
-   "heldout": false,
-   "acc": 0.7713333333333333,
-   "nll": 0.4769253432750702,
-   "ece": 0.06639519715309146
-  },
-  "wic": {
-   "heldout": false,
-   "acc": 0.7429467084639498,
-   "nll": 0.5646101832389832,
-   "ece": 0.09329287749846528
-  },
-  "wiki_qa": {
-   "heldout": false,
-   "acc": 0.9089874857792947,
-   "nll": 0.2449624091386795,
-   "ece": 0.020241137725365718
-  },
-  "winogrande": {
-   "heldout": false,
-   "acc": 0.8089976322020521,
-   "nll": 0.48967406153678894,
-   "ece": 0.08959193594736103
-  },
-  "xstory_cloze": {
-   "heldout": true,
-   "acc": 0.9706666666666667,
-   "nll": 0.07805304229259491,
-   "ece": 0.01724668137232463
-  },
-  "yahoo_topics": {
-   "heldout": false,
-   "acc": 0.7613333333333333,
-   "nll": 0.7648204565048218,
-   "ece": 0.0657656339406967
-  },
-  "yelp": {
-   "heldout": false,
-   "acc": 0.7033333333333334,
-   "nll": 0.7055407166481018,
-   "ece": 0.04224825153748195
+ "protocol": "Regression set: eval half of the mixture, 95 tasks (67 in-task / 28 held-out), plain state-first layout, large label sets sub-sampled to 10 options, ECE 15 bins, per-task mean. Own sets (heldout_jb, test_teacher2, cal_human, guard): stored T=1 logits read at the served temperatures, ECE 10 bins top label. Card lanes (fixtures, games, browser, probes, Bespoke, issue #9) through decider-ai 1.4.0 with the map; parents through 1.3.0 at their released temperatures, same rows and seeds; intervals are 95% paired bootstrap. JevBench public items read in process through 1.4.0 with the map; parents' files read earlier through 1.2.1.",
+ "regression": {
+  "map": {
+   "reg_in": 0.8308,
+   "reg_out": 0.7838,
+   "reg_in_nll": 0.4145,
+   "reg_out_nll": 0.5689,
+   "reg_in_ece": 0.0309,
+   "reg_out_ece": 0.0775
+  },
+  "global_T": {
+   "reg_in": 0.8308,
+   "reg_out": 0.7838,
+   "reg_in_nll": 0.4145,
+   "reg_out_nll": 0.5703,
+   "reg_in_ece": 0.0308,
+   "reg_out_ece": 0.0781
   }
  },
- "other_sets": {
-  "general_validation_847": {
-   "acc": 0.8501,
-   "nll": 0.4186,
-   "ece": 0.0305
-  },
-  "typesafe_102": {
-   "acc": 0.8627,
-   "nll": 0.4101,
-   "ece": 0.0723,
-   "tv_to_reference": 0.1961
-  },
-  "openjev_5252": {
-   "acc": 0.6672,
-   "nll": 0.789,
-   "ece": 0.1309,
-   "macro_acc_3_sources": 0.8098
-  },
-  "mind2web_1770": {
-   "acc": 0.874,
-   "nll": 0.3935,
-   "ece": 0.0512
-  },
-  "jevbench_public_items": {
-   "easy": 1.0,
-   "standard": 0.9861,
-   "hard": 0.6757,
-   "easy_top_label_ece": 0.0078,
-   "standard_top_label_ece": 0.0854,
-   "hard_top_label_ece": 0.0708,
-   "note": "read once at T 1.719 after selection; ECE rescaled to T 1.935 from stored probabilities",
-   "score_items_kept_at_T_1719": 18,
-   "hard_mean_confidence": 0.7401,
-   "hard_conf_ge_0_9_coverage": 0.3063,
-   "hard_conf_ge_0_9_accuracy": 0.9412
-  },
-  "bespoke_public_suite": {
-   "macro": 0.7725,
-   "micro": 0.7809,
-   "macro_untrained": 0.7514,
-   "per_subset": {
-    "vitaminc-dev": 0.778,
-    "massive-en-US": 0.8686,
-    "massive-de-DE": 0.8371,
-    "boolq": 0.86,
-    "squad2": 0.7926,
-    "paws": 0.832,
-    "multinli": 0.9264,
-    "civil_comments": 0.8567,
-    "aegis2": 0.812,
-    "helpsteer2": 0.4659,
-    "summeval-relevance": 0.4625,
-    "summeval-consistency": 0.8264,
-    "pubmedqa": 0.724
-   }
-  },
-  "zero_shot_games_win_rate": {
-   "sampled_all": 0.2244,
-   "greedy_all": 0.2778,
-   "sampled_bags": 0.3789,
-   "greedy_bags": 0.4844
-  },
-  "text_games_zero_shot_5_episodes": {
-   "pong": {
-    "model": -21.0,
-    "teacher": 8.0,
-    "random": -20.4
-   },
-   "freeway": {
-    "model": 1.0,
-    "teacher": 5.0,
-    "random": 0.0
-   },
-   "breakout": {
-    "model": 12.0,
-    "teacher": 22.0,
-    "random": 0.8
+ "own_sets": {
+  "map": {
+   "heldout_jb": {
+    "all": {
+     "n": 5000,
+     "accuracy": 0.556,
+     "nll": 1.2571,
+     "ece": 0.1466,
+     "mean_conf": 0.7026
+    },
+    "by_type": {
+     "choice": {
+      "n": 3293,
+      "accuracy": 0.5056,
+      "nll": 1.4995,
+      "ece": 0.1701,
+      "mean_conf": 0.6757
+     },
+     "noul": {
+      "n": 1289,
+      "accuracy": 0.7114,
+      "nll": 0.6074,
+      "ece": 0.1013,
+      "mean_conf": 0.8127
+     },
+     "score": {
+      "n": 418,
+      "accuracy": 0.4737,
+      "nll": 1.3504,
+      "ece": 0.1081,
+      "mean_conf": 0.5747
+     }
+    }
    },
-   "frozenlake": {
-    "model": 0.0,
-    "teacher": 1.0,
-    "random": 0.0
+   "test_teacher2": {
+    "all": {
+     "n": 449,
+     "accuracy": 0.8196,
+     "nll": 0.4304,
+     "ece": 0.043,
+     "mean_conf": 0.8559
+    },
+    "by_type": {
+     "choice": {
+      "n": 257,
+      "accuracy": 0.8016,
+      "nll": 0.5041,
+      "ece": 0.0569,
+      "mean_conf": 0.8585
+     },
+     "noul": {
+      "n": 130,
+      "accuracy": 0.8692,
+      "nll": 0.2489,
+      "ece": 0.0479,
+      "mean_conf": 0.905
+     },
+     "score": {
+      "n": 62,
+      "accuracy": 0.7903,
+      "nll": 0.5051,
+      "ece": 0.0863,
+      "mean_conf": 0.7423
+     }
+    }
    },
-   "cliffwalking": {
-    "model": -60.0,
-    "teacher": -13.0,
-    "random": -574.8
+   "guard": {
+    "all": {
+     "n": 2994,
+     "accuracy": 0.8186,
+     "nll": 0.4866,
+     "ece": 0.0168,
+     "mean_conf": 0.8344
+    },
+    "by_type": {
+     "choice": {
+      "n": 2994,
+      "accuracy": 0.8186,
+      "nll": 0.4866,
+      "ece": 0.0168,
+      "mean_conf": 0.8344
+     },
+     "noul": null,
+     "score": null
+    }
    },
-   "blackjack": {
-    "model": -0.6,
-    "teacher": -0.6,
-    "random": -1.0
+   "val_jb": {
+    "all": {
+     "n": 5000,
+     "accuracy": 0.796,
+     "nll": 0.5264,
+     "ece": 0.0157,
+     "mean_conf": 0.8117
+    },
+    "by_type": {
+     "choice": {
+      "n": 3374,
+      "accuracy": 0.7647,
+      "nll": 0.6258,
+      "ece": 0.0289,
+      "mean_conf": 0.7935
+     },
+     "noul": {
+      "n": 1273,
+      "accuracy": 0.901,
+      "nll": 0.2233,
+      "ece": 0.0161,
+      "mean_conf": 0.8849
+     },
+     "score": {
+      "n": 353,
+      "accuracy": 0.7167,
+      "nll": 0.6681,
+      "ece": 0.0317,
+      "mean_conf": 0.722
+     }
+    }
    },
-   "minigrid_empty": {
-    "model": 0.0,
-    "teacher": 0.961328125,
-    "random": 0.0
+   "val_teacher": {
+    "all": {
+     "n": 944,
+     "accuracy": 0.8379,
+     "nll": 0.4169,
+     "ece": 0.0369,
+     "mean_conf": 0.8619
+    },
+    "by_type": {
+     "choice": {
+      "n": 540,
+      "accuracy": 0.8315,
+      "nll": 0.4643,
+      "ece": 0.0464,
+      "mean_conf": 0.8547
+     },
+     "noul": {
+      "n": 284,
+      "accuracy": 0.8873,
+      "nll": 0.2931,
+      "ece": 0.0453,
+      "mean_conf": 0.9088
+     },
+     "score": {
+      "n": 120,
+      "accuracy": 0.75,
+      "nll": 0.4969,
+      "ece": 0.0576,
+      "mean_conf": 0.7831
+     }
+    }
    },
-   "minigrid_lavagap": {
-    "model": 0.0,
-    "teacher": 0.19173469387755102,
-    "random": 0.0
+   "val_teacher2": {
+    "all": {
+     "n": 447,
+     "accuracy": 0.8322,
+     "nll": 0.4435,
+     "ece": 0.0627,
+     "mean_conf": 0.8747
+    },
+    "by_type": {
+     "choice": {
+      "n": 265,
+      "accuracy": 0.8377,
+      "nll": 0.4219,
+      "ece": 0.0503,
+      "mean_conf": 0.8858
+     },
+     "noul": {
+      "n": 120,
+      "accuracy": 0.85,
+      "nll": 0.3702,
+      "ece": 0.0827,
+      "mean_conf": 0.9034
+     },
+     "score": {
+      "n": 62,
+      "accuracy": 0.7742,
+      "nll": 0.6773,
+      "ece": 0.1441,
+      "mean_conf": 0.772
+     }
+    }
    },
-   "minigrid_doorkey": {
-    "model": 0.0,
-    "teacher": 0.0,
-    "random": 0.0
+   "cal_human": {
+    "all": {
+     "n": 1595,
+     "accuracy": 0.8564,
+     "nll": 0.3892,
+     "ece": 0.0205,
+     "mean_conf": 0.8717
+    },
+    "by_type": {
+     "choice": {
+      "n": 1595,
+      "accuracy": 0.8564,
+      "nll": 0.3892,
+      "ece": 0.0205,
+      "mean_conf": 0.8717
+     },
+     "noul": null,
+     "score": null
+    }
    },
-   "babyai_goto": {
-    "model": 0.3521875,
-    "teacher": 0.3409375,
-    "random": 0.1184375
+   "val_replay4": {
+    "all": {
+     "n": 1000,
+     "accuracy": 0.889,
+     "nll": 0.2848,
+     "ece": 0.0297,
+     "mean_conf": 0.8702
+    },
+    "by_type": {
+     "choice": {
+      "n": 1000,
+      "accuracy": 0.889,
+      "nll": 0.2848,
+      "ece": 0.0297,
+      "mean_conf": 0.8702
+     },
+     "noul": null,
+     "score": null
+    }
    }
   },
-  "miniwob_live_22x8": {
-   "sampled": 0.8807,
-   "sampled_held_out": 0.75,
-   "greedy": 0.9261,
-   "greedy_held_out": 0.8125
-  },
-  "decision_index_4k_sample": {
-   "note": "read at T 1.719 through decider-ai 1.2.1; calibration rescaled to T 1.935 from stored probabilities (argmax unchanged); sample index measured at T 1.719",
-   "sample_index_T1719": 48.3,
-   "ece_bw": 0.0738,
-   "acc_bw": 0.6373,
-   "conf_bw": 0.6901,
-   "over95_bw": 0.0123,
-   "brier_bw": 0.4788,
-   "pooled_ece": 0.0305,
-   "n": 27662,
-   "benchmarks": 33
-  },
-  "behaviour_probes": {
-   "note": "decider.probes.batteries / applications / isolated through decider-ai 1.2.1; the isolated-level probe reads raw logits (T=1)",
-   "generic": 0.95,
-   "specific": 1.0,
-   "catchall": 0.95,
-   "abstain_battery": "8/8",
-   "router_tier": 0.935,
-   "needs_live_data": 0.839,
-   "command_risk": 0.911,
-   "destructive_called_safe": 0,
-   "touches_outside_project": 0.933,
-   "browser_element": 0.938,
-   "browser_action": 0.938,
-   "isolated_minus_listwise_max_abs_points": 2.0,
-   "per_level_fit_sum_range": [
-    0.92,
-    1.0
-   ],
-   "listwise_ece_five_rating_sets_T1": [
-    0.127,
-    0.265
-   ]
-  },
-  "speed": {
-   "note": "same weights, measured on 2026-09-24 in the candidate readout session (server config T 1.719; the temperature does not change speed); one unshared B300, bf16",
-   "eager_game_state_median_ms": 34.6,
-   "v1_same_session_ms": 32.4,
-   "graphs_compile_single_request_ms": 5.2,
-   "fp8_single_request_ms": 5.0,
-   "batch32_decisions_per_s": 1178,
-   "batch32_fp8_decisions_per_s": 1349,
-   "http_decide_1_client_req_per_s": 72.8,
-   "http_decide_1_client_p50_ms": 13.3,
-   "http_decide_64_clients_req_per_s": 189.7
-  }
- },
- "paired": {
-  "regression_set_per_task_bootstrap": {
-   "v2-v1 T_1.05|in_task": {
-    "acc": {
-     "tasks": 67,
-     "delta": -0.00963326990956669,
-     "ci95": [
-      -0.012258285462742567,
-      -0.007113672993330958
-     ],
-     "higher": 6,
-     "lower": 54,
-     "equal": 7
-    },
-    "nll": {
-     "tasks": 67,
-     "delta": 0.0367151916395428,
-     "ci95": [
-      0.02853969128380196,
-      0.04618769902040932
-     ],
-     "higher": 64,
-     "lower": 3,
-     "equal": 0
-    },
-    "ece": {
-     "tasks": 67,
-     "delta": 0.014469173095775816,
-     "ci95": [
-      0.008371143739489972,
-      0.021221732726272202
-     ],
-     "higher": 49,
-     "lower": 18,
-     "equal": 0
+  "global_T": {
+   "heldout_jb": {
+    "all": {
+     "n": 5000,
+     "accuracy": 0.5562,
+     "nll": 1.2937,
+     "ece": 0.163,
+     "mean_conf": 0.7192
+    },
+    "by_type": {
+     "choice": {
+      "n": 3293,
+      "accuracy": 0.5056,
+      "nll": 1.5059,
+      "ece": 0.1724,
+      "mean_conf": 0.678
+     },
+     "noul": {
+      "n": 1289,
+      "accuracy": 0.7114,
+      "nll": 0.7112,
+      "ece": 0.1463,
+      "mean_conf": 0.8577
+     },
+     "score": {
+      "n": 418,
+      "accuracy": 0.4761,
+      "nll": 1.4183,
+      "ece": 0.1402,
+      "mean_conf": 0.6163
+     }
     }
    },
-   "v2-v1 T_1.05|heldout": {
-    "acc": {
-     "tasks": 28,
-     "delta": -0.008823480116377602,
-     "ci95": [
-      -0.019676785964620195,
-      0.001414172129592744
-     ],
-     "higher": 7,
-     "lower": 21,
-     "equal": 0
-    },
-    "nll": {
-     "tasks": 28,
-     "delta": 0.00806199632851141,
-     "ci95": [
-      -0.024438296351581807,
-      0.03533247855292367
-     ],
-     "higher": 20,
-     "lower": 8,
-     "equal": 0
-    },
-    "ece": {
-     "tasks": 28,
-     "delta": 0.009488148203351523,
-     "ci95": [
-      -0.002820037805815392,
-      0.02130860449786515
-     ],
-     "higher": 20,
-     "lower": 8,
-     "equal": 0
+   "test_teacher2": {
+    "all": {
+     "n": 449,
+     "accuracy": 0.8196,
+     "nll": 0.4381,
+     "ece": 0.0497,
+     "mean_conf": 0.8664
+    },
+    "by_type": {
+     "choice": {
+      "n": 257,
+      "accuracy": 0.8016,
+      "nll": 0.5058,
+      "ece": 0.058,
+      "mean_conf": 0.8596
+     },
+     "noul": {
+      "n": 130,
+      "accuracy": 0.8692,
+      "nll": 0.2755,
+      "ece": 0.0602,
+      "mean_conf": 0.9294
+     },
+     "score": {
+      "n": 62,
+      "accuracy": 0.7903,
+      "nll": 0.4984,
+      "ece": 0.0798,
+      "mean_conf": 0.7623
+     }
     }
    },
-   "v2-v10|in_task": {
-    "acc": {
-     "tasks": 67,
-     "delta": 0.01875302115583182,
-     "ci95": [
-      0.01015619959044516,
-      0.027614901925500205
-     ],
-     "higher": 44,
-     "lower": 21,
-     "equal": 2
-    },
-    "nll": {
-     "tasks": 67,
-     "delta": -0.032433275216773375,
-     "ci95": [
-      -0.05438354930479024,
-      -0.011119020796283635
-     ],
-     "higher": 26,
-     "lower": 41,
-     "equal": 0
-    },
-    "ece": {
-     "tasks": 67,
-     "delta": 0.004320193883140634,
-     "ci95": [
-      -0.0023582249916685393,
-      0.011625304997016925
-     ],
-     "higher": 34,
-     "lower": 33,
-     "equal": 0
+   "guard": {
+    "all": {
+     "n": 2994,
+     "accuracy": 0.8186,
+     "nll": 0.4871,
+     "ece": 0.0194,
+     "mean_conf": 0.836
+    },
+    "by_type": {
+     "choice": {
+      "n": 2994,
+      "accuracy": 0.8186,
+      "nll": 0.4871,
+      "ece": 0.0194,
+      "mean_conf": 0.836
+     },
+     "noul": null,
+     "score": null
     }
    },
-   "v2-v10|heldout": {
-    "acc": {
-     "tasks": 28,
-     "delta": 0.02370397041876381,
-     "ci95": [
-      0.011139798470057867,
-      0.03700944122337325
-     ],
-     "higher": 22,
-     "lower": 6,
-     "equal": 0
-    },
-    "nll": {
-     "tasks": 28,
-     "delta": -0.05593727236347539,
-     "ci95": [
-      -0.093489749350452,
-      -0.023956677370837752
-     ],
-     "higher": 6,
-     "lower": 22,
-     "equal": 0
-    },
-    "ece": {
-     "tasks": 28,
-     "delta": -0.003365505320264168,
-     "ci95": [
-      -0.01585441365540306,
-      0.008293014476259788
-     ],
-     "higher": 13,
-     "lower": 15,
-     "equal": 0
+   "val_jb": {
+    "all": {
+     "n": 5000,
+     "accuracy": 0.796,
+     "nll": 0.5269,
+     "ece": 0.0264,
+     "mean_conf": 0.8224
+    },
+    "by_type": {
+     "choice": {
+      "n": 3374,
+      "accuracy": 0.7647,
+      "nll": 0.6267,
+      "ece": 0.0303,
+      "mean_conf": 0.795
+     },
+     "noul": {
+      "n": 1273,
+      "accuracy": 0.901,
+      "nll": 0.223,
+      "ece": 0.0172,
+      "mean_conf": 0.9146
+     },
+     "score": {
+      "n": 353,
+      "accuracy": 0.7167,
+      "nll": 0.6687,
+      "ece": 0.0363,
+      "mean_conf": 0.7514
+     }
     }
    },
-   "v2-35b|in_task": {
-    "acc": {
-     "tasks": 67,
-     "delta": -0.031028858088253602,
-     "ci95": [
-      -0.039777794555657386,
-      -0.02317882618464841
-     ],
-     "higher": 3,
-     "lower": 61,
-     "equal": 3
-    },
-    "nll": {
-     "tasks": 67,
-     "delta": 0.08398698643544938,
-     "ci95": [
-      0.06702507901571886,
-      0.10335495943019389
-     ],
-     "higher": 67,
-     "lower": 0,
-     "equal": 0
-    },
-    "ece": {
-     "tasks": 67,
-     "delta": 0.015340690748236678,
-     "ci95": [
-      0.009872991235955714,
-      0.021626024452136343
-     ],
-     "higher": 55,
-     "lower": 12,
-     "equal": 0
+   "val_teacher": {
+    "all": {
+     "n": 944,
+     "accuracy": 0.8379,
+     "nll": 0.433,
+     "ece": 0.046,
+     "mean_conf": 0.8722
+    },
+    "by_type": {
+     "choice": {
+      "n": 540,
+      "accuracy": 0.8315,
+      "nll": 0.4654,
+      "ece": 0.0435,
+      "mean_conf": 0.8558
+     },
+     "noul": {
+      "n": 284,
+      "accuracy": 0.8873,
+      "nll": 0.3405,
+      "ece": 0.0634,
+      "mean_conf": 0.9331
+     },
+     "score": {
+      "n": 120,
+      "accuracy": 0.75,
+      "nll": 0.5064,
+      "ece": 0.0603,
+      "mean_conf": 0.8016
+     }
     }
    },
-   "v2-35b|heldout": {
-    "acc": {
-     "tasks": 28,
-     "delta": -0.03173775153445216,
-     "ci95": [
-      -0.045258765155078186,
-      -0.018006466515002026
-     ],
-     "higher": 4,
-     "lower": 24,
-     "equal": 0
-    },
-    "nll": {
-     "tasks": 28,
-     "delta": 0.06927653994145137,
-     "ci95": [
-      0.03497178333179493,
-      0.10177155091660098
-     ],
-     "higher": 24,
-     "lower": 4,
-     "equal": 0
-    },
-    "ece": {
-     "tasks": 28,
-     "delta": 0.01149923111649334,
-     "ci95": [
-      -0.010094514078614029,
-      0.03067631731368639
-     ],
-     "higher": 20,
-     "lower": 8,
-     "equal": 0
-    }
-   }
-  },
-  "fixtures_row_bootstrap_minus_v1": {
-   "general_validation": {
-    "accuracy": {
-     "delta": -0.010625737898465172,
-     "ci95": [
-      -0.025974025974025976,
-      0.004722550177095631
-     ]
-    },
-    "nll": {
-     "delta": 0.0015418035959596215,
-     "ci95": [
-      -0.018830652980342588,
-      0.02145239419471463
-     ]
+   "val_teacher2": {
+    "all": {
+     "n": 447,
+     "accuracy": 0.8277,
+     "nll": 0.4695,
+     "ece": 0.0806,
+     "mean_conf": 0.8852
+    },
+    "by_type": {
+     "choice": {
+      "n": 265,
+      "accuracy": 0.8377,
+      "nll": 0.4233,
+      "ece": 0.0546,
+      "mean_conf": 0.8868
+     },
+     "noul": {
+      "n": 120,
+      "accuracy": 0.85,
+      "nll": 0.4441,
+      "ece": 0.1059,
+      "mean_conf": 0.9311
+     },
+     "score": {
+      "n": 62,
+      "accuracy": 0.7419,
+      "nll": 0.7165,
+      "ece": 0.1556,
+      "mean_conf": 0.7893
+     }
     }
    },
-   "typesafe": {
-    "accuracy": {
-     "delta": 0.049019607843137254,
-     "ci95": [
-      -0.00980392156862745,
-      0.10784313725490197
-     ]
-    },
-    "nll": {
-     "delta": -0.20073007691013034,
-     "ci95": [
-      -0.34779699906684913,
-      -0.07224570606300662
-     ]
+   "cal_human": {
+    "all": {
+     "n": 1595,
+     "accuracy": 0.8564,
+     "nll": 0.3893,
+     "ece": 0.0195,
+     "mean_conf": 0.8733
+    },
+    "by_type": {
+     "choice": {
+      "n": 1595,
+      "accuracy": 0.8564,
+      "nll": 0.3893,
+      "ece": 0.0195,
+      "mean_conf": 0.8733
+     },
+     "noul": null,
+     "score": null
     }
    },
-   "openjev_external": {
-    "accuracy": {
-     "delta": 0.028370144706778372,
-     "ci95": [
-      0.017707539984767706,
-      0.03865194211728865
-     ]
-    },
-    "nll": {
-     "delta": -0.10404728527555326,
-     "ci95": [
-      -0.12081249076065353,
-      -0.08734350246247681
-     ]
+   "val_replay4": {
+    "all": {
+     "n": 1000,
+     "accuracy": 0.889,
+     "nll": 0.2843,
+     "ece": 0.0287,
+     "mean_conf": 0.8713
+    },
+    "by_type": {
+     "choice": {
+      "n": 1000,
+      "accuracy": 0.889,
+      "nll": 0.2843,
+      "ece": 0.0287,
+      "mean_conf": 0.8713
+     },
+     "noul": null,
+     "score": null
     }
+   }
+  }
+ },
+ "fixtures": {
+  "general_validation": {
+   "this": {
+    "rows": 847,
+    "accuracy": 0.8500590318772137,
+    "macro_accuracy": 0.8552272255409907,
+    "groups": 65,
+    "nll": 0.43237405419192004,
+    "ece": 0.03505942904541031,
+    "brier": 0.22353081852050935,
+    "mean_confidence": 0.8244136475475226
    },
-   "mind2web": {
-    "accuracy": {
-     "delta": -0.0096045197740113,
-     "ci95": [
-      -0.01977401129943503,
-      0.0005649717514124294
-     ]
-    },
-    "nll": {
-     "delta": 0.027492282187420017,
-     "ci95": [
-      0.010634569199404512,
-      0.04390014037543978
-     ]
+   "paired": {
+    "cand4b_B4_e2_tt-cand4b_B4_e2": {
+     "accuracy": {
+      "delta": 0.0,
+      "ci95": [
+       0.0,
+       0.0
+      ]
+     },
+     "nll": {
+      "delta": 0.00012970495542040443,
+      "ci95": [
+       -0.00039863639304211823,
+       0.0005942988193083918
+      ]
+     }
+    },
+    "cand4b_B4_e2_tt-v1_4b_p130": {
+     "accuracy": {
+      "delta": -0.010625737898465172,
+      "ci95": [
+       -0.021251475796930343,
+       0.0
+      ]
+     },
+     "nll": {
+      "delta": 0.015307099810435664,
+      "ci95": [
+       0.0030912070436396634,
+       0.028028207832146323
+      ]
+     }
+    },
+    "cand4b_B4_e2_tt-v2_4b_p130": {
+     "accuracy": {
+      "delta": 0.0,
+      "ci95": [
+       -0.015348288075560802,
+       0.015348288075560802
+      ]
+     },
+     "nll": {
+      "delta": 0.013765296214476044,
+      "ci95": [
+       -0.006225581100979938,
+       0.0345539752628619
+      ]
+     }
     }
    }
   },
-  "games_and_browser": {
-   "games": {
-    "v2-v1|sampled|all": {
-     "n": 234,
-     "a": 0.22435897435897437,
-     "b": 0.2777777777777778,
-     "delta": -0.053418803418803416,
-     "ci95": [
-      -0.07585470085470085,
-      -0.030982905982905984
-     ]
-    },
-    "v2-v1|sampled|bag_game": {
-     "n": 64,
-     "a": 0.37890625,
-     "b": 0.56640625,
-     "delta": -0.1875,
-     "ci95": [
-      -0.24609375,
-      -0.12890625
-     ]
-    },
-    "v2-v1|sampled|stochastic_grid": {
-     "n": 64,
-     "a": 0.16015625,
-     "b": 0.1640625,
-     "delta": -0.00390625,
-     "ci95": [
-      -0.02734375,
-      0.01953125
-     ]
-    },
-    "v2-v1|sampled|tic_tac_toe": {
-     "n": 74,
-     "a": 0.24324324324324326,
-     "b": 0.24324324324324326,
-     "delta": 0.0,
-     "ci95": [
-      -0.02702702702702703,
-      0.02702702702702703
-     ]
-    },
-    "v2-v1|sampled|exact_minesweeper": {
-     "n": 32,
-     "a": 0.0,
-     "b": 0.0078125,
-     "delta": -0.0078125,
-     "ci95": [
-      -0.0234375,
-      0.0
-     ]
-    },
-    "v2-v1|greedy|all": {
-     "n": 234,
-     "a": 0.2777777777777778,
-     "b": 0.2905982905982906,
-     "delta": -0.01282051282051282,
-     "ci95": [
-      -0.05555555555555555,
-      0.02564102564102564
-     ]
-    },
-    "v2-v1|greedy|bag_game": {
-     "n": 64,
-     "a": 0.484375,
-     "b": 0.625,
-     "delta": -0.140625,
-     "ci95": [
-      -0.234375,
-      -0.0625
-     ]
-    },
-    "v2-v1|greedy|stochastic_grid": {
-     "n": 64,
-     "a": 0.1875,
-     "b": 0.125,
-     "delta": 0.0625,
-     "ci95": [
-      0.0,
-      0.140625
-     ]
-    },
-    "v2-v1|greedy|tic_tac_toe": {
-     "n": 74,
-     "a": 0.2972972972972973,
-     "b": 0.2702702702702703,
-     "delta": 0.02702702702702703,
-     "ci95": [
-      -0.04054054054054054,
-      0.0945945945945946
-     ]
-    },
-    "v2-v1|greedy|exact_minesweeper": {
-     "n": 32,
-     "a": 0.0,
-     "b": 0.0,
-     "delta": 0.0,
-     "ci95": [
-      0.0,
-      0.0
-     ]
-    },
-    "v2-v2_T1719|sampled|all": {
-     "n": 234,
-     "a": 0.22435897435897437,
-     "b": 0.23397435897435898,
-     "delta": -0.009615384615384616,
-     "ci95": [
-      -0.021367521367521368,
-      0.0010683760683760685
-     ]
-    },
-    "v2-v2_T1719|sampled|bag_game": {
-     "n": 64,
-     "a": 0.37890625,
-     "b": 0.390625,
-     "delta": -0.01171875,
-     "ci95": [
-      -0.04296875,
-      0.01953125
-     ]
-    },
-    "v2-v2_T1719|sampled|stochastic_grid": {
-     "n": 64,
-     "a": 0.16015625,
-     "b": 0.1796875,
-     "delta": -0.01953125,
-     "ci95": [
-      -0.04296875,
-      0.0
-     ]
-    },
-    "v2-v2_T1719|sampled|tic_tac_toe": {
-     "n": 74,
-     "a": 0.24324324324324326,
-     "b": 0.24662162162162163,
-     "delta": -0.0033783783783783786,
-     "ci95": [
-      -0.016891891891891893,
-      0.006756756756756757
-     ]
-    },
-    "v2-v2_T1719|sampled|exact_minesweeper": {
-     "n": 32,
-     "a": 0.0,
-     "b": 0.0,
-     "delta": 0.0,
-     "ci95": [
-      0.0,
-      0.0
-     ]
-    },
-    "v2-v2_T1719|greedy|all": {
-     "n": 234,
-     "a": 0.2777777777777778,
-     "b": 0.2777777777777778,
-     "delta": 0.0,
-     "ci95": [
-      0.0,
-      0.0
-     ]
-    },
-    "v2-v2_T1719|greedy|bag_game": {
-     "n": 64,
-     "a": 0.484375,
-     "b": 0.484375,
-     "delta": 0.0,
-     "ci95": [
-      0.0,
-      0.0
-     ]
-    },
-    "v2-v2_T1719|greedy|stochastic_grid": {
-     "n": 64,
-     "a": 0.1875,
-     "b": 0.1875,
-     "delta": 0.0,
-     "ci95": [
-      0.0,
-      0.0
-     ]
-    },
-    "v2-v2_T1719|greedy|tic_tac_toe": {
-     "n": 74,
-     "a": 0.2972972972972973,
-     "b": 0.2972972972972973,
-     "delta": 0.0,
-     "ci95": [
-      0.0,
-      0.0
-     ]
-    },
-    "v2-v2_T1719|greedy|exact_minesweeper": {
-     "n": 32,
-     "a": 0.0,
-     "b": 0.0,
-     "delta": 0.0,
-     "ci95": [
-      0.0,
-      0.0
-     ]
-    },
-    "v2-v10|sampled|all": {
-     "n": 234,
-     "a": 0.22435897435897437,
-     "b": 0.23717948717948717,
-     "delta": -0.01282051282051282,
-     "ci95": [
-      -0.041666666666666664,
-      0.014957264957264958
-     ]
-    },
-    "v2-v10|sampled|bag_game": {
-     "n": 64,
-     "a": 0.37890625,
-     "b": 0.4140625,
-     "delta": -0.03515625,
-     "ci95": [
-      -0.08984375,
-      0.01953125
-     ]
-    },
-    "v2-v10|sampled|stochastic_grid": {
-     "n": 64,
-     "a": 0.16015625,
-     "b": 0.1875,
-     "delta": -0.02734375,
-     "ci95": [
-      -0.10546875,
-      0.04296875
-     ]
-    },
-    "v2-v10|sampled|tic_tac_toe": {
-     "n": 74,
-     "a": 0.24324324324324326,
-     "b": 0.22972972972972974,
-     "delta": 0.013513513513513514,
-     "ci95": [
-      -0.02364864864864865,
-      0.05405405405405406
-     ]
-    },
-    "v2-v10|sampled|exact_minesweeper": {
-     "n": 32,
-     "a": 0.0,
-     "b": 0.0,
-     "delta": 0.0,
-     "ci95": [
-      0.0,
-      0.0
-     ]
-    },
-    "v2-v10|greedy|all": {
-     "n": 234,
-     "a": 0.2777777777777778,
-     "b": 0.26495726495726496,
-     "delta": 0.01282051282051282,
-     "ci95": [
-      -0.042735042735042736,
-      0.06837606837606838
-     ]
-    },
-    "v2-v10|greedy|bag_game": {
-     "n": 64,
-     "a": 0.484375,
-     "b": 0.578125,
-     "delta": -0.09375,
-     "ci95": [
-      -0.203125,
-      0.015625
-     ]
-    },
-    "v2-v10|greedy|stochastic_grid": {
-     "n": 64,
-     "a": 0.1875,
-     "b": 0.15625,
-     "delta": 0.03125,
-     "ci95": [
-      -0.078125,
-      0.140625
-     ]
-    },
-    "v2-v10|greedy|tic_tac_toe": {
-     "n": 74,
-     "a": 0.2972972972972973,
-     "b": 0.20270270270270271,
-     "delta": 0.0945945945945946,
-     "ci95": [
-      -0.013513513513513514,
-      0.20270270270270271
-     ]
-    },
-    "v2-v10|greedy|exact_minesweeper": {
-     "n": 32,
-     "a": 0.0,
-     "b": 0.0,
-     "delta": 0.0,
-     "ci95": [
-      0.0,
-      0.0
-     ]
-    },
-    "v2-35b|sampled|all": {
-     "n": 234,
-     "a": 0.22435897435897437,
-     "b": 0.24145299145299146,
-     "delta": -0.017094017094017096,
-     "ci95": [
-      -0.03739316239316239,
-      0.002136752136752137
-     ]
-    },
-    "v2-35b|sampled|bag_game": {
-     "n": 64,
-     "a": 0.37890625,
-     "b": 0.41796875,
-     "delta": -0.0390625,
-     "ci95": [
-      -0.0859375,
-      0.0078125
-     ]
-    },
-    "v2-35b|sampled|stochastic_grid": {
-     "n": 64,
-     "a": 0.16015625,
-     "b": 0.16796875,
-     "delta": -0.0078125,
-     "ci95": [
-      -0.04296875,
-      0.02734375
-     ]
-    },
-    "v2-35b|sampled|tic_tac_toe": {
-     "n": 74,
-     "a": 0.24324324324324326,
-     "b": 0.25675675675675674,
-     "delta": -0.013513513513513514,
-     "ci95": [
-      -0.05405405405405406,
-      0.02027027027027027
-     ]
-    },
-    "v2-35b|sampled|exact_minesweeper": {
-     "n": 32,
-     "a": 0.0,
-     "b": 0.0,
-     "delta": 0.0,
-     "ci95": [
-      0.0,
-      0.0
-     ]
-    },
-    "v2-35b|greedy|all": {
-     "n": 234,
-     "a": 0.2777777777777778,
-     "b": 0.3717948717948718,
-     "delta": -0.09401709401709402,
-     "ci95": [
-      -0.14957264957264957,
-      -0.038461538461538464
-     ]
-    },
-    "v2-35b|greedy|bag_game": {
-     "n": 64,
-     "a": 0.484375,
-     "b": 0.625,
-     "delta": -0.140625,
-     "ci95": [
-      -0.234375,
-      -0.0625
-     ]
-    },
-    "v2-35b|greedy|stochastic_grid": {
-     "n": 64,
-     "a": 0.1875,
-     "b": 0.328125,
-     "delta": -0.140625,
-     "ci95": [
-      -0.25,
-      -0.03125
-     ]
-    },
-    "v2-35b|greedy|tic_tac_toe": {
-     "n": 74,
-     "a": 0.2972972972972973,
-     "b": 0.33783783783783783,
-     "delta": -0.04054054054054054,
-     "ci95": [
-      -0.16216216216216217,
-      0.08108108108108109
-     ]
-    },
-    "v2-35b|greedy|exact_minesweeper": {
-     "n": 32,
-     "a": 0.0,
-     "b": 0.03125,
-     "delta": -0.03125,
-     "ci95": [
-      -0.09375,
-      0.0
-     ]
-    }
+  "typesafe": {
+   "this": {
+    "rows": 102,
+    "accuracy": 0.8431372549019608,
+    "macro_accuracy": 0.8464696223316912,
+    "groups": 4,
+    "nll": 0.44128428424617644,
+    "ece": 0.06267209380280735,
+    "brier": 0.24132394423929263,
+    "mean_confidence": 0.8996653276331285,
+    "mean_total_variation_to_reference": 0.19246147475275616
    },
-   "browser": {
-    "v2-v1|greedy|all": {
-     "n": 176,
-     "a": 0.9261363636363636,
-     "b": 0.9147727272727273,
-     "delta": 0.011363636363636364,
-     "ci95": [
-      -0.017045454545454544,
-      0.03977272727272727
-     ]
-    },
-    "v2-v1|greedy|rewarded": {
-     "n": 128,
-     "a": 0.96875,
-     "b": 0.9765625,
-     "delta": -0.0078125,
-     "ci95": [
-      -0.0234375,
-      0.0
-     ]
-    },
-    "v2-v1|greedy|held_out": {
-     "n": 48,
-     "a": 0.8125,
-     "b": 0.75,
-     "delta": 0.0625,
-     "ci95": [
-      -0.020833333333333332,
-      0.16666666666666666
-     ]
-    },
-    "v2-v1|sampled|all": {
-     "n": 176,
-     "a": 0.8806818181818182,
-     "b": 0.9090909090909091,
-     "delta": -0.028409090909090908,
-     "ci95": [
-      -0.07386363636363637,
-      0.017045454545454544
-     ]
-    },
-    "v2-v1|sampled|rewarded": {
-     "n": 128,
-     "a": 0.9296875,
-     "b": 0.9609375,
-     "delta": -0.03125,
-     "ci95": [
-      -0.078125,
-      0.0078125
-     ]
-    },
-    "v2-v1|sampled|held_out": {
-     "n": 48,
-     "a": 0.75,
-     "b": 0.7708333333333334,
-     "delta": -0.020833333333333332,
-     "ci95": [
-      -0.14583333333333334,
-      0.10416666666666667
-     ]
-    },
-    "v2-v2_T1719|greedy|all": {
-     "n": 176,
-     "a": 0.9261363636363636,
-     "b": 0.9261363636363636,
-     "delta": 0.0,
-     "ci95": [
-      0.0,
-      0.0
-     ]
-    },
-    "v2-v2_T1719|greedy|rewarded": {
-     "n": 128,
-     "a": 0.96875,
-     "b": 0.96875,
-     "delta": 0.0,
-     "ci95": [
-      0.0,
-      0.0
-     ]
-    },
-    "v2-v2_T1719|greedy|held_out": {
-     "n": 48,
-     "a": 0.8125,
-     "b": 0.8125,
-     "delta": 0.0,
-     "ci95": [
-      0.0,
-      0.0
-     ]
-    },
-    "v2-v2_T1719|sampled|all": {
-     "n": 176,
-     "a": 0.8806818181818182,
-     "b": 0.8636363636363636,
-     "delta": 0.017045454545454544,
-     "ci95": [
-      -0.022727272727272728,
-      0.0625
-     ]
-    },
-    "v2-v2_T1719|sampled|rewarded": {
-     "n": 128,
-     "a": 0.9296875,
-     "b": 0.8984375,
-     "delta": 0.03125,
-     "ci95": [
-      -0.015625,
-      0.078125
-     ]
-    },
-    "v2-v2_T1719|sampled|held_out": {
-     "n": 48,
-     "a": 0.75,
-     "b": 0.7708333333333334,
-     "delta": -0.020833333333333332,
-     "ci95": [
-      -0.10416666666666667,
-      0.0625
-     ]
-    },
-    "v2-v10|greedy|all": {
-     "n": 176,
-     "a": 0.9261363636363636,
-     "b": 0.9090909090909091,
-     "delta": 0.017045454545454544,
-     "ci95": [
-      -0.03977272727272727,
-      0.07386363636363637
-     ]
-    },
-    "v2-v10|greedy|rewarded": {
-     "n": 128,
-     "a": 0.96875,
-     "b": 0.90625,
-     "delta": 0.0625,
-     "ci95": [
-      0.0078125,
-      0.125
-     ]
-    },
-    "v2-v10|greedy|held_out": {
-     "n": 48,
-     "a": 0.8125,
-     "b": 0.9166666666666666,
-     "delta": -0.10416666666666667,
-     "ci95": [
-      -0.22916666666666666,
-      0.020833333333333332
-     ]
-    },
-    "v2-v10|sampled|all": {
-     "n": 176,
-     "a": 0.8806818181818182,
-     "b": 0.9318181818181818,
-     "delta": -0.05113636363636364,
-     "ci95": [
-      -0.10795454545454546,
-      0.005681818181818182
-     ]
-    },
-    "v2-v10|sampled|rewarded": {
-     "n": 128,
-     "a": 0.9296875,
-     "b": 0.9375,
-     "delta": -0.0078125,
-     "ci95": [
-      -0.0703125,
-      0.0546875
-     ]
-    },
-    "v2-v10|sampled|held_out": {
-     "n": 48,
-     "a": 0.75,
-     "b": 0.9166666666666666,
-     "delta": -0.16666666666666666,
-     "ci95": [
-      -0.3125,
-      -0.041666666666666664
-     ]
-    },
-    "v2-35b|greedy|all": {
-     "n": 176,
-     "a": 0.9261363636363636,
-     "b": 0.9715909090909091,
-     "delta": -0.045454545454545456,
-     "ci95": [
-      -0.07954545454545454,
-      -0.017045454545454544
-     ]
-    },
-    "v2-35b|greedy|rewarded": {
-     "n": 128,
-     "a": 0.96875,
-     "b": 0.96875,
-     "delta": 0.0,
-     "ci95": [
-      0.0,
-      0.0
-     ]
-    },
-    "v2-35b|greedy|held_out": {
-     "n": 48,
-     "a": 0.8125,
-     "b": 0.9791666666666666,
-     "delta": -0.16666666666666666,
-     "ci95": [
-      -0.2708333333333333,
-      -0.0625
-     ]
-    },
-    "v2-35b|sampled|all": {
-     "n": 176,
-     "a": 0.8806818181818182,
-     "b": 0.8636363636363636,
-     "delta": 0.017045454545454544,
-     "ci95": [
-      -0.03409090909090909,
-      0.06832386363636415
-     ]
-    },
-    "v2-35b|sampled|rewarded": {
-     "n": 128,
-     "a": 0.9296875,
-     "b": 0.890625,
-     "delta": 0.0390625,
-     "ci95": [
-      -0.015625,
-      0.09375
-     ]
-    },
-    "v2-35b|sampled|held_out": {
-     "n": 48,
-     "a": 0.75,
-     "b": 0.7916666666666666,
-     "delta": -0.041666666666666664,
-     "ci95": [
-      -0.16666666666666666,
-      0.08333333333333333
-     ]
-    },
-    "per_task|greedy": {
-     "click-button": 1.0,
-     "click-button-sequence": 1.0,
-     "click-checkboxes": 1.0,
-     "click-checkboxes-large": 0.625,
-     "click-checkboxes-soft": 1.0,
-     "click-checkboxes-transfer": 1.0,
-     "click-collapsible": 1.0,
-     "click-collapsible-2": 0.5,
-     "click-color": 1.0,
-     "click-dialog": 1.0,
-     "click-dialog-2": 1.0,
-     "click-link": 1.0,
-     "click-option": 1.0,
-     "click-tab": 1.0,
-     "click-tab-2": 0.875,
-     "click-tab-2-hard": 1.0,
-     "click-test": 1.0,
-     "click-test-2": 1.0,
-     "click-widget": 1.0,
-     "focus-text": 1.0,
-     "focus-text-2": 0.375,
-     "navigate-tree": 1.0
-    },
-    "per_task|sampled": {
-     "click-button": 1.0,
-     "click-button-sequence": 1.0,
-     "click-checkboxes": 1.0,
-     "click-checkboxes-large": 0.75,
-     "click-checkboxes-soft": 0.875,
-     "click-checkboxes-transfer": 0.875,
-     "click-collapsible": 1.0,
-     "click-collapsible-2": 0.75,
-     "click-color": 1.0,
-     "click-dialog": 1.0,
-     "click-dialog-2": 0.5,
-     "click-link": 1.0,
-     "click-option": 0.875,
-     "click-tab": 0.625,
-     "click-tab-2": 0.75,
-     "click-tab-2-hard": 0.625,
-     "click-test": 1.0,
-     "click-test-2": 1.0,
-     "click-widget": 1.0,
-     "focus-text": 1.0,
-     "focus-text-2": 0.75,
-     "navigate-tree": 1.0
+   "paired": {
+    "cand4b_B4_e2_tt-cand4b_B4_e2": {
+     "accuracy": {
+      "delta": 0.0,
+      "ci95": [
+       0.0,
+       0.0
+      ]
+     },
+     "nll": {
+      "delta": -0.0017070072924830098,
+      "ci95": [
+       -0.003544773928864442,
+       -8.629601238631066e-05
+      ]
+     }
+    },
+    "cand4b_B4_e2_tt-v1_4b_p130": {
+     "accuracy": {
+      "delta": 0.029411764705882353,
+      "ci95": [
+       -0.029411764705882353,
+       0.08823529411764706
+      ]
+     },
+     "nll": {
+      "delta": -0.16956755666314244,
+      "ci95": [
+       -0.3236270765220259,
+       -0.035752501984007506
+      ]
+     }
+    },
+    "cand4b_B4_e2_tt-v2_4b_p130": {
+     "accuracy": {
+      "delta": -0.0196078431372549,
+      "ci95": [
+       -0.0784313725490196,
+       0.0392156862745098
+      ]
+     },
+     "nll": {
+      "delta": 0.03116252024698791,
+      "ci95": [
+       -0.06909971743191749,
+       0.13434316002254298
+      ]
+     }
     }
    }
   },
-  "jevbench_item_bootstrap_minus_v1": {
-   "easy": {
-    "n": 48,
-    "a": 1.0,
-    "b": 1.0,
-    "delta": 0.0,
-    "ci95": [
-     0.0,
-     0.0
-    ],
-    "a_only": 0,
-    "b_only": 0,
-    "argmax_differs": 0
+  "openjev_external": {
+   "this": {
+    "rows": 5252,
+    "accuracy": 0.6597486671744097,
+    "macro_accuracy": 0.8063444444444444,
+    "groups": 3,
+    "nll": 0.8457289344223444,
+    "ece": 0.1539857582574297,
+    "brier": 0.5022800176356607,
+    "mean_confidence": 0.8111537299704116
    },
-   "standard": {
-    "n": 72,
-    "a": 0.9861111111111112,
-    "b": 0.9583333333333334,
-    "delta": 0.027777777777777776,
-    "ci95": [
-     0.0,
-     0.06944444444444445
-    ],
-    "a_only": 2,
-    "b_only": 0,
-    "argmax_differs": 2
+   "paired": {
+    "cand4b_B4_e2_tt-cand4b_B4_e2": {
+     "accuracy": {
+      "delta": 0.0,
+      "ci95": [
+       0.0,
+       0.0
+      ]
+     },
+     "nll": {
+      "delta": -0.003617502816960803,
+      "ci95": [
+       -0.003919285763621306,
+       -0.00332506503930084
+      ]
+     }
+    },
+    "cand4b_B4_e2_tt-v1_4b_p130": {
+     "accuracy": {
+      "delta": 0.020944402132520943,
+      "ci95": [
+       0.012376237623762377,
+       0.029322162985529324
+      ]
+     },
+     "nll": {
+      "delta": -0.047333755296254304,
+      "ci95": [
+       -0.059559262901540976,
+       -0.035193633735333484
+      ]
+     }
+    },
+    "cand4b_B4_e2_tt-v2_4b_p130": {
+     "accuracy": {
+      "delta": -0.007425742574257425,
+      "ci95": [
+       -0.015993907083015995,
+       0.0017136329017517135
+      ]
+     },
+     "nll": {
+      "delta": 0.05671352997929895,
+      "ci95": [
+       0.04359541016972144,
+       0.07033520922345497
+      ]
+     }
+    }
+   }
+  },
+  "mind2web": {
+   "this": {
+    "rows": 1770,
+    "accuracy": 0.8751412429378531,
+    "macro_accuracy": 0.8773196126201471,
+    "groups": 3,
+    "nll": 0.37523612083453944,
+    "ece": 0.016833950116135952,
+    "brier": 0.18402882486455008,
+    "mean_confidence": 0.8624463401776923
    },
-   "hard": {
-    "n": 111,
-    "a": 0.6756756756756757,
-    "b": 0.5495495495495496,
-    "delta": 0.12612612612612611,
-    "ci95": [
-     0.04504504504504504,
-     0.20743243243243326
-    ],
-    "a_only": 20,
-    "b_only": 6,
-    "argmax_differs": 32
+   "paired": {
+    "cand4b_B4_e2_tt-cand4b_B4_e2": {
+     "accuracy": {
+      "delta": 0.0,
+      "ci95": [
+       0.0,
+       0.0
+      ]
+     },
+     "nll": {
+      "delta": 0.00026909733615400604,
+      "ci95": [
+       -4.997823592689937e-05,
+       0.0006061414472582659
+      ]
+     }
+    },
+    "cand4b_B4_e2_tt-v1_4b_p130": {
+     "accuracy": {
+      "delta": -0.00847457627118644,
+      "ci95": [
+       -0.01694915254237288,
+       0.0
+      ]
+     },
+     "nll": {
+      "delta": 0.00924747244030173,
+      "ci95": [
+       -0.004092843962589085,
+       0.022512164121517977
+      ]
+     }
+    },
+    "cand4b_B4_e2_tt-v2_4b_p130": {
+     "accuracy": {
+      "delta": 0.0011299435028248588,
+      "ci95": [
+       -0.00847457627118644,
+       0.011299435028248588
+      ]
+     },
+     "nll": {
+      "delta": -0.018244809747118285,
+      "ci95": [
+       -0.032040489157299956,
+       -0.005027723688309381
+      ]
+     }
+    }
+   }
+  }
+ },
+ "games": {
+  "cand4b_B4_e2_tt-cand4b_B4_e2|sampled|all": {
+   "n": 234,
+   "a": 0.2692307692307692,
+   "b": 0.2692307692307692,
+   "delta": 0.0,
+   "ci95": [
+    -0.004273504273504274,
+    0.004273504273504274
+   ]
+  },
+  "cand4b_B4_e2_tt-cand4b_B4_e2|sampled|bag_game": {
+   "n": 64,
+   "a": 0.51953125,
+   "b": 0.51953125,
+   "delta": 0.0,
+   "ci95": [
+    -0.01171875,
+    0.01171875
+   ]
+  },
+  "cand4b_B4_e2_tt-cand4b_B4_e2|sampled|stochastic_grid": {
+   "n": 64,
+   "a": 0.16796875,
+   "b": 0.16796875,
+   "delta": 0.0,
+   "ci95": [
+    -0.01171875,
+    0.01171875
+   ]
+  },
+  "cand4b_B4_e2_tt-cand4b_B4_e2|sampled|tic_tac_toe": {
+   "n": 74,
+   "a": 0.2533783783783784,
+   "b": 0.2533783783783784,
+   "delta": 0.0,
+   "ci95": [
+    0.0,
+    0.0
+   ]
+  },
+  "cand4b_B4_e2_tt-cand4b_B4_e2|sampled|exact_minesweeper": {
+   "n": 32,
+   "a": 0.0078125,
+   "b": 0.0078125,
+   "delta": 0.0,
+   "ci95": [
+    0.0,
+    0.0
+   ]
+  },
+  "cand4b_B4_e2_tt-cand4b_B4_e2|greedy|all": {
+   "n": 234,
+   "a": 0.25213675213675213,
+   "b": 0.25213675213675213,
+   "delta": 0.0,
+   "ci95": [
+    0.0,
+    0.0
+   ]
+  },
+  "cand4b_B4_e2_tt-cand4b_B4_e2|greedy|bag_game": {
+   "n": 64,
+   "a": 0.53125,
+   "b": 0.53125,
+   "delta": 0.0,
+   "ci95": [
+    0.0,
+    0.0
+   ]
+  },
+  "cand4b_B4_e2_tt-cand4b_B4_e2|greedy|stochastic_grid": {
+   "n": 64,
+   "a": 0.109375,
+   "b": 0.109375,
+   "delta": 0.0,
+   "ci95": [
+    0.0,
+    0.0
+   ]
+  },
+  "cand4b_B4_e2_tt-cand4b_B4_e2|greedy|tic_tac_toe": {
+   "n": 74,
+   "a": 0.24324324324324326,
+   "b": 0.24324324324324326,
+   "delta": 0.0,
+   "ci95": [
+    0.0,
+    0.0
+   ]
+  },
+  "cand4b_B4_e2_tt-cand4b_B4_e2|greedy|exact_minesweeper": {
+   "n": 32,
+   "a": 0.0,
+   "b": 0.0,
+   "delta": 0.0,
+   "ci95": [
+    0.0,
+    0.0
+   ]
+  },
+  "cand4b_B4_e2_tt-v1_4b_p130|sampled|all": {
+   "n": 234,
+   "a": 0.2692307692307692,
+   "b": 0.2777777777777778,
+   "delta": -0.008547008547008548,
+   "ci95": [
+    -0.02564102564102564,
+    0.009615384615384616
+   ]
+  },
+  "cand4b_B4_e2_tt-v1_4b_p130|sampled|bag_game": {
+   "n": 64,
+   "a": 0.51953125,
+   "b": 0.56640625,
+   "delta": -0.046875,
+   "ci95": [
+    -0.08984375,
+    -0.00390625
+   ]
+  },
+  "cand4b_B4_e2_tt-v1_4b_p130|sampled|stochastic_grid": {
+   "n": 64,
+   "a": 0.16796875,
+   "b": 0.1640625,
+   "delta": 0.00390625,
+   "ci95": [
+    -0.02734375,
+    0.03515625
+   ]
+  },
+  "cand4b_B4_e2_tt-v1_4b_p130|sampled|tic_tac_toe": {
+   "n": 74,
+   "a": 0.2533783783783784,
+   "b": 0.24324324324324326,
+   "delta": 0.010135135135135136,
+   "ci95": [
+    -0.013513513513513514,
+    0.037162162162162164
+   ]
+  },
+  "cand4b_B4_e2_tt-v1_4b_p130|sampled|exact_minesweeper": {
+   "n": 32,
+   "a": 0.0078125,
+   "b": 0.0078125,
+   "delta": 0.0,
+   "ci95": [
+    -0.0234375,
+    0.0234375
+   ]
+  },
+  "cand4b_B4_e2_tt-v1_4b_p130|greedy|all": {
+   "n": 234,
+   "a": 0.25213675213675213,
+   "b": 0.2905982905982906,
+   "delta": -0.038461538461538464,
+   "ci95": [
+    -0.07264957264957266,
+    -0.004273504273504274
+   ]
+  },
+  "cand4b_B4_e2_tt-v1_4b_p130|greedy|bag_game": {
+   "n": 64,
+   "a": 0.53125,
+   "b": 0.625,
+   "delta": -0.09375,
+   "ci95": [
+    -0.171875,
+    -0.03125
+   ]
+  },
+  "cand4b_B4_e2_tt-v1_4b_p130|greedy|stochastic_grid": {
+   "n": 64,
+   "a": 0.109375,
+   "b": 0.125,
+   "delta": -0.015625,
+   "ci95": [
+    -0.07851562499999987,
+    0.046875
+   ]
+  },
+  "cand4b_B4_e2_tt-v1_4b_p130|greedy|tic_tac_toe": {
+   "n": 74,
+   "a": 0.24324324324324326,
+   "b": 0.2702702702702703,
+   "delta": -0.02702702702702703,
+   "ci95": [
+    -0.0945945945945946,
+    0.04054054054054054
+   ]
+  },
+  "cand4b_B4_e2_tt-v1_4b_p130|greedy|exact_minesweeper": {
+   "n": 32,
+   "a": 0.0,
+   "b": 0.0,
+   "delta": 0.0,
+   "ci95": [
+    0.0,
+    0.0
+   ]
+  },
+  "cand4b_B4_e2_tt-v2_4b_p130|sampled|all": {
+   "n": 234,
+   "a": 0.2692307692307692,
+   "b": 0.22435897435897437,
+   "delta": 0.04487179487179487,
+   "ci95": [
+    0.024572649572649572,
+    0.06623931623931624
+   ]
+  },
+  "cand4b_B4_e2_tt-v2_4b_p130|sampled|bag_game": {
+   "n": 64,
+   "a": 0.51953125,
+   "b": 0.37890625,
+   "delta": 0.140625,
+   "ci95": [
+    0.08203125,
+    0.19921875
+   ]
+  },
+  "cand4b_B4_e2_tt-v2_4b_p130|sampled|stochastic_grid": {
+   "n": 64,
+   "a": 0.16796875,
+   "b": 0.16015625,
+   "delta": 0.0078125,
+   "ci95": [
+    -0.0234375,
+    0.0390625
+   ]
+  },
+  "cand4b_B4_e2_tt-v2_4b_p130|sampled|tic_tac_toe": {
+   "n": 74,
+   "a": 0.2533783783783784,
+   "b": 0.24324324324324326,
+   "delta": 0.010135135135135136,
+   "ci95": [
+    -0.010135135135135136,
+    0.033783783783783786
+   ]
+  },
+  "cand4b_B4_e2_tt-v2_4b_p130|sampled|exact_minesweeper": {
+   "n": 32,
+   "a": 0.0078125,
+   "b": 0.0,
+   "delta": 0.0078125,
+   "ci95": [
+    0.0,
+    0.0234375
+   ]
+  },
+  "cand4b_B4_e2_tt-v2_4b_p130|greedy|all": {
+   "n": 234,
+   "a": 0.25213675213675213,
+   "b": 0.2777777777777778,
+   "delta": -0.02564102564102564,
+   "ci95": [
+    -0.0641025641025641,
+    0.01282051282051282
+   ]
+  },
+  "cand4b_B4_e2_tt-v2_4b_p130|greedy|bag_game": {
+   "n": 64,
+   "a": 0.53125,
+   "b": 0.484375,
+   "delta": 0.046875,
+   "ci95": [
+    -0.03125,
+    0.125
+   ]
+  },
+  "cand4b_B4_e2_tt-v2_4b_p130|greedy|stochastic_grid": {
+   "n": 64,
+   "a": 0.109375,
+   "b": 0.1875,
+   "delta": -0.078125,
+   "ci95": [
+    -0.15625,
+    0.0
+   ]
+  },
+  "cand4b_B4_e2_tt-v2_4b_p130|greedy|tic_tac_toe": {
+   "n": 74,
+   "a": 0.24324324324324326,
+   "b": 0.2972972972972973,
+   "delta": -0.05405405405405406,
+   "ci95": [
+    -0.12162162162162163,
+    0.02702702702702703
+   ]
+  },
+  "cand4b_B4_e2_tt-v2_4b_p130|greedy|exact_minesweeper": {
+   "n": 32,
+   "a": 0.0,
+   "b": 0.0,
+   "delta": 0.0,
+   "ci95": [
+    0.0,
+    0.0
+   ]
+  }
+ },
+ "browser": {
+  "cand4b_B4_e2_tt-cand4b_B4_e2|sampled|all": {
+   "n": 176,
+   "a": 0.9318181818181818,
+   "b": 0.9318181818181818,
+   "delta": 0.0,
+   "ci95": [
+    0.0,
+    0.0
+   ]
+  },
+  "cand4b_B4_e2_tt-cand4b_B4_e2|sampled|rewarded": {
+   "n": 128,
+   "a": 0.953125,
+   "b": 0.953125,
+   "delta": 0.0,
+   "ci95": [
+    0.0,
+    0.0
+   ]
+  },
+  "cand4b_B4_e2_tt-cand4b_B4_e2|sampled|held_out": {
+   "n": 48,
+   "a": 0.875,
+   "b": 0.875,
+   "delta": 0.0,
+   "ci95": [
+    0.0,
+    0.0
+   ]
+  },
+  "cand4b_B4_e2_tt-cand4b_B4_e2|greedy|all": {
+   "n": 176,
+   "a": 0.9375,
+   "b": 0.9375,
+   "delta": 0.0,
+   "ci95": [
+    0.0,
+    0.0
+   ]
+  },
+  "cand4b_B4_e2_tt-cand4b_B4_e2|greedy|rewarded": {
+   "n": 128,
+   "a": 0.9609375,
+   "b": 0.9609375,
+   "delta": 0.0,
+   "ci95": [
+    0.0,
+    0.0
+   ]
+  },
+  "cand4b_B4_e2_tt-cand4b_B4_e2|greedy|held_out": {
+   "n": 48,
+   "a": 0.875,
+   "b": 0.875,
+   "delta": 0.0,
+   "ci95": [
+    0.0,
+    0.0
+   ]
+  },
+  "cand4b_B4_e2_tt-v1_4b_p130|sampled|all": {
+   "n": 176,
+   "a": 0.9318181818181818,
+   "b": 0.9090909090909091,
+   "delta": 0.022727272727272728,
+   "ci95": [
+    -0.017045454545454544,
+    0.0625
+   ]
+  },
+  "cand4b_B4_e2_tt-v1_4b_p130|sampled|rewarded": {
+   "n": 128,
+   "a": 0.953125,
+   "b": 0.9609375,
+   "delta": -0.0078125,
+   "ci95": [
+    -0.0390625,
+    0.0234375
+   ]
+  },
+  "cand4b_B4_e2_tt-v1_4b_p130|sampled|held_out": {
+   "n": 48,
+   "a": 0.875,
+   "b": 0.7708333333333334,
+   "delta": 0.10416666666666667,
+   "ci95": [
+    0.0,
+    0.20833333333333334
+   ]
+  },
+  "cand4b_B4_e2_tt-v1_4b_p130|greedy|all": {
+   "n": 176,
+   "a": 0.9375,
+   "b": 0.9147727272727273,
+   "delta": 0.022727272727272728,
+   "ci95": [
+    -0.005681818181818182,
+    0.056818181818181816
+   ]
+  },
+  "cand4b_B4_e2_tt-v1_4b_p130|greedy|rewarded": {
+   "n": 128,
+   "a": 0.9609375,
+   "b": 0.9765625,
+   "delta": -0.015625,
+   "ci95": [
+    -0.0390625,
+    0.0
+   ]
+  },
+  "cand4b_B4_e2_tt-v1_4b_p130|greedy|held_out": {
+   "n": 48,
+   "a": 0.875,
+   "b": 0.75,
+   "delta": 0.125,
+   "ci95": [
+    0.041666666666666664,
+    0.22916666666666666
+   ]
+  },
+  "cand4b_B4_e2_tt-v2_4b_p130|sampled|all": {
+   "n": 176,
+   "a": 0.9318181818181818,
+   "b": 0.8806818181818182,
+   "delta": 0.05113636363636364,
+   "ci95": [
+    0.005681818181818182,
+    0.09659090909090909
+   ]
+  },
+  "cand4b_B4_e2_tt-v2_4b_p130|sampled|rewarded": {
+   "n": 128,
+   "a": 0.953125,
+   "b": 0.9296875,
+   "delta": 0.0234375,
+   "ci95": [
+    -0.0234375,
+    0.0703125
+   ]
+  },
+  "cand4b_B4_e2_tt-v2_4b_p130|sampled|held_out": {
+   "n": 48,
+   "a": 0.875,
+   "b": 0.75,
+   "delta": 0.125,
+   "ci95": [
+    0.0,
+    0.25
+   ]
+  },
+  "cand4b_B4_e2_tt-v2_4b_p130|greedy|all": {
+   "n": 176,
+   "a": 0.9375,
+   "b": 0.9261363636363636,
+   "delta": 0.011363636363636364,
+   "ci95": [
+    -0.017045454545454544,
+    0.045454545454545456
+   ]
+  },
+  "cand4b_B4_e2_tt-v2_4b_p130|greedy|rewarded": {
+   "n": 128,
+   "a": 0.9609375,
+   "b": 0.96875,
+   "delta": -0.0078125,
+   "ci95": [
+    -0.0234375,
+    0.0
+   ]
+  },
+  "cand4b_B4_e2_tt-v2_4b_p130|greedy|held_out": {
+   "n": 48,
+   "a": 0.875,
+   "b": 0.8125,
+   "delta": 0.0625,
+   "ci95": [
+    -0.041666666666666664,
+    0.16666666666666666
+   ]
+  }
+ },
+ "text_games_greedy": {
+  "pong": -5.0,
+  "freeway": 0.0,
+  "breakout": 35.0,
+  "frozenlake": 0.0,
+  "cliffwalking": -13.0,
+  "blackjack": -0.6,
+  "minigrid_empty": 0.0,
+  "minigrid_lavagap": 0.0,
+  "minigrid_doorkey": 0.0,
+  "babyai_goto": 0.1915625
+ },
+ "probes": {
+  "router_n": 31,
+  "router_tier": 0.968,
+  "needs_live_data": 0.839,
+  "commands_n": 45,
+  "command_risk": 0.889,
+  "destructive_called_safe": 0,
+  "outside_project": 0.933,
+  "browser_element": 0.938,
+  "browser_action": 0.75,
+  "batteries": {
+   "generic": 1.0,
+   "specific": 1.0,
+   "catchall": 0.95,
+   "noul_acc": 1.0,
+   "noul_brier": 0.01,
+   "n_choice": 60,
+   "n_noul": 49,
+   "abstain_battery": "8/8"
+  }
+ },
+ "issue9": {
+  "c_1": {
+   "pred": 26,
+   "gold": 6,
+   "ok": false,
+   "p_gold": 0.198,
+   "p_pred": 0.764,
+   "T": 1.099
+  },
+  "c_2": {
+   "pred": 20,
+   "gold": 20,
+   "ok": true,
+   "p_gold": 0.409,
+   "p_pred": 0.409,
+   "T": 1.099
+  },
+  "b_10": {
+   "pred": 0,
+   "gold": 0,
+   "ok": true,
+   "p_gold": 0.732,
+   "p_pred": 0.732,
+   "T": 1.099
+  },
+  "a2_4": {
+   "pred": 4,
+   "gold": 4,
+   "ok": true,
+   "p_gold": 0.577,
+   "p_pred": 0.577,
+   "T": 1.099
+  }
+ },
+ "bespoke": {
+  "vitaminc-dev": {
+   "n": 599,
+   "type": "choice",
+   "acc": 0.7429,
+   "ece": 0.1456,
+   "brier": 0.3945,
+   "nll": 0.7789,
+   "trained": false,
+   "sec": 11.8
+  },
+  "massive-en-US": {
+   "n": 350,
+   "type": "choice",
+   "acc": 0.8629,
+   "ece": 0.0411,
+   "brier": 0.1936,
+   "nll": 0.4173,
+   "trained": true,
+   "sec": 6.2
+  },
+  "massive-de-DE": {
+   "n": 350,
+   "type": "choice",
+   "acc": 0.84,
+   "ece": 0.0297,
+   "brier": 0.2269,
+   "nll": 0.4947,
+   "trained": false,
+   "sec": 4.5
+  },
+  "boolq": {
+   "n": 300,
+   "type": "noul",
+   "acc": 0.8667,
+   "ece": 0.0319,
+   "brier": 0.1862,
+   "nll": 0.3007,
+   "trained": true,
+   "sec": 6.2
+  },
+  "squad2": {
+   "n": 299,
+   "type": "noul",
+   "acc": 0.7893,
+   "ece": 0.0441,
+   "brier": 0.2955,
+   "nll": 0.452,
+   "trained": false,
+   "sec": 7.2
+  },
+  "paws": {
+   "n": 250,
+   "type": "noul",
+   "acc": 0.764,
+   "ece": 0.1238,
+   "brier": 0.3398,
+   "nll": 0.5538,
+   "trained": true,
+   "sec": 3.0
+  },
+  "multinli": {
+   "n": 299,
+   "type": "choice",
+   "acc": 0.9331,
+   "ece": 0.0103,
+   "brier": 0.1089,
+   "nll": 0.2099,
+   "trained": true,
+   "sec": 3.1
+  },
+  "civil_comments": {
+   "n": 300,
+   "type": "noul",
+   "acc": 0.8733,
+   "ece": 0.0566,
+   "brier": 0.1917,
+   "nll": 0.3199,
+   "trained": true,
+   "sec": 3.2
+  },
+  "aegis2": {
+   "n": 250,
+   "type": "noul",
+   "acc": 0.804,
+   "ece": 0.0467,
+   "brier": 0.2797,
+   "nll": 0.4527,
+   "trained": false,
+   "sec": 6.1
+  },
+  "helpsteer2": {
+   "n": 249,
+   "type": "score",
+   "acc": 0.4498,
+   "ece": 0.142,
+   "brier": 0.6713,
+   "nll": 1.2502,
+   "trained": true,
+   "sec": 25.8,
+   "score_mae": 0.797
+  },
+  "summeval-relevance": {
+   "n": 240,
+   "type": "score",
+   "acc": 0.3958,
+   "ece": 0.0696,
+   "brier": 0.6844,
+   "nll": 1.2541,
+   "trained": false,
+   "sec": 20.1,
+   "score_mae": 0.6172
+  },
+  "summeval-consistency": {
+   "n": 144,
+   "type": "score",
+   "acc": 0.7986,
+   "ece": 0.3095,
+   "brier": 0.4741,
+   "nll": 0.9007,
+   "trained": false,
+   "sec": 12.2,
+   "score_mae": 0.7974
+  },
+  "pubmedqa": {
+   "n": 250,
+   "type": "choice",
+   "acc": 0.708,
+   "ece": 0.0863,
+   "brier": 0.4012,
+   "nll": 0.7205,
+   "trained": true,
+   "sec": 3.8
+  },
+  "_macro": 0.756,
+  "_micro": 0.7652,
+  "_macro_untrained": 0.7284
+ },
+ "jevbench_public": {
+  "easy": {
+   "n": 48,
+   "accuracy": 1.0,
+   "invalid": 0,
+   "ece": 0.0027979166666670663,
+   "by_family": {
+    "extraction": {
+     "n": 12,
+     "accuracy": 1.0
+    },
+    "fact": {
+     "n": 12,
+     "accuracy": 1.0
+    },
+    "intent": {
+     "n": 12,
+     "accuracy": 1.0
+    },
+    "tool_selection": {
+     "n": 12,
+     "accuracy": 1.0
+    }
+   }
+  },
+  "standard": {
+   "n": 72,
+   "accuracy": 0.9861111111111112,
+   "invalid": 0,
+   "ece": 0.0753152777777778,
+   "by_family": {
+    "adequacy": {
+     "n": 12,
+     "accuracy": 0.9166666666666666
+    },
+    "extraction": {
+     "n": 12,
+     "accuracy": 1.0
+    },
+    "intent": {
+     "n": 12,
+     "accuracy": 1.0
+    },
+    "ordinal": {
+     "n": 12,
+     "accuracy": 1.0
+    },
+    "policy": {
+     "n": 12,
+     "accuracy": 1.0
+    },
+    "routing": {
+     "n": 12,
+     "accuracy": 1.0
+    }
+   }
+  },
+  "hard": {
+   "n": 111,
+   "accuracy": 0.6486486486486487,
+   "invalid": 0,
+   "ece": 0.18370720720720732,
+   "by_family": {
+    "adversarial": {
+     "n": 6,
+     "accuracy": 1.0
+    },
+    "ambiguous": {
+     "n": 7,
+     "accuracy": 0.7142857142857143
+    },
+    "judge_hard": {
+     "n": 17,
+     "accuracy": 0.5294117647058824
+    },
+    "long_policy": {
+     "n": 19,
+     "accuracy": 0.47368421052631576
+    },
+    "multi_hop": {
+     "n": 18,
+     "accuracy": 0.8333333333333334
+    },
+    "probability": {
+     "n": 10,
+     "accuracy": 0.8
+    },
+    "routing_hard": {
+     "n": 5,
+     "accuracy": 1.0
+    },
+    "temporal_numeric": {
+     "n": 15,
+     "accuracy": 0.3333333333333333
+    },
+    "tradeoff": {
+     "n": 6,
+     "accuracy": 0.3333333333333333
+    },
+    "trap": {
+     "n": 8,
+     "accuracy": 1.0
+    }
    }
   }
  },
  "references": {
-  "decider_4b_v1": "Mapika/decider-4b revision v1, T 1.05",
-  "decider_2b_v10": "Mapika/decider-2b, T 1.30",
-  "decider_35b_a3b": "Mapika/decider-35b-a3b, T 1.075"
- }
+  "v1_4b": {
+   "reg_in": 0.8338,
+   "reg_out": 0.7876,
+   "reg_in_nll": 0.4044,
+   "reg_out_nll": 0.5583,
+   "reg_in_ece": 0.0269,
+   "reg_out_ece": 0.0708
+  },
+  "v2_4b": {
+   "reg_in": 0.8241,
+   "reg_out": 0.7788,
+   "reg_in_nll": 0.4411,
+   "reg_out_nll": 0.5663,
+   "reg_in_ece": 0.0413,
+   "reg_out_ece": 0.0803
+  }
+ },
+ "speed": "not measured again: the architecture and size equal the parent's, and the temperature map does not change the computation"
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:69e6895461c425c6469cd304838a2e5673141613c2da04782026e37c1481d936
+oid sha256:ee8ce585b3cedd93206dd149b09b4bdd683174874211f85c77a36090b90c9fdd
 size 8411558400

tokenizer_config.json

@@ -10,7 +10,7 @@
   "errors": "replace",
   "image_token": "<|image_pad|>",
   "is_local": true,
-  "local_files_only": true,
+  "local_files_only": false,
   "model_max_length": 262144,
   "model_specific_special_tokens": {
     "audio_bos_token": "<|audio_start|>",