decider-4b v3: text and image decisions (vision/), Noul T 0.05; v2.1 under tag v2.1

README.md

@@ -3,441 +3,152 @@ license: apache-2.0
 base_model: Qwen/Qwen3.5-4B-Base
 language: [en]
 pipeline_tag: text-classification
-tags: [decision-model, calibrated, structured-output, multi-task, system-one, one-pass]
+tags: [decision-model, calibrated, structured-output, multi-task, system-one, one-pass, vision]
 ---
 
-# decider-4b: typed decisions with calibrated probabilities in one forward pass, 4B dense
+# decider-4b: typed decisions with calibrated probabilities in one forward pass, 4B dense, text and images
 
 A language model that does not generate text. It reads a state and one or more typed questions, each with an explicit option
-list, and returns a probability distribution over the options for every question from one forward pass. There is no decoding,
-no parsing and no output outside the options you defined. It is called from software, not chatted with. It is an open
-reproduction of the "System One" model class (TypeSafe AI's Jev).
+list, and returns a probability distribution over the options for every question from one forward pass. It is called from
+software, not chatted with. From v3 the state may include images.
 
-Base model: [Qwen/Qwen3.5-4B-Base](https://huggingface.co/Qwen/Qwen3.5-4B-Base): 4.2B parameters, 32 layers, 8 with full
-attention and 24 with gated delta-net linear attention, hidden size 2,560. Two supervised stages. Stage 1 (decider-4b v1): one
-pass of cross-entropy on the slot readout over mixture v2, the public decision mixture of
-[decider-2b](https://huggingface.co/Mapika/decider-2b) plus 26 further public decision datasets and ten programmatically
-generated families with verifiable gold (742M tokens). Stage 2 (v2.1): a LoRA of rank 64 on the attention and MLP weights,
-trained for 2 epochs on 29,325 rows of harder decisions and replay, with the replay rows trained toward v1's own answer
-distribution, then merged into the weights. There is no reinforcement-learning stage. **This repository holds v2.1**, the bf16
-weights (8.4 GB), with one temperature per answer type. v2 stays available under the Hub tag `v2` and v1 under the tag `v1`
-(see [Changes from v2](#changes-from-v2) for who should keep using them). The other sizes are listed under The decider family.
-`decider/` in this repository is the inference subset of the GitHub package.
+**This repository holds v3 (2026-10-20).** v2.1 stays available under the Hub tag `v2.1`, with its own card; v2 and v1 under
+`v2` and `v1`.
 
-v2.1 keeps most of v2's gain on hard decisions and gets back most of what v2 lost in sampled play. Against v2 on the same rows:
-bag-draw games in sampled play 52.0% against 37.9% wins (v1 56.6%), zero-shot games sampled 26.9% against 22.4% (v1 27.8%), live
-browser tasks sampled 93.2% against 88.1% (v1 90.9%), CliffWalking −13 against −60, the regression set 0.831 / 0.784 against 0.824
-/ 0.779 (v1 0.834 / 0.788); held-out generated decision families 0.556 against 0.560 (v1 0.469) and the JevBench public hard tier
-0.649 against 0.676 (v1 0.550). It is less well calibrated than v2 on hard items (details under
-[Calibration](#calibration)), still answers one of the two form-filling cases of issue #9 wrongly, and is worse than v1 on
-BabyAI-GoTo and on greedy bag-draw play. Against decider-35b-a3b it is 2.4 and 2.6 points lower on the regression set.
+- **Weights:** the text model at the repository root (Qwen3.5-4B-Base architecture, bf16, 8.4 GB), loaded as before by
+  `decider.infer.Decider`, `decider.serve` and the GGUF builds. `vision/` holds the image part: the vision tower of
+  Qwen3.5-4B-Base (unchanged, 0.6 GB) and the config of the image-text model. The image path reuses the text model's weights.
+- **Code:** [decider-ai](https://github.com/Mapika/decider) 1.10.0 for images; text works with 1.4.0 and later.
 
-**Contents:** [Changes from v2](#changes-from-v2) · [The decider family](#the-decider-family) · [Usage](#usage) · [How it works](#how-it-works) · [Training](#training) · [Evaluation](#evaluation) · [Calibration](#calibration) · [Speed](#speed) · [Limitations](#limitations) · [Changelog](#changelog) · [Reproduction](#reproduction)
+## Changes from v2.1
 
-## Changes from v2
-
-v2.1 differs from v2 in two things:
-
-* **Replay rows are trained toward v1's own distribution.** Stage 2 of v2 trained every row, including 6,700 replay rows from v1's
-  training mixture, on its hard label. v2.1 trains the replay rows toward v1's answer distribution instead, with the loss
-  KL(p_v1 ‖ p_model) over the options of each answer, and keeps hard labels on every other row. Everything else is v2's recipe:
-  the same rows (29,325 after removing 31 rows that shared a context with an evaluation file; v2 had 29,356), LoRA rank 64, learning
-  rate 1e-4, 2 epochs. Training the replay rows on hard labels had sharpened the logits everywhere; the fitted temperature then
-  rose to 1.935 and flattened every served answer, which is what cost v2 its sampled play. With the replay rows trained toward
-  v1, the fitted temperature is 1.099 (v1 1.05). On 1,000 held-out replay rows the mean KL(p_v1 ‖ p_model) at temperature 1 is 0.021
-  nats (v2: 0.112), and 96.3% of the argmax answers equal v1's (v2: 94.1%).
-* **One temperature per answer type.** `decider_config.json` has `temperature` 1.099 and `temperature_by_type`
-  `{"choice": 1.110, "noul": 1.560, "score": 1.287}`, fitted by NLL with `decider.calibrate` (decider-ai 1.4.0). decider-ai 1.4.0
-  and later use the map. **decider-ai 1.3.0 and earlier ignore the map and serve every answer at 1.099**; a temperature does not
-  change which option is most probable, so the answers are the same either way (except exact ties between the level rows of an
-  isolated Score answer, where rounding can break the tie differently: 1 of 5,000 held-out generated-family rows and 2 of 447
-  document-question validation rows changed); the
-  probabilities differ, mostly on yes/no and Score answers. The map changes little in practice: sampled
-  play, the fixtures and the probes are the same within noise with and without it, and it lowers the calibration error on
-  held-out yes/no answers (0.146 to 0.101) and Score answers (0.140 to 0.108). It does not fix the overconfidence on hard
-  Choice answers (see Calibration).
-
-All rows below are on identical inputs and seeds: v2.1 through decider-ai 1.4.0 with its map, v1 and v2 through decider-ai 1.3.0
-at their stored temperatures (1.05 and 1.935), measured in the same session on 2026-09-24. Intervals are 95% paired bootstrap
-intervals (rows for the fixtures, boards for the games, task-seed pairs for the browser). The regression set and the two held-out
-sets were read from stored temperature-1 logits at each model's served temperatures. The JevBench files of v1 and v2 were read
-earlier through decider-ai 1.2.1, v2's at the candidate temperature 1.719; accuracy does not depend on the temperature.
-
-| set | v1 (T 1.05) | v2 (T 1.935) | v2.1 (map) | v2.1 minus v1 | v2.1 minus v2 |
-|---|---|---|---|---|---|
-| regression set, 67 in-task tasks, accuracy / NLL / ECE | 0.834 / 0.404 / 0.027 | 0.824 / 0.441 / 0.041 | 0.831 / 0.414 / 0.031 | −0.3 | +0.7 |
-| regression set, 28 held-out tasks | 0.788 / 0.558 / 0.071 | 0.779 / 0.566 / 0.080 | 0.784 / 0.569 / 0.077 | −0.4 | +0.5 |
-| 847 in-task validation rows, accuracy / NLL | 86.1% / 0.417 | 85.0% / 0.419 | 85.0% / 0.432 | −1.1 (−2.1 to +0.0); NLL +0.015 (+0.003 to +0.028) | 0.0 (−1.5 to +1.5); NLL +0.014 (−0.006 to +0.035) |
-| OpenJev, 5,252 rows, accuracy / NLL | 63.9% / 0.893 | 66.7% / 0.789 | 66.0% / 0.846 | +2.1 (+1.2 to +2.9); NLL −0.047 (−0.060 to −0.035) | −0.7 (−1.6 to +0.2); NLL +0.057 (+0.044 to +0.070) |
-| Mind2Web, 1,770 rows, accuracy / NLL | 88.4% / 0.366 | 87.4% / 0.393 | 87.5% / 0.375 | −0.8 (−1.7 to +0.0); NLL +0.009 (−0.004 to +0.023) | +0.1 (−0.8 to +1.1); NLL −0.018 (−0.032 to −0.005) |
-| TypeSafe workflow decisions, 102 rows, accuracy / NLL | 81.4% / 0.611 | 86.3% / 0.410 | 84.3% / 0.441 | +2.9 (−2.9 to +8.8); NLL −0.170 (−0.324 to −0.036) | −2.0 (−7.8 to +3.9); NLL +0.031 (−0.069 to +0.134) |
-| held-out generated families (heldout_jb), 5,000 rows, accuracy / ECE | 0.469 / 0.260 | 0.560 / 0.046 | 0.556 / 0.147 | +8.7 | −0.4 |
-| held-out document questions (test_teacher2), 449 rows, accuracy / ECE | 0.726 / 0.110 | 0.826 / 0.053 | 0.820 / 0.043 | +9.4 | −0.7 |
-| JevBench public items, easy / standard / hard accuracy | 1.000 / 0.958 / 0.550 | 1.000 / 0.986 / 0.676 | 1.000 / 0.986 / 0.649 | hard +11 items | hard −3 items |
-| JevBench hard tier, top-label ECE (v1, v2: files read through 1.2.1, v2 at T 1.719) | 0.288 | 0.104 | 0.184 | | |
-| Bespoke's public suite, macro / micro | 0.757 / 0.765 | 0.773 / 0.781 | 0.756 / 0.765 | −0.1 macro | −1.6 macro |
-| live MiniWoB++, sampled, all 22 tasks | 90.9% | 88.1% | 93.2% | +2.3 (−1.7 to +6.2) | +5.1 (+0.6 to +9.7) |
-| live MiniWoB++, sampled, 16 rewarded tasks | 96.1% | 93.0% | 95.3% | −0.8 (−3.9 to +2.3) | +2.3 (−2.3 to +7.0) |
-| live MiniWoB++, sampled, 6 held-out tasks | 77.1% | 75.0% | 87.5% | +10.4 (0.0 to +20.8) | +12.5 (0.0 to +25.0) |
-| live MiniWoB++, greedy, all 22 tasks | 91.5% | 92.6% | 93.8% | +2.3 (−0.6 to +5.7) | +1.1 (−1.7 to +4.5) |
-| live MiniWoB++, greedy, 16 rewarded tasks | 97.7% | 96.9% | 96.1% | −1.6 (−3.9 to +0.0) | −0.8 (−2.3 to +0.0) |
-| live MiniWoB++, greedy, 6 held-out tasks | 75.0% | 81.2% | 87.5% | +12.5 (+4.2 to +22.9) | +6.2 (−4.2 to +16.7) |
-| zero-shot games, 234 boards, sampled, win rate | 27.8% | 22.4% | 26.9% | −0.9 (−2.6 to +1.0) | +4.5 (+2.5 to +6.6) |
-| bag-draw games, 64 boards, sampled, win rate | 56.6% | 37.9% | 52.0% | −4.7 (−9.0 to −0.4) | +14.1 (+8.2 to +19.9) |
-| slippery-grid games, 64 boards, sampled, win rate | 16.4% | 16.0% | 16.8% | +0.4 (−2.7 to +3.5) | +0.8 (−2.3 to +3.9) |
-| zero-shot games, 234 boards, greedy, win rate | 29.1% | 27.8% | 25.2% | −3.8 (−7.3 to −0.4) | −2.6 (−6.4 to +1.3) |
-| bag-draw games, 64 boards, greedy, win rate | 62.5% | 48.4% | 53.1% | −9.4 (−17.2 to −3.1) | +4.7 (−3.1 to +12.5) |
-| slippery-grid games, 64 boards, greedy, win rate | 12.5% | 18.8% | 10.9% | −1.6 (−7.9 to +4.7) | −7.8 (−15.6 to +0.0) |
-| ten text games, greedy: Pong / Breakout / CliffWalking / BabyAI-GoTo / Freeway / Blackjack | −21 / 14 / −13 / 0.54 / 0 / −0.6 | −21 / 12 / −60 / 0.35 / 1 / −0.6 | −5 / 35 / −13 / 0.19 / 0 / −0.6 | | |
-| behaviour probes: model-router tier / needs-live-data (31 items) | 0.968 / 0.871 | 0.935 / 0.839 | 0.968 / 0.839 | 0 / −1 item | +1 / 0 items |
-| behaviour probes: command risk / touches-outside-project (45 items) | 0.889 / 0.956 | 0.911 / 0.933 | 0.889 / 0.933 | 0 / −1 item | −1 / 0 items |
-| behaviour probes: generic bucket / catch-all / abstention battery / browser element and action | 1.00 / 0.90 / 7 of 8 / 0.875 and 0.875 | 0.95 / 0.95 / 8 of 8 / 0.938 and 0.938 | 1.00 / 0.95 / 8 of 8 / 0.938 and 0.750 | | |
-| issue #9 form cases c_1 / c_2 (probability of the gold option) | right (0.79) / right (0.98) | wrong (0.16) / right (0.28) | wrong (0.20) / right (0.41) | | |
-
-**Regressions, stated plainly.**
-* BabyAI-GoTo (greedy text game): 0.19 against v1's 0.54 and v2's 0.35.
-* Greedy bag-draw play: 53.1% against v1's 62.5% wins (−9.4 points, interval −17.2 to −3.1); greedy zero-shot games overall 25.2%
-  against 29.1% (−3.8, interval −7.3 to −0.4). Sampled bag-draw play is 4.7 points under v1 (interval −9.0 to −0.4).
-* Behaviour probes: needs-live-data 0.839 against v1's 0.871 and touches-outside-project 0.933 against 0.956, one item each (as
-  v2); the browser-agent action choice is 12 of 16 items (0.750), 2 fewer than v1 (0.875) and 3 fewer than v2
-  (0.938).
-* Form filling: issue #9 case c_1 is still answered wrongly. The field is "Degree earned" and the document entity is "Studied:
-  Associate of Arts"; v2.1 chooses "skip" at 0.76 and gives the gold entity 0.20 (v1: gold 0.79; v2: gold 0.16). Case c_2 is
-  answered correctly at 0.41. For form filling we suggest v1.
-* Calibration on hard items: on the held-out generated families the calibration error is 0.147 against v2's 0.046 (see
-  Calibration), and on the JevBench public hard tier 0.184 against v2's 0.104.
-* JevBench public hard tier: 72 of 111 items (0.649) against v2's 75 (0.676).
-* Against v2: TypeSafe −2.0 points and OpenJev −0.7 (intervals include zero), NLL higher on both; Bespoke's suite 0.756 against
-  0.773 macro (v1 0.757); greedy slippery-grid play 10.9% against 18.8%.
-
-**How v2.1 was chosen, and why it is released although it did not pass.** The run had pre-registered release rules. v2.1 passed
-the numeric items (held-out document questions 0.820 against the required 0.816, held-out generated families 0.556 against 0.550,
-regression accuracy and calibration not worse than v2's, sampled bag-draw above the midpoint of v1 and v2, sampled zero-shot
-games and browser play not shown to be below v1's (upper end of the 95% interval at least 0), each probe at most one item below v1) and failed the last item, which required both
-issue #9 form cases to be right: c_1 is wrong. By the rule it was not recommended. A second pre-registered rule for the
-temperature map required the calibration error on the held-out generated families to be at most 0.08; the map reaches 0.147
-(0.163 without it), so the map did not pass either. v2.1 is released with the map on a decision made after reading the full comparison above: it is better than v2 on sampled play, CliffWalking, the model-router probe and the regression set, at v2's
-level on our held-out hard sets, and its failures are listed in this section. The JevBench public items were read once for the
-candidate at its global temperature and once with the map, after the rule decisions; they were not used for training,
-selection or the temperatures.
-
-**Which version to use.**
-* v2.1 (this revision): the default. Sampled play (games, browser agents that sample actions), hard decisions.
-* v2 (`revision="v2"`): if you rely on confidence values on hard multi-step items, where v2 is better calibrated (held-out
-  generated families 0.046, JevBench hard tier 0.104), or on its slightly higher JevBench hard tier and TypeSafe accuracy.
-* v1 (`revision="v1"`): form filling (issue #9), BabyAI-GoTo-like grid navigation, greedy bag-draw play.
-
-The package loads a local folder, so download the revision first:
-
-```python
-from huggingface_hub import snapshot_download
-from decider.infer import Decider
-d = Decider(snapshot_download("Mapika/decider-4b", revision="v2"))      # or revision="v1"
-```
-
-For the HTTP server, set `DECIDER_MODEL` to the same downloaded folder.
+| | v2.1 | v3 |
+|---|---|---|
+| Decision Index 0.3, 4,500-request sample | 41.55 | **51.02** |
+| Image questions, mean of 11 sets | 0.840 (v2.1 + base vision tower) | **0.891** |
+| Visual7W / VSR (never trained on) | 0.857 / 0.813 | 0.873 / 0.833 |
+| JevBench public items, hard tier | 0.649 | 0.640 |
+| JevBench public items, standard tier | 0.986 | 0.986 |
+| Fresh held-out yes/no set, scored the JevBench way | −1.9 | **44.3** |
+
+- **Decision Index:** +9.5 on the sample, from the decider-31b training mixture (train splits of benchmarks the index also tests;
+  see Training).
+- **Images:** new. The gain over the plain base vision tower is mostly on the kinds of image questions trained.
+- **JevBench public items:** level (one hard item fewer).
+- **Yes/no answers:** the config now uses temperature 0.05 for Noul answers, so they come out near 0 or 1. JevBench counts a yes/no
+  probability between 0.2 and 0.8 as wrong; at 0.05 almost none fall there, and our 399-item held-out yes/no set scores 44.3
+  instead of −0.3 at the fitted temperature. The cost: the yes/no probabilities are no longer calibrated (ECE 0.265 instead of
+  0.102 on that set). For calibrated yes/no probabilities use `temperature_by_type={"choice": 2.098, "noul": 2.795, "score":
+  3.152}` (the fitted values).
 
 ## The decider family
 
-All six repositories share one interface (`decider.infer.Decider`, `POST /v1/systemone` in TypeSafe's format) and one
-readout: the letter logits at an answer slot, softmaxed over the options. Pick by size and input.
-
-| model | base | weights | use it for | numbers |
-|---|---|---|---|---|
-| [decider-2b](https://huggingface.co/Mapika/decider-2b) v11 | Qwen3.5-2B-Base | 3.8 GB bf16 | the default: routing, classification, judgments, browser agents; 4 ms per request with CUDA graphs on one GPU; v10 under the tag `v10` | regression set 0.802 in-task / 0.752 held-out; JevBench hard 0.577; live browser 90% sampled; Bespoke suite 0.706 |
-| [decider-4b](https://huggingface.co/Mapika/decider-4b) v2.1 | Qwen3.5-4B-Base | 8.4 GB bf16 | the middle point: knowledge questions and hard judgments above the 2B in a dense 8.4 GB model; no RL stage; v2 and v1 under the tags `v2` and `v1` | 0.831 / 0.784; JevBench hard 0.649; live browser 93% sampled; Bespoke 0.756 |
-| [decider-35b-a3b](https://huggingface.co/Mapika/decider-35b-a3b) v1 | Qwen3.5-35B-A3B-Base (3B active) | 65 GB bf16 | when accuracy is worth 3 to 4 times the cost per decision: knowledge and multi-step questions, long policies | 0.855 / 0.810, above the 2B on 93 of 95 tasks; JevBench hard 0.676; Bespoke 0.774; no RL stage |
-| [decider-35b-a3b-nvfp4](https://huggingface.co/Mapika/decider-35b-a3b-nvfp4) | the 35B in NVFP4 | 19.6 GB | the 35B on Blackwell through vLLM or TensorRT-LLM | 1.0 to 1.5 points under bf16 on the measured fixtures |
-| [decider-0.8b](https://huggingface.co/Mapika/decider-0.8b) | Qwen3.5-0.8B-Base | 1.4 GB bf16 | the smallest: routing, yes/no and short-state lookups within 1 to 4 points of the 2B, 1.5x faster | 0.776 / 0.707 on the single-run protocol (2B: 0.809 / 0.739) |
-| [decider-2b-vision](https://huggingface.co/Mapika/decider-2b-vision) | Qwen3.5-2B vision-language, v5 text weights | 4.1 GB bf16 | decisions from an image plus a question; game frames | Visual7W 0.89; Breakout 41 from pixels |
+| model | base | use it for |
+|---|---|---|
+| [decider-2b](https://huggingface.co/Mapika/decider-2b) v12 | Qwen3.5-2B-Base | the fastest: routing, classification, judgments; text and images |
+| **decider-4b** v3 | Qwen3.5-4B-Base | the middle point; text and images |
+| [decider-12b](https://huggingface.co/Mapika/decider-12b) v2 | Gemma-4-12B-it | our JevBench model; text |
+| [decider-12b-vision](https://huggingface.co/Mapika/decider-12b-vision) | Gemma-4-12B-it | the Decision Index task mix and images; calibrated yes/no |
+| [decider-31b](https://huggingface.co/Mapika/decider-31b) v2 | Gemma-4-31B-it, NVFP4 | the strongest; text and images; Blackwell GPU, vLLM |
+| [decider-35b-a3b](https://huggingface.co/Mapika/decider-35b-a3b) | Qwen3.5-35B-A3B-Base | text |
 
-Code, data registry, training scripts, the changelog and the per-version history: https://github.com/Mapika/decider.
+All share one interface (`POST /v1/systemone` in TypeSafe's format) and one readout. Code and changelog:
+https://github.com/Mapika/decider.
 
 ## Usage
 
+Text, as before:
+
 ```python
-from decider.infer import Decider          # decider/ is included in this repo
+from decider.infer import Decider
 d = Decider("Mapika/decider-4b")
 d.decide("My card was charged twice for the same purchase.",
          [{"question": "Which department should handle this?", "options": ["billing", "technical support", "sales"]},
           {"question": "Does this need a refund action?", "options": ["no", "yes"]}])
-# [{'choice': 'billing', 'confidence': ..., 'probs': {...}}, {'choice': 'yes', 'confidence': ..., 'probs': {...}}]
 ```
 
-The API is the same as decider-2b's: `decide_batch` scores many states with many questions in one call, `abstain_below=t`
-returns `None` under a confidence threshold, a question can have 2 to 255 options, and `system_one` / `decider.serve` accept
-TypeSafe's `POST /v1/systemone` request shape. Every question and every Score level is scored in its own row. The state may be a
-string, object or array of up to 32k tokens. See the decider-2b card for the full description of the request shape, field types
-and the schema cache.
-
-Requirements: `torch`, `transformers>=5`, and `flash-linear-attention` (Triton kernels for the Qwen3.5 linear-attention layers;
-the model runs without it but several times slower). The weights take 8.4 GB in bf16. v2.1 uses the plain prompt layout, as v1 and
-v2 do. The per-type temperatures need decider-ai 1.4.0 or later (or the `decider/` subset in this repository, taken from 1.4.0).
-Checked on CUDA, eager path: decider-ai 1.3.0 loads v2.1 with the name `decider-4b-v2.1` and temperature 1.099 and gives exactly
-the probabilities that 1.4.0 gives with the map switched off; 1.4.0 and the `decider/` subset in this repository load the map
-and give the same probabilities as each other. For Choice and yes/no answers the 1.4.0 output with the map equals
-softmax(log(p) · 1.099 / T_type) of the 1.3.0 output p (to 1e-7 on `decide`, to 8e-5 on the four-decimal `system_one` values);
-an isolated-level Score answer is rescaled per level row (each level's yes/no pair at 1.287 instead of 1.099) before the
-levels are normalised, so it cannot be recomputed from the combined Score probabilities. The 1.4.0 HTTP server with CUDA graphs was checked with `/v1/systemone` and `/decide` requests. Other package
-versions, MPS and the FP8 path were not checked on v2.1. On Blackwell GPUs use 1.0.2 or later (1.0.0 and 1.0.1 have the cuDNN
-attention fault fixed in 1.0.2). The model is dense, so the CUDA-graph engine, `torch.compile` and the FP8 path of the helper
-package apply to it as to decider-2b (`use_graphs=False` selects eager PyTorch).
+Images, through the server (decider-ai 1.10.0; the config's `"images": true` turns image input on):
 
-Without the helper package, the same computation in plain `transformers`:
+```bash
+pip install "decider-ai>=1.10.0" pillow
+DECIDER_MODEL=Mapika/decider-4b uvicorn decider.serve:app --host 127.0.0.1 --port 8000
+```
 
 ```python
-import torch
-from transformers import AutoTokenizer, AutoModelForCausalLM
-tok = AutoTokenizer.from_pretrained(REPO); m = AutoModelForCausalLM.from_pretrained(REPO, dtype=torch.bfloat16).cuda().eval()
-prompt = ("Context:\nMy card was charged twice for the same purchase.\n\n"
-          "Question: Which department should handle this?\nOptions:\n(A) billing\n(B) technical support\n(C) sales\nAnswer: (")
-ids = tok(prompt, return_tensors="pt").to("cuda")
-with torch.no_grad():
-    logits = m(**ids).logits[0, -1]
-letters = [tok.encode(L, add_special_tokens=False)[0] for L in "ABC"]
-probs = torch.softmax(logits[letters].float() / 1.110, -1)     # 1.110 is the stored choice temperature (yes/no 1.560, Score 1.287)
+import base64, httpx
+img = base64.b64encode(open("chart.png", "rb").read()).decode()
+r = httpx.post("http://127.0.0.1:8000/v1/systemone", timeout=60, json={
+    "state": {"chart": "q3-chart", "note": "Escalate if any region fell by more than 10%."},
+    "assets": {"q3-chart": {"type": "image", "format": "png", "data_base64": img}},
+    "questions": {"escalate": {"type": "noul", "instructions": "Does the `chart` call for escalation?",
+                               "criteria": {"true": "escalate", "false": "no escalation"}}}})
 ```
 
-## How it works
-
-The prompt is `Context: ...` followed by, for each question, the question text, the lettered options `(A) ... (B) ...` and an
-answer slot `Answer k: (`. The hidden state at each slot is projected with the option-letter rows of the LM head and softmaxed
-over the valid letters, divided by the temperature in `decider_config.json`. Letters are never generated, so all slots are read
-from one pass. From decider-ai 1.4.0 the config may also hold `temperature_by_type`, one temperature per answer type
-(`choice`, `noul`, `score`; a missing type uses `temperature`). This release's config has such a map: Choice answers use 1.110,
-yes/no (noul) answers 1.560, Score answers 1.287 on each of their level rows. Package versions before 1.4.0 use `temperature`
-(1.099) for every answer. `DECIDER_TEMPERATURE=T` or `Decider(path, temperature=T)` replaces `temperature` with T and switches the
-map off, so every answer then uses T. Large label sets were sub-sampled to at most 10 options per
-training example (gold always kept, order shuffled), so the model conditions on the supplied candidates rather than on a fixed
-head.
+The asset's key is the reference; each place the state names it becomes "the image" in the prompt, and the image goes in front
+of the prompt, as in training. PNG and JPEG; up to 4 images (`DECIDER_MAX_IMAGES`); references in question text and box
+answers are not supported. Image rows are read one at a time, without batching or CUDA graphs. Requirements as before (`torch`,
+`transformers>=5`, `flash-linear-attention`), plus `pillow` for images.
 
 ## Training
 
-**Stage 1 (decider-4b v1).** One supervised pass over **mixture v2**, in the two prompt layouts (state-first and schema-first,
-50/50), with isolated Score levels and 10% abstention rows, 1,892,408 items and 742M tokens, pre-tokenized once and read in the
-same order by both ranks. Three sources:
-
-| source | rows | share of tokens | content |
-|---|---|---|---|
-| the public decision mixture of decider-2b (`scripts/train.sh full` of the GitHub repository) | 1,539,860 | 60% | about 95 public decision datasets, agent trajectories, Mind2Web element choice, teacher-written custom questions, Jev's input shapes |
-| 26 further public datasets with gold labels | 131,318 | 8% | code defect, clone and review-needed judgments; log anomaly and severity (HDFS, BGL); legal (LEDGAR, Unfair-ToS, CaseHOLD, SCOTUS, ECtHR); tables (TabFact, WikiTQ, FeTaQA, InfoTabs, TAT-QA); finance headlines; German credit; symptom diagnosis and medical specialty; SciTail, SciEntsBank, Climate-FEVER; XNLI, PAWS-X, MASSIVE (multilingual), Belebele, XCOPA; MC-TACO, TRACIE, TimeQA; ProofWriter, RuleTaker, LogiQA 2; essay scoring; agent next action |
-| ten programmatic families with verifiable gold, each with a held-out variant | 221,230 | 32% | code, dates and times, logs, long documents, plans, policies, probability, schedules, tables, tools |
-
-Ten of the 26 public datasets (Belebele, code clone, ECtHR, essay scoring, InfoTabs, PAWS-X, RuleTaker, SCOTUS, XCOPA, TRACIE)
-and the held-out variant of every programmatic family were kept out of training; every evaluation row was checked against every
-training row of all three sources and 4,168 overlapping training pairs were dropped. No game rows are in mixture v2 (the
-public mixture rebuilt on this machine has none), so every game result below is zero-shot.
-
-| stage 1 | |
-|---|---|
-| trainable parameters | all 4.2B (426 tensors) |
-| optimizer | `torch.optim.AdamW` applied directly to the bf16 parameters, no FP32 master copy; betas 0.9 / 0.95, no weight decay |
-| schedule | peak learning rate 1e-5, 150 warm-up steps, cosine to zero, 26,729 steps of 32,768 tokens, gradient clip 1.0 |
-| hardware | 2 NVIDIA B300, data parallel, 16,384-token micro-batches per GPU, gradient checkpointing; 577 minutes at about 25,000 tokens per second, 39.6 GB peak per GPU |
-| training cross-entropy | 0.97 over the first 200 steps, 0.42 at 25%, 0.35 at 50%, 0.35 over the last 300 steps |
-
-Why this optimizer: on a controlled quarter-data comparison of the same 4B (`ref_4b` against `ref_4b_bf16opt`, identical data and
-schedule, only the optimizer changed), AdamW on the bf16 parameters beat AdamW with FP32 master weights by 3.3 held-out points and
-0.072 nats (0.791 against 0.758 held-out accuracy), and by 7 points on nine knowledge tasks. The master copy lets every small update
-through and moves the weights further from the base model; without it, updates below the bf16 resolution round away and more of
-the base model's knowledge is kept.
-
-**Stage 2 (v2.1).** A LoRA of rank 64 (alpha 128) on the attention and MLP weights of v1, trained for 2 epochs over 29,325 rows
-in v1's plain state-first layout with isolated Score levels, then merged into the bf16 weights:
+v3 starts from v2.1. Its text weights were placed into the Qwen3.5-4B-Base image-text model (vision tower from the base); the
+language-model weights are identical to the text model's. Then a LoRA stage (rank 64, learning rate 1e-4, 2 epochs, 110 minutes
+on one B300) on the language model only:
 
-| source | rows | content | target |
-|---|---|---|---|
-| generated decision families | 8,000 | ten families (temporal and numeric decisions, subtle answer judgment, long policies, multi-hop lookup, abstention, probability, constrained trade-offs, safety judgment, paraphrase sensitivity, adversarial traps); the answers are computed by the generating code | the label |
-| questions over business documents, written by Qwen3.6-27B with thinking on | 11,356 | in two rounds, one realistic business document (up to 34 domains and 22 document kinds) plus three or four typed questions per writer call; each question was answered twice more by the same model in fresh contexts, with shuffled options and without the writer's answer, and kept only when both answers agreed with the writer's (89% and 91% kept) | the label |
-| human-labelled public sets (training halves) | 3,293 | MMLU, ARC, CommonsenseQA, BoolQ, MNLI, SNLI, Banking77, RACE, OpenBookQA, LogiQA 2, MedQA, Winogrande | the label |
-| replay of mixture v2 | 6,676 | 100 rows from the training half of each of the 67 in-task regression tasks | v1's answer distribution: loss KL(p_v1 ‖ p_model) at temperature 1 |
+- **Text, 39.5k rows:** the decider-31b training mixture (train splits of CLINC150, iSarcasmEval, VAST, ACOS, POP909-CL, GSM8K
+  and NLI4CT, decontaminated against every Decision Index 0.3 suite row; generated smart-home requests; generated
+  state-tracking decisions and decision families; document questions written by Qwen3.6-27B; labelled public rows), plus 6.7k
+  rows replayed from v2.1's training data.
+- **Images, 18,129 rows:** train splits of seven the_cauldron subsets: AI2D, A-OKVQA, Hateful Memes, IconQA, RAVEN, ScienceQA,
+  TQA. Visual7W and VSR were not used.
+- Not used: Decision Index test rows, JevBench or ImageJevBench items.
 
-These are v2's rows without 31 that the v2.1 checks found to share a context with an evaluation file (7 human-labelled rows and
-24 replay rows).
-
-| stage 2 | |
-|---|---|
-| trainable parameters | LoRA rank 64, alpha 128, on the attention and MLP projections; merged after training |
-| loss | cross-entropy on the slot readout for labelled rows; KL(p_v1 ‖ p_model) over the options for replay rows, with p_v1 from the frozen v1 weights; the step loss is the sum over answers divided by the number of answers |
-| schedule | learning rate 1e-4, 5% warm-up then cosine, 1,518 steps of 65,536 tokens (2 epochs), seed 0 |
-| hardware | one NVIDIA B300, 94 minutes |
-
-No JevBench item and no Decision Index item was used for training, for writing the generators or the document questions, for
-selecting the checkpoint, or for the temperatures. The ten generated families and the skill list of the document questions were
-written from the family names that JevBench publishes for its sealed set, not from its items. Every training row was checked
-against every evaluation file used for selection, the canonical examples of the evaluation half of all 153 mixture-v2 evaluation
-tasks, the four request fixtures and the issue #9 cases: there is no exact state-and-question overlap (a few contexts under 40
-characters, such as a short utterance, occur under another task and question). The held-out sets used for selection are
-generated families from held-out templates (heldout_jb) and document questions from business domains that are not in the
-training data (test_teacher2). Two arms were trained to the end (B4 and C4, which differ in the size of the replay; two further arms were
-stopped or cancelled at the user's request); the rule selected B4 after epoch 2.
-
-**Temperatures.** `temperature` 1.099, fitted by NLL (all rows pooled) on the in-task half of the public regression set without
-Banking77, CLINC-OOS, MMLU, ARC, Winogrande and HellaSwag: 61 tasks, 102,804 rows. `temperature_by_type` fitted by NLL per answer
-type with `decider.calibrate.fit_by_type` on a pool of those regression rows and our own validation rows (the validation halves of
-the generated families and document questions, human-labelled validation rows, held-out replay rows): 108,927 Choice answers,
-1,677 yes/no answers and 535 Score answers. The Choice pool is 94% regression rows, so the Choice temperature (1.110) stays close
-to the global one. v2's temperature (1.935) and v1's (1.05) were one value each.
+Temperatures were fitted on our own validation rows: Choice 2.098, Score 3.152, Noul 2.795 (the config uses 0.05 for Noul, see
+above). They are about twice v2.1's because v3's raw answers are sharper.
 
 ## Evaluation
 
-All v2.1 numbers are with the per-type map through decider-ai 1.4.0, except where a paragraph says otherwise. Greedy play and
-accuracy do not depend on the temperature; sampled play and calibration do.
-
-**Public regression set**, rebuilt on this machine (95 tasks: 67 in-task, 28 held-out; large label sets sub-sampled to 10
-options; the temperatures fitted on in-task data). The rows are the same for every model. ECE is the expected calibration
-error with 15 bins, per-task mean. The regression rows are all Choice answers, so v2.1 reads them at the Choice temperature 1.110.
-
-| model | in-task acc / NLL / ECE (67 tasks) | held-out acc / NLL / ECE (28 tasks) |
-|---|---|---|
-| decider-2b v10, T=1.30 | 0.806 / 0.474 / 0.038 | 0.755 / 0.622 / 0.084 |
-| decider-2b v11, per-type map | 0.802 / 0.481 / 0.038 | 0.752 / 0.626 / 0.083 |
-| decider-4b v1, T=1.05 | 0.834 / 0.404 / 0.027 | 0.788 / 0.558 / 0.071 |
-| decider-4b v2, T=1.935 | 0.824 / 0.441 / 0.041 | 0.779 / 0.566 / 0.080 |
-| **decider-4b v2.1 (this repository), per-type map** | **0.831 / 0.414 / 0.031** | **0.784 / 0.569 / 0.077** |
-| decider-35b-a3b v1, T=1.08 | 0.855 / 0.357 / 0.026 | 0.810 / 0.497 / 0.069 |
-
-Per-task paired intervals were not recomputed for v2.1; the v2 card (tag `v2`) has them for v2 against v10 and the 35B, and v2.1
-is within 0.7 points of v2 on both halves.
-
-**The same rows as v1 and v2** are in the table under [Changes from v2](#changes-from-v2): the four fixtures (847 in-task
-validation rows, OpenJev, Mind2Web, TypeSafe), our held-out hard sets, JevBench, Bespoke, live browser tasks, zero-shot games,
-text games, behaviour probes and the issue #9 cases.
-
-**Ten text games, zero-shot** (`decider.games.play`, five episodes per game, greedy, eager; the 2B was trained on the first four,
-the 4B on none): Pong −5 (v1 −21, v2 −21; the 2B 8), Breakout 35 (v1 14, v2 12; the 2B 22), CliffWalking −13 (v1 −13, v2 −60;
-teacher −13), BabyAI-GoTo 0.19 (v1 0.54, v2 0.35), Freeway 0 (v2 1), Blackjack −0.6, FrozenLake, MiniGrid-Empty, LavaGap and DoorKey
-0 as for v1 and v2. Greedy play takes the most probable option, so these results do not depend on the temperature.
-
-**JevBench public items** (231 items of [Benchmark Heaven](https://benchmarkheaven.com/jev-models); argmax over the exact
-label set with the request the harness's TypeSafe adapter builds). Read once for v2.1 at its global temperature (1.3.0) and once
-with the map (1.4.0), both after the rule decisions: easy 1.000, standard 0.986, hard 0.649 (72 of 111) in both reads. Top-label
-ECE on the hard tier is 0.184 with the map (0.210 at the global temperature); by answer type with the map, Choice 0.156 (67 items),
-yes/no 0.259 (38), Score 0.354 (6). Mean confidence on the hard tier is 0.799 at accuracy 0.649. For comparison on the same items:
-v2 0.676 (ECE 0.104 as read at T 1.719; 0.071 recomputed at its release temperature 1.935 with 6 Score items kept at 1.719), v1
-0.550 (ECE 0.288), decider-35b-a3b 0.676, Jev 1.13.0 0.730. The public hard tier is 111 items (95% interval about ±9 points).
-
-**Bespoke's public suite** (13 human-labelled subsets, 3,880 records in Jev's wire format, answered through `system_one` as
-shipped). v1 and v2 through decider-ai 1.3.0, v2.1 through 1.4.0 with the map, same session. Nimble-9B (0.748 / 0.759 macro /
-micro) and Jev 1.13.0 (0.760 / 0.773) are in Bespoke's report.
-
-| subset (type) | decider-4b v1 | decider-4b v2 | decider-4b v2.1 |
-|---|---|---|---|
-| vitaminc-dev (choice) | 0.756 | 0.778 | 0.743 |
-| massive-en-US (choice; trained) | 0.860 | 0.869 | 0.863 |
-| massive-de-DE (choice) | 0.843 | 0.837 | 0.840 |
-| boolq (noul; trained) | 0.873 | 0.860 | 0.867 |
-| squad2 (noul) | 0.706 | 0.793 | 0.789 |
-| paws (noul; trained) | 0.716 | 0.832 | 0.764 |
-| multinli (choice; trained) | 0.933 | 0.926 | 0.933 |
-| civil_comments (noul; trained) | 0.880 | 0.857 | 0.873 |
-| aegis2 (noul) | 0.820 | 0.812 | 0.804 |
-| helpsteer2 (score; trained) | 0.466 | 0.466 | 0.450 |
-| summeval-relevance (score) | 0.425 | 0.463 | 0.396 |
-| summeval-consistency (score) | 0.833 | 0.826 | 0.799 |
-| pubmedqa (choice; trained) | 0.728 | 0.724 | 0.708 |
-| **macro / micro** | 0.757 / 0.765 | 0.773 / 0.781 | 0.756 / 0.765 |
-| macro over the subsets not trained on | 0.731 | 0.751 | 0.728 |
-
-The multinli subset (299 items) is easier than MNLI as a whole. On all 9,815 MNLI validation_matched items, decider-4b
-(revision `eb5fbdf`) scores **0.904** (entailment 0.928, neutral 0.830, contradiction 0.950); on the 299 suite items it
-scores 0.933. An independent run on 120 other validation_matched items measured 0.842
-([discussion #3](https://huggingface.co/Mapika/decider-4b/discussions/3)).
+All numbers measured by us; none were verified by the benchmark maintainers.
 
-v2.1 is at v1's level on this suite and 1.6 points under v2 (macro). The largest differences to v2 are PAWS (0.764 against 0.832),
-SummEval relevance (0.396 against 0.463) and VitaminC (0.743 against 0.778).
-
-**Behaviour probes** (teacher-labelled, same probes as the other releases; v1 / v2 in brackets): generic-versus-specific bucket
-choice 1.00 / 1.00 (1.00 / 1.00; 0.95 / 1.00), catch-all when nothing fits 0.95 (0.90; 0.95), abstention battery 8 of 8 (7 of 8;
-8 of 8); model-router tier 0.968 (0.968; 0.935) and needs-live-data 0.839 (0.871; 0.839); command-risk classification 0.889
-(0.889; 0.911) with no destructive command called safe, touches-outside-project 0.933 (0.956; 0.933); browser-agent element and
-action choice 0.938 / 0.750 (0.875 / 0.875; 0.938 / 0.938). The yes/no Brier score on the probe batteries is 0.010 with the map
-and 0.006 at the global temperature: the yes/no temperature 1.560 flattens easy yes/no answers there.
-
-## Calibration
-
-The per-type map was fitted on everyday regression rows and our own validation rows. It helps where it can: yes/no and Score
-answers on the held-out sets. It does not change Choice answers much, because the Choice temperature fitted on a pool that is
-94% everyday rows stays at 1.110. On the held-out generated families, whose answers are two-thirds Choice, v2.1 is
-overconfident: calibration error 0.147 with the map, where our own release limit is 0.08. v2 is at 0.046 there, only because its
-one temperature of 1.935 flattens every answer, which is also what cost it sampled play. ECE below is 10-bin top-label ECE on
-stored temperature-1 logits read at the served temperatures.
-
-| set | rows | accuracy | ECE at the global T / with the map | choice / noul / score ECE with the map |
-|---|---|---|---|---|
-| heldout_jb | 5,000 | 0.556 | 0.163 / 0.147 | 0.170 / 0.101 / 0.108 |
-| test_teacher2 | 449 | 0.820 | 0.050 / 0.043 | 0.057 / 0.048 / 0.086 |
-| guard | 2,994 | 0.819 | 0.019 / 0.017 | 0.017 / − / − |
-| cal_human | 1,595 | 0.856 | 0.019 / 0.021 | 0.021 / − / − |
-
-On the held-out generated families: v1 (T 1.05) 0.260, v2 (T 1.935) 0.046, v2.1 at T = 1 0.182. A map fitted without the
-regression rows (Choice 1.259) would reach 0.127 there and would make the regression set less calibrated (in-task ECE 0.0375
-against 0.0309); it was not used. On the fixtures (10-bin ECE, map): 847 validation rows 0.035 (v1 0.037, v2 0.031), TypeSafe
-0.063 (v1 0.124, v2 0.072), OpenJev 0.154 (v1 0.158, v2 0.131), Mind2Web 0.017 (v1 0.016, v2 0.051). The Decision Index
-sample was not read for v2.1. If you route on confidence, calibrate on your own labels; `python -m decider.calibrate` fits a
-per-type map from your own answers read at temperature 1.
-
-## Speed
-
-v2.1 has the same architecture and size as v1 and v2, and the temperature map is a division per answer; the speed was not
-measured again. The measurements of v1 and v2 (the same weights layout): in one session on one unshared NVIDIA B300 (bf16), 34.6
-ms (v2) and 32.4 ms (v1) median per decision over 200 game-state decisions of 156 tokens median, batch of one, eager PyTorch
-without CUDA graphs or `torch.compile`, timed around the forward pass with `torch.cuda.synchronize()`. The eager path is
-launch-bound, so host load changes it: v1's first measurement was 24.7 ms (10th to 90th percentile 24.6 to 43.5 ms), with
-decider-2b at 17.9 ms and decider-35b-a3b at 41.4 ms on the same decisions and method. With the helper's CUDA graphs and
-`torch.compile`, one support-ticket request (228 tokens, 3 questions) takes 5.2 ms (FP8 5.0 ms); a batch of 32 such states takes
-81.5 ms, 1,178 decisions per second (FP8 71.2 ms, 1,349 per second). The HTTP server (`/decide`, bf16) answers 72.8 requests per
-second at a median of 13.3 ms with one client and 190 requests per second with 64 clients.
+- **Decision Index 0.3,** 4,500-request sample of the public suite: 51.02 (v2.1 41.55); through `decider.serve` 1.10.0, 50.85.
+  Benchmark-weighted ECE on the sample: 0.160 (v2.1 0.150); with about 88 requests per benchmark this number is inflated for
+  every model. The full public suite was not run.
+- **Image questions,** eval splits of 11 the_cauldron sets, 300 items each: mean 0.891; trained kinds 0.902, never-trained
+  (Visual7W, VSR) 0.853. v2.1 with the base vision tower and no image training: 0.841 / 0.835. Stock Qwen3.5-4B with our
+  readout: 0.827 mean. Through `decider.serve` 1.10.0 with the image sent as an asset: VSR 0.837, Visual7W 0.873.
+- **JevBench public items** (read once, never used for training, fitting or selection): easy 1.000, standard 0.986, hard 0.640.
 
 ## Limitations
 
-* No reinforcement-learning stage: stated beliefs about action outcomes were not trained against exact laws. On live browser
-  tasks v2.1 is at 93.2% sampled and 87.5% on the six held-out tasks; decider-2b v10, which has the RL stage, is at 93.2% and 91.7%.
-* Overconfident on hard multi-step items: calibration error 0.147 on our held-out generated families and 0.184 on the JevBench
-  public hard tier, against v2's 0.046 and 0.104. On those items, do not read a confidence of 0.8 as an 80% chance of being right.
-* Issue #9 form case c_1 is answered wrongly (see Changes from v2); BabyAI-GoTo 0.19 against v1's 0.54; greedy bag-draw play 9.4
-  points under v1.
-* About 0.3 points under v1 on the regression set and 1.1 points under it on the 847 validation rows; needs-live-data and
-  touches-outside-project probes one item under v1 each.
-* The per-type map needs decider-ai 1.4.0 or later. With 1.3.0 or earlier, or with `DECIDER_TEMPERATURE` set, every answer uses
-  1.099: the answers are the same, yes/no answers are sharper (on the held-out generated families their ECE is 0.146 instead of
-  0.101) and Score answers are sharper.
-* Below the 35B by 2.4 / 2.6 points on the regression set and 2.7 on the JevBench public hard tier (3 items).
-* The JevBench public hard tier is 111 items; differences of a few items between versions are within its noise.
-* The stage-2 LoRA was trained only in the plain state-first layout. The schema-first layout (the server's opt-in schema cache)
-  was not measured on v2.1, and `decider_config.json` does not mark it as trained for that layout (`schema_first_trained:
-  false`), so `DECIDER_SCHEMA_CACHE=1` does not turn the schema cache on for this model.
-* Mixture v2's 26 additional public datasets and ten programmatic families, and stage 2's generators and document questions, are
-  described above but their builders are not in the public package; `scripts/train.sh full` reproduces the 60% of stage 1's data
-  that is the public mixture.
-* English is the main language; the multilingual rows (XNLI, PAWS-X, MASSIVE, Belebele, XCOPA) are a small share of the data
-  and were not measured beyond the mixture-v2 evaluation set and Bespoke's German MASSIVE subset.
-* Everything else in the decider-2b card's limitations (packed questions see each other, long JSON arrays by position, full
-  label sets against sampled options, abstention wording, rules in the question) applies; those shapes were not re-measured at
-  this size.
-
-## Changelog
-
-| version | what changed |
-|---|---|
-| **v2.1** (2026-09-24, these weights) | v1 + a merged LoRA on v2's rows (29,325 after removing 31 that overlapped evaluation files), with the replay rows trained toward v1's own answer distribution; temperature 1.099 and `temperature_by_type` {choice 1.110, noul 1.560, score 1.287} (decider-ai 1.4.0; older versions use 1.099). Against v2: sampled play recovered (bag-draw 52.0% against 37.9%, browser 93.2% against 88.1%), CliffWalking −13 against −60, regression set +0.7 / +0.5; hard sets at v2's level; less calibrated on hard items (0.147 against 0.046); issue #9 c_1 still wrong. Did not pass its pre-registered rules; released on the full comparison |
-| v2 (2026-09-24, Hub tag `v2`) | v1 + a merged LoRA (rank 64, attention and MLP, 2 epochs, 29,356 rows: generated decision families, document questions written by Qwen3.6-27B and kept when two independent answers agreed, human-labelled public sets, replay of mixture v2); temperature 1.935. Better on hard judgments, TypeSafe, OpenJev and Bespoke's suite; worse on sampled play, some text games and by about 1 point on everyday tasks |
-| v1 (2026-09-22, Hub tag `v1`) | first release: one pass over mixture v2 on Qwen3.5-4B-Base with AdamW on bf16 parameters, no RL stage; temperature 1.05 |
-
-The GitHub repository's [docs/CHANGELOG.md](https://github.com/Mapika/decider/blob/main/docs/CHANGELOG.md) lists every
-decider release.
-
-## Reproduction
-
-Code, data registry, training and evaluation scripts and the per-version history: https://github.com/Mapika/decider
-(`docs/HISTORY.md`, section "decider-4b"). Stage 1 was trained with the data-parallel trainer of the architecture A/B study
-(`arch_ab/train_dp_optvar.py` in the research repository, optimizer variant `bf16`); stage 2 with a LoRA trainer in the research
-repository. Both were evaluated with the public `decider.evaluate` and the head-to-head tools. `eval_results.json` in this
-repository has the regression metrics with the map and at the global temperature, our held-out sets by answer type, the
-fixtures, games, browser and Bespoke results with the paired comparisons against v1 and v2, the text games, the behaviour
-probes, the issue #9 cases and the JevBench public items.
-
-**Independence.** This is an independent project. It is not affiliated with or endorsed by TypeSafe AI. It is an open
-reproduction of the "System One" model class (TypeSafe AI's Jev); nothing was distilled from Jev. The training data is public
-datasets, programmatically generated rows with verifiable gold, and data written or labelled by local Qwen3.5-27B and
-Qwen3.6-27B models. License: Apache 2.0.
+- **One pass only:** knowledge stays at the base model's level; multi-step tasks are not solved by reading harder.
+- **Yes/no calibration** is traded for JevBench's rule in the default config (see Changes).
+- **Images:** new image question types read about as well as the stock model. Not measured on ImageJevBench.
+- **Latency** was not measured again; the text path is unchanged from v2.1.
+
+## License
+
+The weights are released under Apache 2.0, as the base model. Training sources and their terms:
+
+- Image questions, from the train splits of [the_cauldron](https://huggingface.co/datasets/HuggingFaceM4/the_cauldron) subsets
+  (the_cauldron licenses only its own prompts, CC BY 4.0; each source keeps its own terms):
+  - AI2D (Kembhavi et al., 2016): CC BY-SA 4.0.
+  - A-OKVQA (Schwenk et al., 2022): Apache 2.0 (repository); images from COCO.
+  - IconQA (Lu et al., 2021): CC BY-NC-SA 4.0, non-commercial.
+  - ScienceQA (Lu et al., 2022): CC BY-NC-SA 4.0, non-commercial research use.
+  - TQA (Kembhavi et al., 2017): CC BY-NC 3.0, non-commercial.
+  - RAVEN (Zhang et al., 2019): no data licence stated (the code is GPL-3.0).
+  - Hateful Memes (Kiela et al., 2020): Facebook's Hateful Memes dataset licence. It permits use of the data to research and
+    develop models designed to detect manipulated media, and states that models derived from it may be used for academic and
+    commercial purposes. This is a research project; check the licence for your own use.
+
+- CLINC150: CC BY 3.0 (Larson et al., 2019). Credit: "An Evaluation Dataset for Intent Classification and Out-of-Scope
+  Prediction", clinc/oos-eval.
+- iSarcasmEval: MIT (Abu Farha et al., 2022). The texts are tweets.
+- POP909-CL: MIT; built on POP909 (Wang et al., 2020), MIT.
+- VAST (Allaway and McKeown, 2020) and ACOS (Cai et al., 2021): no licence stated by the authors.
+- GSM8K (Cobbe et al., 2021): MIT.
+- NLI4CT (SemEval-2024 Task 2, Jullien et al.): no licence file in the task repository; released for the shared task.
+- RACE: non-commercial research use. It is a small part of the replayed data (250 rows).
+- ARC, SNLI, LogiQA 2.0 (copy dependent): CC BY-SA 4.0; BoolQ: CC BY-SA 3.0; BANKING77, MedQA: CC BY 4.0; MMLU,
+  CommonsenseQA: MIT; MNLI: mixed per genre.
+- The sources of v2.1 and earlier, listed on the v2.1 card (Hub tag `v2.1`).

decider/batching.py

@@ -1,95 +0,0 @@
-"""Batch planning for the server's queue: which queued rows share one forward.
-
-Pure python, no torch, no engine: `plan_batches` takes the row lengths and two callables that describe the engine's
-bucket grid, and returns the groups to run.  `decider.serve.batcher` calls it once per collection.
-
-Cost model.  A forward of B rows padded to length T costs `overhead + B * T` token-units: `overhead` is the fixed cost of
-one forward (launch, replay, the slot gather and the device-to-host copy) expressed as the number of padded tokens that
-take the same time, and `B * T` is the padded token work.  Two rows of different lengths are therefore worth merging
-into the longer row's bucket exactly when the padding they add is cheaper than a second forward's overhead.
-`DEFAULT_MERGE_OVERHEAD_TOKENS` is measured on decider-2b (docs/SERVING.md, section on the batching policy); the server
-reads `DECIDER_MERGE_OVERHEAD_TOKENS` over it.
-
-The partition is exact, not greedy.  Rows are sorted by padded length descending (stable, so rows of the same bucket keep
-their arrival order) and split into consecutive groups; each group runs at the bucket of its longest member, so a group
-starting at position k costs `overhead + g * T[k]`.  A dynamic program over the sorted sequence takes the cheapest split,
-subject to `g <= min(max_batch, max_rows(T))`, with ties going to the larger group.  The per-bucket grouping the server
-did before is one of the partitions the program may choose (rows of equal length are adjacent in the sorted order), so
-the planned cost is never above it, and with `overhead = 0` the plan is exactly that grouping.
-"""
-
-DEFAULT_MERGE_OVERHEAD_TOKENS = 512
-
-
-def group_cap(T, max_rows, max_batch):
-    """Rows one forward may take at padded length T: the engine's widest captured batch bucket, and the server's cap."""
-    return max(1, min(int(max_batch), int(max_rows(T))))
-
-
-def batch_cost(size, T, overhead=DEFAULT_MERGE_OVERHEAD_TOKENS):
-    """Cost of one forward of `size` rows padded to `T`, in token-units."""
-    return overhead + size * T
-
-
-def plan_cost(groups, overhead=DEFAULT_MERGE_OVERHEAD_TOKENS):
-    """Cost of a plan: the sum over its forwards."""
-    return sum(batch_cost(len(idx), T, overhead) for T, idx in groups)
-
-
-def per_bucket_groups(lengths, pad_len, max_rows, max_batch):
-    """The grouping of 1.1.0: rows of the same padded length, chunked to the cap.  The baseline `plan_batches` improves on."""
-    return _exact_bucket(list(range(len(lengths))), [pad_len(n) for n in lengths], max_rows, max_batch)
-
-
-def _exact_bucket(idx, pads, max_rows, max_batch):
-    buckets = {}
-    for i in idx:
-        buckets.setdefault(pads[i], []).append(i)
-    out = []
-    for T, members in buckets.items():
-        cap = group_cap(T, max_rows, max_batch)
-        for k in range(0, len(members), cap):
-            out.append((T, members[k:k + cap]))
-    return out
-
-
-def plan_batches(lengths, pad_len, max_rows, max_batch, overhead=DEFAULT_MERGE_OVERHEAD_TOKENS, mergeable=None):
-    """Partition queued rows into forwards.
-
-    lengths:    token count of every queued row, in arrival order.
-    pad_len(n): the engine's padded length for a row of n tokens (`EngineV2.pad_len`).
-    max_rows(T):rows the engine runs in one forward at padded length T (`EngineV2.max_rows`).
-    max_batch:  the server's own cap (`DECIDER_MAX_BATCH`).
-    overhead:   fixed cost of a forward in token-units (see the module docstring).
-    mergeable(n): False for a row that must not be padded into another row's bucket.  Rows above the last captured
-                  length bucket run eager at a request-specific shape, so they are grouped by exact length as before.
-
-    -> [(padded length, [row indices]), ...].  Every row appears in exactly one group; a group's padded length is at
-    least every member's own padded length; a group holds at most `min(max_batch, max_rows(T))` rows.
-    """
-    n = len(lengths)
-    if n == 0:
-        return []
-    pads = [pad_len(x) for x in lengths]
-    merge = [True] * n if mergeable is None else [bool(mergeable(x)) for x in lengths]
-    out = _exact_bucket([i for i in range(n) if not merge[i]], pads, max_rows, max_batch)
-
-    order = sorted((i for i in range(n) if merge[i]), key=lambda i: -pads[i])    # stable: equal buckets keep arrival order
-    m = len(order)
-    dp = [0] * (m + 1)                       # dp[k]: cheapest cost of covering order[k:]
-    take = [0] * (m + 1)
-    for k in range(m - 1, -1, -1):
-        T = pads[order[k]]
-        cap = min(group_cap(T, max_rows, max_batch), m - k)
-        best, best_g = None, 1
-        for g in range(1, cap + 1):
-            c = overhead + g * T + dp[k + g]
-            if best is None or c <= best:        # ties go to the larger group: same modelled cost, one fewer launch
-                best, best_g = c, g
-        dp[k], take[k] = best, best_g
-    k = 0
-    while k < m:
-        g = take[k]
-        out.append((pads[order[k]], sorted(order[k:k + g])))                     # inside a group, arrival order
-        k += g
-    return out

decider/calibrate.py

@@ -1,222 +0,0 @@
-"""Fit decider_config.json "temperature_by_type" by NLL, one temperature per answer type (decider.temperature).
-
-    python -m decider.calibrate records.jsonl [--min-rows 50]
-    -> {"temperature_by_type": {"choice": 1.48, "noul": 2.22, "score": 1.38}, "temperature": 1.6, "rows": {...}, "nll": {...}}
-
-A record is one answer with its gold, read at temperature 1:
-    {"type": "choice" | "noul" | "score", "logits": [one value per option], "gold": option index}
-    {"type": "score", "level_logits": [[no, yes], one pair per level], "gold": level index}      Score with isolated levels
-"probs" / "level_probs" (probabilities at temperature 1) may stand in for "logits" / "level_logits": log p is the logit up to a
-constant, which the softmax ignores.  A noul gold is 1 for true, 0 for false.  Every record is checked first (check_record):
-a malformed one (gold out of range or not an integer, wrong shape, NaN, a row without a finite logit, an isolated Score whose
-levels all have P(yes) = 0) raises ValueError naming its position.
-
-An isolated-levels Score answer is fitted through the readout the server uses: every level row is softmax(row / T), and the
-answer is P(yes) of each level divided by their sum (decider.systemone.combine_isolated).  So the "score" temperature is fitted on
-Score answers whichever readout the model serves them with; fit it on records collected with the same isolated_levels setting.
-
-`collect(decider, examples)` produces records from a decider.infer.Decider and labelled /v1/systemone-shaped examples.
-"temperature" in the output is one temperature fitted on all records together, for comparison; a type with fewer than
---min-rows records is left out of the map (it then uses "temperature" of the config).
-"""
-import json, math, sys
-import numpy as np
-
-from decider.temperature import TYPES
-
-GRID = np.exp(np.linspace(math.log(0.05), math.log(20.0), 801))       # 0.05 .. 20, about 0.75 % apart
-
-
-def _log(x):
-    x = np.asarray(x, dtype=np.float64)
-    with np.errstate(divide="ignore"):
-        return np.where(x > 0, np.log(np.clip(x, 1e-300, None)), -np.inf)
-
-
-def _logits(rec, key):
-    if key in rec:
-        return np.asarray(rec[key], dtype=np.float64)
-    alt = {"logits": "probs", "level_logits": "level_probs"}[key]
-    return _log(rec[alt])
-
-
-def _lse(z, axis=-1):
-    m = np.max(z, axis=axis, keepdims=True)
-    m = np.where(np.isfinite(m), m, 0.0)
-    with np.errstate(divide="ignore"):                              # an all -inf row gives -inf, handled by the callers
-        return (m + np.log(np.sum(np.exp(z - m), axis=axis, keepdims=True))).squeeze(axis)
-
-
-def nll(rec, T):
-    """Negative log-likelihood of one record's gold at temperature T."""
-    return float(_Batch([rec]).nll(T))
-
-
-def _isolated(rec):
-    return "level_logits" in rec or "level_probs" in rec
-
-
-def check_record(rec, i=None):
-    """Raise ValueError unless `rec` is a well-formed record (module docstring): an integral gold within the options or levels,
-    a 1-D row of at least two options or an [levels, 2] array of at least two levels, logits that are not NaN or +inf,
-    probabilities in [0, 1] with positive mass, and isolated levels only on a Score record."""
-    where = f"record {i}" if i is not None else "record"
-    if not isinstance(rec, dict):
-        raise ValueError(f"{where}: not a JSON object")
-    if rec.get("type") not in TYPES:
-        raise ValueError(f"{where}: type {rec.get('type')!r}, expected one of {', '.join(TYPES)}")
-    iso = _isolated(rec)
-    keys = ("level_logits", "level_probs") if iso else ("logits", "probs")
-    if sum(k in rec for k in keys) != 1 or (iso and ("logits" in rec or "probs" in rec)):
-        raise ValueError(f"{where}: give exactly one of logits, probs, level_logits, level_probs")
-    if iso and rec["type"] != "score":
-        raise ValueError(f"{where}: isolated levels (level_logits / level_probs) belong to a score record, not {rec['type']!r}")
-    key = keys[0] if keys[0] in rec else keys[1]
-    try:
-        a = np.asarray(rec[key], dtype=np.float64)
-    except (TypeError, ValueError):
-        raise ValueError(f"{where}: {key} is not a numeric array") from None
-    if iso and (a.ndim != 2 or a.shape[1] != 2 or a.shape[0] < 2):
-        raise ValueError(f"{where}: {key} must be one [no, yes] pair per level, at least two levels; got shape {a.shape}")
-    if not iso and (a.ndim != 1 or a.shape[0] < 2):
-        raise ValueError(f"{where}: {key} must be one value per option, at least two options; got shape {a.shape}")
-    if key.endswith("probs"):
-        if not np.all(np.isfinite(a)) or np.any(a < 0) or np.any(a > 1) or np.any(a.sum(-1) <= 0):
-            raise ValueError(f"{where}: {key} must be probabilities in [0, 1] with positive mass per row")
-    elif np.any(np.isnan(a)) or np.any(a == np.inf) or not np.all(np.any(np.isfinite(a), axis=-1)):
-        raise ValueError(f"{where}: {key} contains NaN or +inf, or a row without a finite value")
-    if rec["type"] == "noul" and a.shape[0] != 2:
-        raise ValueError(f"{where}: a noul record has exactly two options (false, true); got {a.shape[0]}")
-    if iso and not np.any(a[:, 1] > 0 if key == "level_probs" else np.isfinite(a[:, 1])):
-        raise ValueError(f"{where}: no level has a yes probability above 0, so the served Score answer has no distribution")
-    g = rec.get("gold")
-    if isinstance(g, bool) or not isinstance(g, (int, np.integer)) or not 0 <= g < a.shape[0]:
-        raise ValueError(f"{where}: gold must be an integer index in 0..{a.shape[0] - 1}, got {g!r}")
-    return rec
-
-
-class _Batch:
-    """Records packed into padded arrays, so one temperature is scored over all of them at once."""
-    def __init__(self, records):
-        records = [check_record(r, i) for i, r in enumerate(records)]
-        lists = [r for r in records if not _isolated(r)]
-        isos = [r for r in records if _isolated(r)]
-        self.n = len(lists) + len(isos)
-        self.L = self.Lg = self.I = self.Im = self.Ig = None
-        if lists:
-            zs = [_logits(r, "logits") for r in lists]
-            K = max(len(z) for z in zs)
-            self.L = np.full((len(zs), K), -np.inf)
-            for i, z in enumerate(zs):
-                self.L[i, :len(z)] = z
-            self.Lg = np.array([int(r["gold"]) for r in lists])
-        if isos:
-            zs = [_logits(r, "level_logits").reshape(-1, 2) for r in isos]
-            n = max(len(z) for z in zs)
-            self.I = np.zeros((len(zs), n, 2)); self.Im = np.zeros((len(zs), n), dtype=bool)
-            for i, z in enumerate(zs):
-                self.I[i, :len(z)] = z; self.Im[i, :len(z)] = True
-            self.Ig = np.array([int(r["gold"]) for r in isos])
-
-    def nll(self, T):
-        """Sum of the records' NLL at temperature T."""
-        tot = 0.0
-        if self.L is not None:
-            z = self.L / T
-            zg = z[np.arange(len(z)), self.Lg]
-            tot += float(np.sum(np.where(np.isfinite(zg), _lse(z) - zg, 690.0)))
-        if self.I is not None:                          # in log space: P(yes) of very confident rows underflows otherwise
-            z = self.I / T
-            with np.errstate(invalid="ignore"):
-                lp = np.where(self.Im, z[..., 1] - _lse(z), -np.inf)          # log P(yes) per level, -inf on padding
-            norm = _lse(lp)                                                  # log of the summed P(yes)
-            lg = lp[np.arange(len(z)), self.Ig]
-            with np.errstate(invalid="ignore"):
-                per = np.where(np.isfinite(lg), norm - lg, 690.0)               # check_record guarantees a finite norm
-            tot += float(np.sum(per))
-        return tot
-
-
-def mean_nll(records, T):
-    b = records if isinstance(records, _Batch) else _Batch(records)
-    return b.nll(T) / b.n
-
-
-def fit(records, grid=GRID):
-    """The temperature with the lowest mean NLL on the grid, refined by a golden-section search between its neighbours."""
-    records = records if isinstance(records, _Batch) else _Batch(list(records))
-    if not records.n:
-        raise ValueError("no records to fit")
-    vals = [mean_nll(records, T) for T in grid]
-    i = int(np.argmin(vals))
-    lo, hi = math.log(grid[max(i - 1, 0)]), math.log(grid[min(i + 1, len(grid) - 1)])
-    r = (math.sqrt(5) - 1) / 2
-    a, b = hi - r * (hi - lo), lo + r * (hi - lo)
-    fa, fb = mean_nll(records, math.exp(a)), mean_nll(records, math.exp(b))
-    for _ in range(40):
-        if fa < fb:
-            hi, b, fb = b, a, fa; a = hi - r * (hi - lo); fa = mean_nll(records, math.exp(a))
-        else:
-            lo, a, fa = a, b, fb; b = lo + r * (hi - lo); fb = mean_nll(records, math.exp(b))
-    T = math.exp((lo + hi) / 2)
-    return T if mean_nll(records, T) <= vals[i] else float(grid[i])
-
-
-def fit_by_type(records, min_rows=50):
-    """-> {"temperature_by_type": {type: T}, "temperature": pooled T, "rows": {type: n}, "nll": {type: {"T=1": .., "fitted": ..}}}"""
-    records = [check_record(r, i) for i, r in enumerate(records)]
-    groups = {t: [] for t in TYPES}
-    for r in records:
-        groups[r["type"]].append(r)
-    out = {"temperature_by_type": {}, "temperature": round(fit(records), 3) if records else None,
-           "rows": {t: len(g) for t, g in groups.items()}, "nll": {}}
-    for t, g in groups.items():
-        if len(g) < max(1, min_rows):
-            continue
-        b = _Batch(g); T = fit(b)
-        out["temperature_by_type"][t] = round(T, 3)
-        out["nll"][t] = {"T=1": round(mean_nll(b, 1.0), 4), "fitted": round(mean_nll(b, T), 4)}
-    return out
-
-
-def collect(decider, examples, independent=True):
-    """Records for fit_by_type from a decider.infer.Decider, read at temperature 1 on the uncached state-first path.
-    examples: iterable of (state, questions, golds) with questions as /v1/systemone takes them and golds {question id: gold},
-    the gold being a Choice option name, a Noul true/false, or a Score level index.  Questions without a gold are skipped.
-    Score questions are read with the Decider's isolated_levels setting, as system_one serves them."""
-    recs = []
-    for state, questions, golds in examples:
-        rqs, index, items = decider._system_one_items(state, questions, independent, layout="state_first")
-        rows = decider._system_one_probs(items, "state_first", temperature=1.0)
-        for k, kind, s, n in index:
-            if k not in golds:
-                continue
-            rq, g = rqs[k], golds[k]
-            if rq["type"] == "noul" and not (isinstance(g, bool) or g in (0, 1)):
-                raise ValueError(f"gold of noul question {k!r} must be true or false, got {g!r}")
-            if rq["type"] == "score" and (isinstance(g, bool) or not isinstance(g, int) or not 0 <= g < len(rq["names"])):
-                raise ValueError(f"gold of score question {k!r} must be a level index in 0..{len(rq['names']) - 1}, got {g!r}")
-            key = bool(g) if rq["type"] == "noul" else g
-            if key not in rq["names"]:
-                raise ValueError(f"gold of choice question {k!r} is not one of its options: {g!r}")
-            gold = rq["names"].index(key)
-            if kind == "iso":
-                recs.append({"type": rq["type"], "level_probs": [rows[s + j][:2] for j in range(n)], "gold": gold})
-            else:
-                recs.append({"type": rq["type"], "probs": rows[s][:len(rq["options"])], "gold": gold})
-    return recs
-
-
-def main(argv=None):
-    import argparse
-    ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
-    ap.add_argument("records", help="JSON lines, one record per line (see the module docstring)")
-    ap.add_argument("--min-rows", type=int, default=50, help="fit a type only when it has at least this many records")
-    a = ap.parse_args(argv)
-    with open(a.records) as f:
-        recs = [json.loads(line) for line in f if line.strip()]
-    print(json.dumps(fit_by_type(recs, a.min_rows), indent=1))
-
-
-if __name__ == "__main__":
-    main(sys.argv[1:])

decider/engine.py

@@ -1,216 +0,0 @@
-"""Low-latency inference engine: shape-bucketed CUDA graphs over the one-pass decision model.
-
-Right padding + causal layers => pad positions never influence earlier slots, so no attention
-mask is needed and every (B, T) bucket can be captured once and replayed.  The graph outputs
-option-letter logits for all positions [B, T, K]; slots are gathered outside.
-"""
-import time, torch, torch._dynamo, torch.nn.functional as F
-from decider.model import DecisionModel, collate
-from decider.prompt import build, MAX_OPTIONS
-from decider.temperature import scaled_softmax, slot_temperatures
-
-T_BUCKETS = [64, 128, 192, 256, 320, 384, 512, 640, 768, 1024, 1280, 1536, 2048]
-B_BUCKETS = [1, 2, 4, 8, 16, 32, 64]
-GRAPH_MAX_T = 2048          # longer inputs (up to the 32k request budget) run eagerly: compute dominates there, and one graph
-LONG_STEP = 1024            # per (B, T) shape would cost a compile + capture for every new length
-
-
-def _bucket(x, buckets):
-    for b in buckets:
-        if x <= b:
-            return b
-    return None
-
-
-def fused_causal_conv1d_fn(hidden_states, weight, bias=None, activation=None, **kwargs):
-    """Depthwise causal conv (kernel k) as k shifted multiply-adds: fuses under torch.compile,
-    unlike the cuDNN grouped conv fallback (which was ~11% of batched GPU time)."""
-    B, C, T = hidden_states.shape; k = weight.shape[-1]
-    x = F.pad(hidden_states.to(weight.dtype), (k - 1, 0))
-    out = x[:, :, k - 1:k - 1 + T] * weight[:, k - 1][None, :, None]
-    for j in range(k - 1):
-        out = out + x[:, :, j:j + T] * weight[:, j][None, :, None]
-    if bias is not None:
-        out = out + bias[None, :, None]
-    if activation == "silu":
-        out = F.silu(out)
-    elif activation is not None:
-        from transformers.activations import ACT2FN
-        out = ACT2FN[activation](out)
-    return out.to(hidden_states.dtype)
-
-
-def patch_conv():
-    from transformers.models.qwen3_5 import modeling_qwen3_5 as mq
-    mq.causal_conv1d_fn = fused_causal_conv1d_fn
-
-
-def read_slots(out, rows, slots, nopts, temperature, n_per_item):
-    """One gather + one softmax + one device-to-host copy for the whole batch (was: three small kernels and a sync per item).
-    out [B, T, K] logits; rows/slots/nopts: flat python lists, one entry per question; n_per_item: questions per item.
-    temperature: a number for every question, or a flat list with one temperature per question (decider.temperature)."""
-    dev = out.device; idx = torch.tensor([rows, slots, nopts], dtype=torch.long).to(dev, non_blocking=True)
-    lg = out[idx[0], idx[1]]                                                                  # [N, K]
-    lg = lg.masked_fill(torch.arange(lg.shape[1], device=dev)[None, :] >= idx[2][:, None], float("-inf"))
-    p = scaled_softmax(lg, temperature).cpu()
-    return list(torch.split(p, n_per_item))
-
-
-def fill_ids(items_ids, B, T, pad):
-    import numpy as np
-    a = np.full((B, T), pad, dtype=np.int64)
-    for b, x in enumerate(items_ids): a[b, :len(x)] = x
-    return torch.from_numpy(a)
-
-
-def set_attention_backend_policy():
-    """Turn off the cuDNN scaled-dot-product-attention backend. On Blackwell with torch 2.14 / CUDA 13 it returns wrong,
-    finite output for masked rectangular attention, which is what the shared-state path (`Engine.score_shared`) and the schema
-    cache run when a suffix is scored against a cached prefix; the math and memory-efficient backends are correct. Measured on
-    a Decision Index row: the cached path answered a wrong option at p=0.93 where the full forward and the corrected cached path
-    both give option_4 at p=0.95 (decider2/SERVING_V2_REVIEW.md in the research notes). Must run before torch.compile and CUDA
-    graph capture: captured graphs keep the backend they were captured with."""
-    if hasattr(torch.backends.cuda, "enable_cudnn_sdp"):
-        torch.backends.cuda.enable_cudnn_sdp(False)
-
-
-class Engine:
-    """compile: torch.compile the forward (needs use_cache=False; ~1.4x batched, fuses elementwise work).
-    fp8: e4m3 weights + per-token activation scaling on the big linears (Hopper tensor cores).
-    conv_patch: fusable depthwise causal conv instead of the cuDNN fallback."""
-    def __init__(self, path, device="cuda", dtype=torch.bfloat16, use_graphs=True, max_ctx_tokens=1536,
-                 compile=True, fp8=False, conv_patch=True):
-        set_attention_backend_policy()
-        if conv_patch:
-            if str(device).startswith("mps"):
-                from decider.mps_ops import patch_mps
-                patch_mps()
-            else:
-                patch_conv()
-        self.m = DecisionModel(path, dtype=dtype, grad_ckpt=False).to(device).eval()
-        use_graphs = use_graphs and torch.device(device).type == "cuda"
-        self.tok = self.m.tok; self.dev = device; self.use_graphs = use_graphs; self.max_ctx = max_ctx_tokens
-        self.core, self.W = self.m.lm.model, self.m.lm.lm_head.weight[self.m.letters].detach().clone()
-        self.cfg = dict(compile=compile, fp8=fp8, conv_patch=conv_patch, graphs=use_graphs)
-        if fp8:
-            from decider.fp8 import convert_to_fp8
-            self.cfg["fp8_layers"] = convert_to_fp8(self.core)
-        if compile:
-            torch._dynamo.config.cache_size_limit = 128
-            self._fwd_impl = torch.compile(self._fwd_eager, dynamic=False)
-        else:
-            self._fwd_impl = self._fwd_eager
-        self.graphs = {}                       # (B, T) -> (static_ids, static_out, graph)
-        self.pool = torch.cuda.graph_pool_handle() if (use_graphs and str(device).startswith("cuda")) else None
-        self.stats = dict(graph_captures=0, forwards=0)
-
-    def _fwd_eager(self, ids):
-        h = self.core(input_ids=ids, use_cache=False).last_hidden_state
-        return F.linear(h, self.W).float()                          # [B, T, K]
-
-    @torch.no_grad()
-    def _fwd(self, ids):
-        return self._fwd_impl(ids)
-
-    def _capture(self, B, T):
-        s_ids = torch.full((B, T), self.tok.pad_token_id, dtype=torch.long, device=self.dev)
-        st = torch.cuda.Stream(); st.wait_stream(torch.cuda.current_stream())
-        with torch.cuda.stream(st):
-            for _ in range(3): self._fwd(s_ids)                     # warm-up: compile / triton autotune
-        torch.cuda.current_stream().wait_stream(st)
-        g = torch.cuda.CUDAGraph()
-        with torch.cuda.graph(g, pool=self.pool):
-            s_out = self._fwd(s_ids)
-        self.stats["graph_captures"] += 1
-        return s_ids, s_out, g
-
-    @torch.no_grad()
-    def logits_all(self, ids):
-        """ids: [B, T] long on device (already right-padded to a bucket). Returns [B, T, K] float."""
-        B, T = ids.shape; self.stats["forwards"] += 1
-        if T > GRAPH_MAX_T:
-            self.stats["long_forwards"] = self.stats.get("long_forwards", 0) + 1
-            return self._fwd_eager(ids)
-        if not self.use_graphs:
-            return self._fwd(ids)
-        key = (B, T)
-        if key not in self.graphs:
-            self.graphs[key] = self._capture(B, T)
-        s_ids, s_out, g = self.graphs[key]
-        s_ids.copy_(ids); g.replay()
-        return s_out
-
-    @torch.no_grad()
-    def score_items(self, items, temperature=1.0):
-        """items: list of dicts from prompt.build. Returns list of [n_q, MAX_OPTIONS] prob tensors (cpu).
-        temperature: a number, or one entry per item (a number or one number per slot; decider.temperature.for_items)."""
-        Tmax = max(len(it["ids"]) for it in items)
-        T = _bucket(Tmax, T_BUCKETS) or -(-Tmax // LONG_STEP) * LONG_STEP
-        B = (_bucket(len(items), B_BUCKETS) or len(items)) if T <= GRAPH_MAX_T else len(items)
-        ids = fill_ids([it["ids"] for it in items], B, T, self.tok.pad_token_id)
-        out = self.logits_all(ids.to(self.dev, non_blocking=True))
-        return read_slots(out, [b for b, it in enumerate(items) for _ in it["slots"]], [s for it in items for s in it["slots"]],
-                          [n for it in items for n in it["nopts"]], slot_temperatures(temperature, items), [len(it["slots"]) for it in items])
-
-    @torch.no_grad()
-    def score_shared(self, items, temperature=1.0, min_prefix=192):
-        """Rows that start with the same tokens (one state, one question per row): run the shared prefix once, fork its
-        cache (attention KV + delta-net conv/recurrent states), and run only the question suffixes.
-        Same answers as score_items up to kernel round-off; cost ~ state + sum(questions) instead of n * state.
-        The fork is made in chunks that fit `DECIDER_SHARED_FORK_GB`, so the peak memory does not grow with the question
-        count; the implementation is decider.shared_prefix, shared with EngineV2."""
-        from decider import shared_prefix                      # imported here: decider.shared_prefix imports this module
-        out = shared_prefix.score_shared(self, items, temperature, min_prefix)
-        if out is None:
-            return self.score_items(items, temperature)
-        self.stats["shared_prefix_calls"] = self.stats.get("shared_prefix_calls", 0) + 1
-        return out
-
-    def warmup(self, shapes=((1, 128), (1, 256), (1, 384), (1, 512), (8, 256), (8, 512), (32, 256), (32, 512))):
-        t = time.time()
-        for B, T in shapes:
-            self.logits_all(torch.full((B, T), self.tok.pad_token_id, dtype=torch.long, device=self.dev))
-        torch.cuda.synchronize(); return time.time() - t
-
-
-if __name__ == "__main__":
-    import sys, random, numpy as np
-    from decider import data as D
-    from decider.infer import Decider
-    path = sys.argv[1] if len(sys.argv) > 1 else "runs/r3_v2/model"
-    cfg = dict(compile="nocompile" not in sys.argv[2:], fp8="fp8" in sys.argv[2:], conv_patch="noconv" not in sys.argv[2:])
-    _, evals = D.load_cache("data/tasks.pkl")
-    eng = Engine(path, **cfg); print("engine cfg", eng.cfg)
-    rng = random.Random(0)
-    exs = evals["support_tickets"][:64] + evals["clinc_oos"][:64] + evals["race"][:32]
-    from decider.prompt import chat_for_model
-    chat = chat_for_model(path, eng.tok)
-    items = [build(e, eng.tok, rng, max_ctx_tokens=1536, chat=chat) for e in exs]
-    # correctness vs eager masked forward (DecisionModel.slot_logits)
-    ref = []
-    with torch.no_grad():
-        for i in range(0, len(items), 16):
-            b = collate(items[i:i + 16], eng.tok.pad_token_id)
-            lg = eng.m.slot_logits(b["input_ids"].cuda(), b["attention_mask"].cuda(), b["slot_idx"].cuda(), b["slot_batch"].cuda(), b["nopts"].cuda())
-            ref.append(torch.softmax(lg, -1).cpu())
-    ref = torch.cat(ref)
-    got = torch.cat(eng.score_items(items))
-    print(f"max |p_graph - p_eager| = {(ref - got).abs().max():.4f} over {len(ref)} questions; argmax agreement {(ref.argmax(1) == got.argmax(1)).float().mean():.4f}")
-    print(f"warmup capture of 8 buckets: {eng.warmup():.1f}s; captures so far {eng.stats['graph_captures']}")
-    # latency: single real requests
-    for name, pool in [("support_tickets", exs[:64]), ("clinc_oos", exs[64:128]), ("race", exs[128:])]:
-        its = [build(e, eng.tok, rng, chat=chat) for e in pool]
-        ts = []
-        for it in its[:40]:
-            torch.cuda.synchronize(); t = time.time(); eng.score_items([it]); torch.cuda.synchronize(); ts.append(time.time() - t)
-        ts = np.array(ts[5:]) * 1000
-        print(f"single request {name:16s}: p50 {np.median(ts):5.1f} ms  p90 {np.percentile(ts, 90):5.1f} ms  (avg {np.mean([len(i['ids']) for i in its]):.0f} tok, {len(its[0]['slots'])} q)")
-        for bs in (8, 32):
-            ts = []
-            for i in range(0, min(len(its), bs * 6), bs):
-                chunk = its[i:i + bs]
-                if len(chunk) < bs: break
-                torch.cuda.synchronize(); t = time.time(); eng.score_items(chunk); torch.cuda.synchronize(); ts.append(time.time() - t)
-            ts = np.array(ts[1:]) * 1000
-            print(f"   batch {bs:2d}: p50 {np.median(ts):6.1f} ms -> {bs/np.median(ts)*1000:6.0f} ctx/s, {bs*len(its[0]['slots'])/np.median(ts)*1000:6.0f} decisions/s")
-    print("stats", eng.stats, "graphs", len(eng.graphs), f"mem {torch.cuda.memory_reserved()/1e9:.1f} GB")

decider/engine_v2.py

@@ -1,213 +0,0 @@
-"""Serving engine: every CUDA graph is keyed on (batch bucket, padded length bucket) only.
-
-Difference from decider.engine.Engine (the library engine behind decider.infer.Decider):
-  * the full bucket grid is captured at start-up by `warmup()`, then `seal()` forbids any further capture, so no request
-    can ever pay a graph capture or a torch.compile;
-  * the length ladder reaches 8192 instead of 2048, so long rows replay a graph instead of running eager;
-  * a batch larger than the widest captured batch bucket for its length is split into captured chunks rather than
-    capturing a new shape; rows longer than the last bucket run eager in chunks of at most `token_budget` padded tokens;
-  * torch.compile and FP8 are off by default.
-
-Numerics are the same as Engine: right padding + causal layers means padded positions never influence earlier slots, so
-the answer for a row does not depend on which (B, T) bucket it was padded into, up to kernel reduction order.
-
-The attention backend policy of decider.engine (cuDNN SDPA off) is applied in __init__, before compile and capture.  It
-is what makes `score_shared` (a suffix scored against a cached prefix) return the same answers as the full forward; see
-docs/CHANGELOG.md 1.0.2 and tests/test_engine_v2_cuda.py.
-"""
-import time, torch, torch.nn.functional as F
-from decider import shared_prefix
-from decider.engine import read_slots, fill_ids, patch_conv, set_attention_backend_policy
-from decider.model import DecisionModel
-from decider.temperature import item_slice, slot_temperatures
-
-T_BUCKETS = [64, 128, 192, 256, 320, 384, 512, 640, 768, 1024, 1280, 1536, 2048, 3072, 4096, 6144, 8192]
-B_BUCKETS = [1, 2, 4, 8, 16, 32]
-TOKEN_BUDGET = 32768        # capture (B, T) only when B * T fits this; B = 1 is always captured
-LONG_STEP = 1024            # rows longer than the last bucket: pad to a multiple of this and run eager
-
-
-class EngineV2:
-    """compile / fp8 default to False.  `model` lets a test inject a stand-in DecisionModel instead of loading one."""
-
-    def __init__(self, path=None, device="cuda", dtype=torch.bfloat16, use_graphs=None, compile=False, fp8=False,
-                 conv_patch=None, t_buckets=None, b_buckets=None, token_budget=TOKEN_BUDGET, max_ctx_tokens=32768,
-                 model=None):
-        set_attention_backend_policy()              # before compile and capture: captured graphs keep their backend
-        if conv_patch is None:
-            conv_patch = bool(compile)          # the unrolled depthwise conv only pays off inside a compiled region
-        if str(device).startswith("mps"):
-            from decider.mps_ops import patch_mps       # the Apple Silicon path of decider.infer.Decider
-            patch_mps()
-        elif conv_patch:
-            patch_conv()
-        self.m = model if model is not None else DecisionModel(path, dtype=dtype, grad_ckpt=False).to(device).eval()
-        self.tok = self.m.tok; self.dev = device; self.max_ctx = max_ctx_tokens
-        self.core = self.m.lm.model
-        self.W = self.m.lm.lm_head.weight[self.m.letters].detach().clone()
-        self.t_buckets = sorted(t_buckets or T_BUCKETS)
-        self.b_buckets = sorted(b_buckets or B_BUCKETS)
-        self.token_budget = int(token_budget)
-        self.use_graphs = (str(device).startswith("cuda") if use_graphs is None else bool(use_graphs))
-        self.cfg = dict(compile=compile, fp8=fp8, conv_patch=conv_patch, graphs=self.use_graphs,
-                        t_buckets=self.t_buckets, b_buckets=self.b_buckets, token_budget=self.token_budget)
-        if fp8:
-            from decider.fp8 import convert_to_fp8
-            self.cfg["fp8_layers"] = convert_to_fp8(self.core)
-        if compile:
-            from torch import _dynamo                    # not `import torch._dynamo`: that rebinds `torch` locally
-            n = len(self.graph_shapes())
-            # one specialisation per captured shape, times the frames inside the model.  Without the accumulated
-            # limits dynamo stops compiling part way through warm-up and silently leaves the rest eager, which with
-            # conv_patch on is slower than not compiling at all.
-            _dynamo.config.cache_size_limit = max(256, n + 16)
-            _dynamo.config.accumulated_cache_size_limit = max(1 << 16, 64 * n)
-            for k in ("accumulated_recompile_limit", "recompile_limit"):
-                if hasattr(_dynamo.config, k):
-                    setattr(_dynamo.config, k, max(1 << 16, 64 * n))
-            self._fwd_impl = torch.compile(self._fwd_eager, dynamic=False)
-        else:
-            self._fwd_impl = self._fwd_eager
-        self.graphs = {}                 # (B, T) -> (static ids, static out, graph)
-        self.b_for_t = {}                # T -> sorted captured batch buckets
-        self.pool = torch.cuda.graph_pool_handle() if (self.use_graphs and str(device).startswith("cuda")) else None
-        self.sealed = False
-        self.stats = dict(graph_captures=0, forwards=0, replays=0, eager_forwards=0, eager_rows=0,
-                          shared_calls=0, unbucketed_requests=0)
-
-    # ---- bucket arithmetic -------------------------------------------------
-    def graph_shapes(self):
-        """The full grid captured at start-up: every (B, T) whose padded token count fits the budget, plus all of B = 1."""
-        return [(B, T) for T in self.t_buckets for B in self.b_buckets if B == 1 or B * T <= self.token_budget]
-
-    def t_bucket(self, n):
-        """Padded length bucket for a row of n tokens, or None when it is longer than the last bucket."""
-        for t in self.t_buckets:
-            if n <= t:
-                return t
-        return None
-
-    def pad_len(self, n):
-        return self.t_bucket(n) or -(-n // LONG_STEP) * LONG_STEP
-
-    def max_rows(self, T):
-        """Rows per forward at padded length T: the widest captured batch bucket, or, when nothing is captured at T, as
-        many rows as fit the token budget (at least 1)."""
-        av = self.b_for_t.get(T)
-        return av[-1] if av else max(1, self.token_budget // max(T, 1))
-
-    def b_plan(self, n, T):
-        """Batch sizes covering n rows at length T, each one a captured bucket.  One chunk when a bucket fits."""
-        av = self.b_for_t.get(T)
-        if not av:
-            return [n]
-        fit = next((b for b in av if b >= n), None)
-        if fit is not None:
-            return [fit]
-        out = []; big = av[-1]
-        while n > big:
-            out.append(big); n -= big
-        out.append(next(b for b in av if b >= n))
-        return out
-
-    # ---- forward -----------------------------------------------------------
-    def _fwd_eager(self, ids):
-        h = self.core(input_ids=ids, use_cache=False).last_hidden_state
-        return F.linear(h, self.W).float()                          # [B, T, K]
-
-    @torch.no_grad()
-    def _fwd(self, ids):
-        return self._fwd_impl(ids)
-
-    def _capture(self, B, T):
-        s_ids = torch.full((B, T), self.tok.pad_token_id, dtype=torch.long, device=self.dev)
-        st = torch.cuda.Stream(); st.wait_stream(torch.cuda.current_stream())
-        with torch.cuda.stream(st):
-            for _ in range(3): self._fwd(s_ids)
-        torch.cuda.current_stream().wait_stream(st)
-        g = torch.cuda.CUDAGraph()
-        with torch.cuda.graph(g, pool=self.pool):
-            s_out = self._fwd(s_ids)
-        self.stats["graph_captures"] += 1
-        return s_ids, s_out, g
-
-    @torch.no_grad()
-    def logits_all(self, ids):
-        """ids: [B, T] long on device, already right-padded to a captured shape where one exists."""
-        B, T = ids.shape; self.stats["forwards"] += 1
-        if self.use_graphs:
-            g = self.graphs.get((B, T))
-            if g is None and not self.sealed:            # only reachable before seal(); a sealed engine never captures
-                g = self.graphs[(B, T)] = self._capture(B, T)
-                self.b_for_t.setdefault(T, [])
-                if B not in self.b_for_t[T]: self.b_for_t[T] = sorted(self.b_for_t[T] + [B])
-            if g is not None:
-                s_ids, s_out, gr = g
-                s_ids.copy_(ids); gr.replay(); self.stats["replays"] += 1
-                return s_out
-        self.stats["eager_forwards"] += 1; self.stats["eager_rows"] += B
-        return self._fwd_eager(ids)                      # never the compiled callable: a new shape must not compile
-
-    # ---- scoring -----------------------------------------------------------
-    @torch.no_grad()
-    def score_items(self, items, temperature=1.0):
-        """items: dicts from prompt.build / build_rows.  -> one [n_q, MAX_OPTIONS] cpu probability tensor per item.
-        temperature: a number, or one entry per item (a number or one number per slot; decider.temperature.for_items)."""
-        if not items:
-            return []
-        slot_temperatures(temperature, items)                      # a length mismatch fails before any forward
-        Tmax = max(len(it["ids"]) for it in items)
-        T = self.t_bucket(Tmax)
-        if T is None:
-            self.stats["unbucketed_requests"] += 1
-            T = -(-Tmax // LONG_STEP) * LONG_STEP
-            per = self.max_rows(T); plan = [per] * (len(items) // per) + ([len(items) % per] if len(items) % per else [])
-        else:
-            plan = self.b_plan(len(items), T)
-        out = []; i = 0
-        for B in plan:
-            chunk = items[i:i + B]; i += len(chunk)
-            ids = fill_ids([it["ids"] for it in chunk], B, T, self.tok.pad_token_id)
-            lg = self.logits_all(ids.to(self.dev, non_blocking=True))
-            out += read_slots(lg, [b for b, it in enumerate(chunk) for _ in it["slots"]],
-                              [s for it in chunk for s in it["slots"]], [n for it in chunk for n in it["nopts"]],
-                              slot_temperatures(item_slice(temperature, i - len(chunk), i), chunk), [len(it["slots"]) for it in chunk])
-        return out
-
-    @torch.no_grad()
-    def score_shared(self, items, temperature=1.0, min_prefix=192, budget_bytes=None, rows_per_fork=None):
-        """Rows that start with the same tokens (one state, one question per row): run the shared prefix once, fork its
-        cache in chunks that fit a byte budget, run only the question suffixes.  The algorithm of Engine.score_shared,
-        shared with it in decider.shared_prefix: it is per request, never per schema, so it adds no state that outlives
-        the request and no shape that depends on the question set.  The prefix and suffix forwards are eager
-        (request-specific shapes).  Correct only with the cuDNN SDPA backend off, which __init__ arranges."""
-        out = shared_prefix.score_shared(self, items, temperature, min_prefix, budget_bytes, rows_per_fork)
-        if out is None:
-            return self.score_items(items, temperature)
-        self.stats["shared_calls"] += 1
-        return out
-
-    # ---- start-up ----------------------------------------------------------
-    def warmup(self, shapes=None, log=None):
-        """Capture the whole grid.  Call seal() afterwards: from then on an unknown shape runs eager, never captures."""
-        t = time.time(); shapes = list(shapes if shapes is not None else self.graph_shapes())
-        for j, (B, T) in enumerate(shapes):
-            self.logits_all(torch.full((B, T), self.tok.pad_token_id, dtype=torch.long, device=self.dev))
-            if log and (j + 1) % 10 == 0:
-                log(f"[engine_v2] {j + 1}/{len(shapes)} graphs, {time.time() - t:.0f}s")
-        if self.use_graphs:
-            torch.cuda.synchronize()
-        return time.time() - t
-
-    def seal(self):
-        self.sealed = True
-        self.b_for_t = {}
-        for B, T in self.graphs:
-            self.b_for_t.setdefault(T, []).append(B)
-        for T in self.b_for_t:
-            self.b_for_t[T].sort()
-        return self
-
-    def describe(self):
-        return dict(self.cfg, graphs=len(self.graphs), sealed=self.sealed,
-                    grid={str(T): self.b_for_t.get(T, []) for T in self.t_buckets})

decider/fp8.py

@@ -1,54 +0,0 @@
-"""FP8 (e4m3) linear layers for Hopper via torch._scaled_mm.
-Weights: per-output-channel scales, quantised once.  Activations: per-token dynamic scales.
-Under torch.compile the quantisation ops fuse into the surrounding elementwise work."""
-import torch, torch.nn as nn
-
-E4M3_MAX = 448.0
-
-
-def _quant_rowwise(x):
-    s = x.abs().amax(dim=-1, keepdim=True).float().clamp(min=1e-12) / E4M3_MAX
-    return (x.float() / s).clamp(-E4M3_MAX, E4M3_MAX).to(torch.float8_e4m3fn), s
-
-
-class FP8Linear(nn.Module):
-    def __init__(self, lin: nn.Linear):
-        super().__init__()
-        wq, sw = _quant_rowwise(lin.weight.detach())          # [N,K] fp8, [N,1]
-        self.register_buffer("wq", wq.contiguous())               # [N,K]; passed as wq.t() -> [K,N] column-major, as _scaled_mm wants
-        self.register_buffer("sw_t", sw.t().contiguous())        # [1,N]
-        self.bias = None if lin.bias is None else nn.Parameter(lin.bias.detach().clone(), requires_grad=False)
-        self.in_features, self.out_features = lin.in_features, lin.out_features
-        self.out_dtype = lin.weight.dtype
-
-    def forward(self, x):
-        shp = x.shape[:-1]
-        x2 = x.reshape(-1, self.in_features)
-        xq, sx = _quant_rowwise(x2)
-        y = torch._scaled_mm(xq, self.wq.t(), scale_a=sx, scale_b=self.sw_t, bias=self.bias, out_dtype=self.out_dtype)
-        return y.reshape(*shp, self.out_features)
-
-
-def convert_to_fp8(model, skip=("lm_head",), min_dim=1024):
-    """Replace nn.Linear (with in/out >= min_dim) by FP8Linear in place. Returns count."""
-    n = 0
-    for name, mod in list(model.named_modules()):
-        for cname, child in list(mod.named_children()):
-            full = f"{name}.{cname}" if name else cname
-            if isinstance(child, nn.Linear) and not any(s in full for s in skip) and min(child.in_features, child.out_features) >= min_dim:
-                setattr(mod, cname, FP8Linear(child)); n += 1
-    return n
-
-
-if __name__ == "__main__":
-    import time
-    lin = nn.Linear(2048, 6144, bias=False).cuda().to(torch.bfloat16)
-    f8 = FP8Linear(lin)
-    x = torch.randn(8192, 2048, device="cuda", dtype=torch.bfloat16)
-    ref = lin(x); got = f8(x)
-    print("rel err", ((ref.float() - got.float()).abs().mean() / ref.float().abs().mean()).item())
-    for f, name in [(lin, "bf16 linear"), (f8, "fp8 linear (eager)"), (torch.compile(f8), "fp8 linear (compiled)")]:
-        for _ in range(3): f(x)
-        torch.cuda.synchronize(); t = time.time()
-        for _ in range(20): f(x)
-        torch.cuda.synchronize(); print(f"{name:24s} {(time.time()-t)/20*1000:.3f} ms")

decider/infer.py

@@ -1,357 +0,0 @@
-"""Usable inference API: typed decisions with probabilities, all from one forward pass.
-
-    from decider.infer import Decider
-    d = Decider("runs/r2_full/model")
-    out = d.decide("My card was charged twice for the same purchase.",
-                   [{"question": "Which department should handle this?", "options": ["billing", "technical", "sales"]},
-                    {"question": "How urgent is this?", "options": ["low", "medium", "high"]}])
-    # -> [{'choice': 'billing', 'confidence': 0.97, 'probs': {...}}, {...}]
-"""
-import logging
-
-import torch
-from decider.model import DecisionModel, collate
-from decider.prompt import build, MAX_OPTIONS, resolve_layout, chat_template
-from decider import temperature as TT
-from dataclasses import dataclass
-
-
-@dataclass
-class Q:
-    text: str; options: list; gold: int = 0
-
-
-@dataclass
-class Example:
-    context: str; qs: list; task: str = "infer"; image: bytes = None
-
-
-logger = logging.getLogger(__name__)
-NEUTRAL_NONE = "not listed here"
-
-
-def neutralize_options(options):
-    """The training augmentation used the literal 'none of the above', and the model learned that exact string as an
-    abstain signal (it abstains even on clear cases when the string is offered). Any option that reads like it is
-    rewritten to a neutral phrasing for the model and mapped back in the output."""
-    out, back = [], {}
-    for o in options:
-        key = o.strip().lower()
-        if key.startswith("none of the above") or key in ("none of the above", "none", "n/a", "none of these"):
-            out.append(NEUTRAL_NONE); back[NEUTRAL_NONE] = o
-        else:
-            out.append(o)
-    return out, back
-
-
-class _NoShuffle:                              # keep option order as given
-    def shuffle(self, x): pass
-    def sample(self, xs, k): return xs[:k]
-
-
-class CompiledSchema:
-    def __init__(self, d, rqs, h, index):
-        from decider.systemone import row_types
-        self.d, self.rqs, self.h, self.index = d, rqs, h, index
-        self.types = row_types(rqs, index)                         # answer type of every schema row (decider.temperature)
-
-    def batch(self, states, max_state_tokens=32768):
-        from decider.systemone import render_state, assemble
-        T = TT.for_types(self.d.T_schema, self.d.T_schema_by_type, self.types)
-        probs = self.d._se.score(self.h, [render_state(s) for s in states], temperature=T, max_ctx_tokens=max_state_tokens)
-        return [{"model": self.d.name, "answers": assemble(self.rqs, self.index, [p.tolist() for p in pr])} for pr in probs]
-
-    def __call__(self, state, max_state_tokens=32768):
-        return self.batch([state], max_state_tokens)[0]
-
-
-class Decider:
-    """One-pass decisions with automatic CUDA, MPS, or CPU device selection.
-
-    CUDA uses shape-bucketed graphs by default. MPS defaults to float16 and uses
-    the optional MPS patch; CPU defaults to bfloat16. Set ``use_graphs=False``
-    for eager execution or debugging.
-    """
-    def __init__(self, path, device=None, dtype=None, temperature=None, abstain_below=0.0, use_graphs=None, temperature_by_type=None):
-        """The prompt layout comes from decider_config.json: "layout": "chat" (chat-trained checkpoints) wraps every prompt in the
-        tokenizer's chat template (decider.prompt.build_chat); no "layout" key is the plain layout of every earlier model.
-        An unknown layout raises ValueError before the weights are loaded.
-
-        Temperatures (decider.temperature): decider_config.json "temperature" and the optional "temperature_by_type"
-        {"choice": T, "noul": T, "score": T}.  temperature= overrides "temperature" and switches the config's by-type map off;
-        temperature_by_type= sets the map explicitly.  An invalid temperature or map key raises ValueError before the weights
-        are loaded."""
-        if device is None:
-            device = "cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu")
-        if dtype is None:
-            dtype = torch.float16 if str(device).startswith("mps") else torch.bfloat16
-        logger.info("Decider device=%s dtype=%s", device, dtype)
-        import json, os
-        cfg = {}
-        try:                                          # model folder may carry decider_config.json (temperature, flags)
-            from huggingface_hub import hf_hub_download
-            cfg_path = os.path.join(path, "decider_config.json") if os.path.isdir(path) else hf_hub_download(path, "decider_config.json")
-            cfg = json.load(open(cfg_path))
-        except Exception:
-            pass
-        self.layout = resolve_layout(cfg)
-        (temperature, self.T_by_type), (T_schema, self.T_schema_by_type) = TT.from_config(cfg, temperature, temperature_by_type)
-        self.neutralize_none = bool(cfg.get("neutralize_none", True))   # v4 and earlier learned the literal string as an abstain signal
-        if use_graphs is None:
-            use_graphs = str(device).startswith("cuda")
-        if use_graphs:
-            from decider.engine import Engine
-            self.eng = Engine(path, device=device, dtype=dtype); self.m = self.eng.m
-        else:
-            if str(device).startswith("mps"):
-                from decider.mps_ops import patch_mps
-                patch_mps()
-            self.eng = None; self.m = DecisionModel(path, dtype=dtype, grad_ckpt=False).to(device).eval()
-        self.chat = chat_template(self.m.tok) if self.layout == "chat" else None      # None: plain layout, prompts as in 1.1.x
-        self.dev = device; self.T = temperature; self.abstain_below = abstain_below
-        self.name = "decider-" + str(cfg.get("version", "dev"))
-        self.schema_first = bool(cfg.get("schema_first", False)) and self.eng is not None      # default layout. Questions-first (the cacheable one) costs accuracy
-        self.T_schema = T_schema                                              # (about 1.5 points on fixed label sets, more elsewhere): opt in with schema()
-        self.isolated_levels = bool(cfg.get("isolated_levels", False))      # Score levels judged one per row (v8+)
-        self._se = None; self._schemas = {}
-
-    def _decide_items(self, requests, max_ctx_tokens=1536):
-        """The prompt rows of decide_batch: -> (requests with neutralized options, one item per request)."""
-        exs = []
-        if self.neutralize_none:
-            requests = [(context, [dict(q, options=neutralize_options(q["options"])[0], _back=neutralize_options(q["options"])[1]) for q in qs]) for context, qs in requests]
-        for context, qs in requests:
-            for q in qs:
-                assert 2 <= len(q["options"]) <= MAX_OPTIONS, f"2..{MAX_OPTIONS} options required"
-            exs.append(Example(context, [Q(q["question"], list(q["options"]), 0) for q in qs], "infer"))
-        items = [build(e, self.m.tok, _NoShuffle(), max_options=MAX_OPTIONS, max_ctx_tokens=max_ctx_tokens, chat=self.chat) for e in exs]
-        for it, (_, qs) in zip(items, requests):                     # answer type per slot: a plain question is "choice"
-            it["types"] = [q.get("_type", "choice") for q in qs]
-        return requests, items
-
-    @torch.no_grad()
-    def decide_batch(self, requests, max_ctx_tokens=1536):
-        """requests: list of (context:str, questions:list[dict(question, options)]). One forward pass for everything."""
-        requests, items = self._decide_items(requests, max_ctx_tokens)
-        T = TT.for_items(self.T, self.T_by_type, items)                  # the scalar self.T when there is no by-type map
-        if self.eng is not None:
-            probs = torch.cat(self.eng.score_items(items, temperature=T))
-        else:
-            b = collate(items, self.m.tok.pad_token_id)
-            logits = self.m.slot_logits(b["input_ids"].to(self.dev), b["attention_mask"].to(self.dev), b["slot_idx"].to(self.dev),
-                                        b["slot_batch"].to(self.dev), b["nopts"].to(self.dev))
-            probs = TT.scaled_softmax(logits, TT.slot_temperatures(T, items)).cpu()
-        out, k = [], 0
-        for context, qs in requests:
-            res = []
-            for q in qs:
-                p = probs[k, :len(q["options"])].tolist(); k += 1
-                j = max(range(len(p)), key=p.__getitem__); back = q.get("_back", {})
-                names = [back.get(o, o) for o in q["options"]]
-                res.append(dict(choice=names[j] if p[j] >= self.abstain_below else None, confidence=p[j],
-                                probs={o: pi for o, pi in zip(names, p)}, probs_list=p))
-            out.append(res)
-        return out
-
-    def decide(self, context, questions, **kw):
-        return self.decide_batch([(context, questions)], **kw)[0]
-
-    # ---- Jev-shaped interface (decider.systemone): state + {id: Choice | Score | Noul with criteria}
-    # ---- schema cache (v7+): the questions are run once, requests only run the state (decider.schema_engine)
-    def schema(self, questions, independent=True, isolated=None, compile=False):
-        """Compile a fixed set of Jev-shaped questions: schema(state) -> answers; schema.batch([state, ...]) -> [answers]."""
-        import json
-        from decider.schema_engine import SchemaEngine
-        from decider.systemone import render_question
-        isolated = self.isolated_levels if isolated is None else isolated
-        key = (json.dumps(questions, sort_keys=True, ensure_ascii=False), independent, isolated)
-        if key not in self._schemas:
-            if self._se is None: self._se = SchemaEngine(self.eng, chat=self.chat)
-            if len(self._schemas) >= 64:                                  # drop the oldest schema and its graphs
-                old = next(iter(self._schemas)); hid = self._schemas.pop(old)[1].id
-                for k in [k for k in self._se.graphs if k[0] == hid]: del self._se.graphs[k]
-            from decider.systemone import plan_rows
-            rqs = {k: render_question(v) for k, v in questions.items()}
-            rows, index = plan_rows(rqs, isolated and independent)
-            h = self._se.prepare(rows, independent=independent, compile=compile)      # compile=True: ~25 s per (batch, length) shape, 1.6x faster after
-            self._schemas[key] = (rqs, h, index)
-        return CompiledSchema(self, *self._schemas[key])
-
-    def system_one(self, state, questions, independent=True, max_state_tokens=32768, max_fwd_tokens=65536, layout=None, isolated=None):
-        layout = layout or ("schema_first" if self.schema_first else "state_first")
-        isolated = (self.isolated_levels if isolated is None else isolated) and independent
-        if layout == "schema_first" and self.eng is not None:
-            return self.schema(questions, independent, isolated)(state, max_state_tokens)
-        """independent=True scores every question in its own row (state + that question only), so adding, removing or
-        reordering questions cannot change any other answer; the state is run once and its cache forked to every
-        question (Engine.score_shared).  independent=False packs all questions behind one copy of the state in one row
-        (later questions can then see earlier question texts)."""
-        from decider.systemone import unique_tokens, assemble
-        rqs, index, items = self._system_one_items(state, questions, independent, max_state_tokens, layout, isolated)
-        flatp = self._system_one_probs(items, layout, max_fwd_tokens, TT.for_items(self.T, self.T_by_type, items))
-        return {"model": self.name, "answers": assemble(rqs, index, flatp),
-                "usage": {"input_tokens": unique_tokens(items), "output_tokens": 0}}
-
-    @torch.no_grad()
-    def _system_one_probs(self, items, layout, max_fwd_tokens=65536, temperature=None):
-        """Score system_one's uncached rows.  temperature: as Engine.score_items takes it (a number, or one entry per item).
-        -> one probability list per question row, in row order."""
-        if self.eng is not None and len(items) > 1 and layout == "state_first":
-            probs = self.eng.score_shared(items, temperature=temperature)
-        else:
-            probs = []; per = max(1, max_fwd_tokens // max(len(it["ids"]) for it in items))
-            for i in range(0, len(items), per):
-                T = TT.item_slice(temperature, i, i + per)
-                if self.eng is not None:
-                    probs += self.eng.score_items(items[i:i + per], temperature=T)
-                else:
-                    bt = collate(items[i:i + per], self.m.tok.pad_token_id)
-                    lg = self.m.slot_logits(*[bt[k].to(self.dev) for k in ("input_ids", "attention_mask", "slot_idx", "slot_batch", "nopts")])
-                    pr = TT.scaled_softmax(lg, TT.slot_temperatures(T, items[i:i + per])).cpu(); c = 0
-                    for it in items[i:i + per]:
-                        probs.append(pr[c:c + len(it["slots"])]); c += len(it["slots"])
-        return [p.tolist() for ps in probs for p in ps]
-
-    def _system_one_items(self, state, questions, independent=True, max_state_tokens=32768, layout=None, isolated=None):
-        """The prompt rows of system_one's uncached path: -> (rendered questions, answer index, items)."""
-        from decider.systemone import render_state, render_question, plan_rows, row_types
-        layout = layout or ("schema_first" if self.schema_first else "state_first")
-        isolated = (self.isolated_levels if isolated is None else isolated) and independent
-        ctx = render_state(state); rqs = {k: render_question(v) for k, v in questions.items()}
-        opts = (lambda r: neutralize_options(r["options"])[0]) if self.neutralize_none else (lambda r: list(r["options"]))
-        flat, index = plan_rows(rqs, isolated)
-        rows = [[r] for r in flat] if independent else [flat]
-        items = [build(Example(ctx, [Q(r["question"], opts(r), 0) for r in row]), self.m.tok, _NoShuffle(), max_options=MAX_OPTIONS,
-                       max_ctx_tokens=max_state_tokens, layout=layout, chat=self.chat) for row in rows]
-        types = row_types(rqs, index)                                 # answer type per row (decider.temperature)
-        for it, ts in zip(items, [[t] for t in types] if independent else [types]):
-            it["types"] = ts
-        return rqs, index, items
-
-    # ---- typed schema interface: {question: {"type": "bool"} | {"type": "choice", "options": [...]}
-    #                                         | {"type": "scale", "legend": {"0": "none", "1": "low", ...}}}
-    SCHEMA_FORM = ('schema: a map {question: field}, each field one of {"type": "choice", "options": [option strings]}, '
-                   '{"type": "bool"} or {"type": "scale", "legend": [descriptions] or {level number: description}}')
-
-    @staticmethod
-    def _check_schema(schema):
-        """Raise ValueError naming the expected form when `schema` does not have it (the HTTP server turns this into a 422).
-        Every schema that 1.1.2 answered stays accepted, except options or a legend given as a bare string (1.1.2 split it into
-        characters); an empty schema is answered with {}."""
-        import json, math
-        form = Decider.SCHEMA_FORM
-        if schema is None:
-            raise ValueError("schema is required; " + form)
-        if not isinstance(schema, dict):
-            raise ValueError(f"schema is a {type(schema).__name__}; " + form)
-        for qtext, spec in schema.items():
-            where = f"schema[{qtext!r}]"
-            if not isinstance(spec, dict):
-                raise ValueError(f"{where} is a {type(spec).__name__}, not a field object; " + form)
-            t = spec.get("type", "choice")
-            if t == "choice":
-                opts = spec.get("options")
-                if not isinstance(opts, (list, tuple, dict)) or not opts or not all(isinstance(o, str) for o in opts):
-                    raise ValueError(f'{where}: "options" must be a non-empty list of strings; ' + form)
-            elif t == "scale":
-                leg = spec.get("legend")
-                if not isinstance(leg, (list, tuple, dict)) or not leg:
-                    raise ValueError(f'{where}: "legend" must be a non-empty list or {{level number: description}} map; ' + form)
-                try:                                           # the legend is echoed in the answer, which must serialise
-                    json.dumps(leg, allow_nan=False)
-                except (TypeError, ValueError):
-                    raise ValueError(f'{where}: the "legend" contains a value that is not finite JSON (NaN or Infinity); ' + form)
-                if isinstance(leg, dict):
-                    try:
-                        ok = all(math.isfinite(float(k)) for k in leg)
-                    except (TypeError, ValueError, OverflowError):
-                        ok = False
-                    if not ok:
-                        raise ValueError(f'{where}: the keys of a "legend" map must be finite numbers, e.g. {{"0": "none", "1": "low"}}; ' + form)
-            elif t != "bool":
-                raise ValueError(f"{where}: unknown field type {t!r}; " + form)
-
-    @staticmethod
-    def _schema_to_questions(schema):
-        Decider._check_schema(schema)
-        qs = []
-        for qtext, spec in schema.items():
-            t = spec.get("type", "choice")
-            if t == "bool":
-                qs.append(dict(question=qtext, options=["no", "yes"], _type="noul"))
-            elif t == "choice":
-                qs.append(dict(question=qtext, options=list(spec["options"]), _type="choice"))
-            elif t == "scale":
-                leg = spec["legend"]
-                keys = sorted(leg, key=lambda k: float(k)) if isinstance(leg, dict) else list(range(len(leg)))
-                labels = [f"{k}: {leg[k]}" if isinstance(leg, dict) else f"{i}: {leg[i]}" for i, k in enumerate(keys)]
-                qs.append(dict(question=qtext, options=labels, _keys=keys, _legend=leg, _type="score"))
-            else:
-                raise ValueError(f"unknown field type {t}")
-        return qs
-
-    def decide_json_batch(self, requests, **kw):
-        """requests: list of (context, schema). Returns one dict per context keyed by question."""
-        qss = [self._schema_to_questions(schema) for _, schema in requests]
-        raw = self.decide_batch([(ctx, qs) for (ctx, _), qs in zip(requests, qss)], **kw)
-        out = []
-        for (ctx, schema), qs, res in zip(requests, qss, raw):
-            o = {}
-            for (qtext, spec), q, r in zip(schema.items(), qs, res):
-                t = spec.get("type", "choice")
-                if t == "bool":
-                    o[qtext] = {"noul": round(r["probs"]["yes"], 4), "type": "noul"}
-                elif t == "choice":
-                    o[qtext] = {"choice": r["choice"], "confidence": round(r["confidence"], 4), "type": "choice",
-                                "probabilities": {k: round(v, 4) for k, v in r["probs"].items()}}
-                else:
-                    p = [r["probs"][lab] for lab in q["options"]]
-                    keys = q["_keys"]; n = len(p)
-                    score = sum(float(k) * pi for k, pi in zip(keys, p))          # expected level on the legend scale
-                    j = max(range(n), key=p.__getitem__)
-                    o[qtext] = {"score": round(score, 2), "confidence": round(p[j], 4), "type": "scale", "legend": q["_legend"],
-                                "probabilities": {str(keys[i]): round(pi, 4) for i, pi in enumerate(p)}}
-            out.append(o)
-        return out
-
-    def decide_json(self, context, schema, **kw):
-        return self.decide_json_batch([(context, schema)], **kw)[0]
-
-
-if __name__ == "__main__":
-    import sys, json, time
-    d = Decider(sys.argv[1] if len(sys.argv) > 1 else "runs/r1_200k/model")
-    demo = [
-        ("My card was charged twice for the same purchase and I want the extra charge refunded.",
-         [{"question": "Which department should handle this?", "options": ["billing", "technical support", "sales"]},
-          {"question": "What is the customer's sentiment?", "options": ["angry", "neutral", "happy"]},
-          {"question": "Does this need a refund action?", "options": ["no", "yes"]}]),
-        ("hey can u turn the lights off in the kitchen",
-         [{"question": "What is the intent?", "options": ["smart home control", "set alarm", "play music", "none of the above"]},
-          {"question": "Is this request toxic?", "options": ["no", "yes"]}]),
-        ("The quarterly report shows revenue fell 12% while costs rose sharply.",
-         [{"question": "What is the financial sentiment?", "options": ["bearish", "neutral", "bullish"]}]),
-    ]
-    t = time.time(); res = d.decide_batch(demo); dt = time.time() - t
-    for (ctx, qs), r in zip(demo, res):
-        print("\n>>", ctx)
-        for q, a in zip(qs, r):
-            print(f"   {q['question']:45s} -> {a['choice']!s:22s} p={a['confidence']:.2f}  " + " ".join(f"{o}:{p:.2f}" for o, p in a['probs'].items()))
-    print(f"\n{sum(len(q) for _, q in demo)} decisions in {dt*1000:.0f} ms (one forward pass)")
-    schema = {
-        "Revenue currently impacted?": {"type": "bool"},
-        "What business impact?": {"type": "choice", "options": ["none", "degraded", "outage"]},
-        "Integration issue present?": {"type": "bool"},
-        "Account health status?": {"type": "choice", "options": ["healthy", "watch", "at risk"]},
-        "Which incident scope?": {"type": "choice", "options": ["single_account", "multi_account", "platform_wide"]},
-        "Security concern present?": {"type": "bool"},
-        "Duplicate charge reported?": {"type": "bool"},
-        "Churn likelihood level?": {"type": "scale", "legend": {"0": "none", "1": "low", "2": "medium", "3": "high"}},
-        "Human attention needed?": {"type": "bool"},
-        "Immediate feature request?": {"type": "bool"},
-    }
-    ctx = ("Hi, since this morning our Stripe webhook integration stopped firing and our checkout is down for all customers. "
-           "We are losing orders every minute and our partner launch is on Thursday. Also I think we got billed twice last week. "
-           "If this is not fixed today we will have to look at other providers.")
-    t = time.time(); js = d.decide_json(ctx, schema); dt = time.time() - t
-    print(f"\n>> {ctx[:80]}...\n" + json.dumps(js, indent=1)[:3000]); print(f"{len(schema)} typed fields in {dt*1000:.0f} ms (one forward pass)")

decider/metrics.py

@@ -1,37 +0,0 @@
-import numpy as np
-
-
-def ece(conf, correct, bins=15):
-    conf = np.asarray(conf); correct = np.asarray(correct, dtype=float)
-    edges = np.linspace(0, 1, bins + 1); e = 0.0
-    for lo, hi in zip(edges[:-1], edges[1:]):
-        m = (conf > lo) & (conf <= hi)
-        if m.any():
-            e += m.mean() * abs(conf[m].mean() - correct[m].mean())
-    return float(e)
-
-
-def aurc(conf, correct):
-    """Area under risk-coverage curve (lower is better)."""
-    order = np.argsort(-np.asarray(conf)); c = np.asarray(correct, dtype=float)[order]
-    risk = np.cumsum(1 - c) / np.arange(1, len(c) + 1)
-    return float(risk.mean())
-
-
-def sel_acc(conf, correct, coverage):
-    order = np.argsort(-np.asarray(conf)); c = np.asarray(correct, dtype=float)[order]
-    n = max(1, int(round(coverage * len(c))))
-    return float(c[:n].mean())
-
-
-def summarize(probs, golds, nopts):
-    """probs [N,K] (masked entries 0), golds [N], nopts [N]."""
-    probs = np.asarray(probs); golds = np.asarray(golds); nopts = np.asarray(nopts)
-    pred = probs.argmax(1); conf = probs.max(1); correct = (pred == golds)
-    p_gold = probs[np.arange(len(golds)), golds]
-    nll = -np.log(np.clip(p_gold, 1e-12, 1)).mean()
-    onehot = np.zeros_like(probs); onehot[np.arange(len(golds)), golds] = 1
-    brier = ((probs - onehot) ** 2).sum(1).mean()
-    return dict(n=int(len(golds)), acc=float(correct.mean()), nll=float(nll), brier=float(brier), ece=ece(conf, correct),
-                aurc=aurc(conf, correct), acc_at_80=sel_acc(conf, correct, 0.8), acc_at_50=sel_acc(conf, correct, 0.5),
-                chance=float((1.0 / nopts).mean()), mean_conf=float(conf.mean()))

decider/model.py

@@ -1,49 +0,0 @@
-"""Backbone -> slot hidden states -> restricted logits over option letters."""
-import torch, torch.nn as nn, torch.nn.functional as F
-from transformers import AutoModelForCausalLM, AutoTokenizer
-from decider.prompt import letter_ids, MAX_OPTIONS
-
-
-class DecisionModel(nn.Module):
-    def __init__(self, name, dtype=torch.bfloat16, grad_ckpt=True):
-        super().__init__()
-        if hasattr(torch.backends.cuda, "enable_cudnn_sdp"):   # as decider.engine.set_attention_backend_policy: the cuDNN SDPA
-            torch.backends.cuda.enable_cudnn_sdp(False)         # backend is wrong for masked attention on Blackwell (torch 2.14)
-        self.tok = AutoTokenizer.from_pretrained(name)
-        self.lm = AutoModelForCausalLM.from_pretrained(name, dtype=dtype)
-        if grad_ckpt:
-            self.lm.gradient_checkpointing_enable()
-        self.register_buffer("letters", torch.tensor(letter_ids(self.tok)), persistent=False)
-
-    def slot_logits(self, input_ids, attention_mask, slot_idx, slot_batch, nopts):
-        """input_ids [B,T]; slot_idx/slot_batch [N] flat slot positions; nopts [N].
-        Returns [N, MAX_OPTIONS] logits with invalid options masked to -inf."""
-        h = self.lm.model(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state
-        hs = h[slot_batch, slot_idx]                                   # [N,H]
-        W = self.lm.lm_head.weight[self.letters]                       # [K,H]
-        logits = F.linear(hs, W).float()                               # [N,K]
-        ar = torch.arange(MAX_OPTIONS, device=logits.device)[None, :]
-        logits = logits.masked_fill(ar >= nopts[:, None], float("-inf"))
-        return logits
-
-    def forward(self, batch):
-        return self.slot_logits(batch["input_ids"], batch["attention_mask"], batch["slot_idx"], batch["slot_batch"], batch["nopts"])
-
-
-def collate(items, pad_id):
-    """items: list of dicts from prompt.build (+ 'task', 'ex_id'). Right-pad."""
-    T = max(len(it["ids"]) for it in items)
-    T = ((T + 63) // 64) * 64          # few distinct shapes -> fewer kernel (re)compiles
-    B = len(items)
-    input_ids = torch.full((B, T), pad_id, dtype=torch.long)
-    attn = torch.zeros((B, T), dtype=torch.long)
-    slot_idx, slot_batch, golds, nopts, tasks, qidx = [], [], [], [], [], []
-    for b, it in enumerate(items):
-        n = len(it["ids"])
-        input_ids[b, :n] = torch.tensor(it["ids"])
-        attn[b, :n] = 1
-        for k, s in enumerate(it["slots"]):
-            slot_idx.append(s); slot_batch.append(b); golds.append(it["golds"][k]); nopts.append(it["nopts"][k])
-            tasks.append(it.get("task", "")); qidx.append(k)
-    return dict(input_ids=input_ids, attention_mask=attn, slot_idx=torch.tensor(slot_idx), slot_batch=torch.tensor(slot_batch),
-                golds=torch.tensor(golds), nopts=torch.tensor(nopts), tasks=tasks, qidx=qidx)

decider/mps_moe.py

@@ -1,102 +0,0 @@
-"""Two MPS slow paths in transformers' Qwen3.5-MoE reference code, replaced in-process (decider-35b-a3b on Apple Silicon).
-
-Contributed by @nassersala in issue #6 under the project's Apache-2.0 licence; adapted for the package (out-of-range expert
-ids are dropped as `histc` drops them, and the replacements are installed from `decider.mps_ops.patch_mps`).
-
-On an M-series Mac (torch 2.14, transformers 5.17) `torch.histc` takes about 45 ms a call on MPS (once per MoE layer, to
-count tokens per expert) and `torch.linalg.solve_triangular` about 17 ms (twice per linear-attention layer): most of a
-2.8-4 s decision. With both replaced the 35B takes 0.23-0.5 s a decision for typical inputs (reported in issue #6).
-The `histc` replacement is an exact count. The block inverse is within 1e-6 of the MPS solver on real systems; on six items
-the probabilities moved at most 0.033, less than switching to the exact CPU solver does (0.042), and no argmax changed.
-
-The replacements are visible only inside those two transformers modules (install() gives each its own view of `torch`), and
-there they act only on MPS tensors of the matching call shape; everything else goes to torch.
-"""
-import torch
-
-_histc, _solve = torch.histc, torch.linalg.solve_triangular
-
-
-def count_ids(x, bins, min=0):
-    """histc for integer-valued ids binned one per bin: a count of ids min..min+bins-1; other values are dropped as histc
-    drops them (transformers passes expert-parallel sentinels >= num_experts and relies on that)."""
-    idx = x.long() - int(min)
-    keep = (idx >= 0) & (idx < bins)
-    out = torch.zeros(bins, device=x.device, dtype=x.dtype)
-    return out.scatter_add_(0, idx.clamp(0, bins - 1), keep.to(x.dtype))
-
-
-def histc(input, bins=100, min=0, max=0, *, out=None):
-    # expert ids 0..n-1 into n bins over [0, n-1]: bin k holds id k exactly, so this is a count (histc: ~45 ms on MPS)
-    if input.device.type == "mps" and input.dim() == 1 and out is None and min == 0 and max > 0 and bins == max - min + 1:
-        return count_ids(input, bins, min)
-    return _histc(input, bins=bins, min=min, max=max) if out is None else _histc(input, bins=bins, min=min, max=max, out=out)
-
-
-def unit_lower_inverse(A):
-    """Inverse of unit lower-triangular A (last dim a power of two) by block doubling:
-    [[A11, 0], [A21, A22]]^-1 = [[X11, 0], [-X22 A21 X11, X22]]."""
-    n = A.shape[-1]
-    inv = torch.ones(*A.shape[:-2], n, 1, 1, device=A.device, dtype=A.dtype)  # n diagonal 1x1 blocks
-    b = 1
-    while b < n:
-        nb = n // (2 * b)
-        blocks = A.reshape(*A.shape[:-2], nb, 2 * b, nb, 2 * b).diagonal(dim1=-4, dim2=-2).movedim(-1, -3)
-        x11, x22, a21 = inv[..., 0::2, :, :], inv[..., 1::2, :, :], blocks[..., b:, :b]
-        new = torch.zeros(*A.shape[:-2], nb, 2 * b, 2 * b, device=A.device, dtype=A.dtype)
-        new[..., :b, :b], new[..., b:, b:], new[..., b:, :b] = x11, x22, -(x22 @ a21 @ x11)
-        inv, b = new, 2 * b
-    return inv[..., 0, :, :]
-
-
-def solve_unit_lower(A, B):
-    """solve_triangular(A, B, upper=False, unitriangular=True): only the strict lower triangle of A is read."""
-    n = A.shape[-1]
-    return unit_lower_inverse(A.tril(-1) + torch.eye(n, device=A.device, dtype=A.dtype)) @ B
-
-
-def solve_triangular(A, B, *, upper, left=True, unitriangular=False, out=None):
-    # unit lower-triangular: invert by block doubling, all batched matmuls (the MPS solver: ~17 ms per call)
-    n = A.shape[-1]
-    if A.device.type == "mps" and not upper and left and unitriangular and out is None and n > 0 and n & (n - 1) == 0:
-        return solve_unit_lower(A, B)
-    return _solve(A, B, upper=upper, left=left, unitriangular=unitriangular, out=out)
-
-
-class _Proxy:
-    """A module's view of `torch` (or `torch.linalg`) with some attributes replaced; everything else is the real one."""
-    def __init__(self, real, **over):
-        self._real, self._over = real, over
-
-    def __getattr__(self, name):
-        return self._over[name] if name in self._over else getattr(self._real, name)
-
-
-_installed = {}
-
-
-def install():
-    """Replace the two operations inside the two transformers modules that call them, and nowhere else: the grouped-MoE
-    expert count in `transformers.integrations.moe` and the gated delta rule in `modeling_qwen3_5_moe` see a `torch` whose
-    `histc` / `linalg.solve_triangular` are the replacements; every other caller keeps torch's.  -> names patched."""
-    import importlib
-    targets = {"transformers.integrations.moe": dict(histc=histc),
-               "transformers.models.qwen3_5_moe.modeling_qwen3_5_moe": dict(linalg=_Proxy(torch.linalg, solve_triangular=solve_triangular))}
-    for name, over in targets.items():
-        if name in _installed:
-            continue
-        try:
-            mod = importlib.import_module(name)
-        except ImportError:
-            continue
-        if getattr(mod, "torch", None) is not torch:
-            continue                                  # not the layout this was written for: leave it alone
-        _installed[name] = mod
-        mod.torch = _Proxy(torch, **over)
-    return sorted(_installed)
-
-
-def uninstall():
-    for name, mod in list(_installed.items()):
-        mod.torch = torch
-        del _installed[name]

decider/mps_ops.py

@@ -1,268 +0,0 @@
-"""MPS kernels for Qwen3.5 gated-delta attention.
-
-The optimization is measured against Transformers' PyTorch reference fallback for
-this kernel; it is not an end-to-end model speedup claim.
-
-The gated-delta and L2-normalization code follows Transformers 5.17's
-``modeling_qwen3_5.py`` under the Apache License 2.0. The MPS inversion and
-backend dispatch are Decider additions.
-"""
-import functools
-import inspect
-import logging
-import warnings
-
-import torch, torch.nn.functional as F
-from decider.engine import fused_causal_conv1d_fn
-
-
-# Adapted from transformers 5.17.0's modeling_qwen3_5.py (Apache-2.0).
-# The original license applies to the gated-delta and normalization portions;
-# the MPS inversion and dispatch below are Decider additions.
-# Native Metal Shading Language (MSL) JIT kernel via MLX/Metal
-_metal_invert_kernel = None
-_metal_failure_reported = False
-_compat_warning_reported = False
-try:
-    import mlx.core as mx
-    import mlx.core.fast as fast
-
-    _METAL_INVERT_SRC = """
-    uint col = thread_position_in_threadgroup.x;
-    uint mat_idx = threadgroup_position_in_grid.x;
-    threadgroup float s_inv[4096];
-
-    for (uint r = 0; r < 64; ++r) {
-        s_inv[r * 64 + col] = (r == col) ? 1.0f : 0.0f;
-    }
-    threadgroup_barrier(mem_flags::mem_threadgroup);
-
-    uint mat_offset = mat_idx * 4096;
-    for (uint r = 1; r < 64; ++r) {
-        float sum = 0.0f;
-        if (col < r) {
-            for (uint k = col; k < r; ++k) {
-                sum += L[mat_offset + r * 64 + k] * s_inv[k * 64 + col];
-            }
-        }
-        threadgroup_barrier(mem_flags::mem_threadgroup);
-        if (col < r) {
-            s_inv[r * 64 + col] = -sum;
-        }
-        threadgroup_barrier(mem_flags::mem_threadgroup);
-    }
-    for (uint r = 0; r < 64; ++r) {
-        out_inv[mat_offset + r * 64 + col] = s_inv[r * 64 + col];
-    }
-    """
-    _metal_invert_kernel = fast.metal_kernel(
-        name="invert_unitriangular_64",
-        input_names=["L"],
-        output_names=["out_inv"],
-        source=_METAL_INVERT_SRC,
-    )
-except (ImportError, OSError, RuntimeError, AttributeError):
-    logging.getLogger(__name__).debug("MLX Metal kernel unavailable; using PyTorch MPS fallback", exc_info=True)
-    _metal_invert_kernel = None
-
-
-def l2norm(x: torch.Tensor, dim: int = -1, eps: float = 1e-6) -> torch.Tensor:
-    return x * torch.rsqrt((x * x).sum(dim=dim, keepdim=True) + eps)
-
-
-def fast_invert_unitriangular_64(L: torch.Tensor) -> torch.Tensor:
-    """Invert batches of 64x64 unit lower-triangular matrices."""
-    global _metal_invert_kernel, _metal_failure_reported
-    if _metal_invert_kernel is not None and L.is_mps and not L.requires_grad:
-        assert L.dtype == torch.float32, "Metal inversion requires float32 input"
-        try:
-            orig_shape = L.shape
-            L_flat = L.reshape(-1, 64, 64).contiguous()
-            num_m = L_flat.shape[0]
-            torch.mps.synchronize()
-            L_mx = mx.from_dlpack(torch.to_dlpack(L_flat))
-            mx.eval(L_mx)
-            out_mx = _metal_invert_kernel(
-                inputs=[L_mx],
-                grid=(num_m * 64, 1, 1),
-                threadgroup=(64, 1, 1),
-                output_shapes=[(num_m, 64, 64)],
-                output_dtypes=[mx.float32],
-            )[0]
-            mx.eval(out_mx)
-            torch.mps.synchronize()
-            return torch.from_dlpack(out_mx).to(L.device).reshape(orig_shape)
-        except Exception:
-            # Optional acceleration must not turn a recoverable backend issue into
-            # a failed request; disable the failing kernel for this process.
-            _metal_invert_kernel = None
-            if not _metal_failure_reported:
-                logging.getLogger(__name__).warning("MLX Metal inversion failed; using the PyTorch MPS fallback", exc_info=True)
-                _metal_failure_reported = True
-
-    N = 64
-    inv = torch.eye(N, device=L.device, dtype=L.dtype).expand_as(L).clone()
-
-    # b = 1: 32 blocks of 2x2. For [[1, 0], [l, 1]], inverse is [[1, 0], [-l, 1]]
-    idx_row = torch.arange(1, 64, 2, device=L.device)
-    idx_col = torch.arange(0, 64, 2, device=L.device)
-    inv[..., idx_row, idx_col] = -L[..., idx_row, idx_col]
-
-    # b = 2: 16 blocks of 4x4
-    for j in range(16):
-        r1, r2, r3 = j * 4, j * 4 + 2, j * 4 + 4
-        inv[..., r2:r3, r1:r2] = -inv[..., r2:r3, r2:r3] @ L[..., r2:r3, r1:r2] @ inv[..., r1:r2, r1:r2]
-
-    # b = 4: 8 blocks of 8x8
-    for j in range(8):
-        r1, r2, r3 = j * 8, j * 8 + 4, j * 8 + 8
-        inv[..., r2:r3, r1:r2] = -inv[..., r2:r3, r2:r3] @ L[..., r2:r3, r1:r2] @ inv[..., r1:r2, r1:r2]
-
-    # b = 8: 4 blocks of 16x16
-    for j in range(4):
-        r1, r2, r3 = j * 16, j * 16 + 8, j * 16 + 16
-        inv[..., r2:r3, r1:r2] = -inv[..., r2:r3, r2:r3] @ L[..., r2:r3, r1:r2] @ inv[..., r1:r2, r1:r2]
-
-    # b = 16: 2 blocks of 32x32
-    for j in range(2):
-        r1, r2, r3 = j * 32, j * 32 + 16, j * 32 + 32
-        inv[..., r2:r3, r1:r2] = -inv[..., r2:r3, r2:r3] @ L[..., r2:r3, r1:r2] @ inv[..., r1:r2, r1:r2]
-
-    # b = 32: 1 block of 64x64
-    inv[..., 32:64, 0:32] = -inv[..., 32:64, 32:64] @ L[..., 32:64, 0:32] @ inv[..., 0:32, 0:32]
-    return inv
-
-
-def mps_chunk_gated_delta_rule(
-    query: torch.Tensor,
-    key: torch.Tensor,
-    value: torch.Tensor,
-    g: torch.Tensor,
-    beta: torch.Tensor,
-    chunk_size: int = 64,
-    initial_state: torch.Tensor | None = None,
-    output_final_state: bool = False,
-    use_qk_l2norm_in_kernel: bool = False,
-    **kwargs,
-) -> tuple[torch.Tensor, torch.Tensor | None]:
-    """Optimized chunk-gated delta rule for Apple Silicon (MPS)."""
-    assert chunk_size == 64, f"MPS optimized delta rule requires chunk_size=64, got {chunk_size}"
-    initial_dtype = query.dtype
-    batch_size, sequence_length, _, k_head_dim = key.shape
-    num_v_heads, v_head_dim = value.shape[-2:]
-    recurrent_state_shape = (batch_size, num_v_heads, k_head_dim, v_head_dim)
-    padded_output_shape = (batch_size, num_v_heads, -1, v_head_dim)
-    decay = g
-
-    query, key, value, beta, decay = [
-        x.transpose(1, 2).to(torch.float32, memory_format=torch.contiguous_format)
-        for x in (query, key, value, beta, decay)
-    ]
-    if use_qk_l2norm_in_kernel:
-        query = l2norm(query, dim=-1, eps=1e-6)
-        key = l2norm(key, dim=-1, eps=1e-6)
-    scaling = query.shape[-1] ** -0.5
-    query = query * scaling
-
-    pad_size = (chunk_size - sequence_length % chunk_size) % chunk_size
-    query = F.pad(query, (0, 0, 0, pad_size))
-    key = F.pad(key, (0, 0, 0, pad_size))
-    value = F.pad(value, (0, 0, 0, pad_size))
-    beta = F.pad(beta, (0, pad_size))
-    decay = F.pad(decay, (0, pad_size))
-
-    v_beta = value * beta.unsqueeze(-1)
-    k_beta = key * beta.unsqueeze(-1)
-
-    query, key, k_beta, v_beta = [
-        x.reshape(x.shape[0], x.shape[1], -1, chunk_size, x.shape[-1])
-        for x in (query, key, k_beta, v_beta)
-    ]
-    decay = decay.reshape(decay.shape[0], decay.shape[1], -1, chunk_size)
-
-    strictly_upper_mask = torch.ones(chunk_size, chunk_size, dtype=torch.bool, device=query.device).triu(1)
-    cum_decay = decay.cumsum(dim=3)
-    pairwise_decay = (cum_decay.unsqueeze(4) - cum_decay.unsqueeze(3)).masked_fill(strictly_upper_mask, float("-inf")).exp()
-
-    ut_system = (k_beta @ key.transpose(-1, -2)) * pairwise_decay
-    intra_chunk_attn = (query @ key.transpose(-1, -2)) * pairwise_decay
-    decayed_k_beta = k_beta * cum_decay.exp().unsqueeze(-1)
-
-    # Fast block divide-and-conquer unitriangular inverse on MPS
-    L = ut_system.tril(-1)
-    inv = fast_invert_unitriangular_64(L)
-    new_values = inv @ v_beta
-    k_cumdecay = inv @ decayed_k_beta
-
-    if initial_state is None:
-        last_recurrent_state = torch.zeros(recurrent_state_shape, dtype=new_values.dtype, device=new_values.device)
-    else:
-        last_recurrent_state = initial_state.to(new_values)
-    core_attn_out = torch.zeros_like(new_values)
-
-    query = query * cum_decay.exp().unsqueeze(-1)
-    key = key * (cum_decay[..., -1:] - cum_decay).exp().unsqueeze(-1)
-    chunk_decay = cum_decay[..., -1].exp()[..., None, None]
-
-    num_chunks = query.shape[2]
-    qk = torch.cat([query, k_cumdecay], dim=3)
-    kt = key.transpose(-1, -2)
-
-    for i in range(num_chunks):
-        qk_state = qk[:, :, i] @ last_recurrent_state
-        inter_chunk_attn = qk_state[:, :, :chunk_size]
-        v_new = new_values[:, :, i] - qk_state[:, :, chunk_size:]
-        core_attn_out[:, :, i] = inter_chunk_attn + intra_chunk_attn[:, :, i] @ v_new
-        last_recurrent_state = last_recurrent_state * chunk_decay[:, :, i] + kt[:, :, i] @ v_new
-
-    last_recurrent_state = None if not output_final_state else last_recurrent_state
-    core_attn_out = core_attn_out.reshape(padded_output_shape)[:, :, :sequence_length]
-    core_attn_out = core_attn_out.transpose(1, 2).to(initial_dtype, memory_format=torch.contiguous_format)
-    return core_attn_out, last_recurrent_state
-
-
-def patch_mps():
-    """Patch Qwen3.5 operations on MPS tensors while preserving other backends."""
-    global _compat_warning_reported
-    if not torch.backends.mps.is_available():
-        return False
-    import os
-    if os.environ.get("DECIDER_MPS_MOE_PATCH", "1") != "0":     # histc + unit-triangular solve replacements (issue #6, @nassersala)
-        from decider import mps_moe
-        mps_moe.install()
-    try:
-        import transformers
-        import transformers.models.qwen3_5.modeling_qwen3_5 as mq
-        version = tuple(int(part) for part in transformers.__version__.split(".")[:2])
-        if not all(callable(getattr(mq, name, None)) for name in ("torch_chunk_gated_delta_rule", "causal_conv1d_fn")):
-            raise AttributeError("unsupported Transformers Qwen3.5 implementation")
-        params = inspect.signature(mq.torch_chunk_gated_delta_rule).parameters
-        required = {"query", "key", "value", "g", "beta", "chunk_size"}
-        if version < (5, 17) or not required.issubset(params):
-            if not _compat_warning_reported:
-                warnings.warn(f"MPS patch requires Transformers >=5.17 with the Qwen3.5 gated-delta signature; got {transformers.__version__}", RuntimeWarning, stacklevel=2)
-                _compat_warning_reported = True
-            return False
-        if getattr(mq.torch_chunk_gated_delta_rule, "_decider_mps_patch", False):
-            return True
-
-        original_delta, original_conv = mq.torch_chunk_gated_delta_rule, mq.causal_conv1d_fn
-        delta_params = tuple(inspect.signature(original_delta).parameters)
-        query_index, chunk_index = delta_params.index("query"), delta_params.index("chunk_size")
-
-        @functools.wraps(original_delta)
-        def delta(*args, **kwargs):
-            query = args[query_index] if len(args) > query_index else kwargs["query"]
-            chunk_size = args[chunk_index] if len(args) > chunk_index else kwargs.get("chunk_size", 64)
-            return mps_chunk_gated_delta_rule(*args, **kwargs) if query.is_mps and chunk_size == 64 else original_delta(*args, **kwargs)
-
-        @functools.wraps(original_conv)
-        def conv(hidden_states, *args, **kwargs):
-            return fused_causal_conv1d_fn(hidden_states, *args, **kwargs) if hidden_states.is_mps else original_conv(hidden_states, *args, **kwargs)
-
-        delta._decider_mps_patch = True
-        mq.torch_chunk_gated_delta_rule, mq.causal_conv1d_fn = delta, conv
-        return True
-    except (ImportError, AttributeError, TypeError, ValueError):
-        logging.getLogger(__name__).debug("MPS patch unavailable; using Transformers fallback", exc_info=True)
-        return False

decider/prompt.py

@@ -1,287 +0,0 @@
-"""Prompt construction.  One context, N typed questions, N answer slots.
-
-All N decisions are read from a single forward pass: the logits at each
-"Answer k: (" slot are restricted to the option-letter tokens.  No answer
-letters are ever inserted, so slot k sees the context and all questions but
-no earlier answers (the decisions are conditionally independent given input).
-"""
-import random
-
-LETTERS = "ABCDEFGHIJ"
-NARROW = len(LETTERS)      # <= NARROW options: the original "(A) .. (J)" rendering, tokenized as a string (unchanged since v1)
-MAX_OPTIONS = 255          # width of the label head. > NARROW options: "wide" rendering, one label token per option:
-                           # A..Z then the first 229 two-letter upper-case strings that are single tokens (AA, AB, ...)
-ABSTAIN_PREFIXES = ("none of the above", "none of these", "not listed", "no suitable", "does not apply", "cannot tell")
-ABSTAIN_EXACT = ("other", "unsure", "something else", "neither of these", "other / not covered")
-
-
-def is_abstain_option(o):
-    o = o.strip().lower()
-    return o.startswith(ABSTAIN_PREFIXES) or o in ABSTAIN_EXACT
-
-
-_LABELS = {}
-_OPT_CACHE = {}
-
-
-def _enc_opt(tok, text):
-    """Token ids of ") <option text>" (cached: fixed label sets repeat the same strings millions of times)."""
-    key = (id(tok), text)
-    v = _OPT_CACHE.get(key)
-    if v is None:
-        v = tok.encode(f") {text}", add_special_tokens=False)
-        if len(_OPT_CACHE) < 2_000_000:
-            _OPT_CACHE[key] = v
-    return v
-
-
-def label_table(tok):
-    """(label strings, label token ids), MAX_OPTIONS entries; the first NARROW are A..J so narrow questions are unchanged."""
-    key = id(tok)
-    if key not in _LABELS:
-        import string
-        U = string.ascii_uppercase
-        names = list(U) + [a + b for a in U for b in U]
-        out = []
-        for n in names:
-            t = tok.encode(n, add_special_tokens=False)
-            if len(t) == 1:
-                out.append((n, t[0]))
-            if len(out) == MAX_OPTIONS:
-                break
-        assert len(out) == MAX_OPTIONS and len({i for _, i in out}) == MAX_OPTIONS
-        _LABELS[key] = ([n for n, _ in out], [i for _, i in out],
-                        tok.encode("\n(", add_special_tokens=False))
-    return _LABELS[key]
-
-
-def _select(q, rng, max_options):
-    opts = list(range(len(q.options)))
-    if len(opts) > max_options:
-        # always keep the gold and any abstain-style option (its mere presence must not carry information)
-        forced = {q.gold} | {i for i, o in enumerate(q.options) if is_abstain_option(o)}
-        others = [i for i in opts if i not in forced]
-        opts = rng.sample(others, max_options - len(forced)) + list(forced)
-    rng.shuffle(opts)
-    return opts
-
-
-def _options_ids(tok, q, opts):
-    if len(opts) <= NARROW:
-        return tok.encode("".join(f"\n({LETTERS[j]}) {q.options[oi]}" for j, oi in enumerate(opts)), add_special_tokens=False)
-    _, lab_ids, open_ids = label_table(tok); out = []
-    for j, oi in enumerate(opts):
-        out += open_ids + [lab_ids[j]] + _enc_opt(tok, q.options[oi])
-    return out
-
-
-def build_schema_first(example, tok, rng=None, max_options=NARROW, max_ctx_tokens=1536, chat=None):
-    """Schema-first layout: all question/option blocks, then the context, then one answer slot per question.
-
-        Question 1: ...\nOptions:\n(A) ...          <- prefix: depends only on the questions, so its cache (attention KV and
-        \n\nQuestion 2: ...                            delta-net states) is computed once per schema and reused for every state
-        \n\nContext:\n<state>\n\nAnswer 1: (\nAnswer 2: (
-
-    The three parts are tokenized separately, so `ids[:prefix_len]` is identical for every state.
-    chat: a ChatTemplate for a chat-layout model (see `build_chat`); None renders the plain layout above."""
-    rng = rng or random
-    perms = [_select(q, rng, max_options) for q in example.qs]
-    pre = schema_prefix_ids(tok, example.qs, perms, chat=chat)
-    suf, slots = schema_suffix_ids(tok, example.context, len(example.qs), max_ctx_tokens, chat=chat)
-    return dict(ids=pre + suf, slots=[len(pre) + s for s in slots], golds=[p.index(q.gold) if q.gold in p else -1 for p, q in zip(perms, example.qs)],
-                nopts=[len(p) for p in perms], perms=perms, prefix_len=len(pre))
-
-
-def schema_prefix_ids(tok, qs, perms=None, chat=None):
-    """Token ids of the question/option blocks (the cacheable part of the schema-first layout).
-    chat: a ChatTemplate; the prefix then starts with the template head (the user turn opens before the first question)."""
-    multi = len(qs) > 1; pre = [] if chat is None else list(chat.head)
-    for k, q in enumerate(qs):
-        opts = perms[k] if perms is not None else list(range(len(q.options)))
-        pre += tok.encode(f"{chr(10) * 2 if k else ''}Question{' ' + str(k + 1) if multi else ''}: {q.text}\nOptions:", add_special_tokens=False) + _options_ids(tok, q, opts)
-    return pre
-
-
-def schema_suffix_ids(tok, context, n_q, max_ctx_tokens=1536, chat=None):
-    """Token ids after the schema prefix: the context and one answer slot per question. Returns (ids, slot positions in ids).
-    chat: a ChatTemplate; the context then ends the user turn, and the template tail and the answer pieces follow."""
-    ids = tok.encode("\n\nContext:\n", add_special_tokens=False) + tok.encode(context, add_special_tokens=False)[:max_ctx_tokens]; slots = []
-    if chat is not None:
-        ids += chat.tail
-        for k in range(n_q):
-            ids += chat.answer_ids(tok, k, n_q > 1); slots.append(len(ids) - 1)
-        return ids, slots
-    for k in range(n_q):
-        ids += tok.encode(f"{chr(10) * 2 if k == 0 else chr(10)}Answer{' ' + str(k + 1) if n_q > 1 else ''}: (", add_special_tokens=False); slots.append(len(ids) - 1)
-    return ids, slots
-
-
-def build(example, tok, rng=None, max_options=NARROW, max_ctx_tokens=1536, layout="state_first", chat=None):
-    """Returns dict(ids=list[int], slots=list[int], golds=list[int], nopts=list[int], perms=list[list[int]]).
-    chat: a ChatTemplate (chat_template(tok)) for a model trained in the chat layout (decider_config.json "layout": "chat");
-    None, the default, is the plain layout every earlier model uses, unchanged."""
-    if layout == "schema_first":
-        return build_schema_first(example, tok, rng, max_options, max_ctx_tokens, chat=chat)
-    if chat is not None:
-        return build_chat(example, tok, chat, rng, max_options, max_ctx_tokens)
-    rng = rng or random
-    ctx_ids = tok.encode("Context:\n" + example.context, add_special_tokens=False)[:max_ctx_tokens]
-    ids = list(ctx_ids)
-    slots, golds, nopts, perms = [], [], [], []
-    multi = len(example.qs) > 1
-    for k, q in enumerate(example.qs):
-        opts = list(range(len(q.options)))
-        if len(opts) > max_options:
-            # always keep the gold and any abstain-style option (its mere presence must not carry information)
-            forced = {q.gold} | {i for i, o in enumerate(q.options) if is_abstain_option(o)}
-            others = [i for i in opts if i not in forced]
-            keep = rng.sample(others, max_options - len(forced)) + list(forced)
-            opts = keep
-        rng.shuffle(opts)
-        head = f"\n\nQuestion{' ' + str(k + 1) if multi else ''}: {q.text}\nOptions:"
-        tail = f"\nAnswer{' ' + str(k + 1) if multi else ''}: ("
-        if len(opts) <= NARROW:
-            lines = [head] + [f"\n({LETTERS[j]}) {q.options[oi]}" for j, oi in enumerate(opts)] + [tail]
-            piece = tok.encode("".join(lines), add_special_tokens=False)
-        else:                                   # wide: "\n(" + <label token> + ") text", built from ids so every label is one token
-            _, lab_ids, open_ids = label_table(tok)
-            piece = tok.encode(head, add_special_tokens=False)
-            for j, oi in enumerate(opts):
-                piece += open_ids + [lab_ids[j]] + _enc_opt(tok, q.options[oi])
-            piece += tok.encode(tail, add_special_tokens=False)
-        ids.extend(piece)
-        slots.append(len(ids) - 1)          # position of " (" token
-        golds.append(opts.index(q.gold) if q.gold in opts else -1)
-        nopts.append(len(opts))
-        perms.append(opts)
-    return dict(ids=ids, slots=slots, golds=golds, nopts=nopts, perms=perms)
-
-
-# ---- chat layout (1.2.0) ----------------------------------------------------------------------------------------------
-# A model whose decider_config.json has "layout": "chat" (or "chat_template": true) was trained with every prompt wrapped
-# in its tokenizer's chat template: one user turn holding the plain content, thinking switched off, and the answer pieces in
-# the assistant turn.  For Qwen3.5, state-first:
-#
-#     <|im_start|>user\n  Context:\n<state>  \n\nQuestion: <q>\nOptions:\n(A) ..\n(B) ..
-#     <|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n  Answer: (
-#
-# With several questions in one row all question blocks come first and the answer pieces ("Answer 1: (", "\nAnswer 2: (",
-# ...) follow the template tail.  Schema-first: the user turn holds the question blocks and then "\n\nContext:\n<state>".
-# Every part is tokenized on its own (head, context, each question header, each option block, tail, each answer piece);
-# the letter is read at the final " (" token as in the plain layout.
-LAYOUTS = ("plain", "chat")
-
-
-def resolve_layout(cfg):
-    """The prompt layout a decider_config.json names: "plain" (no "layout" key and no "chat_template": true, which is every
-    released model) or "chat".  Raises ValueError for any other value, so a model trained in a layout
-    this version does not know is refused instead of being read in the wrong one."""
-    cfg = cfg or {}
-    layout = cfg.get("layout")
-    if layout is None:
-        layout = "chat" if cfg.get("chat_template") is True else "plain"
-    if layout not in LAYOUTS:
-        raise ValueError(f"decider_config.json names the prompt layout {layout!r}; this version of decider-ai knows "
-                         f"{', '.join(repr(x) for x in LAYOUTS)}. Upgrade decider-ai, or check the model's config.")
-    if layout == "plain" and cfg.get("chat_template") is True:
-        raise ValueError('decider_config.json says "layout": "plain" and "chat_template": true; these contradict each other')
-    return layout
-
-
-class ChatTemplate:
-    """Token ids of the tokenizer's chat template around one user turn, and the answer pieces of the assistant turn.
-
-    head/tail come from tok.apply_chat_template([user turn], add_generation_prompt=True) with thinking disabled
-    (enable_thinking=False where the template knows the switch; a thinking block the template leaves open is closed here),
-    with no system prompt.  This is the computation of the chat-layout research server the chat models were trained against."""
-    SENTINEL = "@@DECIDER_USER_CONTENT@@"
-
-    def __init__(self, tok, answer="Answer"):
-        if not getattr(tok, "chat_template", None):
-            raise ValueError('the model is configured for the chat layout ("layout": "chat") but its tokenizer has no chat template')
-        msgs = [{"role": "user", "content": self.SENTINEL}]
-        kw = {"enable_thinking": False} if "enable_thinking" in tok.chat_template else {}
-        s = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True, **kw)
-        head, tail = s.split(self.SENTINEL)
-        for opened, closed in (("<think>\n", "</think>\n\n"), ("<think>", "</think>\n\n"), ("<|channel>thought\n", "<channel|>")):
-            if tail.endswith(opened):
-                tail += closed
-        self.head_text, self.tail_text = head, tail
-        self.head = tok.encode(head, add_special_tokens=False)
-        self.tail = tok.encode(tail, add_special_tokens=False)
-        self.answer = answer
-        names, ids, _ = label_table(tok)
-        pre = tok.encode(f"{answer}: (", add_special_tokens=False)
-        for n, i in list(zip(names, ids))[:NARROW]:
-            if tok.encode(f"{answer}: ({n}", add_special_tokens=False) != pre + [i]:
-                raise ValueError(f"chat layout: the label {n!r} does not tokenize as one token after {answer!r}: (")
-        if tok.encode(tail + f"{answer}: (", add_special_tokens=False) != self.tail + pre:
-            raise ValueError("chat layout: the chat template tail merges with the answer prefix")
-
-    def answer_ids(self, tok, k, multi):
-        """Token ids of answer piece k: "Answer: (" (or "Answer k: (" with several questions), later pieces after a newline."""
-        return tok.encode(f"{'' if k == 0 else chr(10)}{self.answer}{' ' + str(k + 1) if multi else ''}: (", add_special_tokens=False)
-
-
-_CHAT = {}
-
-
-def chat_template(tok):
-    """The ChatTemplate of a tokenizer, cached per tokenizer object."""
-    key = id(tok)
-    if key not in _CHAT:
-        _CHAT[key] = (tok, ChatTemplate(tok))           # keep tok alive so its id is not reused
-    return _CHAT[key][1]
-
-
-def chat_for(tok, cfg):
-    """The ChatTemplate to pass as `chat=` for a model with this decider_config.json, or None for a plain-layout model."""
-    return chat_template(tok) if resolve_layout(cfg) == "chat" else None
-
-
-def load_decider_config(path):
-    """decider_config.json of a model folder or a Hub repository id; {} when the model has none (a raw base model)."""
-    import json, os
-    if os.path.isdir(path):
-        f = os.path.join(path, "decider_config.json")
-        return json.load(open(f)) if os.path.exists(f) else {}
-    try:
-        from huggingface_hub import hf_hub_download
-        return json.load(open(hf_hub_download(path, "decider_config.json")))
-    except Exception:
-        return {}
-
-
-def chat_for_model(path, tok):
-    """chat_for(tok, the model's decider_config.json): what every script that builds prompts for `path` passes as `chat=`."""
-    return chat_for(tok, load_decider_config(path))
-
-
-def build_chat(example, tok, chat, rng=None, max_options=NARROW, max_ctx_tokens=1536):
-    """State-first chat layout (see the comment above LAYOUTS).  Same return fields as `build`.  max_ctx_tokens caps the
-    tokens of "Context:\\n" + state, as in the plain layout; the template head and tail are not counted."""
-    rng = rng or random
-    qs = example.qs; multi = len(qs) > 1
-    ids = list(chat.head) + tok.encode("Context:\n" + example.context, add_special_tokens=False)[:max_ctx_tokens]
-    golds, nopts, perms = [], [], []
-    for k, q in enumerate(qs):
-        opts = _select(q, rng, max_options)
-        ids += tok.encode(f"\n\nQuestion{' ' + str(k + 1) if multi else ''}: {q.text}\nOptions:", add_special_tokens=False) + _options_ids(tok, q, opts)
-        golds.append(opts.index(q.gold) if q.gold in opts else -1); nopts.append(len(opts)); perms.append(opts)
-    ids += chat.tail
-    slots = []
-    for k in range(len(qs)):
-        ids += chat.answer_ids(tok, k, multi); slots.append(len(ids) - 1)
-    return dict(ids=ids, slots=slots, golds=golds, nopts=nopts, perms=perms)
-
-
-def letter_ids(tok):
-    ids = label_table(tok)[1]
-    for j, L in enumerate(LETTERS):
-        assert tok.encode(L, add_special_tokens=False) == [ids[j]], L
-    return ids
-
-
-def render(example, tok, **kw):
-    b = build(example, tok, **kw)
-    return tok.decode(b["ids"])

decider/prompt_fast.py

@@ -1,90 +0,0 @@
-"""Row construction for the systemone path that tokenizes the state once per request instead of once per question.
-
-`prompt.build` encodes `"Context:\\n" + state` for every row it is called with, so an independent request with n
-questions tokenizes the whole state n times.  The question block is encoded as its own string there and simply appended,
-so the ids are exactly `ctx_ids + question_piece`; this module reuses that fact and shares `ctx_ids` across the rows.
-
-`build_rows` returns the same item dicts as
-    [prompt.build(Example(ctx, [Q(text, options, 0) for text, options in row]), tok, <no-shuffle rng>,
-                  max_options=MAX_OPTIONS, max_ctx_tokens=max_ctx_tokens) for row in rows]
-(ids, slots, golds, nopts, perms) and is checked against it in tests/test_prompt_fast.py.
-"""
-from decider.prompt import LETTERS, NARROW, MAX_OPTIONS, label_table, _enc_opt, _options_ids
-
-
-def context_ids(tok, context, max_ctx_tokens=32768):
-    return tok.encode("Context:\n" + context, add_special_tokens=False)[:max_ctx_tokens]
-
-
-def question_piece(tok, text, options, k=0, multi=False):
-    """Token ids of one `\\n\\nQuestion...: <text>\\nOptions:...\\nAnswer...: (` block, independent of the state."""
-    head = f"\n\nQuestion{' ' + str(k + 1) if multi else ''}: {text}\nOptions:"
-    tail = f"\nAnswer{' ' + str(k + 1) if multi else ''}: ("
-    if len(options) <= NARROW:
-        return tok.encode(head + "".join(f"\n({LETTERS[j]}) {o}" for j, o in enumerate(options)) + tail,
-                          add_special_tokens=False)
-    _, lab_ids, open_ids = label_table(tok)
-    piece = tok.encode(head, add_special_tokens=False)
-    for j, o in enumerate(options):
-        piece += open_ids + [lab_ids[j]] + _enc_opt(tok, o)
-    return piece + tok.encode(tail, add_special_tokens=False)
-
-
-class _Opts:
-    def __init__(self, options): self.options = options
-
-
-def _chat_row(tok, chat, ctx, row):
-    """One chat-layout row after the shared ids `ctx` (template head + context): every question block, the template tail,
-    then the answer pieces.  Equal to prompt.build_chat for the same row (tests/test_layout.py)."""
-    multi = len(row) > 1
-    ids = list(ctx)
-    for k, (text, options) in enumerate(row):
-        ids += tok.encode(f"\n\nQuestion{' ' + str(k + 1) if multi else ''}: {text}\nOptions:", add_special_tokens=False)
-        ids += _options_ids(tok, _Opts(options), list(range(len(options))))
-    ids += chat.tail
-    slots = []
-    for k in range(len(row)):
-        ids += chat.answer_ids(tok, k, multi); slots.append(len(ids) - 1)
-    return ids, slots
-
-
-def build_rows(tok, context, rows, max_ctx_tokens=32768, chat=None):
-    """rows: list of rows, each a list of (question text, options).  -> (items, len(ctx_ids)).
-
-    Option order is kept as given (no shuffling, no subsetting): systemone.render_question already caps a choice at
-    MAX_OPTIONS options, so prompt.build's sampling branch is unreachable here.
-    chat: a ChatTemplate (prompt.chat_template) for a chat-layout model; the shared ids are then the template head plus the
-    context, and each row is prompt.build_chat's rendering."""
-    ctx = context_ids(tok, context, max_ctx_tokens)
-    if chat is not None:
-        ctx = list(chat.head) + ctx
-    items = []
-    for row in rows:
-        multi = len(row) > 1
-        ids = list(ctx); slots = []; nopts = []
-        for k, (text, options) in enumerate(row):
-            if not 2 <= len(options) <= MAX_OPTIONS:
-                raise ValueError(f"2..{MAX_OPTIONS} options required")
-            if chat is None:
-                ids.extend(question_piece(tok, text, options, k, multi))
-                slots.append(len(ids) - 1)
-            nopts.append(len(options))
-        if chat is not None:
-            ids, slots = _chat_row(tok, chat, ctx, row)
-        items.append(dict(ids=ids, slots=slots, golds=[0] * len(row), nopts=nopts,
-                          perms=[list(range(len(o))) for _, o in row]))
-    return items, len(ctx)
-
-
-def unique_tokens(items, ctx_len):
-    """systemone.unique_tokens(items) without re-walking the shared context (all rows start with the same ctx_len ids).
-    No rows (a request with no questions) counts 0, as systemone.unique_tokens does."""
-    if not items:
-        return 0
-    sufs = [it["ids"][ctx_len:] for it in items]
-    if len(sufs) < 2:
-        return ctx_len + sum(len(s) for s in sufs)
-    lcp = 0; short = min(len(s) for s in sufs); s0 = sufs[0]
-    while lcp < short and all(s[lcp] == s0[lcp] for s in sufs): lcp += 1
-    return ctx_len + lcp + sum(len(s) - lcp for s in sufs)

decider/schema_engine.py

@@ -1,150 +0,0 @@
-"""Schema cache: compute a question schema once, then score states against it.
-
-In production the questions are fixed and only the state changes.  With the schema-first prompt layout
-(prompt.build_schema_first) the question/option blocks are a prefix that does not depend on the state, so their
-cache - attention K/V for the 6 full-attention layers, conv + recurrent state for the 18 delta-net layers - is computed
-once (`prepare`).  A request then runs only "Context: <state>" plus one answer slot per question, as a CUDA graph per
-(batch, length) bucket.  The prefix cache is read-only during a request (nothing is written back), so one copy serves
-every batch and every graph.
-
-    se = SchemaEngine(engine); h = se.prepare([{"question": ..., "options": [...]}, ...])
-    probs = se.score(h, ["state 1", "state 2", ...])          # list of [n_questions, MAX_OPTIONS] tensors
-"""
-import time, types, torch, torch.nn.functional as F
-from decider.prompt import schema_prefix_ids, schema_suffix_ids, MAX_OPTIONS
-from decider.engine import read_slots, fill_ids
-
-TS_BUCKETS = [32, 48, 64, 96, 128, 192, 256, 384, 512, 768, 1024]
-B_BUCKETS = [1, 2, 4, 8, 16, 32, 64]
-
-
-class _Q:
-    def __init__(self, text, options): self.text, self.options = text, options
-
-
-class PrefixCache:
-    """Duck-typed transformers Cache over fixed, read-only prefixes, for one suffix forward pass.
-    A handle holds P prefixes (P = 1: all questions packed in one prefix; P = n_questions: one prefix per question, so every
-    question is scored independently).  A batch of R states has R * P rows; row r * P + p continues prefix p."""
-    def __init__(self, h, R):
-        rep = (lambda t: t.expand(R, *t.shape[1:])) if h.P == 1 else (lambda t: t.repeat(R, *([1] * (t.dim() - 1))))
-        self.tp = h.tpmax; self.k = {i: rep(k) for i, k in h.k.items()}; self.v = {i: rep(v) for i, v in h.v.items()}
-        self.conv = {i: rep(c).contiguous() for i, c in h.conv.items()}
-        self.layers = {i: types.SimpleNamespace(record_past=False, recurrent_states={0: rep(r).contiguous()}) for i, r in h.rec.items()}
-
-    def has_previous_state(self, layer_idx=None, state_idx=None): return True
-    def get_seq_length(self, *a, **k): return self.tp
-    def update(self, key, value, layer_idx, *a, **k): return torch.cat([self.k[layer_idx], key], 2), torch.cat([self.v[layer_idx], value], 2)
-    def update_conv_state(self, x, layer_idx, **k): return torch.cat([self.conv[layer_idx].to(x.dtype), x], -1)
-    def update_recurrent_state(self, s, layer_idx, **k): return s
-
-
-class SchemaEngine:
-    def __init__(self, engine, use_graphs=True, chat=None):
-        """chat: the ChatTemplate of a chat-layout model (decider.prompt.chat_template); the prefix then starts with the template
-        head and the suffix ends with the template tail and the answer pieces (decider.prompt.build_schema_first with chat)."""
-        self.chat = chat
-        self.e = engine; self.core = engine.core; self.W = engine.W; self.tok = engine.tok; self.dev = engine.dev
-        self.use_graphs = use_graphs and engine.use_graphs; self.graphs = {}; self.stats = dict(prepared=0, captures=0, replays=0, eager=0)
-        self.compile = bool(engine.cfg.get("compile")); self._compiled = {}
-        if self.compile:                        # every compiled schema graph specialises the model frames again (its cache tensors are constants)
-            import torch._dynamo
-            torch._dynamo.config.cache_size_limit = 4096; torch._dynamo.config.accumulated_cache_size_limit = 1 << 16
-
-    @torch.no_grad()
-    def prepare(self, questions, independent=False, compile=False):
-        """questions: [{"question": str, "options": [str]}] in the order answers are wanted.  Runs the prefix(es) once.
-        independent=False: one prefix holding every question (cheapest: a request costs state + n slots).
-        independent=True:  one prefix per question, one row per question (a request costs n * (state + 1 slot); no question
-                           can influence another)."""
-        qs = [_Q(q["question"], list(q["options"])) for q in questions]
-        groups = [[q] for q in qs] if independent else [qs]; pres = [schema_prefix_ids(self.tok, g, chat=self.chat) for g in groups]
-        h = types.SimpleNamespace(P=len(groups), nq=len(qs), slots_per_row=1 if independent else len(qs), nopts=[len(q.options) for q in qs], tps=[len(p) for p in pres],
-                                  tpmax=max(len(p) for p in pres), k={}, v={}, conv={}, rec={}, id=self.stats["prepared"],
-                                  compile=bool(compile and self.compile))
-        parts = []
-        for pre in pres:
-            out = self.core(input_ids=torch.tensor(pre, device=self.dev)[None], use_cache=True).past_key_values; d = dict(k={}, v={}, conv={}, rec={})
-            for i, layer in enumerate(out.layers):
-                if getattr(layer, "recurrent_states", None) is not None and layer.recurrent_states.get(0) is not None:
-                    d["conv"][i] = layer.conv_states[0]; d["rec"][i] = layer.recurrent_states[0]
-                else:                                                   # right-pad every prefix's K/V to the longest; the mask hides the padding
-                    pad = (0, 0, 0, h.tpmax - len(pre)); d["k"][i] = F.pad(layer.keys, pad); d["v"][i] = F.pad(layer.values, pad)
-            parts.append(d)
-        for name in ("k", "v", "conv", "rec"):
-            getattr(h, name).update({i: torch.cat([d[name][i] for d in parts], 0).clone() for i in parts[0][name]})
-        self.stats["prepared"] += 1
-        return h
-
-    def _fwd(self, ids, cache, mask, pos):
-        hs = self.core(input_ids=ids, past_key_values=cache, attention_mask={"full_attention": mask, "linear_attention": None}, position_ids=pos, use_cache=True).last_hidden_state
-        return F.linear(hs, self.W).float()
-
-    def _static(self, h, R, Ts):
-        """R request slots -> R * P rows.  Mask: a row sees its own prefix (not the padding up to tpmax) and the causal suffix."""
-        ar = torch.arange(Ts, device=self.dev); tps = torch.tensor(h.tps, device=self.dev).repeat(R)                                # [R*P]
-        pre = (torch.arange(h.tpmax, device=self.dev)[None, :] < tps[:, None])[:, None, None, :].expand(-1, 1, Ts, -1)             # [B,1,Ts,tpmax]
-        mask = torch.cat([pre, (ar[:, None] >= ar[None, :])[None, None].expand(len(tps), 1, -1, -1)], 3).contiguous()
-        return PrefixCache(h, R), mask, (tps[:, None] + ar[None, :]).contiguous()
-
-    def _capture(self, h, R, Ts):
-        B = R * h.P
-        ids = torch.full((B, Ts), self.tok.pad_token_id, dtype=torch.long, device=self.dev); cache, mask, pos = self._static(h, R, Ts)
-        fwd = self._fwd
-        if h.compile:                                       # one compiled function per graph (20-30 s each: only for preloaded schemas): the cache tensors are constants of that graph
-            fwd = torch.compile(lambda i: self._fwd(i, cache, mask, pos), dynamic=False)
-            call = lambda: fwd(ids)
-        else:
-            call = lambda: fwd(ids, cache, mask, pos)
-        st = torch.cuda.Stream(); st.wait_stream(torch.cuda.current_stream())
-        with torch.cuda.stream(st):
-            for _ in range(3): call()
-        torch.cuda.current_stream().wait_stream(st)
-        g = torch.cuda.CUDAGraph()
-        with torch.cuda.graph(g, pool=self.e.pool):
-            out = call()
-        self.stats["captures"] += 1
-        return ids, out, g, (cache, mask, pos)
-
-    def warmup(self, h, batch_sizes=(1, 8, 32), state_tokens=(64, 128, 256)):
-        """Capture (and, for a compiled schema, compile) the graphs for these request-batch sizes and suffix lengths ahead of traffic."""
-        t = time.time()
-        for R in batch_sizes:
-            for Ts in state_tokens:
-                Ts = next((x for x in TS_BUCKETS if x >= Ts), TS_BUCKETS[-1])
-                if (h.id, R, Ts) not in self.graphs: self.graphs[(h.id, R, Ts)] = self._capture(h, R, Ts)
-        torch.cuda.synchronize(); return time.time() - t
-
-    def tokenize(self, h, context, max_ctx_tokens=1536):
-        """CPU part of a request (do it outside any GPU lock): -> (suffix ids, slot positions)."""
-        return schema_suffix_ids(self.tok, context, h.slots_per_row, max_ctx_tokens, chat=self.chat)
-
-    @staticmethod
-    def bucket(n_tokens):
-        return next((t for t in TS_BUCKETS if t >= n_tokens), -(-n_tokens // 256) * 256)
-
-    def score(self, h, contexts, temperature=1.0, max_ctx_tokens=1536):
-        """-> one [n_questions, MAX_OPTIONS] probability tensor per context.  temperature: as in score_rows."""
-        return self.score_rows(h, [self.tokenize(h, c, max_ctx_tokens) for c in contexts], temperature)
-
-    @torch.no_grad()
-    def score_rows(self, h, rows, temperature=1.0):
-        """rows: [(suffix ids, slots)] from tokenize().  temperature: a number, or a list with one temperature per schema row
-        (h.nq values, in the order of prepare's questions), applied to every request."""
-        if isinstance(temperature, (list, tuple)) and len(temperature) != h.nq:
-            raise ValueError(f"temperature: {len(temperature)} values for a schema with {h.nq} rows")
-        Tmax = max(len(r[0]) for r in rows); Ts = next((t for t in TS_BUCKETS if t >= Tmax), None); n = len(rows)
-        R = next((b for b in B_BUCKETS if b >= n), n) if Ts else n; Ts = Ts or -(-Tmax // 256) * 256
-        ids = fill_ids([x for x, _ in rows for _ in range(h.P)], R * h.P, Ts, self.tok.pad_token_id).to(self.dev, non_blocking=True)
-        if self.use_graphs and Ts <= TS_BUCKETS[-1]:
-            key = (h.id, R, Ts)
-            if key not in self.graphs: self.graphs[key] = self._capture(h, R, Ts)
-            s_ids, s_out, g, _ = self.graphs[key]; s_ids.copy_(ids); g.replay(); out = s_out; self.stats["replays"] += 1
-        else:
-            out = self._fwd(ids, *self._static(h, R, Ts)); self.stats["eager"] += 1
-        if h.P == 1:                                           # packed: n slots in one row per request
-            rws = [r for r in range(n) for _ in range(h.nq)]; sls = [x for _, sl in rows for x in sl]
-        else:                                                  # independent: one slot in each of the request's P rows
-            rws = [r * h.P + p for r in range(n) for p in range(h.P)]; sls = [sl[0] for _, sl in rows for _ in range(h.P)]
-        temps = list(temperature) * n if isinstance(temperature, (list, tuple)) else temperature
-        return read_slots(out, rws, sls, h.nopts * n, temps, [h.nq] * n)

decider/serve.py

@@ -1,562 +0,0 @@
-"""HTTP server.
-  POST /decide        {"context": str, "schema": {...}}                     -> typed JSON decisions (all questions packed in one row)
-                      schema: {question: {"type": "choice", "options": [...]} | {"type": "bool"} | {"type": "scale", "legend": [...]}};
-                      a missing or malformed schema is a 422 naming this form (decider.infer.Decider._check_schema)
-  POST /v1/systemone  {"state": str|object|array, "questions": {id: {...}}} -> the TypeSafe/Jev wire format (decider.systemone):
-                      Choice (up to 255 described options), Score, Noul; every question is scored in its own row, so answers
-                      are independent of each other ("independent": false packs them behind one copy of the state instead).
-  GET  /v1/models  /health  /stats
-
-   uvicorn decider.serve:app --host 0.0.0.0 --port 8000        (env: DECIDER_MODEL and the variables below)
-
-Execution (1.1.1; docs/SERVING.md has the design and the measurements):
-  * decider.engine_v2.EngineV2: CUDA graphs keyed on (batch bucket, padded length bucket) only.  The whole grid is captured
-    during start-up and the engine is then sealed, so no request pays a graph capture or a torch.compile.  Rows longer than the
-    last length bucket (8,192 tokens) run eager, in chunks bounded by DECIDER_GRAPH_TOKEN_BUDGET padded tokens.
-  * one GPU thread runs every forward; tokenisation runs on a CPU pool; rows waiting at the same moment are partitioned by
-    decider.batching.plan_batches, which pads a shorter row into a longer row's bucket when that costs less than a second
-    forward (cost model: DECIDER_MERGE_OVERHEAD_TOKENS + rows * padded length).  When a collection held rows from more than
-    one live request the next one waits up to DECIDER_BATCH_ADAPTIVE_WAIT_MS for more rows, unless its own first row took
-    more than 50 ms to arrive (a quiet queue drops the window again); after a single-request collection it does not wait.
-  * an independent request with more than one question over a state of at least DECIDER_SHARED_MIN_TOKENS tokens runs the state
-    once and forks its cache per question (EngineV2.score_shared, decider.shared_prefix).  The fork is made in chunks that fit
-    DECIDER_SHARED_FORK_GB, so the peak memory does not grow with the question count.  The cuDNN SDPA backend is off
-    (decider.engine.set_attention_backend_policy), which is what makes that path agree with the full forward.
-  * the schema cache (questions-first layout, prefix cached per question set) is honoured exactly as before: on when
-    decider_config.json has "schema_first": true, or with DECIDER_SCHEMA_CACHE=1 on a model trained for that layout.  It is
-    the one path that does GPU work after start-up that is not a graph replay: the first request with a new schema runs its
-    prefix and captures one graph per (schema, batch bucket, state bucket) as in 1.0.x (/stats -> schema_cache.captures).
-    Its requests are bounded and admitted like the others, on prefix plus suffix tokens, before any GPU preparation.
-  * prompt layout (1.2.0): a model whose decider_config.json has "layout": "chat" (a chat-trained checkpoint) is read in the chat layout
-    (decider.prompt.build_chat: the tokenizer's chat template around one user turn, thinking off, "Answer: (" in the assistant
-    turn) on every route, including the shared-prefix fork and the schema cache.  Every other model is read in the plain
-    layout, with the same token ids as 1.1.x.  A config naming an unknown layout stops start-up with a ValueError.
-  * requests are bounded before they reach the GPU: HTTP 413 when a row, the request or the expanded question count exceeds the
-    limits below, HTTP 503 when the outstanding work exceeds DECIDER_MAX_QUEUE_ROWS.
-
-Variables (default):  DECIDER_DEVICE (auto: cuda, else mps, else cpu)  DECIDER_COMPILE (0)  DECIDER_FP8 (0)  DECIDER_SHARED (1)  DECIDER_SHARED_MIN_TOKENS (768)
-  DECIDER_SHARED_FORK_GB (8)  DECIDER_MERGE_OVERHEAD_TOKENS (512)  DECIDER_BATCH_ADAPTIVE_WAIT_MS (2)
-  DECIDER_MAX_BATCH (32)  DECIDER_BATCH_WAIT_MS (0; DECIDER_MAX_WAIT_MS is an alias)  DECIDER_MAX_STATE_TOKENS (32768)
-  DECIDER_T_BUCKETS  DECIDER_B_BUCKETS  DECIDER_GRAPH_TOKEN_BUDGET (32768)  DECIDER_WARMUP (1)  DECIDER_TOKENIZE_THREADS (8)
-  DECIDER_MAX_ROWS (1024)  DECIDER_MAX_ROW_TOKENS (DECIDER_MAX_STATE_TOKENS + 4096)  DECIDER_MAX_REQUEST_TOKENS (1048576)
-  DECIDER_MAX_QUEUE_ROWS (4096)  DECIDER_TEMPERATURE  DECIDER_SCHEMA_CACHE (0)  DECIDER_SCHEMA_MIN_SEEN (2)  DECIDER_SCHEMAS
-
-Temperatures (1.4.0, decider.temperature): decider_config.json "temperature", and optionally "temperature_by_type"
-{"choice": T, "noul": T, "score": T} (a /decide "bool" field is "noul", a "scale" field is "score"; a missing type uses
-"temperature").  Every row carries the answer type of each of its slots, so a batch that mixes requests and types still
-applies each answer's own temperature.  DECIDER_TEMPERATURE replaces "temperature" and switches the by-type map off.
-/health and the ready line report the temperature every answer type gets.
-"""
-import asyncio, json, os, time
-from concurrent.futures import ThreadPoolExecutor
-from contextlib import asynccontextmanager
-from fastapi import FastAPI, HTTPException
-from pydantic import BaseModel
-from decider import systemone as S1
-from decider import temperature as TT
-from decider.batching import DEFAULT_MERGE_OVERHEAD_TOKENS, plan_batches
-from decider.prompt import build, MAX_OPTIONS, resolve_layout, chat_template
-from decider.prompt_fast import build_rows, unique_tokens
-
-
-def _env_int(name, default):
-    return int(os.environ.get(name, default))
-
-
-MODEL = os.environ.get("DECIDER_MODEL", "runs/r3_v2/model")
-MAX_BATCH = _env_int("DECIDER_MAX_BATCH", 32)
-BATCH_WAIT_MS = float(os.environ.get("DECIDER_BATCH_WAIT_MS", os.environ.get("DECIDER_MAX_WAIT_MS", "0")))
-ADAPTIVE_WAIT_MS = float(os.environ.get("DECIDER_BATCH_ADAPTIVE_WAIT_MS", "2"))   # window after a collection that held >1 request; 0 disables
-ADAPTIVE_IDLE_RESET_MS = 50.0                                               # an idler queue than this drops the adaptive window again
-MERGE_OVERHEAD_TOKENS = _env_int("DECIDER_MERGE_OVERHEAD_TOKENS", DEFAULT_MERGE_OVERHEAD_TOKENS)
-MAX_STATE_TOKENS = _env_int("DECIDER_MAX_STATE_TOKENS", 32768)
-SHARED = os.environ.get("DECIDER_SHARED", "1") == "1"
-SHARED_MIN_TOKENS = _env_int("DECIDER_SHARED_MIN_TOKENS", 768)
-DEVICE = os.environ.get("DECIDER_DEVICE", "auto")                          # auto | cuda[:i] | mps | cpu
-COMPILE = os.environ.get("DECIDER_COMPILE", "0") == "1"
-FP8 = os.environ.get("DECIDER_FP8", "0") == "1"
-WARMUP = os.environ.get("DECIDER_WARMUP", "1") == "1"
-TOKENIZE_THREADS = _env_int("DECIDER_TOKENIZE_THREADS", 8)
-GRAPH_TOKEN_BUDGET = _env_int("DECIDER_GRAPH_TOKEN_BUDGET", 32768)
-# request bounds, checked after tokenisation and before anything is queued
-MAX_ROWS = _env_int("DECIDER_MAX_ROWS", 1024)                               # scoring rows per request (questions, expanded score levels)
-MAX_ROW_TOKENS = _env_int("DECIDER_MAX_ROW_TOKENS", MAX_STATE_TOKENS + 4096)  # tokens in one row: truncated state + question block
-MAX_REQUEST_TOKENS = _env_int("DECIDER_MAX_REQUEST_TOKENS", 1 << 20)       # sum of row lengths of one request
-MAX_QUEUE_ROWS = _env_int("DECIDER_MAX_QUEUE_ROWS", 4096)                   # rows admitted and not yet scored, over all requests
-DECIDE_MAX_CTX_TOKENS = 1536                                                # /decide context cap, unchanged from 1.0.x
-
-MODEL_NAME = "decider"; TEMP = 1.0; TEMP_SCHEMA = 1.0; TEMP_BY_TYPE = {}; TEMP_SCHEMA_BY_TYPE = {}; RELEASE_DATE = "2026-09-17"; ISOLATED = False; NEUTRALIZE_NONE = True
-SCHEMA_FIRST = False; LAYOUT = "plain"; CHAT = None; se = None; squeue = None; schemas = {}; seen = {}
-eng = None; queue = None; gpu = None; cpu = None; batcher_task = None; schema_task = None
-outstanding = 0; REQ_SEQ = 0
-stats = dict(requests=0, batches=0, decisions=0, rows=0, shared_prefix_requests=0, errors=0, rejected_too_large=0,
-             rejected_overloaded=0, batch_hist={}, bucket_hist={})
-
-
-def _ints(name):
-    v = os.environ.get(name)
-    return [int(x) for x in v.split(",")] if v else None
-
-
-def load_config(path):
-    """decider_config.json from a model folder or a Hub repository (the way decider.infer.Decider resolves it)."""
-    try:
-        if os.path.isdir(path):
-            return json.load(open(os.path.join(path, "decider_config.json")))
-        from huggingface_hub import hf_hub_download
-        return json.load(open(hf_hub_download(path, "decider_config.json")))
-    except Exception:
-        return {}
-
-
-# ---- request preparation (CPU) -------------------------------------------
-class _NoShuffle:
-    def shuffle(self, x): pass
-    def sample(self, xs, k): return xs[:k]
-
-
-def prepare(tok, state, questions, independent, isolated=False, max_state_tokens=32768, chat=None):
-    """Render, plan the rows, tokenize the state once.  -> (rqs, index, items, ctx_len).  The rows are those
-    `prompt.build` produces for the same request (tests/test_prompt_fast.py, tests/test_serve_prepare.py); chat: the
-    ChatTemplate of a chat-layout model, None for the plain layout."""
-    ctx = S1.render_state(state)
-    rqs = {k: S1.render_question(v) for k, v in questions.items()}
-    flat, index = S1.plan_rows(rqs, isolated and independent)
-    pairs = [(r["question"], list(r["options"])) for r in flat]
-    rows = [[p] for p in pairs] if independent else [pairs]
-    items, ctx_len = build_rows(tok, ctx, rows, max_ctx_tokens=max_state_tokens, chat=chat)
-    types = S1.row_types(rqs, index)                                 # answer type per slot (decider.temperature)
-    for it, ts in zip(items, [[t] for t in types] if independent else [types]):
-        it["types"] = ts
-    return rqs, index, items, ctx_len
-
-
-def _prepare_s1(state, questions, independent):
-    return prepare(eng.tok, state, questions, independent, ISOLATED, MAX_STATE_TOKENS, chat=CHAT)
-
-
-def _prepare_decide(context, schema):
-    from decider.infer import Decider, Example, Q, neutralize_options
-    qs = Decider._schema_to_questions(schema)
-    for q in qs:
-        if NEUTRALIZE_NONE:
-            q["options"], q["_back"] = neutralize_options(q["options"])
-    ex = Example(context, [Q(q["question"], list(q["options"]), 0) for q in qs])
-    it = build(ex, eng.tok, _NoShuffle(), max_options=MAX_OPTIONS, max_ctx_tokens=DECIDE_MAX_CTX_TOKENS, chat=CHAT)
-    it["types"] = [q["_type"] for q in qs]                            # bool -> noul, scale -> score (decider.temperature)
-    return qs, it
-
-
-def _format_decide(schema, qs, probs):
-    o = {}
-    for (qtext, spec), q, p in zip(schema.items(), qs, probs):
-        p = p[:len(q["options"])].tolist(); t = spec.get("type", "choice"); j = max(range(len(p)), key=p.__getitem__)
-        back = q.get("_back", {}); names = [back.get(x, x) for x in q["options"]]
-        if t == "bool":
-            o[qtext] = {"noul": round(p[1], 4), "type": "noul"}
-        elif t == "choice":
-            o[qtext] = {"choice": names[j], "confidence": round(p[j], 4), "type": "choice",
-                        "probabilities": {k: round(v, 4) for k, v in zip(names, p)}}
-        else:
-            keys = q["_keys"]; score = sum(float(k) * pi for k, pi in zip(keys, p))
-            o[qtext] = {"score": round(score, 2), "confidence": round(p[j], 4), "type": "scale", "legend": q["_legend"],
-                        "probabilities": {str(keys[i]): round(pi, 4) for i, pi in enumerate(p)}}
-    return o
-
-
-# ---- limits ---------------------------------------------------------------
-def check_size(row_lengths, max_rows=None, max_row_tokens=None, max_request_tokens=None):
-    """Raise HTTPException(413) when a request exceeds the row count, per-row token or total token limit."""
-    max_rows = MAX_ROWS if max_rows is None else max_rows
-    max_row_tokens = MAX_ROW_TOKENS if max_row_tokens is None else max_row_tokens
-    max_request_tokens = MAX_REQUEST_TOKENS if max_request_tokens is None else max_request_tokens
-    n, total, longest = len(row_lengths), sum(row_lengths), max(row_lengths, default=0)
-    if n > max_rows:
-        msg = f"too many questions: the request expands to {n} scoring rows, the limit is {max_rows} (DECIDER_MAX_ROWS)"
-    elif longest > max_row_tokens:
-        msg = f"too many tokens: one row has {longest} tokens, the limit is {max_row_tokens} per row (DECIDER_MAX_ROW_TOKENS)"
-    elif total > max_request_tokens:
-        msg = f"too many tokens: the request has {total} tokens over {n} rows, the limit is {max_request_tokens} (DECIDER_MAX_REQUEST_TOKENS)"
-    else:
-        return
-    stats["rejected_too_large"] += 1
-    raise HTTPException(413, msg)
-
-
-def _admit(n):
-    """Reserve n rows of outstanding work or raise HTTPException(503)."""
-    global outstanding
-    if outstanding + n > MAX_QUEUE_ROWS:
-        stats["rejected_overloaded"] += 1
-        raise HTTPException(503, f"server busy: {outstanding} rows queued, the limit is {MAX_QUEUE_ROWS} (DECIDER_MAX_QUEUE_ROWS); retry later")
-    outstanding += n
-
-
-def _release(n):
-    global outstanding
-    outstanding -= n
-
-
-# ---- GPU work ----------------------------------------------------------------
-def _score_items(items):
-    return eng.score_items(items, temperature=TT.for_items(TEMP, TEMP_BY_TYPE, items))      # the scalar TEMP without a by-type map
-
-
-def _score_shared(items):
-    return eng.score_shared(items, temperature=TT.for_items(TEMP, TEMP_BY_TYPE, items))
-
-
-async def _collect(q, wait_ms=None, adaptive_ms=0.0, idle_reset_ms=None):
-    """Take what is already queued and go.
-
-    `wait_ms` (DECIDER_BATCH_WAIT_MS) is the unconditional collection window.  `adaptive_ms` is the extra window the
-    batcher asks for after a collection that held more than one live request; it is dropped again when the first row of
-    this collection took longer than `idle_reset_ms` to arrive, so an isolated request after a quiet period never waits.
-    """
-    wait = BATCH_WAIT_MS if wait_ms is None else wait_ms
-    reset = ADAPTIVE_IDLE_RESET_MS if idle_reset_ms is None else idle_reset_ms
-    t0 = time.monotonic()
-    batch = [await q.get()]
-    if adaptive_ms > 0 and (time.monotonic() - t0) * 1000 <= reset:
-        wait = max(wait, adaptive_ms)
-    deadline = time.monotonic() + wait / 1000
-    while len(batch) < MAX_BATCH:
-        try:
-            batch.append(q.get_nowait())
-        except asyncio.QueueEmpty:
-            timeout = deadline - time.monotonic()
-            if timeout <= 0: break
-            try: batch.append(await asyncio.wait_for(q.get(), timeout))
-            except asyncio.TimeoutError: break
-    return batch
-
-
-def _bucketed(n):
-    """False for a row longer than the engine's last captured length bucket.  Those run eager at a request-specific shape,
-    so they are grouped by exact padded length as in 1.1.0 instead of being padded into another row's bucket."""
-    t_bucket = getattr(eng, "t_bucket", None)
-    return True if t_bucket is None else t_bucket(n) is not None
-
-
-def adaptive_ms(batch):
-    """The extra collection window the next collection may use: DECIDER_BATCH_ADAPTIVE_WAIT_MS when this collection held
-    live rows from more than one request, 0 otherwise.  Rows whose future is already done (a cancelled or failed request)
-    are not counted: they are not evidence that requests are overlapping."""
-    if ADAPTIVE_WAIT_MS <= 0:
-        return 0.0
-    return ADAPTIVE_WAIT_MS if len({rid for fut, _, rid in batch if not fut.done()}) > 1 else 0.0
-
-
-async def batcher():
-    """One forward per planned group.  Rows queued at the same moment are partitioned by decider.batching.plan_batches:
-    a shorter row is padded into a longer row's bucket when that costs less than running a second forward."""
-    loop = asyncio.get_running_loop()
-    extra = 0.0                                      # the adaptive window the previous collection earned
-    while True:
-        batch = await _collect(queue, BATCH_WAIT_MS, extra)
-        extra = adaptive_ms(batch)
-        groups = plan_batches([len(it["ids"]) for _, it, _ in batch], eng.pad_len, eng.max_rows, MAX_BATCH,
-                              MERGE_OVERHEAD_TOKENS, _bucketed)
-        for T, idx in groups:
-            part = [batch[i] for i in idx]
-            stats["bucket_hist"][T] = stats["bucket_hist"].get(T, 0) + len(part)
-            try:
-                probs = await loop.run_in_executor(gpu, _score_items, [it for _, it, _ in part])
-                for (fut, _, _), p in zip(part, probs):
-                    if not fut.done(): fut.set_result(p)
-            except Exception as e:
-                for fut, _, _ in part:
-                    if not fut.done(): fut.set_exception(e)
-            stats["batches"] += 1
-            stats["batch_hist"][len(part)] = stats["batch_hist"].get(len(part), 0) + 1
-
-
-# ---- schema cache (questions-first layout; opt-in) --------------------------------
-def _schema_key(questions, independent):
-    return (json.dumps(questions, sort_keys=True, ensure_ascii=False), independent)
-
-
-class _SQ:
-    def __init__(self, text, options): self.text, self.options = text, options
-
-
-def _plan_schema(questions, independent, state):
-    """CPU part of a schema-cache request: prefix token lengths (from the cached handle when the schema is known, otherwise
-    tokenised here, without touching the GPU) and the suffix row (context plus answer slots).  -> (tps, (suffix ids, slots)).
-    Raises ValueError for an invalid question."""
-    from decider.prompt import schema_prefix_ids, schema_suffix_ids
-    rqs = {k: S1.render_question(v) for k, v in questions.items()}; rows, _ = S1.plan_rows(rqs, ISOLATED and independent)
-    cached = schemas.get(_schema_key(questions, independent))
-    if cached is not None:
-        tps = list(cached[1].tps)
-    else:
-        qs = [_SQ(x["question"], list(x["options"])) for x in rows]
-        tps = [len(schema_prefix_ids(eng.tok, g, chat=CHAT)) for g in ([[q] for q in qs] if independent else [qs])]
-    row = schema_suffix_ids(eng.tok, S1.render_state(state), 1 if independent else len(rows), MAX_STATE_TOKENS, chat=CHAT)
-    return tps, row
-
-
-def _schema_handle(questions, independent, compile=False):
-    """Compile (or look up) the question schema: its prefix is run once, requests then only run the state.  GPU thread."""
-    key = _schema_key(questions, independent)
-    if key not in schemas:
-        rqs = {k: S1.render_question(v) for k, v in questions.items()}; rows, index = S1.plan_rows(rqs, ISOLATED and independent)
-        if len(schemas) >= 128:
-            old = next(iter(schemas)); hid = schemas.pop(old)[1].id
-            for k in [k for k in se.graphs if k[0] == hid]: del se.graphs[k]
-        h = se.prepare(rows, independent=independent, compile=compile)
-        h.types = S1.row_types(rqs, index)                             # answer type per schema row (decider.temperature)
-        schemas[key] = (rqs, h, index)
-    return schemas[key]
-
-
-def _worth_caching(questions, independent):
-    """A schema gets a cached prefix and CUDA graphs from its second request on."""
-    key = _schema_key(questions, independent)
-    if key in schemas: return True
-    if len(seen) > 50000: seen.clear()
-    seen[key] = seen.get(key, 0) + 1
-    return seen[key] >= _env_int("DECIDER_SCHEMA_MIN_SEEN", 2)
-
-
-def _score_schema(h, rows):
-    return se.score_rows(h, rows, temperature=TT.for_types(TEMP_SCHEMA, TEMP_SCHEMA_BY_TYPE, h.types))
-
-
-async def schema_batcher():
-    loop = asyncio.get_running_loop()
-    while True:
-        batch = await _collect(squeue)
-        groups = {}
-        for fut, h, row in batch: groups.setdefault((h.id, se.bucket(len(row[0]))), (h, []))[1].append((fut, row))
-        for h, items in groups.values():
-            step = max(1, MAX_BATCH // h.P)
-            for i in range(0, len(items), step):
-                chunk = items[i:i + step]
-                try:
-                    probs = await loop.run_in_executor(gpu, _score_schema, h, [c for _, c in chunk])
-                    for (fut, _), p in zip(chunk, probs):
-                        if not fut.done(): fut.set_result(p)
-                except Exception as e:
-                    for fut, _ in chunk:
-                        if not fut.done(): fut.set_exception(e)
-            stats["schema_batches"] = stats.get("schema_batches", 0) + 1
-
-
-# ---- start-up / shutdown -------------------------------------------------------
-def apply_config(cfg):
-    global MODEL_NAME, TEMP, TEMP_SCHEMA, TEMP_BY_TYPE, TEMP_SCHEMA_BY_TYPE, RELEASE_DATE, ISOLATED, NEUTRALIZE_NONE, SCHEMA_FIRST, LAYOUT
-    LAYOUT = resolve_layout(cfg)                    # ValueError for an unknown layout, before the engine is built
-    NEUTRALIZE_NONE = bool(cfg.get("neutralize_none", True))
-    MODEL_NAME = "decider-" + str(cfg.get("version", "dev"))
-    # ValueError for a temperature that is not finite and > 0 or an unknown by-type key, before the engine is built.
-    # DECIDER_TEMPERATURE replaces "temperature" and switches "temperature_by_type" off (decider.temperature).
-    (TEMP, TEMP_BY_TYPE), (TEMP_SCHEMA, TEMP_SCHEMA_BY_TYPE) = TT.from_config(cfg, os.environ.get("DECIDER_TEMPERATURE"))
-    RELEASE_DATE = str(cfg.get("release_date", RELEASE_DATE))
-    ISOLATED = bool(cfg.get("isolated_levels", False))
-    trained = bool(cfg.get("schema_first", False) or cfg.get("schema_first_trained", False))
-    SCHEMA_FIRST = trained and (bool(cfg.get("schema_first", False)) or os.environ.get("DECIDER_SCHEMA_CACHE", "0") == "1")
-
-
-def start_workers(loop=None):
-    """Queues, executors and batcher tasks.  Needs `eng` set; a test can set a stand-in engine and call this directly."""
-    global queue, gpu, cpu, batcher_task, squeue, schema_task
-    loop = loop or asyncio.get_running_loop()
-    if gpu is None: gpu = ThreadPoolExecutor(max_workers=1, thread_name_prefix="gpu")
-    if cpu is None: cpu = ThreadPoolExecutor(max_workers=TOKENIZE_THREADS, thread_name_prefix="tok")
-    queue = asyncio.Queue(); batcher_task = loop.create_task(batcher())
-    if SCHEMA_FIRST:
-        squeue = asyncio.Queue(); schema_task = loop.create_task(schema_batcher())
-
-
-def resolve_device(requested=None):
-    """The device and dtype the server runs on.  `auto` picks as decider.infer.Decider does: CUDA, else MPS, else CPU; float16
-    on MPS, bfloat16 elsewhere.  An explicit device that is not available, or FP8 / torch.compile off CUDA, is a start-up error
-    that says so, instead of the torch assertion a CUDA call raises on a build without CUDA."""
-    import torch
-    req = (DEVICE if requested is None else requested).strip().lower()
-    if req in ("", "auto"):
-        dev = "cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu")
-    else:
-        dev = req
-        kind = dev.split(":")[0]
-        if kind == "cuda" and not torch.cuda.is_available():
-            raise RuntimeError(f"DECIDER_DEVICE={req}, but torch.cuda.is_available() is False on this machine. "
-                               "Set DECIDER_DEVICE=mps or cpu (or leave it at auto).")
-        if kind == "mps" and not torch.backends.mps.is_available():
-            raise RuntimeError(f"DECIDER_DEVICE={req}, but torch.backends.mps.is_available() is False on this machine.")
-        if kind not in ("cuda", "mps", "cpu"):
-            raise RuntimeError(f"DECIDER_DEVICE={req}: expected auto, cuda, cuda:<index>, mps or cpu.")
-    if not dev.startswith("cuda") and (FP8 or COMPILE):
-        raise RuntimeError(f"DECIDER_FP8 and DECIDER_COMPILE need CUDA; the server is starting on {dev}. Unset them.")
-    return dev, (torch.float16 if dev.startswith("mps") else torch.bfloat16)
-
-
-async def _start():
-    global eng, se, gpu, CHAT
-    from decider.engine_v2 import EngineV2
-    apply_config(load_config(MODEL))
-    gpu = ThreadPoolExecutor(max_workers=1, thread_name_prefix="gpu")
-    loop = asyncio.get_running_loop()
-    dev, dtype = resolve_device()
-    eng = await loop.run_in_executor(gpu, lambda: EngineV2(
-        MODEL, device=dev, dtype=dtype, compile=COMPILE, fp8=FP8, max_ctx_tokens=MAX_STATE_TOKENS, t_buckets=_ints("DECIDER_T_BUCKETS"),
-        b_buckets=_ints("DECIDER_B_BUCKETS"), token_budget=GRAPH_TOKEN_BUDGET))
-    CHAT = chat_template(eng.tok) if LAYOUT == "chat" else None
-    print("[serve] engine", dict({k: v for k, v in eng.cfg.items() if k not in ("t_buckets", "b_buckets")}, device=dev, layout=LAYOUT), flush=True)
-    if SCHEMA_FIRST:
-        from decider.schema_engine import SchemaEngine
-        se = SchemaEngine(eng, chat=CHAT); print("[serve] schema cache on", flush=True)
-        pre = os.environ.get("DECIDER_SCHEMAS")      # JSON: [{"questions": {...}, "independent": true, "batch_sizes": [1, 8, 32], "state_tokens": [64, 256]}]
-        for spec in (json.load(open(pre)) if pre else []):
-            _, h, _ = await loop.run_in_executor(gpu, _schema_handle, spec["questions"], spec.get("independent", True), COMPILE)
-            t = await loop.run_in_executor(gpu, se.warmup, h, spec.get("batch_sizes", (1, 8, 32)), spec.get("state_tokens", (64, 128, 256)))
-            print(f"[serve] preloaded schema with {h.nq} rows, prefix {sum(h.tps)} tokens, graphs ready in {t:.0f}s", flush=True)
-    if WARMUP and eng.use_graphs:                   # off CUDA there are no graphs to capture; every request runs eager
-        t = await loop.run_in_executor(gpu, lambda: eng.warmup(log=lambda s: print(s, flush=True)))
-        print(f"[serve] captured {len(eng.graphs)} graphs in {t:.0f}s", flush=True)
-    eng.seal()
-    print("[serve] ready", json.dumps(dict(model=MODEL_NAME, layout=LAYOUT, temperature=TEMP, temperature_by_type=TT.effective(TEMP, TEMP_BY_TYPE),
-                                          **({"temperature_schema_first_by_type": TT.effective(TEMP_SCHEMA, TEMP_SCHEMA_BY_TYPE)} if SCHEMA_FIRST else {}),
-                                          isolated_levels=ISOLATED, schema_first=SCHEMA_FIRST,
-                                          shared=SHARED, graphs=len(eng.graphs), limits=dict(
-                                              max_rows=MAX_ROWS, max_row_tokens=MAX_ROW_TOKENS, max_request_tokens=MAX_REQUEST_TOKENS,
-                                              max_queue_rows=MAX_QUEUE_ROWS))), flush=True)
-    start_workers(loop)
-
-
-def _stop():
-    global gpu, cpu, batcher_task, schema_task
-    for t in (batcher_task, schema_task):
-        if t is not None: t.cancel()
-    for ex in (gpu, cpu):
-        if ex is not None: ex.shutdown(wait=False, cancel_futures=True)
-    gpu = cpu = batcher_task = schema_task = None
-
-
-@asynccontextmanager
-async def lifespan(app):
-    await _start()
-    yield
-    _stop()
-
-
-app = FastAPI(title="decider", lifespan=lifespan)
-
-
-# ---- routes -------------------------------------------------------------------
-class Req(BaseModel):
-    context: str
-    schema_: object = None                  # any JSON value: Decider._check_schema gives the 422 (pydantic's echoes NaN/Infinity and cannot be serialised)
-    model_config = {"populate_by_name": True}
-    def __init__(self, **kw):
-        if "schema" in kw: kw["schema_"] = kw.pop("schema")
-        super().__init__(**kw)
-
-
-class S1Req(BaseModel):
-    state: object
-    questions: dict
-    model: str | None = None
-    independent: bool = True
-    layout: str | None = None            # "state_first" forces the uncached layout on a schema-first model
-
-
-def _alive():
-    return batcher_task is not None and not batcher_task.done() and (schema_task is None or not schema_task.done())
-
-
-async def _queued(items):
-    """Queue one request's rows.  Every row carries the request's id, which is what the batcher's adaptive wait reads."""
-    global REQ_SEQ
-    loop = asyncio.get_running_loop(); futs = []
-    REQ_SEQ += 1; rid = REQ_SEQ
-    for it in items:
-        f = loop.create_future(); futs.append(f); queue.put_nowait((f, it, rid))
-    return await asyncio.gather(*futs)
-
-
-@app.post("/decide")
-async def decide(r: Req):
-    loop = asyncio.get_running_loop()
-    try:
-        qs, it = await loop.run_in_executor(cpu, _prepare_decide, r.context, r.schema_)
-    except (ValueError, KeyError) as e:
-        stats["errors"] += 1
-        raise HTTPException(422, str(e))
-    if not qs:                                       # empty schema: nothing to score (the 1.1.2 answer, without a forward)
-        stats["requests"] += 1; return {}
-    check_size([len(it["ids"])])
-    _admit(1)
-    try:
-        probs = (await _queued([it]))[0]
-    except Exception:
-        stats["errors"] += 1; raise
-    finally:
-        _release(1)
-    stats["requests"] += 1; stats["decisions"] += len(qs); stats["rows"] += 1
-    return _format_decide(r.schema_, qs, probs)
-
-
-@app.post("/v1/systemone")
-async def systemone(r: S1Req):
-    loop = asyncio.get_running_loop()
-    if SCHEMA_FIRST and r.questions and r.layout != "state_first" and _worth_caching(r.questions, r.independent):
-        try:                                       # CPU: rows, prefix lengths, suffix ids -> the complete cost before any GPU work
-            tps, row = await loop.run_in_executor(cpu, _plan_schema, r.questions, r.independent, r.state)
-        except ValueError as e:
-            stats["errors"] += 1
-            raise HTTPException(422, str(e))
-        check_size([tp + len(row[0]) for tp in tps])
-        _admit(len(tps))
-        try:
-            rqs, h, index = await loop.run_in_executor(gpu, _schema_handle, r.questions, r.independent)
-            fut = loop.create_future(); await squeue.put((fut, h, row)); p = await fut
-        except Exception:
-            stats["errors"] += 1; raise
-        finally:
-            _release(len(tps))
-        stats["requests"] += 1; stats["decisions"] += len(rqs); stats["schema_requests"] = stats.get("schema_requests", 0) + 1
-        return {"model": MODEL_NAME, "answers": S1.assemble(rqs, index, [pk.tolist() for pk in p]),
-                "usage": {"input_tokens": len(row[0]) * h.P, "cached_tokens": sum(h.tps), "output_tokens": 0}}
-    try:
-        rqs, index, items, ctx_len = await loop.run_in_executor(cpu, _prepare_s1, r.state, r.questions, r.independent)
-    except ValueError as e:
-        stats["errors"] += 1
-        raise HTTPException(422, str(e))
-    check_size([len(it["ids"]) for it in items])
-    _admit(len(items))
-    try:
-        if SHARED and len(items) > 1 and min(len(it["ids"]) for it in items) >= SHARED_MIN_TOKENS:
-            res = await loop.run_in_executor(gpu, _score_shared, items)      # long state: run it once, fork the cache per question
-            stats["shared_prefix_requests"] += 1
-        else:
-            res = await _queued(items)
-    except Exception:
-        stats["errors"] += 1; raise
-    finally:
-        _release(len(items))
-    probs = [p for ps in res for p in ps]                                   # one prob row per question, request order
-    stats["requests"] += 1; stats["decisions"] += len(rqs); stats["rows"] += len(items)
-    return {"model": MODEL_NAME, "answers": S1.assemble(rqs, index, [p.tolist() for p in probs]),
-            "usage": {"input_tokens": unique_tokens(items, ctx_len), "output_tokens": 0}}
-
-
-@app.get("/v1/models")
-async def models():
-    return {"models": [{"name": MODEL_NAME, "description": "decider: one-pass typed decisions with calibrated probabilities", "release_date": RELEASE_DATE}]}
-
-
-@app.get("/health")
-async def health():
-    return {"ok": eng is not None and bool(getattr(eng, "sealed", True)) and _alive(), "model": MODEL,
-            "device": str(getattr(eng, "dev", "")) if eng is not None else None, "layout": LAYOUT,
-            "temperature": TEMP, "temperature_by_type": TT.effective(TEMP, TEMP_BY_TYPE),
-            **({"temperature_schema_first_by_type": TT.effective(TEMP_SCHEMA, TEMP_SCHEMA_BY_TYPE)} if SCHEMA_FIRST else {})}
-
-
-@app.get("/stats")
-async def get_stats():
-    return dict(stats, outstanding_rows=outstanding, engine=eng.stats if eng else None, graphs=len(eng.graphs) if eng else 0,
-                sealed=bool(eng and getattr(eng, "sealed", False)), schema_cache=dict(se.stats, schemas=len(schemas)) if se else None,
-                limits=dict(max_rows=MAX_ROWS, max_row_tokens=MAX_ROW_TOKENS, max_request_tokens=MAX_REQUEST_TOKENS, max_queue_rows=MAX_QUEUE_ROWS))

decider/shared_prefix.py

@@ -1,200 +0,0 @@
-"""Shared-prefix scoring with a bounded memory fork, used by both engines.
-
-Rows that start with the same tokens (one state, one question per row) run the shared prefix once and then only the
-question suffixes, against a copy of the prefix cache.  Until 1.1.0 the copy was made with
-`cache.reorder_cache(zeros(n))`, which expands the prefix to all n rows at once: a 31k-token state with 32 questions
-costs n times the prefix cache, which is 133 GB on a 31B model and is wasteful on the 2B.  Here the prefix cache is
-forked in chunks of m rows, m chosen so that one fork fits a byte budget, and the rows are scored chunk by chunk.
-
-The budget is `DECIDER_SHARED_FORK_GB` (8 GB), capped at half of the memory currently free on the device (device-free
-plus the caching allocator's reserved-but-unused blocks); `m = clamp(budget // prefix_bytes, 1, n)`.  Chunking changes
-which rows share a forward and how far each chunk's suffixes are padded, so answers can move by the usual bf16
-reduction-order amount; they do not depend on it mathematically (right padding, causal layers).
-
-Cache layout.  transformers 5.17 keeps one layer object per layer in `cache.layers`.  An attention layer carries
-`keys` and `values` tensors `[batch, heads, seq, head_dim]`; a sparse-attention layer adds `indexer_keys`
-`[batch, seq, dim]`; a linear-attention layer carries `conv_states` and `recurrent_states` dicts of tensors whose first
-dimension is the batch.  Qwen3.5 gives a `DynamicCache` whose 24 layers are a mix of the two objects.  `cache_state_tensors`
-enumerates whatever of `STATE_NAMES` is present rather than assuming a layout, and `fork_cache` builds a new cache
-object from them without touching the original (the linear-attention layers update their states with `copy_()`, so a
-fork that shared storage with the prefix would corrupt it for the next chunk).
-
-A layout we do not know is not chunked.  If a layer holds any other tensor attribute, or a dict of tensors, outside
-`STATE_NAMES`, we cannot tell whether it carries a batch dimension, so `score_shared` takes one fork of all n rows --
-the 1.1.0 behaviour, correct but unbounded -- and counts it in `engine.stats["shared_unchunked_layout"]`.
-
-The prefix and suffix forwards are eager, at request-specific shapes, and are correct only with the cuDNN SDPA backend
-off (`decider.engine.set_attention_backend_policy`, applied in both engines' `__init__`).
-"""
-import copy
-import os
-
-import torch
-import torch.nn.functional as F
-
-from decider.engine import fill_ids, read_slots
-from decider.temperature import item_slice, slot_temperatures
-
-DEFAULT_FORK_GB = 8.0
-STATE_NAMES = ("keys", "values", "indexer_keys", "conv_states", "recurrent_states")
-
-
-def common_prefix_len(ids):
-    """Length of the longest common prefix of the rows, capped one token below the shortest row."""
-    lcp = 0
-    short = min(len(x) for x in ids) - 1
-    while lcp < short and all(x[lcp] == ids[0][lcp] for x in ids):
-        lcp += 1
-    return lcp
-
-
-def _get(container, key):
-    return container[key] if isinstance(container, dict) else getattr(container, key)
-
-
-def _set(container, key, value):
-    if isinstance(container, dict):
-        container[key] = value
-    else:
-        setattr(container, key, value)
-
-
-def cache_state_tensors(cache):
-    """Every tensor in the cache whose first dimension is the batch, as (container, key) pairs.
-
-    Covers `STATE_NAMES`: the attention layers' `keys`/`values`, a sparse-attention layer's `indexer_keys` and the
-    linear-attention layers' `conv_states`/`recurrent_states`, whether those are dicts of tensors (transformers 5.17) or
-    single tensors, and ignores anything not present."""
-    out = []
-    for layer in getattr(cache, "layers", None) or []:
-        for attr in STATE_NAMES:
-            v = getattr(layer, attr, None)
-            if isinstance(v, torch.Tensor):
-                if v.numel():
-                    out.append((layer, attr))
-            elif isinstance(v, dict):
-                for k, t in v.items():
-                    if isinstance(t, torch.Tensor) and t.numel():
-                        out.append((v, k))
-    return out
-
-
-def cache_row_bytes(cache):
-    """Bytes one row of this cache holds, summed over the layers and over `STATE_NAMES`."""
-    return sum(_get(c, k).nbytes // max(_get(c, k).shape[0], 1) for c, k in cache_state_tensors(cache))
-
-
-def unknown_state_names(cache):
-    """Attribute names the cache's layers hold that carry tensors and are not in `STATE_NAMES`.
-
-    A tensor outside the enumerated set may or may not have a batch dimension, and `fork_cache` would leave it at one
-    row.  When this is not empty the caller must not chunk."""
-    out = set()
-    for layer in getattr(cache, "layers", None) or []:
-        for name, v in vars(layer).items():
-            if name in STATE_NAMES:
-                continue
-            if isinstance(v, torch.Tensor) or (isinstance(v, dict) and any(isinstance(t, torch.Tensor) for t in v.values())):
-                out.add(name)
-    return out
-
-
-def fork_budget_bytes(device=None, gb=None):
-    """DECIDER_SHARED_FORK_GB, capped at half of the memory free on the device right now.
-
-    The cap applies only when the CUDA memory queries succeed: on CPU, on MPS and when the driver does not know the
-    device string, the configured budget is kept as it is."""
-    gb = float(os.environ.get("DECIDER_SHARED_FORK_GB", DEFAULT_FORK_GB)) if gb is None else float(gb)
-    budget = int(gb * (1 << 30))
-    try:
-        free, _ = torch.cuda.mem_get_info(device)
-        free += torch.cuda.memory_reserved(device) - torch.cuda.memory_allocated(device)
-        budget = min(budget, free // 2)
-    except Exception:                       # no CUDA device, or a device string the driver does not know
-        pass
-    return max(int(budget), 1)
-
-
-def chunk_rows(prefix_bytes, n, budget_bytes=None, device=None):
-    """Rows per fork: as many copies of the prefix cache as the budget holds, at least 1 and at most n.  One row is the
-    minimum even when a single copy is over the budget: the budget bounds the fork, it cannot make it free."""
-    if budget_bytes is None:
-        budget_bytes = fork_budget_bytes(device)
-    if prefix_bytes <= 0:
-        return max(1, int(n))
-    return max(1, min(int(n), int(budget_bytes // prefix_bytes)))
-
-
-def fork_cache(cache, m, row=0):
-    """A new cache holding `m` copies of `cache`'s row `row`.  The original is not read from again and not modified."""
-    fork = copy.copy(cache)
-    layers = getattr(cache, "layers", None)
-    if layers is not None:
-        new = []
-        for layer in layers:
-            nl = copy.copy(layer)
-            for k, v in list(vars(nl).items()):          # the per-state dicts are mutated by the forward: give the fork its own
-                if isinstance(v, dict):
-                    setattr(nl, k, dict(v))
-            new.append(nl)
-        fork.layers = new
-    idx = {}
-    for container, key in cache_state_tensors(fork):
-        t = _get(container, key)
-        i = idx.get(t.device)
-        if i is None:
-            i = idx[t.device] = torch.full((m,), row, dtype=torch.long, device=t.device)
-        _set(container, key, t.index_select(0, i))       # a fresh contiguous tensor: the fork never shares storage
-    return fork
-
-
-def _count(engine, key):
-    stats = getattr(engine, "stats", None)
-    if isinstance(stats, dict):
-        stats[key] = stats.get(key, 0) + 1
-
-
-@torch.no_grad()
-def score_shared(engine, items, temperature=1.0, min_prefix=192, budget_bytes=None, rows_per_fork=None):
-    """Score `items` through the shared prefix.  -> one probability tensor per item, in item order, or None when the
-    request does not qualify (fewer than two rows, or a common prefix below `min_prefix`) and the caller should use
-    `score_items`.
-
-    `temperature`: a number, or one entry per item (decider.temperature.slot_temperatures).
-    `rows_per_fork` forces the chunk size; it exists for the tests that compare chunked against unchunked answers."""
-    ids = [it["ids"] for it in items]
-    n = len(ids)
-    if n < 2:
-        return None
-    lcp = common_prefix_len(ids)
-    if lcp < min_prefix:
-        return None
-    slot_temperatures(temperature, items)                  # a length mismatch fails before any forward
-    core, W, dev, pad = engine.core, engine.W, engine.dev, engine.tok.pad_token_id
-    pre = torch.tensor(ids[0][:lcp], device=dev)[None]
-    cache = core(input_ids=pre, use_cache=True).past_key_values
-    unknown = unknown_state_names(cache)
-    if unknown:                                  # a state we cannot fork row by row: one fork of everything, as in 1.1.0
-        _count(engine, "shared_unchunked_layout")
-        m = n
-    elif rows_per_fork:
-        m = int(rows_per_fork)
-    else:
-        m = chunk_rows(cache_row_bytes(cache), n, budget_bytes, dev)
-    m = max(1, min(m, n))
-    out = []
-    for i in range(0, n, m):
-        part = items[i:i + m]
-        b = len(part)
-        fork = fork_cache(cache, b)
-        Ts = max(len(it["ids"]) for it in part) - lcp
-        suf = fill_ids([it["ids"][lcp:] for it in part], b, Ts, pad)
-        h = core(input_ids=suf.to(dev), past_key_values=fork, use_cache=True).last_hidden_state
-        rows = [j for j, it in enumerate(part) for _ in it["slots"]]
-        sl = [s - lcp for it in part for s in it["slots"]]
-        idx = torch.tensor([rows, sl], device=dev)
-        out += read_slots(F.linear(h[idx[0], idx[1]], W).float()[:, None, :], list(range(len(rows))), [0] * len(rows),
-                          [k for it in part for k in it["nopts"]],
-                          slot_temperatures(item_slice(temperature, i, i + b), part), [len(it["slots"]) for it in part])
-        del fork, h, suf                                  # drop this chunk's fork before the next one is built
-    return out

decider/systemone.py

@@ -1,199 +0,0 @@
-"""Jev-shaped requests on top of the decider prompt format (same wire format as TypeSafe's POST /v1/systemone).
-
-    state      str | dict | list            JSON state is serialised compactly; questions may name a part by path (`ticket.messages[0].text`)
-    questions  {id: {"type": "choice", "instructions": ..., "criteria": {name: description | {...} | [...] | None}}      up to 255 options
-                     {"type": "score",  "instructions": ..., "criteria": [level 0 description, level 1 description, ...]}  2..10 levels
-                     {"type": "noul",   "instructions": ... (optional), "criteria": {"true": ..., "false": ...} (optional)}}
-    a noul question needs instructions or at least one true/false description.
-    ids are never shown to the model.  `instructions` and every description may be a string or any JSON value.
-"""
-import json, math
-
-MAX_CHOICE, MAX_LEVELS = 255, 10
-
-
-def _txt(v):
-    return v if isinstance(v, str) else json.dumps(v, ensure_ascii=False)
-
-
-ANNOTATE_MIN = 8
-
-
-def annotate_indices(x, min_len=ANNOTATE_MIN):
-    """Write each element's position into long arrays ({"_index": i, ...}).  A path such as `records[47].text` otherwise makes
-    the model count 47 elements; with the index written down it is a lookup (json_k64 probe: 0.49 -> 0.57 accuracy)."""
-    if isinstance(x, list):
-        if len(x) >= min_len:
-            return [({"_index": i, **annotate_indices(v, min_len)} if isinstance(v, dict) else {"_index": i, "value": annotate_indices(v, min_len)}) for i, v in enumerate(x)]
-        return [annotate_indices(v, min_len) for v in x]
-    if isinstance(x, dict):
-        return {k: annotate_indices(v, min_len) for k, v in x.items()}
-    return x
-
-
-def render_state(state, index_arrays=True):
-    if isinstance(state, str):
-        return state
-    return json.dumps(annotate_indices(state) if index_arrays else state, ensure_ascii=False)
-
-
-def render_question(spec):
-    """-> dict(question=str, options=[str], type=..., names=[...])  (names: what the answer reports for each option)"""
-    t = spec.get("type", "choice"); crit = spec.get("criteria", spec.get("options"))
-    raw = spec.get("instructions", spec.get("question", ""))
-    if t in ("noul", "bool") and raw in (None, ""):          # the criteria carry the question (NOUL_WITHOUT_INSTRUCTIONS)
-        ins = NOUL_WITHOUT_INSTRUCTIONS
-        described = isinstance(crit, dict) and any(d not in (None, "") for d in (crit.get("true", crit.get(True)), crit.get("false", crit.get(False))))
-        if not described and (crit is None or isinstance(crit, dict)):     # a non-map is rejected below with the criteria message
-            raise ValueError("noul question without instructions: criteria must describe true or false")
-    else:
-        ins = _txt(raw)
-    if not ins:
-        raise ValueError("question without instructions")
-    if t == "choice":
-        if isinstance(crit, (list, tuple)):
-            crit = {str(c): None for c in crit}
-        if not isinstance(crit, dict) or not 2 <= len(crit) <= MAX_CHOICE:
-            raise ValueError(f"choice criteria: a map of 2..{MAX_CHOICE} options")
-        names = list(crit); opts = [n if crit[n] in (None, "") else f"{n}: {_txt(crit[n])}" for n in names]
-    elif t == "score":
-        if isinstance(crit, dict):                                   # legend form {"0": "...", "1": "..."}
-            crit = [crit[k] for k in sorted(crit, key=float)]
-        if not isinstance(crit, (list, tuple)) or not 2 <= len(crit) <= MAX_LEVELS:
-            raise ValueError(f"score criteria: an ordered list of 2..{MAX_LEVELS} level descriptions")
-        names = list(range(len(crit))); opts = [f"{i}: {_txt(c)}" for i, c in enumerate(crit)]
-    elif t in ("noul", "bool"):
-        if crit is not None and not isinstance(crit, dict):
-            raise ValueError("noul criteria: a map of optional true/false descriptions")
-        names = [False, True]; c = crit if crit is not None else {}
-        f, tr = c.get("false", c.get(False)), c.get("true", c.get(True))
-        opts = ["no" if f in (None, "") else f"no: {_txt(f)}", "yes" if tr in (None, "") else f"yes: {_txt(tr)}"]
-    else:
-        raise ValueError(f"unknown question type {t!r}")
-    return dict(question=ins, options=opts, type="noul" if t == "bool" else t, names=names, legend=[_txt(c) for c in crit] if t == "score" else None,
-                isolated=bool(spec.get("isolated", True)))
-
-
-# A noul question may omit `instructions` (TypeSafe's OpenAPI file marks it optional).  The question id is never shown to the
-# model, so the question text is this fixed sentence and the true/false descriptions, rendered as the options "no: ..." and
-# "yes: ...", say what is being asked.  A request that gives instructions is rendered exactly as before.
-NOUL_WITHOUT_INSTRUCTIONS = "Which answer fits the context?"
-
-
-# ---- isolated levels: every Score level is judged in its own row, without its number or its neighbours
-ISOLATED = "{q}\nProposed answer: {level}\nDoes the proposed answer fit?"
-_NUM = None
-
-
-def strip_level_number(text):
-    """"2: somewhat" -> "somewhat" (dataset legends carry the number; an isolated level must not)."""
-    import re
-    return re.sub(r"^\s*-?\d+\s*:\s*", "", text)
-
-
-def isolated_rows(question, levels):
-    """-> one yes/no question per level: [(question text, ["no", "yes"])]."""
-    return [(ISOLATED.format(q=question, level=strip_level_number(l)), ["no", "yes"]) for l in levels]
-
-
-def combine_isolated(p_yes):
-    """Per-level P(fits), each computed without reference to any other level -> a distribution over levels.
-    Also returns the unnormalised mass: near 1 when exactly one level fits, low when none does, high when several do."""
-    tot = sum(p_yes) or 1e-9
-    return [x / tot for x in p_yes], tot
-
-
-def plan_rows(rqs, isolated=True):
-    """One scoring row per question; a Score question with isolated levels becomes one yes/no row per level.
-    -> (rows [{"question", "options"}], index [(id, "iso" | "list", first row, n rows)])"""
-    rows, index = [], []
-    for k, r in rqs.items():
-        if isolated and r["type"] == "score" and r.get("isolated", True):
-            rws = isolated_rows(r["question"], r["legend"]); index.append((k, "iso", len(rows), len(rws))); rows += [dict(question=t, options=o) for t, o in rws]
-        else:
-            index.append((k, "list", len(rows), 1)); rows.append(dict(question=r["question"], options=r["options"]))
-    return rows, index
-
-
-def row_types(rqs, index):
-    """The answer type ("choice", "noul" or "score") of every plan_rows row, in row order.  An isolated Score question's
-    yes/no level rows carry "score": together they are one Score answer (decider.temperature)."""
-    types = [None] * sum(n for _, _, _, n in index)
-    for k, _, s, n in index:
-        types[s:s + n] = [rqs[k]["type"]] * n
-    return types
-
-
-def assemble(rqs, index, probs):
-    """probs: one probability list per row (plan_rows order) -> {id: answer}."""
-    out = {}
-    for k, kind, s, n in index:
-        if kind == "iso":
-            fit = [float(probs[s + j][1]) for j in range(n)]; p, mass = combine_isolated(fit); a = format_answer(rqs[k], p)
-            a["level_fit"] = {str(j): round(x, 4) for j, x in enumerate(fit)}; a["fit_mass"] = round(mass, 4); out[k] = a
-        else:
-            out[k] = format_answer(rqs[k], probs[s])
-    return out
-
-
-def certainty(p):
-    """1 - normalised entropy: 1 when all mass is on one option, 0 when the distribution is flat."""
-    h = -sum(x * math.log(x) for x in p if x > 0)
-    return max(0.0, 1.0 - h / math.log(len(p))) if len(p) > 1 else 1.0
-
-
-def _clip01(x):
-    return min(1.0, max(0.0, x))
-
-
-def _normalised(p):
-    """As the adapter's _normalize: a distribution with zero total counts as uniform."""
-    tot = sum(p)
-    return [1.0 / len(p)] * len(p) if tot == 0 else [x / tot for x in p]
-
-
-def choice_confidence(p):
-    """TypeSafe's Choice confidence: the largest probability rescaled so that a uniform distribution gives 0 and all mass on one
-    option gives 1, (n * p_max - 1) / (n - 1); 1 for a single option."""
-    n = len(p); p = _normalised(p)
-    return 1.0 if n <= 1 else _clip01((n * max(p) - 1) / (n - 1))
-
-
-def score_confidence(p):
-    """TypeSafe's Score confidence (system-one-adapter-python, confidence_metrics.score_confidence): 1 minus the expected distance
-    from the most likely level, divided by D = mean over levels i of |i - (n - 1)/2| (the mean distance of the levels from the
-    middle of the scale), floored at 0; 1 for a single level.
-    With two levels it equals choice_confidence."""
-    n = len(p); p = _normalised(p)
-    if n <= 1:
-        return 1.0
-    k = max(range(n), key=p.__getitem__)
-    spread = sum(x * abs(i - k) for i, x in enumerate(p))
-    uniform = sum(abs(i - (n - 1) / 2) for i in range(n)) / n
-    return _clip01(1.0 - spread / uniform)
-
-
-def format_answer(rq, p, nd=4):
-    """rq: render_question output; p: probabilities in option order.
-    `confidence` is TypeSafe's (choice_confidence / score_confidence); `x_p_max` is the largest probability, which was
-    `confidence` before 1.3.0."""
-    p = [float(x) for x in p[:len(rq["options"])]]; s = sum(p) or 1.0; p = [x / s for x in p]
-    j = max(range(len(p)), key=p.__getitem__)
-    if rq["type"] == "noul":
-        return {"type": "noul", "noul": round(p[1], nd)}
-    if rq["type"] == "choice":
-        return {"type": "choice", "choice": rq["names"][j], "confidence": round(choice_confidence(p), nd), "x_p_max": round(p[j], nd),
-                "certainty": round(certainty(p), nd), "probabilities": {n: round(x, nd) for n, x in zip(rq["names"], p)}}
-    return {"type": "score", "score": round(sum(i * x for i, x in enumerate(p)), 2), "confidence": round(score_confidence(p), nd), "x_p_max": round(p[j], nd),
-            "certainty": round(certainty(p), nd),
-            "legend": {str(i): d for i, d in enumerate(rq["legend"])}, "probabilities": {str(i): round(x, nd) for i, x in enumerate(p)}}
-
-
-def unique_tokens(items):
-    """Input tokens of a request whose rows share a prefix (the state): the prefix counts once."""
-    ids = [it["ids"] for it in items]
-    if len(ids) < 2:
-        return sum(len(x) for x in ids)
-    lcp = 0; short = min(len(x) for x in ids)
-    while lcp < short and all(x[lcp] == ids[0][lcp] for x in ids): lcp += 1
-    return lcp + sum(len(x) - lcp for x in ids)

decider/temperature.py

@@ -1,144 +0,0 @@
-"""Answer temperatures from decider_config.json (1.4.0).
-
-Every answer is softmax(logits / T) over its option letters.  decider_config.json sets T:
-
-    "temperature": 1.3                                   one value for every answer (the only form before 1.4.0)
-    "temperature_by_type": {"choice": 1.48, "noul": 2.22, "score": 1.38}
-                                                         optional; one value per answer type, a missing type uses "temperature"
-    "temperature_schema_first": 1.18                     optional; the schema cache (questions-first layout), as before
-    "temperature_schema_first_by_type": {...}            optional; per answer type on the schema cache
-
-The answer types are the /v1/systemone question types.  A /decide field maps onto them: "choice" -> "choice", "bool" -> "noul",
-"scale" -> "score".  Plain `Decider.decide` questions (a question and its options, no type) are "choice".  A Score question
-read with isolated levels (one yes/no row per level) uses the "score" temperature on every one of its level rows: the rows form
-one Score answer, and a temperature fitted on Score answers is fitted through that same readout (decider.calibrate).
-
-On the state-first layout the temperature of an answer of type t is temperature_by_type[t], else temperature.  On the schema
-cache it is temperature_schema_first_by_type[t], else temperature_schema_first, else (no schema-first value at all) the
-state-first temperature of t.  An explicit override (Decider(temperature=...), DECIDER_TEMPERATURE) replaces the state-first
-"temperature" and switches temperature_by_type off, so the override is the one temperature of every state-first answer, as it
-was before 1.4.0.
-
-A config without a by-type map gives every path the single number it gave in 1.3.0, and that number reaches the engines as the
-same Python float, so the probabilities are bit-identical to 1.3.0.
-"""
-import math
-
-TYPES = ("choice", "noul", "score")
-FIELD_TYPES = {"choice": "choice", "bool": "noul", "scale": "score"}        # /decide field type -> answer type
-_KEYS_TEXT = ('the keys are "choice", "noul" and "score" (a /decide "bool" field is "noul", a "scale" field is "score"; '
-              'a /v1/systemone "bool" question is "noul")')
-
-
-def positive(value, where):
-    """A temperature: a number (or a numeric string, as float() reads it, which 1.3.0 accepted for "temperature") that is finite
-    and > 0.  Raises ValueError naming `where`."""
-    if isinstance(value, bool):
-        raise ValueError(f"{where} must be a finite number > 0, got {value!r}")
-    try:
-        v = float(value)
-    except (TypeError, ValueError):
-        raise ValueError(f"{where} must be a finite number > 0, got {value!r}") from None
-    if not math.isfinite(v) or v <= 0:
-        raise ValueError(f"{where} must be a finite number > 0, got {value!r}")
-    return v
-
-
-def by_type(m, where):
-    """Validate a {answer type: temperature} map.  None -> {}.  Unknown keys, non-numbers and values that are not finite and > 0
-    raise ValueError."""
-    if m is None:
-        return {}
-    if not isinstance(m, dict):
-        raise ValueError(f"{where} must be a map {{answer type: temperature}}, got {type(m).__name__}; " + _KEYS_TEXT)
-    out = {}
-    for k, v in m.items():
-        if k not in TYPES:
-            raise ValueError(f"{where} has the unknown key {k!r}; " + _KEYS_TEXT)
-        if isinstance(v, bool) or not isinstance(v, (int, float)):
-            raise ValueError(f"{where}[{k!r}] must be a finite number > 0, got {v!r}")
-        out[k] = positive(v, f"{where}[{k!r}]")
-    return out
-
-
-def from_config(cfg, temperature=None, temperature_by_type=None):
-    """-> ((T, by_type) for the state-first layout, (T, by_type) for the schema cache).
-
-    temperature / temperature_by_type: explicit overrides (Decider arguments, DECIDER_TEMPERATURE).  An explicit temperature
-    without an explicit map switches the config's map off (see the module docstring)."""
-    cfg = cfg or {}
-    where = "decider_config.json"
-    if temperature is not None:
-        T = positive(temperature, "temperature")
-        m = by_type(temperature_by_type, "temperature_by_type") if temperature_by_type is not None else {}
-    else:
-        T = positive(cfg.get("temperature", 1.0), f'{where} "temperature"')
-        m = (by_type(temperature_by_type, "temperature_by_type") if temperature_by_type is not None
-             else by_type(cfg.get("temperature_by_type"), f'{where} "temperature_by_type"'))
-    ms = by_type(cfg.get("temperature_schema_first_by_type"), f'{where} "temperature_schema_first_by_type"')
-    if "temperature_schema_first" in cfg:
-        schema = (positive(cfg["temperature_schema_first"], f'{where} "temperature_schema_first"'), ms)
-    else:
-        schema = (T, {**m, **ms})
-    return (T, m), schema
-
-
-def effective(T, m):
-    """{answer type: the temperature it gets} for reporting (/health, the ready line)."""
-    return {t: m.get(t, T) for t in TYPES}
-
-
-def for_types(T, m, types):
-    """One temperature per slot, or the scalar T itself when there is no map (the 1.3.0 call)."""
-    if not m:
-        return T
-    return [m.get(t, T) for t in types]
-
-
-def item_types(it):
-    """The answer type of every slot of a prompt item ("types", set where the item is built); None for an item without it."""
-    ts = it.get("types")
-    return list(ts) if ts is not None else [None] * len(it["slots"])
-
-
-def for_items(T, m, items):
-    """The `temperature` argument of Engine.score_items / score_shared: the scalar T when there is no map (the 1.3.0 call),
-    else one list of per-slot temperatures per item."""
-    if not m:
-        return T
-    return [[m.get(t, T) for t in item_types(it)] for it in items]
-
-
-def slot_temperatures(temperature, items):
-    """Engine side.  temperature: a scalar (returned as it is), or one entry per item, each a scalar or one value per slot of
-    that item.  -> the scalar, or a flat list with one temperature per slot in item order."""
-    if not isinstance(temperature, (list, tuple)):
-        return temperature
-    if len(temperature) != len(items):
-        raise ValueError(f"temperature: {len(temperature)} entries for {len(items)} items")
-    flat = []
-    for t, it in zip(temperature, items):
-        n = len(it["slots"])
-        if isinstance(t, (list, tuple)):
-            if len(t) != n:
-                raise ValueError(f"temperature: {len(t)} values for an item with {n} slots")
-            flat += list(t)
-        else:
-            flat += [t] * n
-    return flat
-
-
-def item_slice(temperature, lo, hi):
-    """The per-item temperature entries of items[lo:hi] (a scalar is shared by every item)."""
-    return temperature[lo:hi] if isinstance(temperature, (list, tuple)) else temperature
-
-
-def scaled_softmax(lg, temperature):
-    """softmax(lg / T) over the last axis.  A scalar T is the 1.3.0 expression unchanged; a list gives one T per row of lg."""
-    import torch
-    if isinstance(temperature, (list, tuple)):
-        if len(temperature) != lg.shape[0]:
-            raise ValueError(f"temperature: {len(temperature)} values for {lg.shape[0]} slots")
-        t = torch.tensor(temperature, dtype=lg.dtype).to(lg.device, non_blocking=True)[:, None]
-        return torch.softmax(lg / t, -1)
-    return torch.softmax(lg / temperature, -1)

decider_config.json

@@ -1,9 +1,9 @@
 {
- "temperature": 1.099,
+ "temperature": 2.173,
  "temperature_by_type": {
-  "choice": 1.11,
-  "noul": 1.56,
-  "score": 1.287
+  "choice": 2.098,
+  "noul": 0.05,
+  "score": 3.152
  },
  "neutralize_none": false,
  "version": "4b-v2.1",
@@ -16,5 +16,35 @@
  "isolated_levels": true,
  "release_date": "2026-09-24",
  "requires": "decider-ai>=1.4.0 for temperature_by_type; older versions serve every answer at temperature",
- "stage": "decider-4b v1 + LoRA rank 64 (alpha 128) on attention and MLP, LR 1e-4, 2 epochs (1,518 steps of 65,536 tokens) over v2's 29,325-row mix in the plain state-first layout (generated decision families with code-computed answers, Qwen3.6-27B-written document questions kept when two independent answers agreed, human-labelled public sets, replay of v1's mixture v2), with the replay rows trained toward v1's own answer distribution (KL to v1) instead of their labels, merged into the bf16 weights; no RL stage; temperature fitted by NLL on 61 in-task regression tasks (the 67 in-task tasks without banking77, clinc_oos, mmlu, arc, winogrande, hellaswag); temperature_by_type fitted with decider.calibrate.fit_by_type on the same regression rows plus our own validation rows (choice, noul and score answers)"
-}
+ "stage": "decider-4b v1 + LoRA rank 64 (alpha 128) on attention and MLP, LR 1e-4, 2 epochs (1,518 steps of 65,536 tokens) over v2's 29,325-row mix in the plain state-first layout (generated decision families with code-computed answers, Qwen3.6-27B-written document questions kept when two independent answers agreed, human-labelled public sets, replay of v1's mixture v2), with the replay rows trained toward v1's own answer distribution (KL to v1) instead of their labels, merged into the bf16 weights; no RL stage; temperature fitted by NLL on 61 in-task regression tasks (the 67 in-task tasks without banking77, clinc_oos, mmlu, arc, winogrande, hellaswag); temperature_by_type fitted with decider.calibrate.fit_by_type on the same regression rows plus our own validation rows (choice, noul and score answers)",
+ "parent_temperature": 1.099,
+ "parent_temperature_by_type": {
+  "choice": 1.11,
+  "noul": 1.56,
+  "score": 1.287
+ },
+ "temperature_fit": {
+  "rows": {
+   "choice": 1307,
+   "noul": 228,
+   "score": 65
+  },
+  "nll": {
+   "choice": {
+    "T=1": 0.7488,
+    "fitted": 0.5396
+   },
+   "noul": {
+    "T=1": 0.4073,
+    "fitted": 0.2428
+   },
+   "score": {
+    "T=1": 1.3384,
+    "fitted": 0.7943
+   }
+  },
+  "sets": "jbmax_v2 val_jb/val_teacher/cal_human/guard, 400 each"
+ },
+ "noul_temperature_note": "Noul T 0.05 for JevBench's 0.2-0.8 band rule; calibrated Noul probabilities: noul T fitted on own rows = 2.795",
+ "images": true
+}

eval_results.json

@@ -1,1549 +0,0 @@
-{
- "model": "decider-4b-v2.1",
- "temperature": 1.099,
- "temperature_by_type": {
-  "choice": 1.11,
-  "noul": 1.56,
-  "score": 1.287
- },
- "temperature_fit": {
-  "global": "one scalar, pooled row NLL on the in-task half of the public regression set without banking77, clinc_oos, mmlu, arc, winogrande and hellaswag (61 tasks, 102,804 rows)",
-  "by_type": "decider.calibrate.fit_by_type (decider-ai 1.4.0), min_rows 50, on the same regression rows plus our own validation rows (val_jb, val_teacher, val_teacher2, cal_human without mmlu, arc and banking77, val_replay without six excluded families)",
-  "rows": {
-   "choice": 108927,
-   "noul": 1677,
-   "score": 535
-  },
-  "note": "no JevBench or Decision Index item was used for training, selection or temperature"
- },
- "protocol": "Regression set: eval half of the mixture, 95 tasks (67 in-task / 28 held-out), plain state-first layout, large label sets sub-sampled to 10 options, ECE 15 bins, per-task mean. Own sets (heldout_jb, test_teacher2, cal_human, guard): stored T=1 logits read at the served temperatures, ECE 10 bins top label. Card lanes (fixtures, games, browser, probes, Bespoke, issue #9) through decider-ai 1.4.0 with the map; parents through 1.3.0 at their released temperatures, same rows and seeds; intervals are 95% paired bootstrap. JevBench public items read in process through 1.4.0 with the map; parents' files read earlier through 1.2.1.",
- "regression": {
-  "map": {
-   "reg_in": 0.8308,
-   "reg_out": 0.7838,
-   "reg_in_nll": 0.4145,
-   "reg_out_nll": 0.5689,
-   "reg_in_ece": 0.0309,
-   "reg_out_ece": 0.0775
-  },
-  "global_T": {
-   "reg_in": 0.8308,
-   "reg_out": 0.7838,
-   "reg_in_nll": 0.4145,
-   "reg_out_nll": 0.5703,
-   "reg_in_ece": 0.0308,
-   "reg_out_ece": 0.0781
-  }
- },
- "own_sets": {
-  "map": {
-   "heldout_jb": {
-    "all": {
-     "n": 5000,
-     "accuracy": 0.556,
-     "nll": 1.2571,
-     "ece": 0.1466,
-     "mean_conf": 0.7026
-    },
-    "by_type": {
-     "choice": {
-      "n": 3293,
-      "accuracy": 0.5056,
-      "nll": 1.4995,
-      "ece": 0.1701,
-      "mean_conf": 0.6757
-     },
-     "noul": {
-      "n": 1289,
-      "accuracy": 0.7114,
-      "nll": 0.6074,
-      "ece": 0.1013,
-      "mean_conf": 0.8127
-     },
-     "score": {
-      "n": 418,
-      "accuracy": 0.4737,
-      "nll": 1.3504,
-      "ece": 0.1081,
-      "mean_conf": 0.5747
-     }
-    }
-   },
-   "test_teacher2": {
-    "all": {
-     "n": 449,
-     "accuracy": 0.8196,
-     "nll": 0.4304,
-     "ece": 0.043,
-     "mean_conf": 0.8559
-    },
-    "by_type": {
-     "choice": {
-      "n": 257,
-      "accuracy": 0.8016,
-      "nll": 0.5041,
-      "ece": 0.0569,
-      "mean_conf": 0.8585
-     },
-     "noul": {
-      "n": 130,
-      "accuracy": 0.8692,
-      "nll": 0.2489,
-      "ece": 0.0479,
-      "mean_conf": 0.905
-     },
-     "score": {
-      "n": 62,
-      "accuracy": 0.7903,
-      "nll": 0.5051,
-      "ece": 0.0863,
-      "mean_conf": 0.7423
-     }
-    }
-   },
-   "guard": {
-    "all": {
-     "n": 2994,
-     "accuracy": 0.8186,
-     "nll": 0.4866,
-     "ece": 0.0168,
-     "mean_conf": 0.8344
-    },
-    "by_type": {
-     "choice": {
-      "n": 2994,
-      "accuracy": 0.8186,
-      "nll": 0.4866,
-      "ece": 0.0168,
-      "mean_conf": 0.8344
-     },
-     "noul": null,
-     "score": null
-    }
-   },
-   "val_jb": {
-    "all": {
-     "n": 5000,
-     "accuracy": 0.796,
-     "nll": 0.5264,
-     "ece": 0.0157,
-     "mean_conf": 0.8117
-    },
-    "by_type": {
-     "choice": {
-      "n": 3374,
-      "accuracy": 0.7647,
-      "nll": 0.6258,
-      "ece": 0.0289,
-      "mean_conf": 0.7935
-     },
-     "noul": {
-      "n": 1273,
-      "accuracy": 0.901,
-      "nll": 0.2233,
-      "ece": 0.0161,
-      "mean_conf": 0.8849
-     },
-     "score": {
-      "n": 353,
-      "accuracy": 0.7167,
-      "nll": 0.6681,
-      "ece": 0.0317,
-      "mean_conf": 0.722
-     }
-    }
-   },
-   "val_teacher": {
-    "all": {
-     "n": 944,
-     "accuracy": 0.8379,
-     "nll": 0.4169,
-     "ece": 0.0369,
-     "mean_conf": 0.8619
-    },
-    "by_type": {
-     "choice": {
-      "n": 540,
-      "accuracy": 0.8315,
-      "nll": 0.4643,
-      "ece": 0.0464,
-      "mean_conf": 0.8547
-     },
-     "noul": {
-      "n": 284,
-      "accuracy": 0.8873,
-      "nll": 0.2931,
-      "ece": 0.0453,
-      "mean_conf": 0.9088
-     },
-     "score": {
-      "n": 120,
-      "accuracy": 0.75,
-      "nll": 0.4969,
-      "ece": 0.0576,
-      "mean_conf": 0.7831
-     }
-    }
-   },
-   "val_teacher2": {
-    "all": {
-     "n": 447,
-     "accuracy": 0.8322,
-     "nll": 0.4435,
-     "ece": 0.0627,
-     "mean_conf": 0.8747
-    },
-    "by_type": {
-     "choice": {
-      "n": 265,
-      "accuracy": 0.8377,
-      "nll": 0.4219,
-      "ece": 0.0503,
-      "mean_conf": 0.8858
-     },
-     "noul": {
-      "n": 120,
-      "accuracy": 0.85,
-      "nll": 0.3702,
-      "ece": 0.0827,
-      "mean_conf": 0.9034
-     },
-     "score": {
-      "n": 62,
-      "accuracy": 0.7742,
-      "nll": 0.6773,
-      "ece": 0.1441,
-      "mean_conf": 0.772
-     }
-    }
-   },
-   "cal_human": {
-    "all": {
-     "n": 1595,
-     "accuracy": 0.8564,
-     "nll": 0.3892,
-     "ece": 0.0205,
-     "mean_conf": 0.8717
-    },
-    "by_type": {
-     "choice": {
-      "n": 1595,
-      "accuracy": 0.8564,
-      "nll": 0.3892,
-      "ece": 0.0205,
-      "mean_conf": 0.8717
-     },
-     "noul": null,
-     "score": null
-    }
-   },
-   "val_replay4": {
-    "all": {
-     "n": 1000,
-     "accuracy": 0.889,
-     "nll": 0.2848,
-     "ece": 0.0297,
-     "mean_conf": 0.8702
-    },
-    "by_type": {
-     "choice": {
-      "n": 1000,
-      "accuracy": 0.889,
-      "nll": 0.2848,
-      "ece": 0.0297,
-      "mean_conf": 0.8702
-     },
-     "noul": null,
-     "score": null
-    }
-   }
-  },
-  "global_T": {
-   "heldout_jb": {
-    "all": {
-     "n": 5000,
-     "accuracy": 0.5562,
-     "nll": 1.2937,
-     "ece": 0.163,
-     "mean_conf": 0.7192
-    },
-    "by_type": {
-     "choice": {
-      "n": 3293,
-      "accuracy": 0.5056,
-      "nll": 1.5059,
-      "ece": 0.1724,
-      "mean_conf": 0.678
-     },
-     "noul": {
-      "n": 1289,
-      "accuracy": 0.7114,
-      "nll": 0.7112,
-      "ece": 0.1463,
-      "mean_conf": 0.8577
-     },
-     "score": {
-      "n": 418,
-      "accuracy": 0.4761,
-      "nll": 1.4183,
-      "ece": 0.1402,
-      "mean_conf": 0.6163
-     }
-    }
-   },
-   "test_teacher2": {
-    "all": {
-     "n": 449,
-     "accuracy": 0.8196,
-     "nll": 0.4381,
-     "ece": 0.0497,
-     "mean_conf": 0.8664
-    },
-    "by_type": {
-     "choice": {
-      "n": 257,
-      "accuracy": 0.8016,
-      "nll": 0.5058,
-      "ece": 0.058,
-      "mean_conf": 0.8596
-     },
-     "noul": {
-      "n": 130,
-      "accuracy": 0.8692,
-      "nll": 0.2755,
-      "ece": 0.0602,
-      "mean_conf": 0.9294
-     },
-     "score": {
-      "n": 62,
-      "accuracy": 0.7903,
-      "nll": 0.4984,
-      "ece": 0.0798,
-      "mean_conf": 0.7623
-     }
-    }
-   },
-   "guard": {
-    "all": {
-     "n": 2994,
-     "accuracy": 0.8186,
-     "nll": 0.4871,
-     "ece": 0.0194,
-     "mean_conf": 0.836
-    },
-    "by_type": {
-     "choice": {
-      "n": 2994,
-      "accuracy": 0.8186,
-      "nll": 0.4871,
-      "ece": 0.0194,
-      "mean_conf": 0.836
-     },
-     "noul": null,
-     "score": null
-    }
-   },
-   "val_jb": {
-    "all": {
-     "n": 5000,
-     "accuracy": 0.796,
-     "nll": 0.5269,
-     "ece": 0.0264,
-     "mean_conf": 0.8224
-    },
-    "by_type": {
-     "choice": {
-      "n": 3374,
-      "accuracy": 0.7647,
-      "nll": 0.6267,
-      "ece": 0.0303,
-      "mean_conf": 0.795
-     },
-     "noul": {
-      "n": 1273,
-      "accuracy": 0.901,
-      "nll": 0.223,
-      "ece": 0.0172,
-      "mean_conf": 0.9146
-     },
-     "score": {
-      "n": 353,
-      "accuracy": 0.7167,
-      "nll": 0.6687,
-      "ece": 0.0363,
-      "mean_conf": 0.7514
-     }
-    }
-   },
-   "val_teacher": {
-    "all": {
-     "n": 944,
-     "accuracy": 0.8379,
-     "nll": 0.433,
-     "ece": 0.046,
-     "mean_conf": 0.8722
-    },
-    "by_type": {
-     "choice": {
-      "n": 540,
-      "accuracy": 0.8315,
-      "nll": 0.4654,
-      "ece": 0.0435,
-      "mean_conf": 0.8558
-     },
-     "noul": {
-      "n": 284,
-      "accuracy": 0.8873,
-      "nll": 0.3405,
-      "ece": 0.0634,
-      "mean_conf": 0.9331
-     },
-     "score": {
-      "n": 120,
-      "accuracy": 0.75,
-      "nll": 0.5064,
-      "ece": 0.0603,
-      "mean_conf": 0.8016
-     }
-    }
-   },
-   "val_teacher2": {
-    "all": {
-     "n": 447,
-     "accuracy": 0.8277,
-     "nll": 0.4695,
-     "ece": 0.0806,
-     "mean_conf": 0.8852
-    },
-    "by_type": {
-     "choice": {
-      "n": 265,
-      "accuracy": 0.8377,
-      "nll": 0.4233,
-      "ece": 0.0546,
-      "mean_conf": 0.8868
-     },
-     "noul": {
-      "n": 120,
-      "accuracy": 0.85,
-      "nll": 0.4441,
-      "ece": 0.1059,
-      "mean_conf": 0.9311
-     },
-     "score": {
-      "n": 62,
-      "accuracy": 0.7419,
-      "nll": 0.7165,
-      "ece": 0.1556,
-      "mean_conf": 0.7893
-     }
-    }
-   },
-   "cal_human": {
-    "all": {
-     "n": 1595,
-     "accuracy": 0.8564,
-     "nll": 0.3893,
-     "ece": 0.0195,
-     "mean_conf": 0.8733
-    },
-    "by_type": {
-     "choice": {
-      "n": 1595,
-      "accuracy": 0.8564,
-      "nll": 0.3893,
-      "ece": 0.0195,
-      "mean_conf": 0.8733
-     },
-     "noul": null,
-     "score": null
-    }
-   },
-   "val_replay4": {
-    "all": {
-     "n": 1000,
-     "accuracy": 0.889,
-     "nll": 0.2843,
-     "ece": 0.0287,
-     "mean_conf": 0.8713
-    },
-    "by_type": {
-     "choice": {
-      "n": 1000,
-      "accuracy": 0.889,
-      "nll": 0.2843,
-      "ece": 0.0287,
-      "mean_conf": 0.8713
-     },
-     "noul": null,
-     "score": null
-    }
-   }
-  }
- },
- "fixtures": {
-  "general_validation": {
-   "this": {
-    "rows": 847,
-    "accuracy": 0.8500590318772137,
-    "macro_accuracy": 0.8552272255409907,
-    "groups": 65,
-    "nll": 0.43237405419192004,
-    "ece": 0.03505942904541031,
-    "brier": 0.22353081852050935,
-    "mean_confidence": 0.8244136475475226
-   },
-   "paired": {
-    "cand4b_B4_e2_tt-cand4b_B4_e2": {
-     "accuracy": {
-      "delta": 0.0,
-      "ci95": [
-       0.0,
-       0.0
-      ]
-     },
-     "nll": {
-      "delta": 0.00012970495542040443,
-      "ci95": [
-       -0.00039863639304211823,
-       0.0005942988193083918
-      ]
-     }
-    },
-    "cand4b_B4_e2_tt-v1_4b_p130": {
-     "accuracy": {
-      "delta": -0.010625737898465172,
-      "ci95": [
-       -0.021251475796930343,
-       0.0
-      ]
-     },
-     "nll": {
-      "delta": 0.015307099810435664,
-      "ci95": [
-       0.0030912070436396634,
-       0.028028207832146323
-      ]
-     }
-    },
-    "cand4b_B4_e2_tt-v2_4b_p130": {
-     "accuracy": {
-      "delta": 0.0,
-      "ci95": [
-       -0.015348288075560802,
-       0.015348288075560802
-      ]
-     },
-     "nll": {
-      "delta": 0.013765296214476044,
-      "ci95": [
-       -0.006225581100979938,
-       0.0345539752628619
-      ]
-     }
-    }
-   }
-  },
-  "typesafe": {
-   "this": {
-    "rows": 102,
-    "accuracy": 0.8431372549019608,
-    "macro_accuracy": 0.8464696223316912,
-    "groups": 4,
-    "nll": 0.44128428424617644,
-    "ece": 0.06267209380280735,
-    "brier": 0.24132394423929263,
-    "mean_confidence": 0.8996653276331285,
-    "mean_total_variation_to_reference": 0.19246147475275616
-   },
-   "paired": {
-    "cand4b_B4_e2_tt-cand4b_B4_e2": {
-     "accuracy": {
-      "delta": 0.0,
-      "ci95": [
-       0.0,
-       0.0
-      ]
-     },
-     "nll": {
-      "delta": -0.0017070072924830098,
-      "ci95": [
-       -0.003544773928864442,
-       -8.629601238631066e-05
-      ]
-     }
-    },
-    "cand4b_B4_e2_tt-v1_4b_p130": {
-     "accuracy": {
-      "delta": 0.029411764705882353,
-      "ci95": [
-       -0.029411764705882353,
-       0.08823529411764706
-      ]
-     },
-     "nll": {
-      "delta": -0.16956755666314244,
-      "ci95": [
-       -0.3236270765220259,
-       -0.035752501984007506
-      ]
-     }
-    },
-    "cand4b_B4_e2_tt-v2_4b_p130": {
-     "accuracy": {
-      "delta": -0.0196078431372549,
-      "ci95": [
-       -0.0784313725490196,
-       0.0392156862745098
-      ]
-     },
-     "nll": {
-      "delta": 0.03116252024698791,
-      "ci95": [
-       -0.06909971743191749,
-       0.13434316002254298
-      ]
-     }
-    }
-   }
-  },
-  "openjev_external": {
-   "this": {
-    "rows": 5252,
-    "accuracy": 0.6597486671744097,
-    "macro_accuracy": 0.8063444444444444,
-    "groups": 3,
-    "nll": 0.8457289344223444,
-    "ece": 0.1539857582574297,
-    "brier": 0.5022800176356607,
-    "mean_confidence": 0.8111537299704116
-   },
-   "paired": {
-    "cand4b_B4_e2_tt-cand4b_B4_e2": {
-     "accuracy": {
-      "delta": 0.0,
-      "ci95": [
-       0.0,
-       0.0
-      ]
-     },
-     "nll": {
-      "delta": -0.003617502816960803,
-      "ci95": [
-       -0.003919285763621306,
-       -0.00332506503930084
-      ]
-     }
-    },
-    "cand4b_B4_e2_tt-v1_4b_p130": {
-     "accuracy": {
-      "delta": 0.020944402132520943,
-      "ci95": [
-       0.012376237623762377,
-       0.029322162985529324
-      ]
-     },
-     "nll": {
-      "delta": -0.047333755296254304,
-      "ci95": [
-       -0.059559262901540976,
-       -0.035193633735333484
-      ]
-     }
-    },
-    "cand4b_B4_e2_tt-v2_4b_p130": {
-     "accuracy": {
-      "delta": -0.007425742574257425,
-      "ci95": [
-       -0.015993907083015995,
-       0.0017136329017517135
-      ]
-     },
-     "nll": {
-      "delta": 0.05671352997929895,
-      "ci95": [
-       0.04359541016972144,
-       0.07033520922345497
-      ]
-     }
-    }
-   }
-  },
-  "mind2web": {
-   "this": {
-    "rows": 1770,
-    "accuracy": 0.8751412429378531,
-    "macro_accuracy": 0.8773196126201471,
-    "groups": 3,
-    "nll": 0.37523612083453944,
-    "ece": 0.016833950116135952,
-    "brier": 0.18402882486455008,
-    "mean_confidence": 0.8624463401776923
-   },
-   "paired": {
-    "cand4b_B4_e2_tt-cand4b_B4_e2": {
-     "accuracy": {
-      "delta": 0.0,
-      "ci95": [
-       0.0,
-       0.0
-      ]
-     },
-     "nll": {
-      "delta": 0.00026909733615400604,
-      "ci95": [
-       -4.997823592689937e-05,
-       0.0006061414472582659
-      ]
-     }
-    },
-    "cand4b_B4_e2_tt-v1_4b_p130": {
-     "accuracy": {
-      "delta": -0.00847457627118644,
-      "ci95": [
-       -0.01694915254237288,
-       0.0
-      ]
-     },
-     "nll": {
-      "delta": 0.00924747244030173,
-      "ci95": [
-       -0.004092843962589085,
-       0.022512164121517977
-      ]
-     }
-    },
-    "cand4b_B4_e2_tt-v2_4b_p130": {
-     "accuracy": {
-      "delta": 0.0011299435028248588,
-      "ci95": [
-       -0.00847457627118644,
-       0.011299435028248588
-      ]
-     },
-     "nll": {
-      "delta": -0.018244809747118285,
-      "ci95": [
-       -0.032040489157299956,
-       -0.005027723688309381
-      ]
-     }
-    }
-   }
-  }
- },
- "games": {
-  "cand4b_B4_e2_tt-cand4b_B4_e2|sampled|all": {
-   "n": 234,
-   "a": 0.2692307692307692,
-   "b": 0.2692307692307692,
-   "delta": 0.0,
-   "ci95": [
-    -0.004273504273504274,
-    0.004273504273504274
-   ]
-  },
-  "cand4b_B4_e2_tt-cand4b_B4_e2|sampled|bag_game": {
-   "n": 64,
-   "a": 0.51953125,
-   "b": 0.51953125,
-   "delta": 0.0,
-   "ci95": [
-    -0.01171875,
-    0.01171875
-   ]
-  },
-  "cand4b_B4_e2_tt-cand4b_B4_e2|sampled|stochastic_grid": {
-   "n": 64,
-   "a": 0.16796875,
-   "b": 0.16796875,
-   "delta": 0.0,
-   "ci95": [
-    -0.01171875,
-    0.01171875
-   ]
-  },
-  "cand4b_B4_e2_tt-cand4b_B4_e2|sampled|tic_tac_toe": {
-   "n": 74,
-   "a": 0.2533783783783784,
-   "b": 0.2533783783783784,
-   "delta": 0.0,
-   "ci95": [
-    0.0,
-    0.0
-   ]
-  },
-  "cand4b_B4_e2_tt-cand4b_B4_e2|sampled|exact_minesweeper": {
-   "n": 32,
-   "a": 0.0078125,
-   "b": 0.0078125,
-   "delta": 0.0,
-   "ci95": [
-    0.0,
-    0.0
-   ]
-  },
-  "cand4b_B4_e2_tt-cand4b_B4_e2|greedy|all": {
-   "n": 234,
-   "a": 0.25213675213675213,
-   "b": 0.25213675213675213,
-   "delta": 0.0,
-   "ci95": [
-    0.0,
-    0.0
-   ]
-  },
-  "cand4b_B4_e2_tt-cand4b_B4_e2|greedy|bag_game": {
-   "n": 64,
-   "a": 0.53125,
-   "b": 0.53125,
-   "delta": 0.0,
-   "ci95": [
-    0.0,
-    0.0
-   ]
-  },
-  "cand4b_B4_e2_tt-cand4b_B4_e2|greedy|stochastic_grid": {
-   "n": 64,
-   "a": 0.109375,
-   "b": 0.109375,
-   "delta": 0.0,
-   "ci95": [
-    0.0,
-    0.0
-   ]
-  },
-  "cand4b_B4_e2_tt-cand4b_B4_e2|greedy|tic_tac_toe": {
-   "n": 74,
-   "a": 0.24324324324324326,
-   "b": 0.24324324324324326,
-   "delta": 0.0,
-   "ci95": [
-    0.0,
-    0.0
-   ]
-  },
-  "cand4b_B4_e2_tt-cand4b_B4_e2|greedy|exact_minesweeper": {
-   "n": 32,
-   "a": 0.0,
-   "b": 0.0,
-   "delta": 0.0,
-   "ci95": [
-    0.0,
-    0.0
-   ]
-  },
-  "cand4b_B4_e2_tt-v1_4b_p130|sampled|all": {
-   "n": 234,
-   "a": 0.2692307692307692,
-   "b": 0.2777777777777778,
-   "delta": -0.008547008547008548,
-   "ci95": [
-    -0.02564102564102564,
-    0.009615384615384616
-   ]
-  },
-  "cand4b_B4_e2_tt-v1_4b_p130|sampled|bag_game": {
-   "n": 64,
-   "a": 0.51953125,
-   "b": 0.56640625,
-   "delta": -0.046875,
-   "ci95": [
-    -0.08984375,
-    -0.00390625
-   ]
-  },
-  "cand4b_B4_e2_tt-v1_4b_p130|sampled|stochastic_grid": {
-   "n": 64,
-   "a": 0.16796875,
-   "b": 0.1640625,
-   "delta": 0.00390625,
-   "ci95": [
-    -0.02734375,
-    0.03515625
-   ]
-  },
-  "cand4b_B4_e2_tt-v1_4b_p130|sampled|tic_tac_toe": {
-   "n": 74,
-   "a": 0.2533783783783784,
-   "b": 0.24324324324324326,
-   "delta": 0.010135135135135136,
-   "ci95": [
-    -0.013513513513513514,
-    0.037162162162162164
-   ]
-  },
-  "cand4b_B4_e2_tt-v1_4b_p130|sampled|exact_minesweeper": {
-   "n": 32,
-   "a": 0.0078125,
-   "b": 0.0078125,
-   "delta": 0.0,
-   "ci95": [
-    -0.0234375,
-    0.0234375
-   ]
-  },
-  "cand4b_B4_e2_tt-v1_4b_p130|greedy|all": {
-   "n": 234,
-   "a": 0.25213675213675213,
-   "b": 0.2905982905982906,
-   "delta": -0.038461538461538464,
-   "ci95": [
-    -0.07264957264957266,
-    -0.004273504273504274
-   ]
-  },
-  "cand4b_B4_e2_tt-v1_4b_p130|greedy|bag_game": {
-   "n": 64,
-   "a": 0.53125,
-   "b": 0.625,
-   "delta": -0.09375,
-   "ci95": [
-    -0.171875,
-    -0.03125
-   ]
-  },
-  "cand4b_B4_e2_tt-v1_4b_p130|greedy|stochastic_grid": {
-   "n": 64,
-   "a": 0.109375,
-   "b": 0.125,
-   "delta": -0.015625,
-   "ci95": [
-    -0.07851562499999987,
-    0.046875
-   ]
-  },
-  "cand4b_B4_e2_tt-v1_4b_p130|greedy|tic_tac_toe": {
-   "n": 74,
-   "a": 0.24324324324324326,
-   "b": 0.2702702702702703,
-   "delta": -0.02702702702702703,
-   "ci95": [
-    -0.0945945945945946,
-    0.04054054054054054
-   ]
-  },
-  "cand4b_B4_e2_tt-v1_4b_p130|greedy|exact_minesweeper": {
-   "n": 32,
-   "a": 0.0,
-   "b": 0.0,
-   "delta": 0.0,
-   "ci95": [
-    0.0,
-    0.0
-   ]
-  },
-  "cand4b_B4_e2_tt-v2_4b_p130|sampled|all": {
-   "n": 234,
-   "a": 0.2692307692307692,
-   "b": 0.22435897435897437,
-   "delta": 0.04487179487179487,
-   "ci95": [
-    0.024572649572649572,
-    0.06623931623931624
-   ]
-  },
-  "cand4b_B4_e2_tt-v2_4b_p130|sampled|bag_game": {
-   "n": 64,
-   "a": 0.51953125,
-   "b": 0.37890625,
-   "delta": 0.140625,
-   "ci95": [
-    0.08203125,
-    0.19921875
-   ]
-  },
-  "cand4b_B4_e2_tt-v2_4b_p130|sampled|stochastic_grid": {
-   "n": 64,
-   "a": 0.16796875,
-   "b": 0.16015625,
-   "delta": 0.0078125,
-   "ci95": [
-    -0.0234375,
-    0.0390625
-   ]
-  },
-  "cand4b_B4_e2_tt-v2_4b_p130|sampled|tic_tac_toe": {
-   "n": 74,
-   "a": 0.2533783783783784,
-   "b": 0.24324324324324326,
-   "delta": 0.010135135135135136,
-   "ci95": [
-    -0.010135135135135136,
-    0.033783783783783786
-   ]
-  },
-  "cand4b_B4_e2_tt-v2_4b_p130|sampled|exact_minesweeper": {
-   "n": 32,
-   "a": 0.0078125,
-   "b": 0.0,
-   "delta": 0.0078125,
-   "ci95": [
-    0.0,
-    0.0234375
-   ]
-  },
-  "cand4b_B4_e2_tt-v2_4b_p130|greedy|all": {
-   "n": 234,
-   "a": 0.25213675213675213,
-   "b": 0.2777777777777778,
-   "delta": -0.02564102564102564,
-   "ci95": [
-    -0.0641025641025641,
-    0.01282051282051282
-   ]
-  },
-  "cand4b_B4_e2_tt-v2_4b_p130|greedy|bag_game": {
-   "n": 64,
-   "a": 0.53125,
-   "b": 0.484375,
-   "delta": 0.046875,
-   "ci95": [
-    -0.03125,
-    0.125
-   ]
-  },
-  "cand4b_B4_e2_tt-v2_4b_p130|greedy|stochastic_grid": {
-   "n": 64,
-   "a": 0.109375,
-   "b": 0.1875,
-   "delta": -0.078125,
-   "ci95": [
-    -0.15625,
-    0.0
-   ]
-  },
-  "cand4b_B4_e2_tt-v2_4b_p130|greedy|tic_tac_toe": {
-   "n": 74,
-   "a": 0.24324324324324326,
-   "b": 0.2972972972972973,
-   "delta": -0.05405405405405406,
-   "ci95": [
-    -0.12162162162162163,
-    0.02702702702702703
-   ]
-  },
-  "cand4b_B4_e2_tt-v2_4b_p130|greedy|exact_minesweeper": {
-   "n": 32,
-   "a": 0.0,
-   "b": 0.0,
-   "delta": 0.0,
-   "ci95": [
-    0.0,
-    0.0
-   ]
-  }
- },
- "browser": {
-  "cand4b_B4_e2_tt-cand4b_B4_e2|sampled|all": {
-   "n": 176,
-   "a": 0.9318181818181818,
-   "b": 0.9318181818181818,
-   "delta": 0.0,
-   "ci95": [
-    0.0,
-    0.0
-   ]
-  },
-  "cand4b_B4_e2_tt-cand4b_B4_e2|sampled|rewarded": {
-   "n": 128,
-   "a": 0.953125,
-   "b": 0.953125,
-   "delta": 0.0,
-   "ci95": [
-    0.0,
-    0.0
-   ]
-  },
-  "cand4b_B4_e2_tt-cand4b_B4_e2|sampled|held_out": {
-   "n": 48,
-   "a": 0.875,
-   "b": 0.875,
-   "delta": 0.0,
-   "ci95": [
-    0.0,
-    0.0
-   ]
-  },
-  "cand4b_B4_e2_tt-cand4b_B4_e2|greedy|all": {
-   "n": 176,
-   "a": 0.9375,
-   "b": 0.9375,
-   "delta": 0.0,
-   "ci95": [
-    0.0,
-    0.0
-   ]
-  },
-  "cand4b_B4_e2_tt-cand4b_B4_e2|greedy|rewarded": {
-   "n": 128,
-   "a": 0.9609375,
-   "b": 0.9609375,
-   "delta": 0.0,
-   "ci95": [
-    0.0,
-    0.0
-   ]
-  },
-  "cand4b_B4_e2_tt-cand4b_B4_e2|greedy|held_out": {
-   "n": 48,
-   "a": 0.875,
-   "b": 0.875,
-   "delta": 0.0,
-   "ci95": [
-    0.0,
-    0.0
-   ]
-  },
-  "cand4b_B4_e2_tt-v1_4b_p130|sampled|all": {
-   "n": 176,
-   "a": 0.9318181818181818,
-   "b": 0.9090909090909091,
-   "delta": 0.022727272727272728,
-   "ci95": [
-    -0.017045454545454544,
-    0.0625
-   ]
-  },
-  "cand4b_B4_e2_tt-v1_4b_p130|sampled|rewarded": {
-   "n": 128,
-   "a": 0.953125,
-   "b": 0.9609375,
-   "delta": -0.0078125,
-   "ci95": [
-    -0.0390625,
-    0.0234375
-   ]
-  },
-  "cand4b_B4_e2_tt-v1_4b_p130|sampled|held_out": {
-   "n": 48,
-   "a": 0.875,
-   "b": 0.7708333333333334,
-   "delta": 0.10416666666666667,
-   "ci95": [
-    0.0,
-    0.20833333333333334
-   ]
-  },
-  "cand4b_B4_e2_tt-v1_4b_p130|greedy|all": {
-   "n": 176,
-   "a": 0.9375,
-   "b": 0.9147727272727273,
-   "delta": 0.022727272727272728,
-   "ci95": [
-    -0.005681818181818182,
-    0.056818181818181816
-   ]
-  },
-  "cand4b_B4_e2_tt-v1_4b_p130|greedy|rewarded": {
-   "n": 128,
-   "a": 0.9609375,
-   "b": 0.9765625,
-   "delta": -0.015625,
-   "ci95": [
-    -0.0390625,
-    0.0
-   ]
-  },
-  "cand4b_B4_e2_tt-v1_4b_p130|greedy|held_out": {
-   "n": 48,
-   "a": 0.875,
-   "b": 0.75,
-   "delta": 0.125,
-   "ci95": [
-    0.041666666666666664,
-    0.22916666666666666
-   ]
-  },
-  "cand4b_B4_e2_tt-v2_4b_p130|sampled|all": {
-   "n": 176,
-   "a": 0.9318181818181818,
-   "b": 0.8806818181818182,
-   "delta": 0.05113636363636364,
-   "ci95": [
-    0.005681818181818182,
-    0.09659090909090909
-   ]
-  },
-  "cand4b_B4_e2_tt-v2_4b_p130|sampled|rewarded": {
-   "n": 128,
-   "a": 0.953125,
-   "b": 0.9296875,
-   "delta": 0.0234375,
-   "ci95": [
-    -0.0234375,
-    0.0703125
-   ]
-  },
-  "cand4b_B4_e2_tt-v2_4b_p130|sampled|held_out": {
-   "n": 48,
-   "a": 0.875,
-   "b": 0.75,
-   "delta": 0.125,
-   "ci95": [
-    0.0,
-    0.25
-   ]
-  },
-  "cand4b_B4_e2_tt-v2_4b_p130|greedy|all": {
-   "n": 176,
-   "a": 0.9375,
-   "b": 0.9261363636363636,
-   "delta": 0.011363636363636364,
-   "ci95": [
-    -0.017045454545454544,
-    0.045454545454545456
-   ]
-  },
-  "cand4b_B4_e2_tt-v2_4b_p130|greedy|rewarded": {
-   "n": 128,
-   "a": 0.9609375,
-   "b": 0.96875,
-   "delta": -0.0078125,
-   "ci95": [
-    -0.0234375,
-    0.0
-   ]
-  },
-  "cand4b_B4_e2_tt-v2_4b_p130|greedy|held_out": {
-   "n": 48,
-   "a": 0.875,
-   "b": 0.8125,
-   "delta": 0.0625,
-   "ci95": [
-    -0.041666666666666664,
-    0.16666666666666666
-   ]
-  }
- },
- "text_games_greedy": {
-  "pong": -5.0,
-  "freeway": 0.0,
-  "breakout": 35.0,
-  "frozenlake": 0.0,
-  "cliffwalking": -13.0,
-  "blackjack": -0.6,
-  "minigrid_empty": 0.0,
-  "minigrid_lavagap": 0.0,
-  "minigrid_doorkey": 0.0,
-  "babyai_goto": 0.1915625
- },
- "probes": {
-  "router_n": 31,
-  "router_tier": 0.968,
-  "needs_live_data": 0.839,
-  "commands_n": 45,
-  "command_risk": 0.889,
-  "destructive_called_safe": 0,
-  "outside_project": 0.933,
-  "browser_element": 0.938,
-  "browser_action": 0.75,
-  "batteries": {
-   "generic": 1.0,
-   "specific": 1.0,
-   "catchall": 0.95,
-   "noul_acc": 1.0,
-   "noul_brier": 0.01,
-   "n_choice": 60,
-   "n_noul": 49,
-   "abstain_battery": "8/8"
-  }
- },
- "issue9": {
-  "c_1": {
-   "pred": 26,
-   "gold": 6,
-   "ok": false,
-   "p_gold": 0.198,
-   "p_pred": 0.764,
-   "T": 1.099
-  },
-  "c_2": {
-   "pred": 20,
-   "gold": 20,
-   "ok": true,
-   "p_gold": 0.409,
-   "p_pred": 0.409,
-   "T": 1.099
-  },
-  "b_10": {
-   "pred": 0,
-   "gold": 0,
-   "ok": true,
-   "p_gold": 0.732,
-   "p_pred": 0.732,
-   "T": 1.099
-  },
-  "a2_4": {
-   "pred": 4,
-   "gold": 4,
-   "ok": true,
-   "p_gold": 0.577,
-   "p_pred": 0.577,
-   "T": 1.099
-  }
- },
- "bespoke": {
-  "vitaminc-dev": {
-   "n": 599,
-   "type": "choice",
-   "acc": 0.7429,
-   "ece": 0.1456,
-   "brier": 0.3945,
-   "nll": 0.7789,
-   "trained": false,
-   "sec": 11.8
-  },
-  "massive-en-US": {
-   "n": 350,
-   "type": "choice",
-   "acc": 0.8629,
-   "ece": 0.0411,
-   "brier": 0.1936,
-   "nll": 0.4173,
-   "trained": true,
-   "sec": 6.2
-  },
-  "massive-de-DE": {
-   "n": 350,
-   "type": "choice",
-   "acc": 0.84,
-   "ece": 0.0297,
-   "brier": 0.2269,
-   "nll": 0.4947,
-   "trained": false,
-   "sec": 4.5
-  },
-  "boolq": {
-   "n": 300,
-   "type": "noul",
-   "acc": 0.8667,
-   "ece": 0.0319,
-   "brier": 0.1862,
-   "nll": 0.3007,
-   "trained": true,
-   "sec": 6.2
-  },
-  "squad2": {
-   "n": 299,
-   "type": "noul",
-   "acc": 0.7893,
-   "ece": 0.0441,
-   "brier": 0.2955,
-   "nll": 0.452,
-   "trained": false,
-   "sec": 7.2
-  },
-  "paws": {
-   "n": 250,
-   "type": "noul",
-   "acc": 0.764,
-   "ece": 0.1238,
-   "brier": 0.3398,
-   "nll": 0.5538,
-   "trained": true,
-   "sec": 3.0
-  },
-  "multinli": {
-   "n": 299,
-   "type": "choice",
-   "acc": 0.9331,
-   "ece": 0.0103,
-   "brier": 0.1089,
-   "nll": 0.2099,
-   "trained": true,
-   "sec": 3.1
-  },
-  "civil_comments": {
-   "n": 300,
-   "type": "noul",
-   "acc": 0.8733,
-   "ece": 0.0566,
-   "brier": 0.1917,
-   "nll": 0.3199,
-   "trained": true,
-   "sec": 3.2
-  },
-  "aegis2": {
-   "n": 250,
-   "type": "noul",
-   "acc": 0.804,
-   "ece": 0.0467,
-   "brier": 0.2797,
-   "nll": 0.4527,
-   "trained": false,
-   "sec": 6.1
-  },
-  "helpsteer2": {
-   "n": 249,
-   "type": "score",
-   "acc": 0.4498,
-   "ece": 0.142,
-   "brier": 0.6713,
-   "nll": 1.2502,
-   "trained": true,
-   "sec": 25.8,
-   "score_mae": 0.797
-  },
-  "summeval-relevance": {
-   "n": 240,
-   "type": "score",
-   "acc": 0.3958,
-   "ece": 0.0696,
-   "brier": 0.6844,
-   "nll": 1.2541,
-   "trained": false,
-   "sec": 20.1,
-   "score_mae": 0.6172
-  },
-  "summeval-consistency": {
-   "n": 144,
-   "type": "score",
-   "acc": 0.7986,
-   "ece": 0.3095,
-   "brier": 0.4741,
-   "nll": 0.9007,
-   "trained": false,
-   "sec": 12.2,
-   "score_mae": 0.7974
-  },
-  "pubmedqa": {
-   "n": 250,
-   "type": "choice",
-   "acc": 0.708,
-   "ece": 0.0863,
-   "brier": 0.4012,
-   "nll": 0.7205,
-   "trained": true,
-   "sec": 3.8
-  },
-  "_macro": 0.756,
-  "_micro": 0.7652,
-  "_macro_untrained": 0.7284
- },
- "jevbench_public": {
-  "easy": {
-   "n": 48,
-   "accuracy": 1.0,
-   "invalid": 0,
-   "ece": 0.0027979166666670663,
-   "by_family": {
-    "extraction": {
-     "n": 12,
-     "accuracy": 1.0
-    },
-    "fact": {
-     "n": 12,
-     "accuracy": 1.0
-    },
-    "intent": {
-     "n": 12,
-     "accuracy": 1.0
-    },
-    "tool_selection": {
-     "n": 12,
-     "accuracy": 1.0
-    }
-   }
-  },
-  "standard": {
-   "n": 72,
-   "accuracy": 0.9861111111111112,
-   "invalid": 0,
-   "ece": 0.0753152777777778,
-   "by_family": {
-    "adequacy": {
-     "n": 12,
-     "accuracy": 0.9166666666666666
-    },
-    "extraction": {
-     "n": 12,
-     "accuracy": 1.0
-    },
-    "intent": {
-     "n": 12,
-     "accuracy": 1.0
-    },
-    "ordinal": {
-     "n": 12,
-     "accuracy": 1.0
-    },
-    "policy": {
-     "n": 12,
-     "accuracy": 1.0
-    },
-    "routing": {
-     "n": 12,
-     "accuracy": 1.0
-    }
-   }
-  },
-  "hard": {
-   "n": 111,
-   "accuracy": 0.6486486486486487,
-   "invalid": 0,
-   "ece": 0.18370720720720732,
-   "by_family": {
-    "adversarial": {
-     "n": 6,
-     "accuracy": 1.0
-    },
-    "ambiguous": {
-     "n": 7,
-     "accuracy": 0.7142857142857143
-    },
-    "judge_hard": {
-     "n": 17,
-     "accuracy": 0.5294117647058824
-    },
-    "long_policy": {
-     "n": 19,
-     "accuracy": 0.47368421052631576
-    },
-    "multi_hop": {
-     "n": 18,
-     "accuracy": 0.8333333333333334
-    },
-    "probability": {
-     "n": 10,
-     "accuracy": 0.8
-    },
-    "routing_hard": {
-     "n": 5,
-     "accuracy": 1.0
-    },
-    "temporal_numeric": {
-     "n": 15,
-     "accuracy": 0.3333333333333333
-    },
-    "tradeoff": {
-     "n": 6,
-     "accuracy": 0.3333333333333333
-    },
-    "trap": {
-     "n": 8,
-     "accuracy": 1.0
-    }
-   }
-  }
- },
- "references": {
-  "v1_4b": {
-   "reg_in": 0.8338,
-   "reg_out": 0.7876,
-   "reg_in_nll": 0.4044,
-   "reg_out_nll": 0.5583,
-   "reg_in_ece": 0.0269,
-   "reg_out_ece": 0.0708
-  },
-  "v2_4b": {
-   "reg_in": 0.8241,
-   "reg_out": 0.7788,
-   "reg_in_nll": 0.4411,
-   "reg_out_nll": 0.5663,
-   "reg_in_ece": 0.0413,
-   "reg_out_ece": 0.0803
-  }
- },
- "speed": "not measured again: the architecture and size equal the parent's, and the temperature map does not change the computation"
-}

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ee8ce585b3cedd93206dd149b09b4bdd683174874211f85c77a36090b90c9fdd
+oid sha256:e83c74cc305ef4643c61a3c44cb71ed03aef7560ece63a64c14c91875f5ea47e
 size 8411558400

vision/chat_template.jinja

@@ -0,0 +1,154 @@
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain images.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+            {%- elif 'video' in item or item.type == 'video' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain videos.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Video ' ~ video_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+            {%- elif 'text' in item %}
+                {{- item.text }}
+            {%- else %}
+                {{- raise_exception('Unexpected item type in content.') }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- if tools and tools is iterable and tools is not mapping %}
+    {{- '<|im_start|>system\n' }}
+    {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>" }}
+    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {%- if content %}
+            {{- '\n\n' + content }}
+        {%- endif %}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" %}
+        {%- set content = render_content(message.content, false)|trim %}
+        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if ns.multi_step_tool %}
+    {{- raise_exception('No user query found in messages.') }}
+{%- endif %}
+{%- for message in messages %}
+    {%- set content = render_content(message.content, true)|trim %}
+    {%- if message.role == "system" %}
+        {%- if not loop.first %}
+            {{- raise_exception('System message must be at the beginning.') }}
+        {%- endif %}
+    {%- elif message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content|trim %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {%- if loop.first %}
+                    {%- if content|trim %}
+                        {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- else %}
+                        {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- endif %}
+                {%- else %}
+                    {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                {%- endif %}
+                {%- if tool_call.arguments is defined %}
+                    {%- for args_name, args_value in tool_call.arguments|items %}
+                        {{- '<parameter=' + args_name + '>\n' }}
+                        {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
+                        {{- args_value }}
+                        {{- '\n</parameter>\n' }}
+                    {%- endfor %}
+                {%- endif %}
+                {{- '</function>\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>\n' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- else %}
+        {{- raise_exception('Unexpected message role.') }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- else %}
+        {{- '<think>\n' }}
+    {%- endif %}
+{%- endif %}

vision/config.json

@@ -0,0 +1,114 @@
+{
+  "architectures": [
+    "Qwen3_5ForConditionalGeneration"
+  ],
+  "dtype": "bfloat16",
+  "image_token_id": 248056,
+  "model_type": "qwen3_5",
+  "text_config": {
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attn_output_gate": true,
+    "bos_token_id": null,
+    "dtype": "bfloat16",
+    "eos_token_id": 248044,
+    "full_attention_interval": 4,
+    "head_dim": 256,
+    "hidden_act": "silu",
+    "hidden_size": 2560,
+    "initializer_range": 0.02,
+    "intermediate_size": 9216,
+    "layer_types": [
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention"
+    ],
+    "linear_conv_kernel_dim": 4,
+    "linear_key_head_dim": 128,
+    "linear_num_key_heads": 16,
+    "linear_num_value_heads": 32,
+    "linear_value_head_dim": 128,
+    "mamba_ssm_dtype": "float32",
+    "max_position_embeddings": 262144,
+    "mlp_only_layers": [],
+    "model_type": "qwen3_5_text",
+    "mtp_num_hidden_layers": 1,
+    "mtp_use_dedicated_embeddings": false,
+    "num_attention_heads": 16,
+    "num_hidden_layers": 32,
+    "num_key_value_heads": 4,
+    "pad_token_id": null,
+    "partial_rotary_factor": 0.25,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "mrope_interleaved": true,
+      "mrope_section": [
+        11,
+        11,
+        10
+      ],
+      "partial_rotary_factor": 0.25,
+      "rope_theta": 10000000,
+      "rope_type": "default"
+    },
+    "tie_word_embeddings": true,
+    "use_cache": true,
+    "vocab_size": 248320
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.17.0",
+  "use_cache": true,
+  "video_token_id": 248057,
+  "vision_config": {
+    "deepstack_visual_indexes": [],
+    "depth": 24,
+    "dtype": "bfloat16",
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1024,
+    "in_channels": 3,
+    "initializer_range": 0.02,
+    "intermediate_size": 4096,
+    "model_type": "qwen3_5_vision",
+    "num_heads": 16,
+    "num_position_embeddings": 2304,
+    "out_hidden_size": 2560,
+    "patch_size": 16,
+    "rope_parameters": {
+      "rope_theta": 10000.0,
+      "rope_type": "axial"
+    },
+    "spatial_merge_size": 2,
+    "temporal_patch_size": 2
+  },
+  "vision_end_token_id": 248054,
+  "vision_start_token_id": 248053
+}

vision/processor_config.json

@@ -0,0 +1,61 @@
+{
+  "image_processor": {
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_processor_type": "Qwen2VLImageProcessor",
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "merge_size": 2,
+    "patch_size": 16,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "size": {
+      "longest_edge": 16777216,
+      "shortest_edge": 65536
+    },
+    "temporal_patch_size": 2
+  },
+  "processor_class": "Qwen3VLProcessor",
+  "video_processor": {
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "do_sample_frames": true,
+    "fps": 2,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "max_frames": 768,
+    "max_video_tokens": 768,
+    "merge_size": 2,
+    "min_frames": 4,
+    "patch_size": 16,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "return_metadata": false,
+    "size": {
+      "longest_edge": 234881024,
+      "shortest_edge": 4096
+    },
+    "temporal_patch_size": 2,
+    "video_processor_type": "Qwen3VLVideoProcessor"
+  }
+}

vision/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523
+size 19989325

vision/tokenizer_config.json

@@ -0,0 +1,32 @@
+{
+  "add_prefix_space": false,
+  "audio_bos_token": "<|audio_start|>",
+  "audio_eos_token": "<|audio_end|>",
+  "audio_token": "<|audio_pad|>",
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "image_token": "<|image_pad|>",
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 262144,
+  "model_specific_special_tokens": {
+    "audio_bos_token": "<|audio_start|>",
+    "audio_eos_token": "<|audio_end|>",
+    "audio_token": "<|audio_pad|>",
+    "image_token": "<|image_pad|>",
+    "video_token": "<|video_pad|>",
+    "vision_bos_token": "<|vision_start|>",
+    "vision_eos_token": "<|vision_end|>"
+  },
+  "pad_token": "<|endoftext|>",
+  "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null,
+  "video_token": "<|video_pad|>",
+  "vision_bos_token": "<|vision_start|>",
+  "vision_eos_token": "<|vision_end|>"
+}

vision/visual.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:87f0f1e2be1c073a49761003a1a635da88fd5e00d3a79dd4a5e3099b4e9a9306
+size 667061480