decider-12b v2: merged state-tracking LoRA (rank 32) on Gemma-4-12B-it; T choice 1.5 / noul 0.05 / score 1.0; v1 kept under tag v1

README.md

@@ -8,8 +8,8 @@ tags: [decision-model, calibrated, structured-output, system-one, one-pass]
 
 # decider-12b: typed decisions with calibrated probabilities in one forward pass, on Gemma-4-12B-it
 
-decider-12b is [google/gemma-4-12B-it](https://huggingface.co/google/gemma-4-12B-it) (revision `707f0a3b`) with its weights
-unchanged, read through the decider readout. It is not trained further.
+decider-12b is [google/gemma-4-12B-it](https://huggingface.co/google/gemma-4-12B-it) (revision `707f0a3b`) read through
+the decider readout.
 
 - **Input:** a state and one or more typed questions (Choice, Noul yes/no, Score), each with an explicit option list.
 - **Output:** a probability distribution over the options for every question, from one forward pass. There is no decoding and
@@ -18,7 +18,11 @@ unchanged, read through the decider readout. It is not trained further.
   final-logit softcapping applied.
 - **Temperatures:** one per answer type, in `decider_config.json`.
 
-This repository holds the bf16 weights as published by Google, the chat template, and `decider_config.json`.
+**Version 2 (2026-09-29, this revision):** a merged LoRA fine-tune (state tracking) on top of the stock weights.
+- Only the 328 attention and MLP projection matrices differ from Google's weights. The config, tokenizer and chat template are
+  Google's files unchanged.
+- **Version 1** is stock Gemma-4-12B-it with no training. It is kept under the tag `v1`:
+  `Decider("Mapika/decider-12b", revision="v1")`.
 
 ## Usage
 
@@ -33,51 +37,62 @@ d.system_one(state, {"refund": {"type": "noul", "instructions": "Is the refund a
 
 Serving: `DECIDER_MODEL=Mapika/decider-12b uvicorn decider.serve:app` (POST `/v1/systemone`, the System One wire format).
 
-## Temperatures
-
-| type | T | fitted on (our own rows, no JevBench or Decision Index items) |
-|---|---|---|
-| Choice | 4.0 | NLL and top-label ECE over 2,113 Choice rows (human-labelled public sets, held-out generated decision families, teacher-written document questions) |
-| Noul | 1.0 | 685 Noul rows plus the 204-item fit half of a fresh authored yes/no set |
-| Score | 3.5 | 246 Score rows |
-
-Choice accuracy does not depend on T, so its temperature is chosen for calibration only:
-- ECE 0.243 at T 1.0, 0.026 at T 4.0.
-
-Noul is kept sharp. JevBench v1.5 counts a yes/no answer with P(yes) between 0.2 and 0.8 as wrong:
-- on the fresh set, T 2.0 would lower ECE from 0.140 to 0.096 but cost 16 chance-corrected points.
+## Training (version 2)
+
+- **Method:** LoRA rank 32, alpha 64, on q/k/v/o/gate/up/down. One epoch of 10,000 items (10.6M tokens), LR 5e-5 cosine,
+  one B300, 45 minutes. The loss is cross-entropy over the option letters at the answer slot, in the same chat layout as serving.
+- **Data:**
+  - **6,000 generated state-tracking decisions:** event logs over seven invented domains (library loans, parking permits, hotel
+    rooms, sprint boards, vehicle fleets, licence seats, lab samples). They include rejected entries, VOID and CORRECTION
+    entries, aliases and out-of-order exports, rendered as logs, tables, prose and JSON.
+    - The gold answers are computed by replaying the log.
+    - Of 104 yes/no rows answered blind by GLM-5.3-Flash, 101 agree with the computed gold.
+  - **4,000 rows from our earlier training data, replayed:**
+    - 836 human-labelled public datasets;
+    - 2,047 from our own generated decision families, written without reading JevBench items;
+    - 1,117 teacher-written document questions.
+- **Not used for training or selection:** JevBench items, public or sealed; Decision Index rows. The checkpoint and the
+  temperatures were chosen on our own held-out sets only.
 
-Score is read as the expected level:
-- T 3.5 gives chance-corrected competence 59.9 (60.4 at best, 57.1 at T 1.0), with top-label ECE 0.044 (0.241 at T 1.0).
-
-## Measurements (decider-ai 1.7.0, one B300)
+## Temperatures
 
-| set | decider-12b | decider-4b v2 | decider-35b-a3b |
+| type | v2 T | v1 T | fitted on (our own rows) |
 |---|---|---|---|
-| JevBench public items, accuracy easy / standard / hard | 1.000 / 0.986 / 0.730 | 1.000 / 0.986 / 0.676 | 1.000 / 0.972 / 0.676 |
-| JevBench public hard tier, top-label ECE | 0.106 | 0.104 | 0.157 |
-| fresh 399-item yes/no set, v1.5 chance-corrected Noul score | 61.2 | 1.0 | 21.0 |
-| the same, argmax accuracy / share of answers between 0.2 and 0.8 | 0.822 / 4 % | 0.717 / 38 % | 0.737 / 21 % |
-| latency per decision, serial, p50 / p95 | 20 ms / 155 ms | 33 ms / 55 ms | |
+| Choice | 1.5 | 4.0 | NLL and top-label ECE over 2,113 held-out Choice rows |
+| Noul | 0.05 | 1.0 | chance-corrected v1.5 Noul score on 685 rows plus the 204-item fit half of a fresh yes/no set |
+| Score | 1.0 | 3.5 | chance-corrected Score competence on 246 rows |
 
-About the fresh yes/no set:
-- 319 items were written and reviewed for this model family's evaluation. 80 are generated state-tracking logs with
-  computed answers.
-- The set was not used for anything but the Noul temperature, which was fitted on its fit half.
+- The fine-tune leaves the model less confident, so every temperature is lower than in v1.
+- Noul is read sharp. JevBench v1.5 counts a yes/no answer with P(yes) between 0.2 and 0.8 as wrong, and at T 0.05 about 1% of
+  answers fall in that band.
 
-About the JevBench public items:
-- They were read for measurement only.
-- They were not used for training, temperatures or any choice in this repository.
+## Measurements (decider-ai 1.8.0, one B300, each version at its own temperatures)
+
+| set | v2 | v1 |
+|---|---|---|
+| fresh 399-item yes/no set, test half (195), v1.5 chance-corrected Noul score | 71.6 | 58.8 |
+| same, argmax accuracy / share of answers between 0.2 and 0.8 / ECE | 0.856 / 1 % / 0.144 | 0.805 / 6 % / 0.181 |
+| same set, 80 generated state-tracking logs (another generator than training), argmax | 0.750 | 0.675 |
+| same set, 319 written items, argmax | 0.871 | 0.859 |
+| held-out generated decisions (1,500) and teacher questions (944): Choice accuracy | 0.615 | 0.588 |
+| same rows: Noul accuracy / chance-corrected Noul score | 0.790 / 57.1 | 0.740 / 38.7 |
+| same rows: chance-corrected Score competence | 61.2 | 60.1 |
+| human-labelled Choice rows (600): accuracy / ECE | 0.847 / 0.015 | 0.845 / 0.024 |
+| JevBench public items, accuracy easy / standard / hard | 1.000 / 0.986 / 0.712 | 1.000 / 0.986 / 0.730 |
+| JevBench public hard tier, top-label ECE | 0.147 | 0.098 |
+
+- The public JevBench items were read for measurement only.
+- On the hard tier (111 items), v2 answers 79 correctly and v1 answers 81.
+- Hard-tier calibration is worse in v2.
 
 ## Limitations
 
-- The model is stock Gemma-4-12B-it. Its strengths and failure modes are the base model's. The one change is the readout: one
-  forward pass with the probabilities read at a fixed answer slot.
-- Choice probabilities at T 4.0 are calibrated on our rows. Per-dataset calibration still varies (knowledge multiple choice is
-  underconfident, hard reasoning overconfident).
-- Weakest measured area: state-tracking logs, with argmax accuracy 0.69 on the fresh set.
+- The base model's strengths and failure modes carry over.
+- The fine-tune targets state tracking over event logs. Gains elsewhere come from the replayed rows and are 1-5 points.
+- On the JevBench public hard tier, v2 is not better than v1: 79 vs 81 correct of 111, and ECE 0.147 vs 0.098.
+- Per-dataset calibration still varies: knowledge multiple choice is underconfident, and hard reasoning is overconfident.
 
 ## Licence
 
-Weights: google/gemma-4-12B-it, Apache-2.0 as published by Google.
+Weights: derived from google/gemma-4-12B-it, Apache-2.0 as published by Google.
 Readout code: Apache-2.0 (github.com/Mapika/decider).

decider_config.json

@@ -1,10 +1,10 @@
 {
- "version": "12b-v1",
+ "version": "12b-v2",
  "layout": "chat",
- "temperature": 4.0,
- "temperature_by_type": {"choice": 4.0, "noul": 1.0, "score": 3.5},
+ "temperature": 1.5,
+ "temperature_by_type": {"choice": 1.5, "noul": 0.05, "score": 1.0},
  "neutralize_none": false,
  "isolated_levels": false,
- "base": "google/gemma-4-12B-it (revision 707f0a3b), unmodified weights",
+ "base": "google/gemma-4-12B-it (revision 707f0a3b) + merged LoRA (rank 32, attention and MLP), state-tracking fine-tune",
  "requires": "decider-ai>=1.7.0 (final-logit softcapping)"
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5a84cb313260ac447237b890387116dfa8682e49a6b44bc585ae8353abbff18d
+oid sha256:efb90aaf7521515851584f673ff4513027abb84df97a9ad2aa4fe27bc250770a
 size 23919549408