Expand public evaluation set and regenerate frontier charts

.gitattributes

@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+assets/balance.png filter=lfs diff=lfs merge=lfs -text

MODEL_CARD.md

@@ -61,7 +61,7 @@ model-index:
 
 # Pluto-2.1
 
-### A 1.7B decision model that beats d1-2B on both measured decision benchmarks.
+### A 1.7B decision model that clears both d1 checkpoints and owns the balance.
 
 Pluto-2.1 reads a state and typed questions, then returns a calibrated probability
 distribution for every candidate. It does not generate an answer, so there is
@@ -70,33 +70,42 @@ nothing to parse, nothing to retry, and nothing to hallucinate.
 Built solo on one **RTX 5060 Ti 16GB**.
 
 **Headline:** **0.6695** on Typed Decisions and **0.7208** on the matched
-DecisionBench comparison, versus **0.6325** and **0.6339** for d1-2B. One
-standalone 1.7B checkpoint, **2–255 candidates**, **choice / noul / score**,
-**Apache-2.0**.
+DecisionBench comparison. It clears both d1-2B and d1-3B on both suites and posts
+the **highest mean accuracy** in the comparison at **0.6952** — from a 1.738B
+standalone checkpoint with **2–255 candidates**, **choice / noul / score**, and
+**Apache-2.0** weights.
 
 ---
 
 ## The receipts
 
-Same rows and same scorer for both models. Higher is better.
+Same rows and same scorer for every locally measured model. Mean is the mean of
+Typed Decisions and DecisionBench matched accuracy.
 
-| Model | Params | Typed Decisions | DecisionBench matched | Brier | KL |
-|---|---:|---:|---:|---:|---:|
-| **Pluto-2.1** | **1.738B** | **0.6695** | **0.7208** | **0.1261** | **0.2139** |
-| d1-2B | 1.918B | 0.6325 | 0.6339 | 0.1440 | 0.2958 |
+| Model | Params | Typed Decisions | DecisionBench matched | Mean |
+|---|---:|---:|---:|---:|
+| **Pluto-2.1** | **1.738B** | **0.6695** | **0.7208** | **0.6952** |
+| Pluto-2 400M | 0.397B | **0.7835** | 0.5789 | 0.6812 |
+| Bosun 3.1 1.7B (initializer) | 1.738B | 0.4980 | **0.8288** | 0.6634 |
+| d1-3B | 3.123B | 0.6495 | 0.6486 | 0.6490 |
+| d1-2B | 1.918B | 0.6325 | 0.6339 | 0.6332 |
 
 **What this means**
 
-- **+3.7 points** on the full 2,000-decision Typed Decisions test.
-- **+8.7 points** on the matched DecisionBench comparison.
-- **9.4% fewer parameters** than d1-2B.
-- **Sharper distributions**: lower Brier and KL on Typed Decisions, and lower
-  NLL and Brier on DecisionBench.
+- **+3.7 points** over d1-2B and **+2.0 points** over d1-3B on Typed Decisions.
+- **+8.7 points** over d1-2B and **+7.2 points** over d1-3B on DecisionBench.
+- **Highest mean accuracy** across the two suites.
+- **1.738B**: 9.4% smaller than d1-2B and 44% smaller than d1-3B.
+- **Sharper distributions** than both d1 checkpoints: lower Typed Brier and KL,
+  lower DecisionBench NLL and Brier.
 - **One forward pass.** No autoregressive decoding and no token bill.
+- **Bosun leads broad-only; Pluto-2 leads Typed-only; Pluto-2.1 owns the balance.**
 
 ![Accuracy](assets/benchmark_accuracy.png)
 
-![Uplift](assets/uplift.png)
+![Mean accuracy](assets/average_accuracy.png)
+
+![Balance](assets/balance.png)
 
 ![Calibration](assets/calibration.png)
 
@@ -104,11 +113,11 @@ Same rows and same scorer for both models. Higher is better.
 
 ### Comparison hygiene
 
-Pluto-2.1 is a standalone merged checkpoint. The full 2,000-decision Typed
-Decisions test and a 1,565-row DecisionBench matched subset are reported. The
-DecisionBench subset is the same ≤16-candidate, ≤2,048-token accessibility slice
-used for the d1-2B comparison; it is not the full 23,900-row suite. The full
-suite is not claimed here.
+The Typed Decisions result is the full 2,000-decision test split. The
+DecisionBench result is the 1,565-row matched subset used for every model in the
+table: ≤16 candidates and ≤2,048-token prompts, not the full 23,900-row suite.
+Bosun, Pluto-2, d1-2B, and d1-3B were all run locally on the same rows and
+scored with the same code. Raw summaries are in enchmarks/. The full DecisionBench suite is not claimed.
 
 ---
 
@@ -190,8 +199,8 @@ The runtime returns candidate-aligned probabilities. It never generates text.
 ## Training
 
 - 2,000 LoRA steps on Pluto’s Typed Decisions and JEV decision data.
-- Selected checkpoint: 2,000 steps, the only tested checkpoint that beats d1-2B
-  on both measured benchmarks.
+- Selected checkpoint: 2,000 steps, the only tested checkpoint that clears both
+  d1-2B and d1-3B on both measured benchmarks.
 - Trained and merged on one 16GB consumer GPU.
 - The adapter is merged into the base; no PEFT dependency at inference.
 - The Typed Decisions test split was never used for training or checkpoint

README.md

@@ -61,7 +61,7 @@ model-index:
 
 # Pluto-2.1
 
-### A 1.7B decision model that beats d1-2B on both measured decision benchmarks.
+### A 1.7B decision model that clears both d1 checkpoints and owns the balance.
 
 Pluto-2.1 reads a state and typed questions, then returns a calibrated probability
 distribution for every candidate. It does not generate an answer, so there is
@@ -70,33 +70,42 @@ nothing to parse, nothing to retry, and nothing to hallucinate.
 Built solo on one **RTX 5060 Ti 16GB**.
 
 **Headline:** **0.6695** on Typed Decisions and **0.7208** on the matched
-DecisionBench comparison, versus **0.6325** and **0.6339** for d1-2B. One
-standalone 1.7B checkpoint, **2–255 candidates**, **choice / noul / score**,
-**Apache-2.0**.
+DecisionBench comparison. It clears both d1-2B and d1-3B on both suites and posts
+the **highest mean accuracy** in the comparison at **0.6952** — from a 1.738B
+standalone checkpoint with **2–255 candidates**, **choice / noul / score**, and
+**Apache-2.0** weights.
 
 ---
 
 ## The receipts
 
-Same rows and same scorer for both models. Higher is better.
+Same rows and same scorer for every locally measured model. Mean is the mean of
+Typed Decisions and DecisionBench matched accuracy.
 
-| Model | Params | Typed Decisions | DecisionBench matched | Brier | KL |
-|---|---:|---:|---:|---:|---:|
-| **Pluto-2.1** | **1.738B** | **0.6695** | **0.7208** | **0.1261** | **0.2139** |
-| d1-2B | 1.918B | 0.6325 | 0.6339 | 0.1440 | 0.2958 |
+| Model | Params | Typed Decisions | DecisionBench matched | Mean |
+|---|---:|---:|---:|---:|
+| **Pluto-2.1** | **1.738B** | **0.6695** | **0.7208** | **0.6952** |
+| Pluto-2 400M | 0.397B | **0.7835** | 0.5789 | 0.6812 |
+| Bosun 3.1 1.7B (initializer) | 1.738B | 0.4980 | **0.8288** | 0.6634 |
+| d1-3B | 3.123B | 0.6495 | 0.6486 | 0.6490 |
+| d1-2B | 1.918B | 0.6325 | 0.6339 | 0.6332 |
 
 **What this means**
 
-- **+3.7 points** on the full 2,000-decision Typed Decisions test.
-- **+8.7 points** on the matched DecisionBench comparison.
-- **9.4% fewer parameters** than d1-2B.
-- **Sharper distributions**: lower Brier and KL on Typed Decisions, and lower
-  NLL and Brier on DecisionBench.
+- **+3.7 points** over d1-2B and **+2.0 points** over d1-3B on Typed Decisions.
+- **+8.7 points** over d1-2B and **+7.2 points** over d1-3B on DecisionBench.
+- **Highest mean accuracy** across the two suites.
+- **1.738B**: 9.4% smaller than d1-2B and 44% smaller than d1-3B.
+- **Sharper distributions** than both d1 checkpoints: lower Typed Brier and KL,
+  lower DecisionBench NLL and Brier.
 - **One forward pass.** No autoregressive decoding and no token bill.
+- **Bosun leads broad-only; Pluto-2 leads Typed-only; Pluto-2.1 owns the balance.**
 
 ![Accuracy](assets/benchmark_accuracy.png)
 
-![Uplift](assets/uplift.png)
+![Mean accuracy](assets/average_accuracy.png)
+
+![Balance](assets/balance.png)
 
 ![Calibration](assets/calibration.png)
 
@@ -104,11 +113,11 @@ Same rows and same scorer for both models. Higher is better.
 
 ### Comparison hygiene
 
-Pluto-2.1 is a standalone merged checkpoint. The full 2,000-decision Typed
-Decisions test and a 1,565-row DecisionBench matched subset are reported. The
-DecisionBench subset is the same ≤16-candidate, ≤2,048-token accessibility slice
-used for the d1-2B comparison; it is not the full 23,900-row suite. The full
-suite is not claimed here.
+The Typed Decisions result is the full 2,000-decision test split. The
+DecisionBench result is the 1,565-row matched subset used for every model in the
+table: ≤16 candidates and ≤2,048-token prompts, not the full 23,900-row suite.
+Bosun, Pluto-2, d1-2B, and d1-3B were all run locally on the same rows and
+scored with the same code. Raw summaries are in enchmarks/. The full DecisionBench suite is not claimed.
 
 ---
 
@@ -190,8 +199,8 @@ The runtime returns candidate-aligned probabilities. It never generates text.
 ## Training
 
 - 2,000 LoRA steps on Pluto’s Typed Decisions and JEV decision data.
-- Selected checkpoint: 2,000 steps, the only tested checkpoint that beats d1-2B
-  on both measured benchmarks.
+- Selected checkpoint: 2,000 steps, the only tested checkpoint that clears both
+  d1-2B and d1-3B on both measured benchmarks.
 - Trained and merged on one 16GB consumer GPU.
 - The adapter is merged into the base; no PEFT dependency at inference.
 - The Typed Decisions test split was never used for training or checkpoint

RESULTS.md

@@ -1,21 +1,33 @@
 # Pluto-2.1 results
 
-## Head-to-head
+## General-model comparison
 
-Same rows and same scorer for both models. Higher accuracy is better; lower
-error and divergence are better.
+Same rows and same scorer for every locally measured model. `Mean` is the mean
+of Typed Decisions and DecisionBench matched accuracy.
 
-| Benchmark | Pluto-2.1 | d1-2B | Rows |
-|---|---:|---:|---:|
-| Typed Decisions accuracy | **0.6695** | 0.6325 | 2,000 decisions |
-| Typed Decisions Brier | **0.1261** | 0.1440 | 2,000 |
-| Typed Decisions KL | **0.2139** | 0.2958 | 2,000 |
-| DecisionBench matched accuracy | **0.7208** | 0.6339 | 1,565 |
-| DecisionBench matched NLL | **0.7439** | 0.9333 | 1,565 |
-| DecisionBench matched Brier | **0.3730** | 0.4510 | 1,565 |
+| Model | Params | Typed Decisions | DecisionBench matched | Mean |
+|---|---:|---:|---:|---:|
+| **Pluto-2.1** | **1.738B** | **0.6695** | **0.7208** | **0.6952** |
+| Pluto-2 400M | 0.397B | **0.7835** | 0.5789 | 0.6812 |
+| Bosun 3.1 1.7B (initializer) | 1.738B | 0.4980 | **0.8288** | 0.6634 |
+| d1-3B | 3.123B | 0.6495 | 0.6486 | 0.6490 |
+| d1-2B | 1.918B | 0.6325 | 0.6339 | 0.6332 |
+
+Pluto-2.1 clears d1-2B by **+3.7** Typed and **+8.7** DecisionBench points, and
+clears d1-3B by **+2.0** and **+7.2** points. It has the highest mean accuracy
+of the five models. Bosun retains the broad-only lead; Pluto-2 400M retains the
+Typed-only lead. Pluto-2.1 owns the balance.
+
+## Distribution quality
 
-Pluto-2.1 uses **1.738B** parameters versus **1.918B** for d1-2B, a 9.4%
-reduction.
+| Benchmark | Model | Brier | KL / NLL |
+|---|---|---:|---:|
+| Typed Decisions | **Pluto-2.1** | **0.1261** | **0.2139** |
+| Typed Decisions | d1-3B | 0.1416 | 0.2854 |
+| Typed Decisions | d1-2B | 0.1440 | 0.2958 |
+| DecisionBench matched | **Pluto-2.1** | **0.3730** | **0.7439** |
+| DecisionBench matched | d1-3B | 0.4937 | 1.0345 |
+| DecisionBench matched | d1-2B | 0.4510 | 0.9333 |
 
 ## Typed Decisions by primitive
 
@@ -37,9 +49,9 @@ reduction.
 
 ## Evaluation scope
 
-Typed Decisions is the full 2,000-decision test split. The DecisionBench result
-is a 1,565-row matched subset limited to ≤16 candidates and ≤2,048-token prompts;
-it is not the full 23,900-row suite. The full suite is not claimed.
+Typed Decisions is the full 2,000-decision test split. DecisionBench is a
+1,565-row matched subset limited to ≤16 candidates and ≤2,048-token prompts; it
+is not the full 23,900-row suite. The full suite is not claimed.
 
 ## Checkpoint selection
 
@@ -50,4 +62,4 @@ it is not the full 23,900-row suite. The full suite is not claimed.
 | **2,000** | **0.6695** | **0.7208** |
 
 The 2,000-step checkpoint is shipped because it is the only tested checkpoint
-that beats d1-2B on both measured benchmarks.
+that clears both d1-2B and d1-3B on both measured benchmarks.

benchmarks/README.md

@@ -0,0 +1,28 @@
+# Benchmark summaries
+
+All numbers here were measured locally on one RTX 5060 Ti 16GB using the same
+rows and scorer for every model.
+
+## Typed Decisions
+
+- Full independent test split: 400 held-out cases, 2,000 probabilistic decisions.
+- Scored with the benchmark metric code: argmax accuracy, KL, Brier, ECE.
+- Models: Pluto-2.1, Pluto-2, d1-2B, d1-3B. d1 checkpoints are zero-shot.
+
+## DecisionBench
+
+- Public `Hanno-Labs/decision-bench` evaluation split.
+- Matched accessibility subset: 1,565 of 23,900 rows, limited to ≤16 candidates
+  and ≤2,048-token prompts.
+- Models: Pluto-2.1, Pluto-2, Bosun 3.1 1.7B, d1-2B, d1-3B.
+- This is not the full 23,900-row suite. No full-suite score is claimed.
+
+## Files
+
+- `pluto21_*.json` — Pluto-2.1 step 2,000
+- `pluto2_*.json` — Pluto-2 400M
+- `bosun_base_decisionbench_1565.json` — Bosun 3.1 1.7B initializer
+- `d1_2b_*.json` — Liquid d1-2B
+- `d1_3b_*.json` — Liquid d1-3B
+
+The model card charts are generated from these values.

benchmarks/bosun_base_decisionbench_1565.json

@@ -0,0 +1,35 @@
+{
+  "model": "Hanno-Labs/bosun-v3.1-1.7b",
+  "benchmark": "Hanno-Labs/decision-bench",
+  "split": "eval",
+  "rows": 1565,
+  "n": 1565,
+  "accuracy": 0.8287539936102236,
+  "nll": 0.4682318288769702,
+  "brier": 0.22878205160952753,
+  "ece": 0.06104286726143513,
+  "by_primitive": {
+    "ordinal_scoring": {
+      "n": 142,
+      "accuracy": 0.5633802816901409,
+      "nll": 0.92196065616788,
+      "brier": 0.4060187358413133,
+      "ece": 0.21783173873551476
+    },
+    "candidate_selection": {
+      "n": 893,
+      "accuracy": 0.8208286674132139,
+      "nll": 0.5566041466056967,
+      "brier": 0.2688005828349878,
+      "ece": 0.05700985998286913
+    },
+    "binary_classification": {
+      "n": 530,
+      "accuracy": 0.9132075471698113,
+      "nll": 0.1977677284862872,
+      "brier": 0.11386854680716972,
+      "ece": 0.040639265530327866
+    }
+  },
+  "latency_s": 192.5441007000045
+}

benchmarks/d1_2b_decisionbench_1565.json

@@ -0,0 +1,36 @@
+{
+  "model": "LiquidAI/d1-2B",
+  "benchmark": "Hanno-Labs/decision-bench",
+  "split": "eval",
+  "rows": 1565,
+  "n": 1565,
+  "accuracy": 0.6338658146964856,
+  "nll": 0.9333471682463929,
+  "brier": 0.45101948246027257,
+  "ece": 0.044773390533129946,
+  "by_primitive": {
+    "ordinal_scoring": {
+      "n": 142,
+      "accuracy": 0.4295774647887324,
+      "nll": 1.8746878255118757,
+      "brier": 0.7679486561148359,
+      "ece": 0.2915928996178949
+    },
+    "candidate_selection": {
+      "n": 893,
+      "accuracy": 0.5363941769316909,
+      "nll": 1.1348712329104207,
+      "brier": 0.543667454690635,
+      "ece": 0.051484785687556094
+    },
+    "binary_classification": {
+      "n": 530,
+      "accuracy": 0.8528301886792453,
+      "nll": 0.3415898794224763,
+      "brier": 0.21000329027034487,
+      "ece": 0.04181434994420001
+    }
+  },
+  "latency_s": 103.77784200001042,
+  "attention": "sdpa-fallback (flash attention unavailable on Windows PyTorch build)"
+}

benchmarks/d1_2b_typed_test.json

@@ -0,0 +1,70 @@
+{
+  "model": "LiquidAI/d1-2B",
+  "benchmark": "LocalLLaMA/typed-decisions",
+  "split": "test",
+  "mode": "zero-shot",
+  "n_cases": 400,
+  "n_decisions": 2000,
+  "accuracy": 0.6325,
+  "kl_from_gold": 0.2957730492630638,
+  "brier": 0.1439612976496623,
+  "ece": 0.038993063986546,
+  "by_kind": {
+    "choice": {
+      "n_decisions": 600,
+      "accuracy": 0.6233333333333333,
+      "kl_from_gold": 0.32338565053955876,
+      "brier": 0.14524759307617283,
+      "ece": 0.04631447069924948
+    },
+    "noul": {
+      "n_decisions": 600,
+      "accuracy": 0.7683333333333333,
+      "kl_from_gold": 0.22672597347530257,
+      "brier": 0.13186409573157287,
+      "ece": 0.06072048330627751
+    },
+    "score": {
+      "n_decisions": 800,
+      "accuracy": 0.5375,
+      "kl_from_gold": 0.32684890514651355,
+      "brier": 0.1520694775183465,
+      "ece": 0.037082597492069366
+    }
+  },
+  "by_workflow": {
+    "agent_trace_observability": {
+      "n_decisions": 500,
+      "accuracy": 0.554,
+      "kl_from_gold": 0.2749524761253962,
+      "brier": 0.14702185932793024,
+      "ece": 0.07304823447088081
+    },
+    "customer_service": {
+      "n_decisions": 500,
+      "accuracy": 0.642,
+      "kl_from_gold": 0.320992938711665,
+      "brier": 0.15642483940562715,
+      "ece": 0.08308475343289316
+    },
+    "invoice_processing": {
+      "n_decisions": 500,
+      "accuracy": 0.712,
+      "kl_from_gold": 0.3098222087702867,
+      "brier": 0.13534537250903614,
+      "ece": 0.06076691123801671
+    },
+    "security_incidents": {
+      "n_decisions": 500,
+      "accuracy": 0.622,
+      "kl_from_gold": 0.2773245734449073,
+      "brier": 0.13705311935605577,
+      "ece": 0.13615814902686033
+    }
+  },
+  "latency_ms_per_case_amortized": 81.20556874993781,
+  "latency_ms_single_p50": 82.20564998919144,
+  "latency_ms_single_p95": 96.76197000808314,
+  "params_m": 1918.18,
+  "attention": "sdpa-fallback (flash attention unavailable on Windows PyTorch build)"
+}

benchmarks/d1_3b_decisionbench_1565.json

@@ -0,0 +1,36 @@
+{
+  "model": "d1-3B",
+  "benchmark": "Hanno-Labs/decision-bench",
+  "split": "eval",
+  "rows": 1565,
+  "n": 1565,
+  "accuracy": 0.6485623003194888,
+  "nll": 1.0344951306950594,
+  "brier": 0.4937204370168296,
+  "ece": 0.11743752614111892,
+  "by_primitive": {
+    "ordinal_scoring": {
+      "n": 142,
+      "accuracy": 0.43661971830985913,
+      "nll": 2.1940984014033558,
+      "brier": 0.8601191725325842,
+      "ece": 0.3753595340789747
+    },
+    "candidate_selection": {
+      "n": 893,
+      "accuracy": 0.5599104143337066,
+      "nll": 1.2599422288282047,
+      "brier": 0.6028164026417906,
+      "ece": 0.15850576690027032
+    },
+    "binary_classification": {
+      "n": 530,
+      "accuracy": 0.8547169811320755,
+      "nll": 0.34395187961302803,
+      "brier": 0.21173681862753252,
+      "ece": 0.020618797239968317
+    }
+  },
+  "latency_s": 142.38109049998457,
+  "attention": "sdpa-fallback (flash attention unavailable on Windows PyTorch build)"
+}

benchmarks/d1_3b_typed_test.json

@@ -0,0 +1,36 @@
+{
+  "model": "d1-3B",
+  "benchmark": "LocalLLaMA/typed-decisions",
+  "split": "test",
+  "n": 2000,
+  "accuracy": 0.6495,
+  "kl": 0.28536580145857754,
+  "brier": 0.14156798412867302,
+  "ece": 0.03433960353524046,
+  "by_kind": {
+    "choice": {
+      "n": 600,
+      "accuracy": 0.6033333333333334,
+      "kl": 0.38308227977572346,
+      "brier": 0.18394841063198902,
+      "ece": 0.06404703890449398
+    },
+    "noul": {
+      "n": 600,
+      "accuracy": 0.7933333333333333,
+      "kl": 0.14307420163370385,
+      "brier": 0.09547091383837804,
+      "ece": 0.032054926390584396
+    },
+    "score": {
+      "n": 800,
+      "accuracy": 0.57625,
+      "kl": 0.3187971425893734,
+      "brier": 0.14435546696890725,
+      "ece": 0.04930493936736953
+    }
+  },
+  "latency_ms_single_p50": 63.934250007150695,
+  "latency_s": 120.74863280000864,
+  "attention": "sdpa-fallback (flash attention unavailable on Windows PyTorch build)"
+}

benchmarks/pluto21_decisionbench_1565.json

@@ -0,0 +1,37 @@
+{
+  "model": "pluto-2.1-1.7b-step2000",
+  "benchmark": "Hanno-Labs/decision-bench",
+  "split": "eval",
+  "rows": 1565,
+  "n": 1565,
+  "accuracy": 0.7207667731629392,
+  "nll": 0.743893769561289,
+  "brier": 0.37302688896067215,
+  "ece": 0.05858011190501126,
+  "by_primitive": {
+    "ordinal_scoring": {
+      "n": 142,
+      "accuracy": 0.29577464788732394,
+      "nll": 1.5536898785809998,
+      "brier": 0.699555331279626,
+      "ece": 0.155038637254273
+    },
+    "candidate_selection": {
+      "n": 893,
+      "accuracy": 0.7491601343784995,
+      "nll": 0.7901911483528816,
+      "brier": 0.3715472431253504,
+      "ece": 0.09278790573727526
+    },
+    "binary_classification": {
+      "n": 530,
+      "accuracy": 0.7867924528301887,
+      "nll": 0.44892281344489054,
+      "brier": 0.28803497371850384,
+      "ece": 0.04888898737294541
+    }
+  },
+  "latency_s": 191.43742590001784,
+  "params_b": 1.738,
+  "lineage": "Qwen3-1.7B -> Bosun 3.1 1.7B -> Pluto-2.1 2000-step fine-tune"
+}

benchmarks/pluto21_decisionbench_390.json

@@ -0,0 +1,36 @@
+{
+  "model": "pluto-2.1-1.7b-step2000",
+  "benchmark": "Hanno-Labs/decision-bench",
+  "split": "eval",
+  "rows": 390,
+  "n": 390,
+  "accuracy": 0.7307692307692307,
+  "nll": 0.782475716850374,
+  "brier": 0.38374544733999405,
+  "ece": 0.07872579764729119,
+  "by_primitive": {
+    "ordinal_scoring": {
+      "n": 36,
+      "accuracy": 0.3055555555555556,
+      "nll": 1.5345522148320503,
+      "brier": 0.6649819753906753,
+      "ece": 0.14113543406196793
+    },
+    "candidate_selection": {
+      "n": 222,
+      "accuracy": 0.7297297297297297,
+      "nll": 0.8837282527930221,
+      "brier": 0.41591003706917623,
+      "ece": 0.0940529449476535
+    },
+    "binary_classification": {
+      "n": 132,
+      "accuracy": 0.8484848484848485,
+      "nll": 0.4070755887700082,
+      "brier": 0.2529495842361836,
+      "ece": 0.10125961085499781
+    }
+  },
+  "latency_s": 47.35091979999561,
+  "params_b": 1.738
+}

benchmarks/pluto21_typed_test.json

@@ -0,0 +1,37 @@
+{
+  "model": "pluto-2.1-1.7b-step2000",
+  "benchmark": "LocalLLaMA/typed-decisions",
+  "split": "test",
+  "mode": "fitted",
+  "n": 2000,
+  "accuracy": 0.6695,
+  "kl": 0.21389316512223397,
+  "brier": 0.12613537464071567,
+  "ece": 0.05558588827160893,
+  "by_kind": {
+    "choice": {
+      "n": 600,
+      "accuracy": 0.725,
+      "kl": 0.2037330221309278,
+      "brier": 0.10235709260413375,
+      "ece": 0.10109364924647721
+    },
+    "noul": {
+      "n": 600,
+      "accuracy": 0.735,
+      "kl": 0.15611870023697572,
+      "brier": 0.12411358328130827,
+      "ece": 0.05807213359097196
+    },
+    "score": {
+      "n": 800,
+      "accuracy": 0.57875,
+      "kl": 0.26484412102965726,
+      "brier": 0.14548542968770764,
+      "ece": 0.036768776402162195
+    }
+  },
+  "latency_ms_single_p50": 86.26855001784861,
+  "params_b": 1.738,
+  "lineage": "Qwen3-1.7B -> Bosun 3.1 1.7B -> Pluto-2.1 2000-step fine-tune"
+}

benchmarks/pluto2_decisionbench_1565.json

@@ -0,0 +1,157 @@
+{
+  "model": "C:\\Users\\henry\\new-omida-project\\pluto-2\\artifacts\\pluto-2",
+  "benchmark": "Hanno-Labs/decision-bench",
+  "split": "eval",
+  "requested_rows": 1992,
+  "included_rows": 1565,
+  "skipped_options": 403,
+  "skipped_prompt": 24,
+  "max_len": 2048,
+  "max_options": 16,
+  "batch_size": 8,
+  "n": 1565,
+  "accuracy": 0.5789137380191693,
+  "nll": 1.086787512493374,
+  "brier": 0.5476896818997071,
+  "ece": 0.03040878511978975,
+  "by_primitive": {
+    "ordinal_scoring": {
+      "n": 142,
+      "accuracy": 0.31690140845070425,
+      "nll": 1.702245548212995,
+      "brier": 0.7919354885271652,
+      "ece": 0.23922968191870858
+    },
+    "candidate_selection": {
+      "n": 893,
+      "accuracy": 0.5139977603583427,
+      "nll": 1.3175748448567437,
+      "brier": 0.6268954549540386,
+      "ece": 0.05335862384965334
+    },
+    "binary_classification": {
+      "n": 530,
+      "accuracy": 0.7584905660377359,
+      "nll": 0.5330363259411564,
+      "brier": 0.34879598401552414,
+      "ece": 0.06172943167517076
+    }
+  },
+  "by_candidate_count_bucket": {
+    "2": {
+      "n": 571,
+      "accuracy": 0.7758318739054291,
+      "nll": 0.5050864657848015,
+      "brier": 0.327164851226164,
+      "ece": 0.06657381248932939
+    },
+    "3": {
+      "n": 52,
+      "accuracy": 0.7307692307692307,
+      "nll": 0.8230841673369436,
+      "brier": 0.4656287957639696,
+      "ece": 0.1768807057572377
+    },
+    "4": {
+      "n": 417,
+      "accuracy": 0.38369304556354916,
+      "nll": 1.3917934761391375,
+      "brier": 0.7077339030970196,
+      "ece": 0.1366008560847453
+    },
+    "5": {
+      "n": 53,
+      "accuracy": 0.41509433962264153,
+      "nll": 1.3906180619397066,
+      "brier": 0.6965652674513569,
+      "ece": 0.12229648836302624
+    },
+    "6": {
+      "n": 62,
+      "accuracy": 0.5161290322580645,
+      "nll": 1.4950352941490053,
+      "brier": 0.7103141629730844,
+      "ece": 0.20879640991101237
+    },
+    "7": {
+      "n": 45,
+      "accuracy": 0.4444444444444444,
+      "nll": 1.6899349194297915,
+      "brier": 0.7423954014990986,
+      "ece": 0.12905421665120007
+    },
+    "8": {
+      "n": 84,
+      "accuracy": 0.7738095238095238,
+      "nll": 0.9060157136010754,
+      "brier": 0.3785204070880192,
+      "ece": 0.24301153029444017
+    },
+    "9": {
+      "n": 204,
+      "accuracy": 0.36764705882352944,
+      "nll": 1.8435587409953924,
+      "brier": 0.812923101907946,
+      "ece": 0.22095318662488653
+    },
+    "10": {
+      "n": 20,
+      "accuracy": 0.55,
+      "nll": 1.3466658370996674,
+      "brier": 0.581430754465156,
+      "ece": 0.19840851353183617
+    },
+    "11": {
+      "n": 11,
+      "accuracy": 0.45454545454545453,
+      "nll": 1.742375978409565,
+      "brier": 0.7057783122595841,
+      "ece": 0.14119367689162138
+    },
+    "12": {
+      "n": 5,
+      "accuracy": 0.6,
+      "nll": 1.60027353448161,
+      "brier": 0.7113295784136134,
+      "ece": 0.3569272335501148
+    },
+    "13": {
+      "n": 2,
+      "accuracy": 0.0,
+      "nll": 2.7291098611081326,
+      "brier": 1.0117958954704067,
+      "ece": 0.26010035168011614
+    },
+    "14": {
+      "n": 5,
+      "accuracy": 0.2,
+      "nll": 2.0200694744604966,
+      "brier": 0.8290373434160234,
+      "ece": 0.08302967212398918
+    },
+    "15": {
+      "n": 4,
+      "accuracy": 0.75,
+      "nll": 1.4792958719146407,
+      "brier": 0.6844986526248836,
+      "ece": 0.4951474916901011
+    },
+    "16": {
+      "n": 30,
+      "accuracy": 0.9333333333333333,
+      "nll": 0.6343312692816507,
+      "brier": 0.23775440952147306,
+      "ece": 0.2986224605056543
+    }
+  },
+  "prompt_state_truncated": 48,
+  "latency_s": 21.23680189999868,
+  "latency_ms_per_row": 13.56984146964772,
+  "length_tokens": {
+    "p50": 250,
+    "p90": 662,
+    "p95": 1086,
+    "p99": 2048,
+    "max": 2048
+  }
+}

benchmarks/pluto2_typed_test.json

@@ -0,0 +1,69 @@
+{
+  "model": "artifacts/pluto-2-typed",
+  "benchmark": "LocalLLaMA/typed-decisions",
+  "split": "test",
+  "mode": "fitted",
+  "n_cases": 400,
+  "n_decisions": 2000,
+  "accuracy": 0.803,
+  "kl_from_gold": 0.11923040849736305,
+  "brier": 0.0672323907787711,
+  "ece": 0.07477404832751905,
+  "by_kind": {
+    "choice": {
+      "n_decisions": 600,
+      "accuracy": 0.78,
+      "kl_from_gold": 0.15241561932803213,
+      "brier": 0.081281756228142,
+      "ece": 0.07988290969168543
+    },
+    "noul": {
+      "n_decisions": 600,
+      "accuracy": 0.8516666666666667,
+      "kl_from_gold": 0.06329251245082283,
+      "brier": 0.04614648872056481,
+      "ece": 0.04405483458049834
+    },
+    "score": {
+      "n_decisions": 800,
+      "accuracy": 0.78375,
+      "kl_from_gold": 0.13629492240926638,
+      "brier": 0.07250979323539765,
+      "ece": 0.10546664498729107
+    }
+  },
+  "by_workflow": {
+    "agent_trace_observability": {
+      "n_decisions": 500,
+      "accuracy": 0.764,
+      "kl_from_gold": 0.1149915606592059,
+      "brier": 0.06838644564523914,
+      "ece": 0.09413051276996476
+    },
+    "customer_service": {
+      "n_decisions": 500,
+      "accuracy": 0.83,
+      "kl_from_gold": 0.12424451907243046,
+      "brier": 0.0661457304954968,
+      "ece": 0.07617299813519858
+    },
+    "invoice_processing": {
+      "n_decisions": 500,
+      "accuracy": 0.846,
+      "kl_from_gold": 0.1235060928782072,
+      "brier": 0.06848787497824028,
+      "ece": 0.05575175429481842
+    },
+    "security_incidents": {
+      "n_decisions": 500,
+      "accuracy": 0.772,
+      "kl_from_gold": 0.11417946137960859,
+      "brier": 0.06590951199610816,
+      "ece": 0.08430837784308685
+    }
+  },
+  "latency_ms_per_case_amortized": 50.90324725002574,
+  "latency_ms_single_p50": 42.323649977333844,
+  "latency_ms_single_p95": 43.27478500053985,
+  "params_m": 395.85
+}