| # Benchmark summaries |
|
|
| All numbers here were measured locally on one RTX 5060 Ti 16GB using the same |
| rows and scorer for every model. |
|
|
| ## Typed Decisions |
|
|
| - Full independent test split: 400 held-out cases, 2,000 probabilistic decisions. |
| - Scored with the benchmark metric code: argmax accuracy, KL, Brier, ECE. |
| - Models: Pluto-2.1, Pluto-2, d1-2B, d1-3B. d1 checkpoints are zero-shot. |
|
|
| ## DecisionBench |
|
|
| - Public `Hanno-Labs/decision-bench` evaluation split. |
| - Matched accessibility subset: 1,565 of 23,900 rows, limited to ≤16 candidates |
| and ≤2,048-token prompts. |
| - Models: Pluto-2.1, Pluto-2, Bosun 3.1 1.7B, d1-2B, d1-3B. |
| - This is not the full 23,900-row suite. No full-suite score is claimed. |
|
|
| ## Files |
|
|
| - `pluto21_*.json` — Pluto-2.1 step 2,000 |
| - `pluto2_*.json` — Pluto-2 400M |
| - `bosun_base_decisionbench_1565.json` — Bosun 3.1 1.7B initializer |
| - `d1_2b_*.json` — Liquid d1-2B |
| - `d1_3b_*.json` — Liquid d1-3B |
|
|
| The model card charts are generated from these values. |
|
|
