| # Pluto-2.1 results |
|
|
| ## General-model comparison |
|
|
| Same rows and same scorer for every locally measured model. `Mean` is the mean |
| of Typed Decisions and DecisionBench matched accuracy. |
|
|
| | Model | Params | Typed Decisions | DecisionBench matched | Mean | |
| |---|---:|---:|---:|---:| |
| | **Pluto-2.1** | **1.738B** | **0.6695** | **0.7208** | **0.6952** | |
| | Pluto-2 400M | 0.397B | **0.7835** | 0.5789 | 0.6812 | |
| | Bosun 3.1 1.7B (initializer) | 1.738B | 0.4980 | **0.8288** | 0.6634 | |
| | d1-3B | 3.123B | 0.6495 | 0.6486 | 0.6490 | |
| | d1-2B | 1.918B | 0.6325 | 0.6339 | 0.6332 | |
|
|
| Pluto-2.1 clears d1-2B by **+3.7** Typed and **+8.7** DecisionBench points, and |
| clears d1-3B by **+2.0** and **+7.2** points. It has the highest mean accuracy |
| of the five models. Bosun retains the broad-only lead; Pluto-2 400M retains the |
| Typed-only lead. Pluto-2.1 owns the balance. |
|
|
| ## Distribution quality |
|
|
| | Benchmark | Model | Brier | KL / NLL | |
| |---|---|---:|---:| |
| | Typed Decisions | **Pluto-2.1** | **0.1261** | **0.2139** | |
| | Typed Decisions | d1-3B | 0.1416 | 0.2854 | |
| | Typed Decisions | d1-2B | 0.1440 | 0.2958 | |
| | DecisionBench matched | **Pluto-2.1** | **0.3730** | **0.7439** | |
| | DecisionBench matched | d1-3B | 0.4937 | 1.0345 | |
| | DecisionBench matched | d1-2B | 0.4510 | 0.9333 | |
|
|
| ## Typed Decisions by primitive |
|
|
| | Primitive | n | Accuracy | Brier | ECE | |
| |---|---:|---:|---:|---:| |
| | choice | 600 | 0.7250 | 0.1024 | 0.1011 | |
| | noul | 600 | 0.7350 | 0.1241 | 0.0581 | |
| | score | 800 | 0.5788 | 0.1455 | 0.0368 | |
| | **all** | **2,000** | **0.6695** | **0.1261** | **0.0556** | |
|
|
| ## DecisionBench by primitive |
|
|
| | Primitive | n | Accuracy | NLL | ECE | |
| |---|---:|---:|---:|---:| |
| | binary classification | 530 | 0.7868 | 0.4489 | 0.0489 | |
| | candidate selection | 893 | 0.7492 | 0.7902 | 0.0928 | |
| | ordinal scoring | 142 | 0.2958 | 1.5537 | 0.1550 | |
| | **all** | **1,565** | **0.7208** | **0.7439** | **0.0586** | |
|
|
| ## Evaluation scope |
|
|
| Typed Decisions is the full 2,000-decision test split. DecisionBench is a |
| 1,565-row matched subset limited to ≤16 candidates and ≤2,048-token prompts; it |
| is not the full 23,900-row suite. The full suite is not claimed. |
|
|
| ## Checkpoint selection |
|
|
| | Step | Typed Decisions | DecisionBench 1,565 | |
| |---:|---:|---:| |
| | 1,000 | 0.6095 | 0.7534 | |
| | 1,500 | 0.5550 | 0.7655 | |
| | **2,000** | **0.6695** | **0.7208** | |
|
|
| The 2,000-step checkpoint is shipped because it is the only tested checkpoint |
| that clears both d1-2B and d1-3B on both measured benchmarks. |
|
|
