Pluto-2.1 results
General-model comparison
Same rows and same scorer for every locally measured model. Mean is the mean
of Typed Decisions and DecisionBench matched accuracy.
| Model |
Params |
Typed Decisions |
DecisionBench matched |
Mean |
| Pluto-2.1 |
1.738B |
0.6695 |
0.7208 |
0.6952 |
| Pluto-2 400M |
0.397B |
0.7835 |
0.5789 |
0.6812 |
| Bosun 3.1 1.7B (initializer) |
1.738B |
0.4980 |
0.8288 |
0.6634 |
| d1-3B |
3.123B |
0.6495 |
0.6486 |
0.6490 |
| d1-2B |
1.918B |
0.6325 |
0.6339 |
0.6332 |
Pluto-2.1 clears d1-2B by +3.7 Typed and +8.7 DecisionBench points, and
clears d1-3B by +2.0 and +7.2 points. It has the highest mean accuracy
of the five models. Bosun retains the broad-only lead; Pluto-2 400M retains the
Typed-only lead. Pluto-2.1 owns the balance.
Distribution quality
| Benchmark |
Model |
Brier |
KL / NLL |
| Typed Decisions |
Pluto-2.1 |
0.1261 |
0.2139 |
| Typed Decisions |
d1-3B |
0.1416 |
0.2854 |
| Typed Decisions |
d1-2B |
0.1440 |
0.2958 |
| DecisionBench matched |
Pluto-2.1 |
0.3730 |
0.7439 |
| DecisionBench matched |
d1-3B |
0.4937 |
1.0345 |
| DecisionBench matched |
d1-2B |
0.4510 |
0.9333 |
Typed Decisions by primitive
| Primitive |
n |
Accuracy |
Brier |
ECE |
| choice |
600 |
0.7250 |
0.1024 |
0.1011 |
| noul |
600 |
0.7350 |
0.1241 |
0.0581 |
| score |
800 |
0.5788 |
0.1455 |
0.0368 |
| all |
2,000 |
0.6695 |
0.1261 |
0.0556 |
DecisionBench by primitive
| Primitive |
n |
Accuracy |
NLL |
ECE |
| binary classification |
530 |
0.7868 |
0.4489 |
0.0489 |
| candidate selection |
893 |
0.7492 |
0.7902 |
0.0928 |
| ordinal scoring |
142 |
0.2958 |
1.5537 |
0.1550 |
| all |
1,565 |
0.7208 |
0.7439 |
0.0586 |
Evaluation scope
Typed Decisions is the full 2,000-decision test split. DecisionBench is a
1,565-row matched subset limited to ≤16 candidates and ≤2,048-token prompts; it
is not the full 23,900-row suite. The full suite is not claimed.
Checkpoint selection
| Step |
Typed Decisions |
DecisionBench 1,565 |
| 1,000 |
0.6095 |
0.7534 |
| 1,500 |
0.5550 |
0.7655 |
| 2,000 |
0.6695 |
0.7208 |
The 2,000-step checkpoint is shipped because it is the only tested checkpoint
that clears both d1-2B and d1-3B on both measured benchmarks.
