File size: 2,469 Bytes
5b31db0 4649798 5b31db0 4649798 5b31db0 4649798 5b31db0 4649798 5b31db0 4649798 5b31db0 4649798 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 | # Pluto-2.1 results
## General-model comparison
Same rows and same scorer for every locally measured model. `Mean` is the mean
of Typed Decisions and DecisionBench matched accuracy.
| Model | Params | Typed Decisions | DecisionBench matched | Mean |
|---|---:|---:|---:|---:|
| **Pluto-2.1** | **1.738B** | **0.6695** | **0.7208** | **0.6952** |
| Pluto-2 400M | 0.397B | **0.7835** | 0.5789 | 0.6812 |
| Bosun 3.1 1.7B (initializer) | 1.738B | 0.4980 | **0.8288** | 0.6634 |
| d1-3B | 3.123B | 0.6495 | 0.6486 | 0.6490 |
| d1-2B | 1.918B | 0.6325 | 0.6339 | 0.6332 |
Pluto-2.1 clears d1-2B by **+3.7** Typed and **+8.7** DecisionBench points, and
clears d1-3B by **+2.0** and **+7.2** points. It has the highest mean accuracy
of the five models. Bosun retains the broad-only lead; Pluto-2 400M retains the
Typed-only lead. Pluto-2.1 owns the balance.
## Distribution quality
| Benchmark | Model | Brier | KL / NLL |
|---|---|---:|---:|
| Typed Decisions | **Pluto-2.1** | **0.1261** | **0.2139** |
| Typed Decisions | d1-3B | 0.1416 | 0.2854 |
| Typed Decisions | d1-2B | 0.1440 | 0.2958 |
| DecisionBench matched | **Pluto-2.1** | **0.3730** | **0.7439** |
| DecisionBench matched | d1-3B | 0.4937 | 1.0345 |
| DecisionBench matched | d1-2B | 0.4510 | 0.9333 |
## Typed Decisions by primitive
| Primitive | n | Accuracy | Brier | ECE |
|---|---:|---:|---:|---:|
| choice | 600 | 0.7250 | 0.1024 | 0.1011 |
| noul | 600 | 0.7350 | 0.1241 | 0.0581 |
| score | 800 | 0.5788 | 0.1455 | 0.0368 |
| **all** | **2,000** | **0.6695** | **0.1261** | **0.0556** |
## DecisionBench by primitive
| Primitive | n | Accuracy | NLL | ECE |
|---|---:|---:|---:|---:|
| binary classification | 530 | 0.7868 | 0.4489 | 0.0489 |
| candidate selection | 893 | 0.7492 | 0.7902 | 0.0928 |
| ordinal scoring | 142 | 0.2958 | 1.5537 | 0.1550 |
| **all** | **1,565** | **0.7208** | **0.7439** | **0.0586** |
## Evaluation scope
Typed Decisions is the full 2,000-decision test split. DecisionBench is a
1,565-row matched subset limited to ≤16 candidates and ≤2,048-token prompts; it
is not the full 23,900-row suite. The full suite is not claimed.
## Checkpoint selection
| Step | Typed Decisions | DecisionBench 1,565 |
|---:|---:|---:|
| 1,000 | 0.6095 | 0.7534 |
| 1,500 | 0.5550 | 0.7655 |
| **2,000** | **0.6695** | **0.7208** |
The 2,000-step checkpoint is shipped because it is the only tested checkpoint
that clears both d1-2B and d1-3B on both measured benchmarks.
|