File size: 4,942 Bytes
2778af7 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 | # Pluto results
All Pluto numbers below were measured on this machine: **RTX 5060 Ti 16GB,
Windows, PyTorch 2.11.0+cu128, bf16**. The Laya numbers marked *measured* were
run by us with `laya 0.4.17` on the same test rows and scored with the same
metric code. Published numbers are cited as published.
## TL;DR
| Model | Params | Typed Decisions accuracy | KL from gold | Brier | ECE | p50 |
|---|---:|---:|---:|---:|---:|---:|
| **Pluto-Ensemble** | 332.8M | **0.7515** | **0.1183** | **0.0649** | **0.1400** | **54.2 ms** |
| **Pluto-typed (single)** | 149.1M | 0.7275 | 0.1391 | 0.0765 | 0.1067 | 26.9 ms |
| Laya-typed *(measured here)* | 421.3M | 0.7680 | 0.1171 | 0.0615 | 0.2152 | 66.2 ms |
| Laya base *(measured here)* | 421.3M | 0.3620 | 0.5724 | 0.3157 | 0.1742 | 66.0 ms |
| Jev 1.13 *(published)* | 27B-class API | 0.727 | 1.442 | 0.148 | 0.144 | ~710 ms |
| Liquid d1 *(published)* | API | 0.742 | 0.475 | 0.155 | 0.124 | ~525 ms |
Pluto-Ensemble beats Jev 1.13 and d1 on accuracy while being orders of magnitude
smaller and faster. It is **1.65 accuracy points behind Laya-typed**, but has
**35% lower ECE**, comparable KL, 18% lower latency, and 21% fewer parameters.
Pluto-typed (single checkpoint) is 2.5x faster than Laya-typed and 2.8x smaller,
at 0.7275 accuracy.
## Independent Typed Decisions benchmark
Source: [`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions),
`all/test`, 400 held-out cases / 2,000 probabilistic decisions across four
workflows. The benchmark's own scoring definition is used: argmax accuracy, mean
KL(gold || prediction), mean Brier, and 10-bin ECE over the top probability.
### Accuracy by primitive
| Model | choice (600) | noul (600) | score (800) | Overall |
|---|---:|---:|---:|---:|
| **Pluto-Ensemble** | 0.7167 | **0.8417** | 0.7100 | **0.7515** |
| **Pluto-typed** | 0.7283 | 0.8333 | 0.6475 | 0.7275 |
| Laya-typed *(measured)* | **0.7350** | 0.8567 | **0.7263** | **0.7680** |
| Laya base *(measured)* | 0.2883 | 0.4850 | 0.3250 | 0.3620 |
| Jev 1.13 *(published overall)* | — | — | — | 0.727 |
| d1 *(published overall)* | — | — | — | 0.742 |
The specialist models (`Pluto-typed`, `Pluto-Ensemble`, `Laya-typed`) are fitted
on the benchmark's `train` split. Pluto's test split was never used for training,
early stopping, model selection, or calibration.
### Per-workflow accuracy
| Workflow | Pluto-Ensemble | Pluto-typed | Laya-typed |
|---|---:|---:|---:|
| agent_trace_observability | 0.700 | 0.712 | 0.724 |
| customer_service | 0.754 | 0.702 | 0.760 |
| invoice_processing | 0.834 | 0.830 | 0.814 |
| security_incidents | 0.718 | 0.666 | 0.774 |
### Latency
Single-case p50 on the RTX 5060 Ti, five questions per case, tokenization
included:
- Pluto-typed: **26.9 ms** (p95 36.1 ms)
- Pluto-Ensemble: **54.2 ms** (p95 66.7 ms)
- Laya-typed: **66.2 ms** (p95 89.4 ms)
## General decision engine
The general checkpoint (`pluto-149m`) is trained on 655,806 calibrated decisions
across 66 domains from `SargeDev/jev-distill-corpus-v3`. Evaluation is on the
corpus's held-out `test_set_30k` split (29,955 decisions; never trained on).
| Primitive | Decisions | Accuracy | ECE |
|---|---:|---:|---:|
| choice | 9,199 | 0.5982 | 0.0422 |
| noul | 12,229 | 0.8275 | 0.1074 |
| score | 8,527 | 0.6074 | 0.0857 |
| **all** | **29,955** | **0.6944** | **0.0811** |
Soft agreement with the teacher distribution: 0.537. Score MAE: 0.413.
Latency for a 3-question request: **25.8 ms p50** on the same GPU.
The broad-corpus abstention head is weak (AUROC 0.590), so the general checkpoint
does **not** claim a certified 5% risk gate. The specialist models use calibrated
top-1 probabilities and should be validated on their target domain before
high-stakes use.
## Reproduce
```bash
# General held-out corpus
python -m eval.evaluate --model artifacts/pluto-149m --split test30k
# Independent Typed Decisions benchmark
python -m eval.typed_decisions --engine task \
--model artifacts/pluto-typed-task --out eval/typed_task.json
python -m eval.typed_decisions --engine ensemble \
--model artifacts/pluto-typed-ensemble/config.json --out eval/typed_ensemble.json
```
Prediction files are written next to the summaries and contain every decision,
so the numbers can be re-scored without a GPU.
## Honest limits
- Laya-typed remains 1.65 points ahead on raw accuracy on the same test split.
- The ensemble needs two checkpoints (333M total) and is 2x slower than the single
specialist.
- The general 66-domain checkpoint is a System-1 model; it is not a reasoning
model and is not evaluated on MMLU-Pro, GPQA, HLE or the full Decision Index.
- All models are English-only.
- The Typed Decisions benchmark is synthetic and public. Results should be
validated on production data before operational use.
|