Pluto results
All Pluto numbers below were measured on this machine: RTX 5060 Ti 16GB,
Windows, PyTorch 2.11.0+cu128, bf16. The Laya numbers marked measured were
run by us with laya 0.4.17 on the same test rows and scored with the same
metric code. Published numbers are cited as published.
TL;DR
| Model | Params | Typed Decisions accuracy | KL from gold | Brier | ECE | p50 |
|---|---|---|---|---|---|---|
| Pluto-Ensemble | 332.8M | 0.7515 | 0.1183 | 0.0649 | 0.1400 | 54.2 ms |
| Pluto-typed (single) | 149.1M | 0.7275 | 0.1391 | 0.0765 | 0.1067 | 26.9 ms |
| Laya-typed (measured here) | 421.3M | 0.7680 | 0.1171 | 0.0615 | 0.2152 | 66.2 ms |
| Laya base (measured here) | 421.3M | 0.3620 | 0.5724 | 0.3157 | 0.1742 | 66.0 ms |
| Jev 1.13 (published) | 27B-class API | 0.727 | 1.442 | 0.148 | 0.144 | ~710 ms |
| Liquid d1 (published) | API | 0.742 | 0.475 | 0.155 | 0.124 | ~525 ms |
Pluto-Ensemble beats Jev 1.13 and d1 on accuracy while being orders of magnitude smaller and faster. It is 1.65 accuracy points behind Laya-typed, but has 35% lower ECE, comparable KL, 18% lower latency, and 21% fewer parameters. Pluto-typed (single checkpoint) is 2.5x faster than Laya-typed and 2.8x smaller, at 0.7275 accuracy.
Independent Typed Decisions benchmark
Source: LocalLLaMA/typed-decisions,
all/test, 400 held-out cases / 2,000 probabilistic decisions across four
workflows. The benchmark's own scoring definition is used: argmax accuracy, mean
KL(gold || prediction), mean Brier, and 10-bin ECE over the top probability.
Accuracy by primitive
| Model | choice (600) | noul (600) | score (800) | Overall |
|---|---|---|---|---|
| Pluto-Ensemble | 0.7167 | 0.8417 | 0.7100 | 0.7515 |
| Pluto-typed | 0.7283 | 0.8333 | 0.6475 | 0.7275 |
| Laya-typed (measured) | 0.7350 | 0.8567 | 0.7263 | 0.7680 |
| Laya base (measured) | 0.2883 | 0.4850 | 0.3250 | 0.3620 |
| Jev 1.13 (published overall) | — | — | — | 0.727 |
| d1 (published overall) | — | — | — | 0.742 |
The specialist models (Pluto-typed, Pluto-Ensemble, Laya-typed) are fitted
on the benchmark's train split. Pluto's test split was never used for training,
early stopping, model selection, or calibration.
Per-workflow accuracy
| Workflow | Pluto-Ensemble | Pluto-typed | Laya-typed |
|---|---|---|---|
| agent_trace_observability | 0.700 | 0.712 | 0.724 |
| customer_service | 0.754 | 0.702 | 0.760 |
| invoice_processing | 0.834 | 0.830 | 0.814 |
| security_incidents | 0.718 | 0.666 | 0.774 |
Latency
Single-case p50 on the RTX 5060 Ti, five questions per case, tokenization included:
- Pluto-typed: 26.9 ms (p95 36.1 ms)
- Pluto-Ensemble: 54.2 ms (p95 66.7 ms)
- Laya-typed: 66.2 ms (p95 89.4 ms)
General decision engine
The general checkpoint (pluto-149m) is trained on 655,806 calibrated decisions
across 66 domains from SargeDev/jev-distill-corpus-v3. Evaluation is on the
corpus's held-out test_set_30k split (29,955 decisions; never trained on).
| Primitive | Decisions | Accuracy | ECE |
|---|---|---|---|
| choice | 9,199 | 0.5982 | 0.0422 |
| noul | 12,229 | 0.8275 | 0.1074 |
| score | 8,527 | 0.6074 | 0.0857 |
| all | 29,955 | 0.6944 | 0.0811 |
Soft agreement with the teacher distribution: 0.537. Score MAE: 0.413. Latency for a 3-question request: 25.8 ms p50 on the same GPU.
The broad-corpus abstention head is weak (AUROC 0.590), so the general checkpoint does not claim a certified 5% risk gate. The specialist models use calibrated top-1 probabilities and should be validated on their target domain before high-stakes use.
Reproduce
# General held-out corpus
python -m eval.evaluate --model artifacts/pluto-149m --split test30k
# Independent Typed Decisions benchmark
python -m eval.typed_decisions --engine task \
--model artifacts/pluto-typed-task --out eval/typed_task.json
python -m eval.typed_decisions --engine ensemble \
--model artifacts/pluto-typed-ensemble/config.json --out eval/typed_ensemble.json
Prediction files are written next to the summaries and contain every decision, so the numbers can be re-scored without a GPU.
Honest limits
- Laya-typed remains 1.65 points ahead on raw accuracy on the same test split.
- The ensemble needs two checkpoints (333M total) and is 2x slower than the single specialist.
- The general 66-domain checkpoint is a System-1 model; it is not a reasoning model and is not evaluated on MMLU-Pro, GPQA, HLE or the full Decision Index.
- All models are English-only.
- The Typed Decisions benchmark is synthetic and public. Results should be validated on production data before operational use.