Pluto-1 private release candidate: dual-encoder ensemble, calibrated Typed Decisions specialist
2778af7 verified | # Pluto | |
| **A compact, calibrated, typed decision model.** Give Pluto a state and typed | |
| questions; it returns a probability distribution for every option in a single | |
| forward pass. No generation, no JSON parsing, no retries. | |
| - **160.6M parameter general model** (ModernBERT-base encoder + a decision tower) | |
| - **149.1M parameter Typed Decisions specialist** | |
| - **One forward pass** for `choice`, `noul` and `score` questions | |
| - **Calibrated** probabilities and measurable ECE | |
| - **Fast**: 26.9 ms p50 for five questions on an RTX 5060 Ti | |
| - **Apache 2.0**, small enough to run on a 16GB consumer GPU | |
| ## Headline results | |
| Independent [`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) | |
| benchmark, 400 held-out cases / 2,000 decisions, scored with the benchmark's own | |
| metric definition. Laya numbers were measured by us on the same rows. | |
| | Model | Params | Accuracy | KL | Brier | ECE | p50 | | |
| |---|---:|---:|---:|---:|---:|---:| | |
| | **Pluto-Ensemble** | 332.8M | **0.7515** | **0.118** | **0.0649** | 0.140 | **54.2 ms** | | |
| | **Pluto-typed** | 149.1M | 0.7275 | 0.139 | 0.0765 | **0.107** | **26.9 ms** | | |
| | Laya-typed (measured) | 421.3M | 0.7680 | 0.117 | 0.0615 | 0.215 | 66.2 ms | | |
| | Jev 1.13 (published) | ~27B API | 0.727 | 1.442 | 0.148 | 0.144 | ~710 ms | | |
| | Liquid d1 (published) | API | 0.742 | 0.475 | 0.155 | 0.124 | ~525 ms | | |
| Pluto-Ensemble beats Jev 1.13 and d1 on accuracy while being 9-81x smaller and | |
| 10-13x faster. It is 1.65 points behind Laya-typed on raw accuracy, but has 35% | |
| lower ECE, comparable KL, 18% lower latency, and 21% fewer parameters. | |
| On the general 66-domain held-out corpus (29,955 decisions, never trained on): | |
| **0.694 accuracy, 0.081 ECE**. Full breakdowns and methodology: [`RESULTS.md`](RESULTS.md). | |
| ## Quickstart | |
| ```bash | |
| pip install -e ".[serve]" | |
| ``` | |
| ```python | |
| from pluto import PlutoEngine | |
| engine = PlutoEngine("artifacts/pluto-149m") | |
| state = "Hi, we were billed twice for March. Refund the duplicate or we cancel." | |
| questions = { | |
| "department": { | |
| "type": "choice", | |
| "instructions": "Which department should handle this?", | |
| "criteria": { | |
| "billing": "invoices, payments and refunds", | |
| "technical": "bugs and outages", | |
| "other": "everything else", | |
| }, | |
| }, | |
| "churn_risk": { | |
| "type": "noul", | |
| "instructions": "Does the customer threaten to cancel?", | |
| }, | |
| "urgency": { | |
| "type": "score", | |
| "instructions": "How urgent is this?", | |
| "criteria": ["not urgent", "soon", "blocking"], | |
| }, | |
| } | |
| result = engine.predict(state, questions) | |
| print(result["answers"]["department"]["choice"]) | |
| print(result["answers"]["churn_risk"]["noul"]) # P(true) | |
| print(result["answers"]["urgency"]["score"]) # expected level | |
| print(result["escalate"]) | |
| ``` | |
| CLI: | |
| ```bash | |
| pluto --model artifacts/pluto-149m predict \ | |
| --state "Invoice #4411 was charged twice." \ | |
| --questions questions.json | |
| ``` | |
| Jev-compatible server: | |
| ```bash | |
| pluto --model artifacts/pluto-149m serve --port 8000 | |
| curl -s localhost:8000/v1/systemone -H 'Content-Type: application/json' \ | |
| -d '{"state":"refund my duplicate charge","questions":{"billing":{"type":"noul","instructions":"Is this about billing?"}}}' | |
| ``` | |
| ### Specialist checkpoints | |
| ```python | |
| from pluto.task_specialist import TaskSpecialistEngine | |
| from pluto.ensemble import EnsembleEngine | |
| single = TaskSpecialistEngine("artifacts/pluto-typed-task") | |
| best = EnsembleEngine("artifacts/pluto-typed-ensemble/config.json") | |
| ``` | |
| Specialist model card: [`MODEL_CARD_TYPED.md`](MODEL_CARD_TYPED.md). | |
| The specialist supports the Typed Decisions question names (`action`, | |
| `needs_review`, `outcome`, `risk`, `urgency`, `category`, ...). The general model | |
| supports arbitrary typed questions. | |
| ## Typed questions | |
| | Type | Input | Output | | |
| |---|---|---| | |
| | `choice` | `criteria`: keyed or ordered option descriptions | chosen key + distribution | | |
| | `noul` | no criteria | `P(true)` + distribution over false/true | | |
| | `score` | `criteria`: ordered levels (2-12) | expected level + distribution | | |
| The answer space is defined at request time. New schemas need no retraining for | |
| the general model. | |
| ## Architecture | |
| ``` | |
| state + question + options | |
| | | |
| ModernBERT-base (149M, bidirectional, 640-token budget) | |
| | | |
| one hidden state per option marker | |
| | | |
| 2-layer permutation-equivariant option tower | |
| | | |
| scalar scorer --> option distribution | |
| confidence head --> escalation signal | |
| ``` | |
| The decision tower is trained from scratch. The backbone is initialized from | |
| `answerdotai/ModernBERT-base` and fine-tuned end to end. The training objective | |
| combines KL distillation from calibrated soft labels, top-1 cross-entropy, | |
| ordinal RPS for `score`, and a confidence BCE auxiliary. | |
| The Typed Decisions specialist adds 13 per-question linear heads on the same | |
| backbone; the ensemble averages the specialist with a DeBERTa-v3-base sibling. | |
| Both are shipped as separate checkpoints. | |
| ## Calibration and abstention | |
| - The general checkpoint is calibrated on 30k held-out decisions (ECE **0.081**). | |
| - The Typed Decisions specialist is temperature-calibrated per primitive | |
| (ECE **0.107**, versus Laya-typed's **0.215** on the same rows). | |
| - The general checkpoint has a confidence head but its broad-corpus gate is not | |
| certified at 5% risk (AUROC 0.590); do not treat `escalate` as a guarantee. | |
| - Validate calibration and escalation thresholds on your own domain before | |
| high-stakes automation. | |
| ## Training | |
| ```bash | |
| python -m train.prepare --source data/jev-distill-v3 --out data/pluto-corpus | |
| python -m train.train --data data/pluto-corpus --out artifacts/pluto-149m \ | |
| --max-steps 4000 --batch-size 32 --freeze-layers 8 | |
| python scripts/make_typed_dataset.py --replay 0 --out data/pluto-typed-only | |
| python -m train.task_specialist --data data/pluto-typed-only \ | |
| --out artifacts/pluto-typed-task --init artifacts/pluto-149m | |
| ``` | |
| Full commands and hardware notes are in [`RESULTS.md`](RESULTS.md). | |
| ## Repository layout | |
| ``` | |
| pluto/ inference engine, schema, general model, specialist, ensemble, server, CLI | |
| train/ data preparation, general/specialist training, metrics | |
| eval/ held-out evaluator, Typed Decisions harness, latency, Laya comparison | |
| scripts/ dataset builders, calibration/finalization, HF publisher, launch copy | |
| space/ Gradio demo for Hugging Face Spaces | |
| tests/ schema and calibration tests | |
| ``` | |
| ## Limitations | |
| - English only. | |
| - A System-1 decision model, not a reasoning model; it is not evaluated on | |
| MMLU-Pro, GPQA, HLE or the full Decision Index. | |
| - Laya-typed remains ahead on raw accuracy on the Typed Decisions test split. | |
| - The general broad-corpus abstention head is weak; use the specialist for | |
| risk-sensitive workflows and validate the gate on production data. | |
| - Benchmark results are synthetic and public; validate on your own domain. | |
| ## License | |
| Apache 2.0. The distillation corpus and the Typed Decisions benchmark are | |
| Apache 2.0. See `NOTICE` for attribution. | |
| Built by **Omida**. | |