Blame for code/README.md

# Pluto

**A compact, calibrated, typed decision model.** Give Pluto a state and typed
questions; it returns a probability distribution for every option in a single
forward pass. No generation, no JSON parsing, no retries.

- **160.6M parameter general model** (ModernBERT-base encoder + a decision tower)
- **149.1M parameter Typed Decisions specialist**
- **One forward pass** for `choice`, `noul` and `score` questions
- **Calibrated** probabilities and measurable ECE
- **Fast**: 26.9 ms p50 for five questions on an RTX 5060 Ti
- **Apache 2.0**, small enough to run on a 16GB consumer GPU

## Headline results

Independent [`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions)
benchmark, 400 held-out cases / 2,000 decisions, scored with the benchmark's own
metric definition. Laya numbers were measured by us on the same rows.

| Model | Params | Accuracy | KL | Brier | ECE | p50 |
|---|---:|---:|---:|---:|---:|---:|
| **Pluto-Ensemble** | 332.8M | **0.7515** | **0.118** | **0.0649** | 0.140 | **54.2 ms** |
| **Pluto-typed** | 149.1M | 0.7275 | 0.139 | 0.0765 | **0.107** | **26.9 ms** |
| Laya-typed (measured) | 421.3M | 0.7680 | 0.117 | 0.0615 | 0.215 | 66.2 ms |
| Jev 1.13 (published) | ~27B API | 0.727 | 1.442 | 0.148 | 0.144 | ~710 ms |
| Liquid d1 (published) | API | 0.742 | 0.475 | 0.155 | 0.124 | ~525 ms |

Pluto-Ensemble beats Jev 1.13 and d1 on accuracy while being 9-81x smaller and
10-13x faster. It is 1.65 points behind Laya-typed on raw accuracy, but has 35%
lower ECE, comparable KL, 18% lower latency, and 21% fewer parameters.

On the general 66-domain held-out corpus (29,955 decisions, never trained on):
**0.694 accuracy, 0.081 ECE**. Full breakdowns and methodology: [`RESULTS.md`](RESULTS.md).

## Quickstart

```bash

pip install -e ".[serve]"

```

```python

from pluto import PlutoEngine



engine = PlutoEngine("artifacts/pluto-149m")



state = "Hi, we were billed twice for March. Refund the duplicate or we cancel."

questions = {

    "department": {

        "type": "choice",

        "instructions": "Which department should handle this?",

        "criteria": {

            "billing": "invoices, payments and refunds",

            "technical": "bugs and outages",

            "other": "everything else",

        },

    },

    "churn_risk": {

        "type": "noul",

        "instructions": "Does the customer threaten to cancel?",

    },

    "urgency": {

        "type": "score",

        "instructions": "How urgent is this?",

        "criteria": ["not urgent", "soon", "blocking"],

    },

}



result = engine.predict(state, questions)

print(result["answers"]["department"]["choice"])

print(result["answers"]["churn_risk"]["noul"])       # P(true)

print(result["answers"]["urgency"]["score"])          # expected level

print(result["escalate"])

```

CLI:

```bash

pluto --model artifacts/pluto-149m predict \

  --state "Invoice #4411 was charged twice." \

  --questions questions.json

```

Jev-compatible server:

```bash

pluto --model artifacts/pluto-149m serve --port 8000

curl -s localhost:8000/v1/systemone -H 'Content-Type: application/json' \

  -d '{"state":"refund my duplicate charge","questions":{"billing":{"type":"noul","instructions":"Is this about billing?"}}}'

```

### Specialist checkpoints

```python

from pluto.task_specialist import TaskSpecialistEngine

from pluto.ensemble import EnsembleEngine



single = TaskSpecialistEngine("artifacts/pluto-typed-task")

best = EnsembleEngine("artifacts/pluto-typed-ensemble/config.json")

```

Specialist model card: [`MODEL_CARD_TYPED.md`](MODEL_CARD_TYPED.md).

The specialist supports the Typed Decisions question names (`action`,
`needs_review`, `outcome`, `risk`, `urgency`, `category`, ...). The general model
supports arbitrary typed questions.

## Typed questions

| Type | Input | Output |
|---|---|---|
| `choice` | `criteria`: keyed or ordered option descriptions | chosen key + distribution |
| `noul` | no criteria | `P(true)` + distribution over false/true |
| `score` | `criteria`: ordered levels (2-12) | expected level + distribution |

The answer space is defined at request time. New schemas need no retraining for
the general model.

## Architecture

```

state + question + options

        |

  ModernBERT-base (149M, bidirectional, 640-token budget)

        |

  one hidden state per option marker

        |

  2-layer permutation-equivariant option tower

        |

  scalar scorer --> option distribution

  confidence head --> escalation signal

```

The decision tower is trained from scratch. The backbone is initialized from
`answerdotai/ModernBERT-base` and fine-tuned end to end. The training objective
combines KL distillation from calibrated soft labels, top-1 cross-entropy,
ordinal RPS for `score`, and a confidence BCE auxiliary.

The Typed Decisions specialist adds 13 per-question linear heads on the same
backbone; the ensemble averages the specialist with a DeBERTa-v3-base sibling.
Both are shipped as separate checkpoints.

## Calibration and abstention

- The general checkpoint is calibrated on 30k held-out decisions (ECE **0.081**).
- The Typed Decisions specialist is temperature-calibrated per primitive
  (ECE **0.107**, versus Laya-typed's **0.215** on the same rows).
- The general checkpoint has a confidence head but its broad-corpus gate is not
  certified at 5% risk (AUROC 0.590); do not treat `escalate` as a guarantee.
- Validate calibration and escalation thresholds on your own domain before
  high-stakes automation.

## Training

```bash

python -m train.prepare --source data/jev-distill-v3 --out data/pluto-corpus

python -m train.train --data data/pluto-corpus --out artifacts/pluto-149m \

  --max-steps 4000 --batch-size 32 --freeze-layers 8



python scripts/make_typed_dataset.py --replay 0 --out data/pluto-typed-only

python -m train.task_specialist --data data/pluto-typed-only \

  --out artifacts/pluto-typed-task --init artifacts/pluto-149m

```

Full commands and hardware notes are in [`RESULTS.md`](RESULTS.md).

## Repository layout

```

pluto/        inference engine, schema, general model, specialist, ensemble, server, CLI

train/        data preparation, general/specialist training, metrics

eval/         held-out evaluator, Typed Decisions harness, latency, Laya comparison

scripts/      dataset builders, calibration/finalization, HF publisher, launch copy

space/        Gradio demo for Hugging Face Spaces

tests/        schema and calibration tests

```

## Limitations

- English only.
- A System-1 decision model, not a reasoning model; it is not evaluated on
  MMLU-Pro, GPQA, HLE or the full Decision Index.
- Laya-typed remains ahead on raw accuracy on the Typed Decisions test split.
- The general broad-corpus abstention head is weak; use the specialist for
  risk-sensitive workflows and validate the gate on production data.
- Benchmark results are synthetic and public; validate on your own domain.

## License

Apache 2.0. The distillation corpus and the Typed Decisions benchmark are
Apache 2.0. See `NOTICE` for attribution.

Built by **Omida**.