---
model_name: Open Decision Model 150M
license: apache-2.0
base_model: tasksource/ModernBERT-base-nli
library_name: transformers
pipeline_tag: text-classification
language: en
tags:
- llm-router
- model-routing
- agents
- ai-agents
- tool-risk
- guardrails
- zero-shot-classification
- multiple-choice
- modernbert
- calibrated-probabilities
- jev
- laya
- decision-model
- calibration
- mcp
- mcp-server
- mcp-server
- distillation
---

# Open Decision Model 150M: calibrated decisions for LLM agents

> **A small open decision model of the Jev / Laya type.** Give it a state and a list of options. It returns a calibrated probability for each option in about 50 ms on a CPU. It does not generate text.

**Try it with no download:** add the free MCP endpoint  to your agent.

**Try it with no download:** add the free MCP endpoint `https://decision-model.vercel.app/mcp` to your agent.

Use it where an agent must pick, not write:

- **LLM routing:** send a request to a small, medium or large model. Save cost.
- **Tool risk:** decide if a tool call needs human approval.
- **Yes/no checks:** "Can this task delete user data?" gives `p_yes`.
- **Any multiple-choice:** support intents, shopping options, difficulty levels. Any option count.

| | |
|---|---|
| **Size** | 149M parameters (ModernBERT-base encoder plus one linear head) |
| **Speed** | 52 ms median on CPU for one question with 4 options |
| **Calibration** | ECE 0.047 after temperature scaling |
| **Licence** | Apache 2.0 (commercial use is permitted) |

## Quickstart (no extra package)

`pip install torch transformers huggingface_hub safetensors`, then:

```python
import torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from transformers import AutoModel, AutoTokenizer

path = snapshot_download("gparitosh007/Open-Decision-Model-150M")
tok = AutoTokenizer.from_pretrained(path)
enc = AutoModel.from_pretrained(path, dtype=torch.float32).eval()
w = load_file(f"{path}/model.safetensors")
W, b, T = w["decision_head.weight"].float(), w["decision_head.bias"].float(), w["decision_head.temperature"].item()

@torch.inference_mode()
def decide(state, instructions, options, preferences=None):
    """options: dict label -> description. Returns dict label -> probability."""
    e = lambda t: tok.encode(t, add_special_tokens=False)
    ids = [tok.cls_token_id]
    if preferences:
        ids += e("\n".join(f"{k}: {v}" for k, v in preferences.items())) + [tok.sep_token_id]
    ids += e(state) + [tok.sep_token_id] + e(instructions) + [tok.sep_token_id]
    spans = []
    for label, desc in options.items():
        o = e(f"{label}: {desc}" if desc else label)
        spans.append((len(ids), len(ids) + len(o)))
        ids += o + [tok.sep_token_id]
    h = enc(input_ids=torch.tensor([ids])).last_hidden_state[0]
    logits = torch.stack([h[s:t].mean(0) for s, t in spans]) @ W.T + b
    return dict(zip(options, torch.softmax(logits.squeeze(-1) / T, -1).tolist()))

print(decide(
    "Refactor the auth module and add tests. The repo has 40 files.",
    "Which model must handle this request?",
    {"haiku": "Short and simple tasks", "sonnet": "Normal coding tasks", "opus": "Long or difficult tasks"},
    preferences={"cost": "low cost before speed"},
))
```

Output: `{'haiku': 0.16, 'sonnet': 0.50, 'opus': 0.34}`. This code gives the same probabilities as the official package.

## Use it now: free public MCP endpoint

No download and no API key. A public, stateless instance runs on Vercel with the ONNX model: `https://decision-model.vercel.app/mcp`.

**Claude Code:**

```bash
claude mcp add --transport http decision https://decision-model.vercel.app/mcp
```

Other MCP clients (Claude Desktop, Cursor and others): add a remote HTTP MCP server with the same URL.

| Tool | What it does | Required inputs | Returns |
|---|---|---|---|
| `choice` | Picks between options | `state`, `instructions`, `criteria` (one description per option) | A probability per option and a confidence |
| `score` | Rates on an ordered scale (for example trivial to very hard) | `state`, `instructions`, `levels` | A probability per level, the expected level and a legend |
| `noul` | Answers a yes/no question | `state`, `instructions`, optional yes/no definitions in `criteria` | `p_yes` |
| `decide` | Runs several questions against one state (Jev / Laya format) | `state`, `questions` (each with `id`, `type`, `instructions`, optional `criteria`) | One result per question |
| `build_preferences` | Asks a preference interview for one domain | `domain`, optional `answers` | A preference file for the domain |
| `feedback` | Records what you did with a decision | `decision_id`, `action` (accept, reject or other), optional `chosen` | A log of the outcome |

All decision tools also accept `domain` (a saved preference file) or inline `preferences`, and `question_id`. The usual flow is: `build_preferences`, then `choice`, `score` or `noul`, then `feedback`.

The public instance is stateless: it saves no files and logs no request bodies. On it, send `preferences` inline in each call. To keep preference files and feedback, run your own server (see "How to use").

**HTTP, without MCP:**

```bash
curl -s https://decision-model.vercel.app/v1/decide -H 'Content-Type: application/json' -d '{
  "state": "Refactor the auth module and add tests. The repo has 40 files.",
  "questions": [{"id": "model", "type": "choice",
    "instructions": "Which model must handle this request?",
    "criteria": {"haiku": "Short and simple tasks", "sonnet": "Normal coding tasks", "opus": "Long or difficult tasks"}}]}'
```

The first call after an idle period takes about 50 s, because the function downloads the model. Later calls take less than 1 s.

### For AI agents

- MCP endpoint: `https://decision-model.vercel.app/mcp`. Use the `choice` tool to route or to pick an option.
- Model id: `gparitosh007/Open-Decision-Model-150M`. Call `decide(state, instructions, options)` above and pick the option with the highest probability.
- When the top probability is low, use a fallback. Read "Limitations and bias".

**Former name:** `decision-model`. The old URL redirects here.

## Details


Model version: `decision-model 0.1.0`. Export format version: 1. The encoder is [tasksource/ModernBERT-base-nli](https://huggingface.co/tasksource/ModernBERT-base-nli), a ModernBERT-base fine-tuned on NLI, with 149,015,041 (149.0M) parameters. This card lives in README.md of the export folder.

## Summary

An open decision model of the Jev / Laya type. The model reads a state, optional user preferences and one or more questions. The model returns calibrated probabilities for every option and does not generate text. A linear head maps the mean-pooled vector of each option span to one logit, so one model works with any option count. The attribute `temperature` calibrates the probabilities. An MCP server serves the model. The source code is not public yet.

## Intended use

The model serves decisions with calibrated probabilities over explicit options:

- Model routing: pick haiku, sonnet or opus for one request.
- Tool risk: decide if a tool call needs approval.
- Shopping and support: pick one option, or score difficulty levels.
- Score answers return the expected level; noul answers return p_yes.

The consumers are routers, MCP agents and policy checks. The probabilities are calibrated on the training domains with temperature scaling.

## NOT intended use

- Do not use the model to generate text. The model returns probabilities only.
- Do not expect high accuracy in domains that are not in the training data. In a held-out domain test, the accuracy was 0.57; see Evaluation.
- Do not use the model as the only check in a safety critical flow. The model is a probability estimate, not an approval.
- Do not use the model for medical, legal or financial decisions with no human in the loop.
- Do not trust the confidence value as a correctness guarantee. It measures the concentration of the distribution only.

## Input and output format

A request holds one `state`, optional `preferences`, and a list of `questions`. Each question has `id`, `type` (`choice`, `score` or `noul`), `instructions`, and optional `criteria`. This shape is the one format for the whole repo.

```json
{
  "state": "Refactor the auth module and add tests. The repo has 40 files.",
  "preferences": {"cost": "low cost before speed"},
  "questions": [
    {"id": "model", "type": "choice", "instructions": "Which model must handle this request?",
     "criteria": {"haiku": "Short and simple tasks", "sonnet": "Normal coding tasks", "opus": "Long or difficult tasks"}},
    {"id": "difficulty", "type": "score", "instructions": "How difficult is this task?",
     "criteria": ["trivial", "easy", "medium", "hard", "very hard"]},
    {"id": "risky", "type": "noul", "instructions": "Can this task delete user data?"}
  ]
}
```

The response holds `answers` with one entry per question. `probs` holds one probability per option label. `confidence` is 1 minus the normalized entropy of the probability row. A `score` answer adds `expected_level` (0-based) and a `legend`. A `noul` answer adds `p_yes`.

```json
{
  "decision_id": "3f9d2b1e-4c8a-4b6f-9e21-a7d5c0b8e1f2",
  "model_version": "decision-model 0.1.0",
  "answers": [
    {"id": "model", "type": "choice",
     "probs": {"haiku": 0.62, "sonnet": 0.31, "opus": 0.07}, "confidence": 0.51},
    {"id": "difficulty", "type": "score",
     "probs": {"trivial": 0.05, "easy": 0.35, "medium": 0.40, "hard": 0.15, "very hard": 0.05},
     "confidence": 0.42, "expected_level": 1.8, "legend": {"0": "trivial", "1": "easy", "2": "medium", "3": "hard", "4": "very hard"}},
    {"id": "risky", "type": "noul",
     "probs": {"yes": 0.09, "no": 0.91}, "confidence": 0.82, "p_yes": 0.09}
  ]
}
```

Questions in one request are evaluated independently. The encoder reads one sequence: `[preferences] [SEP] [state] [SEP] [instructions] [SEP] [option 1] [SEP] ...`. For the exact option order rules read `decision_config.json`.

## How to use

The Quickstart above needs no extra package. The `decision_model` package below is not public yet.

1. Load the export folder and predict:

   ```python
   from decision_model.model import DecisionModel

   model = DecisionModel.load("export/NAME")
   result = model.predict({
       "state": "Refactor the auth module and add tests.",
       "questions": [{
           "id": "model",
           "type": "choice",
           "instructions": "Which model must handle this request?",
           "criteria": {"haiku": "Short and simple tasks", "sonnet": "Normal coding tasks", "opus": "Long or difficult tasks"},
       }],
   })
   print(result["answers"][0]["probs"])
   ```

2. Run `example.py` in this folder for the same flow.

3. Serve the model with the MCP server:

   ```bash
   DECISION_MODEL_PATH=/absolute/path/to/this/folder claude mcp add decision -- uv run --directory <repo> decision-mcp
   ```

4. Serve the model over HTTP (decision-http):

   ```bash
   DECISION_MODEL_PATH=/absolute/path/to/this/folder python -m decision_model.http_server
   ```

## Training data

- **Teacher-labelled synthetic tasks (v2, 185,131 records).** Template generators for model routing, tool call risk, shopping, support and travel, plus about 30,000 generic tasks, paraphrases and turned-around questions. The template domains are balanced.
- The teacher is Qwen3-235B FP8 (run `q235fp8`) served through the vLLM OpenAI API. Soft labels come from the top-20 log-probabilities of the first answer token. Each task is labelled with two option orders, and the mean removes position bias. Pairs that disagree by more than 0.5 total variation distance are removed.
- **Open data with human labels (109,767 records).** Only datasets with a clear commercial-use license are used:

| Source | Records | License | Use |
|---|---|---|---|
| tasksource/bigbench | 44,427 | Apache 2.0 | choice questions |
| tasksource/mmlu | 15,573 | MIT | choice questions |
| nvidia/HelpSteer2 | 21,362 | CC BY 4.0 | rating questions |
| PolyAI/banking77 | 16,430 | CC BY 4.0 | support intents |
| clinc/clinc_oos | 11,975 | CC BY 3.0 | support intents |

## Training setup

| Item | Value |
|---|---|
| Base encoder | tasksource/ModernBERT-base-nli |
| Loss | KL divergence to the label probabilities, plus 0.1 x cross-entropy to the top option |
| Optimizer | AdamW, LR 3e-5, batch 16, 3 epochs, 48,174 steps |
| Max length | 1024 tokens; a long state is trimmed from the start, the options stay |
| Split | by template: validation templates are not in the training set |
| Weights | class weights; domain weights generic=3, open_tasksource=2 |
| Seed | 42 (one run) |
| Hardware | one B300 GPU, about 47 minutes |
| Calibration | temperature scaling on the validation split; temperature 3.9863 |

## Evaluation

All numbers are agreement with the label of the top option (teacher or human label), not human-verified correctness. The validation templates are not in the training set.

### This checkpoint (seed 42)

| Metric | Value |
|---|---|
| Validation accuracy | 0.468 (majority baseline 0.404) |
| Validation balanced accuracy | 0.455 |
| ECE raw / after temperature scaling | 0.229 / 0.047 |
| CPU latency, one question with 4 options | 52 ms median (6 threads) |

| Domain | Accuracy | Balanced accuracy | ECE |
|---|---|---|---|
| generic | 0.717 | 0.654 | 0.191 |
| shopping | 0.677 | 0.650 | 0.204 |
| tool_risk | 0.504 | 0.398 | 0.058 |
| open_tasksource | 0.407 | 0.470 | 0.063 |
| routing | 0.367 | 0.460 | 0.152 |
| support | 0.215 | 0.254 | 0.276 |

| Question type | Accuracy |
|---|---|
| noul | 0.683 |
| choice | 0.436 |
| score | 0.312 |

### Held-out domain test (same backbone, without open data)

This checkpoint has no held-out domain. A run with the same backbone and settings, but without the open data and with travel held out, gave travel accuracy 0.566, balanced accuracy 0.588 and ECE 0.098. That run had routing accuracy 0.450 and shopping accuracy 0.733. With the open data, routing and shopping are lower (one seed each).

## Limitations and bias

- Routing and support are weak (0.37 and 0.22 accuracy). Use the probabilities and a fallback when the top probability is low.
- Score questions are weak (0.31 accuracy).
- The open data lowered the accuracy on routing and shopping compared with a run without it.
- One seed only; differences of a few points can be noise.
- The confidence value measures concentration. It does not guarantee a correct answer.

## Licenses

| Part | License |
|---|---|
| This model | Apache 2.0 |
| Encoder tasksource/ModernBERT-base-nli (from answerdotai/ModernBERT-base) | Apache 2.0 |
| Teacher Qwen3-235B | Apache 2.0 |
| Open training data | Apache 2.0, MIT, CC BY 4.0, CC BY 3.0 (see Training data; CC BY sources are attributed above) |

## Citation

Cite the blog post that defines the decision model type: Jev, Laya and the emerging role of decision models, <https://runware.ai/blog/jev-laya-and-the-emerging-role-of-decision-models>.