license: apache-2.0
base_model: tasksource/ModernBERT-base-nli
library_name: transformers
pipeline_tag: text-classification
language: en
tags:
- decision-model
- calibration
- mcp
- distillation
decision-model v2
Model version: decision-model 0.1.0. Export format version: 1. The encoder is tasksource/ModernBERT-base-nli, a ModernBERT-base fine-tuned on NLI, with 149,015,041 (149.0M) parameters. This card lives in README.md of the export folder.
Summary
An open decision model of the Jev / Laya type. The model reads a state, optional user preferences and one or more questions. The model returns calibrated probabilities for every option and does not generate text. A linear head maps the mean-pooled vector of each option span to one logit, so one model works with any option count. The attribute temperature calibrates the probabilities. An MCP server serves the model. The source code is not public yet.
Intended use
The model serves decisions with calibrated probabilities over explicit options:
- Model routing: pick haiku, sonnet or opus for one request.
- Tool risk: decide if a tool call needs approval.
- Shopping and support: pick one option, or score difficulty levels.
- Score answers return the expected level; noul answers return p_yes.
The consumers are routers, MCP agents and policy checks. The probabilities are calibrated on the training domains with temperature scaling.
NOT intended use
- Do not use the model to generate text. The model returns probabilities only.
- Do not expect high accuracy in domains that are not in the training data. In a held-out domain test, the accuracy was 0.57; see Evaluation.
- Do not use the model as the only check in a safety critical flow. The model is a probability estimate, not an approval.
- Do not use the model for medical, legal or financial decisions with no human in the loop.
- Do not trust the confidence value as a correctness guarantee. It measures the concentration of the distribution only.
Input and output format
A request holds one state, optional preferences, and a list of questions. Each question has id, type (choice, score or noul), instructions, and optional criteria. This shape is the one format for the whole repo.
{
"state": "Refactor the auth module and add tests. The repo has 40 files.",
"preferences": {"cost": "low cost before speed"},
"questions": [
{"id": "model", "type": "choice", "instructions": "Which model must handle this request?",
"criteria": {"haiku": "Short and simple tasks", "sonnet": "Normal coding tasks", "opus": "Long or difficult tasks"}},
{"id": "difficulty", "type": "score", "instructions": "How difficult is this task?",
"criteria": ["trivial", "easy", "medium", "hard", "very hard"]},
{"id": "risky", "type": "noul", "instructions": "Can this task delete user data?"}
]
}
The response holds answers with one entry per question. probs holds one probability per option label. confidence is 1 minus the normalized entropy of the probability row. A score answer adds expected_level (0-based) and a legend. A noul answer adds p_yes.
{
"decision_id": "3f9d2b1e-4c8a-4b6f-9e21-a7d5c0b8e1f2",
"model_version": "decision-model 0.1.0",
"answers": [
{"id": "model", "type": "choice",
"probs": {"haiku": 0.62, "sonnet": 0.31, "opus": 0.07}, "confidence": 0.51},
{"id": "difficulty", "type": "score",
"probs": {"trivial": 0.05, "easy": 0.35, "medium": 0.40, "hard": 0.15, "very hard": 0.05},
"confidence": 0.42, "expected_level": 1.8, "legend": {"0": "trivial", "1": "easy", "2": "medium", "3": "hard", "4": "very hard"}},
{"id": "risky", "type": "noul",
"probs": {"yes": 0.09, "no": 0.91}, "confidence": 0.82, "p_yes": 0.09}
]
}
Questions in one request are evaluated independently. The encoder reads one sequence: [preferences] [SEP] [state] [SEP] [instructions] [SEP] [option 1] [SEP] .... For the exact option order rules read decision_config.json.
How to use
The source code is not public yet. This card ships the model weights and the files of this folder.
Load the export folder and predict:
from decision_model.model import DecisionModel model = DecisionModel.load("export/NAME") result = model.predict({ "state": "Refactor the auth module and add tests.", "questions": [{ "id": "model", "type": "choice", "instructions": "Which model must handle this request?", "criteria": {"haiku": "Short and simple tasks", "sonnet": "Normal coding tasks", "opus": "Long or difficult tasks"}, }], }) print(result["answers"][0]["probs"])Run
example.pyin this folder for the same flow.Serve the model with the MCP server:
DECISION_MODEL_PATH=/absolute/path/to/this/folder claude mcp add decision -- uv run --directory <repo> decision-mcpServe the model over HTTP (decision-http):
DECISION_MODEL_PATH=/absolute/path/to/this/folder python -m decision_model.http_server
Training data
- Teacher-labelled synthetic tasks (v2, 185,131 records). Template generators for model routing, tool call risk, shopping, support and travel, plus about 30,000 generic tasks, paraphrases and turned-around questions. The template domains are balanced.
- The teacher is Qwen3-235B FP8 (run
q235fp8) served through the vLLM OpenAI API. Soft labels come from the top-20 log-probabilities of the first answer token. Each task is labelled with two option orders, and the mean removes position bias. Pairs that disagree by more than 0.5 total variation distance are removed. - Open data with human labels (109,767 records). Only datasets with a clear commercial-use license are used:
| Source | Records | License | Use |
|---|---|---|---|
| tasksource/bigbench | 44,427 | Apache 2.0 | choice questions |
| tasksource/mmlu | 15,573 | MIT | choice questions |
| nvidia/HelpSteer2 | 21,362 | CC BY 4.0 | rating questions |
| PolyAI/banking77 | 16,430 | CC BY 4.0 | support intents |
| clinc/clinc_oos | 11,975 | CC BY 3.0 | support intents |
Training setup
| Item | Value |
|---|---|
| Base encoder | tasksource/ModernBERT-base-nli |
| Loss | KL divergence to the label probabilities, plus 0.1 x cross-entropy to the top option |
| Optimizer | AdamW, LR 3e-5, batch 16, 3 epochs, 48,174 steps |
| Max length | 1024 tokens; a long state is trimmed from the start, the options stay |
| Split | by template: validation templates are not in the training set |
| Weights | class weights; domain weights generic=3, open_tasksource=2 |
| Seed | 42 (one run) |
| Hardware | one B300 GPU, about 47 minutes |
| Calibration | temperature scaling on the validation split; temperature 3.9863 |
Evaluation
All numbers are agreement with the label of the top option (teacher or human label), not human-verified correctness. The validation templates are not in the training set.
This checkpoint (seed 42)
| Metric | Value |
|---|---|
| Validation accuracy | 0.468 (majority baseline 0.404) |
| Validation balanced accuracy | 0.455 |
| ECE raw / after temperature scaling | 0.229 / 0.047 |
| CPU latency, one question with 4 options | 52 ms median (6 threads) |
| Domain | Accuracy | Balanced accuracy | ECE |
|---|---|---|---|
| generic | 0.717 | 0.654 | 0.191 |
| shopping | 0.677 | 0.650 | 0.204 |
| tool_risk | 0.504 | 0.398 | 0.058 |
| open_tasksource | 0.407 | 0.470 | 0.063 |
| routing | 0.367 | 0.460 | 0.152 |
| support | 0.215 | 0.254 | 0.276 |
| Question type | Accuracy |
|---|---|
| noul | 0.683 |
| choice | 0.436 |
| score | 0.312 |
Held-out domain test (same backbone, without open data)
This checkpoint has no held-out domain. A run with the same backbone and settings, but without the open data and with travel held out, gave travel accuracy 0.566, balanced accuracy 0.588 and ECE 0.098. That run had routing accuracy 0.450 and shopping accuracy 0.733. With the open data, routing and shopping are lower (one seed each).
Limitations and bias
- Routing and support are weak (0.37 and 0.22 accuracy). Use the probabilities and a fallback when the top probability is low.
- Score questions are weak (0.31 accuracy).
- The open data lowered the accuracy on routing and shopping compared with a run without it.
- One seed only; differences of a few points can be noise.
- The confidence value measures concentration. It does not guarantee a correct answer.
Licenses
| Part | License |
|---|---|
| This model | Apache 2.0 |
| Encoder tasksource/ModernBERT-base-nli (from answerdotai/ModernBERT-base) | Apache 2.0 |
| Teacher Qwen3-235B | Apache 2.0 |
| Open training data | Apache 2.0, MIT, CC BY 4.0, CC BY 3.0 (see Training data; CC BY sources are attributed above) |
Citation
Cite the blog post that defines the decision model type: Jev, Laya and the emerging role of decision models, https://runware.ai/blog/jev-laya-and-the-emerging-role-of-decision-models.