v2: ModernBERT-base-nli, v2 data + open data, seed 42

2400eca verified about 23 hours ago

9.09 kB

---
license: apache-2.0
base_model: tasksource/ModernBERT-base-nli
library_name: transformers
pipeline_tag: text-classification
language: en
tags:
- decision-model
- calibration
- mcp
- distillation
---
# decision-model v2

Model version: `decision-model 0.1.0`. Export format version: 1. The encoder is [tasksource/ModernBERT-base-nli](https://huggingface.co/tasksource/ModernBERT-base-nli), a ModernBERT-base fine-tuned on NLI, with 149,015,041 (149.0M) parameters. This card lives in README.md of the export folder.

## Summary

An open decision model of the Jev / Laya type. The model reads a state, optional user preferences and one or more questions. The model returns calibrated probabilities for every option and does not generate text. A linear head maps the mean-pooled vector of each option span to one logit, so one model works with any option count. The attribute `temperature` calibrates the probabilities. An MCP server serves the model. The source code is not public yet.

## Intended use

The model serves decisions with calibrated probabilities over explicit options:

- Model routing: pick haiku, sonnet or opus for one request.
- Tool risk: decide if a tool call needs approval.
- Shopping and support: pick one option, or score difficulty levels.
- Score answers return the expected level; noul answers return p_yes.
The consumers are routers, MCP agents and policy checks. The probabilities are calibrated on the training domains with temperature scaling.
## NOT intended use
- Do not use the model to generate text. The model returns probabilities only.
- Do not expect high accuracy in domains that are not in the training data. In a held-out domain test, the accuracy was 0.57; see Evaluation.
- Do not use the model as the only check in a safety critical flow. The model is a probability estimate, not an approval.
- Do not use the model for medical, legal or financial decisions with no human in the loop.
- Do not trust the confidence value as a correctness guarantee. It measures the concentration of the distribution only.
## Input and output format
A request holds one `state`, optional `preferences`, and a list of `questions`. Each question has `id`, `type` (`choice`, `score` or `noul`), `instructions`, and optional `criteria`. This shape is the one format for the whole repo.
```json
{
  "state": "Refactor the auth module and add tests. The repo has 40 files.",
  "preferences": {"cost": "low cost before speed"},
  "questions": [
    {"id": "model", "type": "choice", "instructions": "Which model must handle this request?",
     "criteria": {"haiku": "Short and simple tasks", "sonnet": "Normal coding tasks", "opus": "Long or difficult tasks"}},
    {"id": "difficulty", "type": "score", "instructions": "How difficult is this task?",
     "criteria": ["trivial", "easy", "medium", "hard", "very hard"]},
    {"id": "risky", "type": "noul", "instructions": "Can this task delete user data?"}
  ]
}
```
The response holds `answers` with one entry per question. `probs` holds one probability per option label. `confidence` is 1 minus the normalized entropy of the probability row. A `score` answer adds `expected_level` (0-based) and a `legend`. A `noul` answer adds `p_yes`.
```json
{
  "decision_id": "3f9d2b1e-4c8a-4b6f-9e21-a7d5c0b8e1f2",
  "model_version": "decision-model 0.1.0",
  "answers": [
    {"id": "model", "type": "choice",
     "probs": {"haiku": 0.62, "sonnet": 0.31, "opus": 0.07}, "confidence": 0.51},
    {"id": "difficulty", "type": "score",
     "probs": {"trivial": 0.05, "easy": 0.35, "medium": 0.40, "hard": 0.15, "very hard": 0.05},
     "confidence": 0.42, "expected_level": 1.8, "legend": {"0": "trivial", "1": "easy", "2": "medium", "3": "hard", "4": "very hard"}},
    {"id": "risky", "type": "noul",
     "probs": {"yes": 0.09, "no": 0.91}, "confidence": 0.82, "p_yes": 0.09}
  ]
}
```
Questions in one request are evaluated independently. The encoder reads one sequence: `[preferences] [SEP] [state] [SEP] [instructions] [SEP] [option 1] [SEP] ...`. For the exact option order rules read `decision_config.json`.

## How to use

The source code is not public yet. This card ships the model weights and the files of this folder.

1. Load the export folder and predict:

   ```python
   from decision_model.model import DecisionModel
   model = DecisionModel.load("export/NAME")
   result = model.predict({
       "state": "Refactor the auth module and add tests.",
       "questions": [{
           "id": "model",
           "type": "choice",
           "instructions": "Which model must handle this request?",
           "criteria": {"haiku": "Short and simple tasks", "sonnet": "Normal coding tasks", "opus": "Long or difficult tasks"},
       }],
   })
   print(result["answers"][0]["probs"])
   ```

2. Run `example.py` in this folder for the same flow.

3. Serve the model with the MCP server:

   ```bash
   DECISION_MODEL_PATH=/absolute/path/to/this/folder claude mcp add decision -- uv run --directory <repo> decision-mcp
   ```

4. Serve the model over HTTP (decision-http):

   ```bash
   DECISION_MODEL_PATH=/absolute/path/to/this/folder python -m decision_model.http_server
   ```

## Training data

- **Teacher-labelled synthetic tasks (v2, 185,131 records).** Template generators for model routing, tool call risk, shopping, support and travel, plus about 30,000 generic tasks, paraphrases and turned-around questions. The template domains are balanced.
- The teacher is Qwen3-235B FP8 (run `q235fp8`) served through the vLLM OpenAI API. Soft labels come from the top-20 log-probabilities of the first answer token. Each task is labelled with two option orders, and the mean removes position bias. Pairs that disagree by more than 0.5 total variation distance are removed.
- **Open data with human labels (109,767 records).** Only datasets with a clear commercial-use license are used:

| Source | Records | License | Use |
|---|---|---|---|
| tasksource/bigbench | 44,427 | Apache 2.0 | choice questions |
| tasksource/mmlu | 15,573 | MIT | choice questions |
| nvidia/HelpSteer2 | 21,362 | CC BY 4.0 | rating questions |
| PolyAI/banking77 | 16,430 | CC BY 4.0 | support intents |
| clinc/clinc_oos | 11,975 | CC BY 3.0 | support intents |
## Training setup
| Item | Value |
|---|---|
| Base encoder | tasksource/ModernBERT-base-nli |
| Loss | KL divergence to the label probabilities, plus 0.1 x cross-entropy to the top option |
| Optimizer | AdamW, LR 3e-5, batch 16, 3 epochs, 48,174 steps |
| Max length | 1024 tokens; a long state is trimmed from the start, the options stay |
| Split | by template: validation templates are not in the training set |
| Weights | class weights; domain weights generic=3, open_tasksource=2 |
| Seed | 42 (one run) |
| Hardware | one B300 GPU, about 47 minutes |
| Calibration | temperature scaling on the validation split; temperature 3.9863 |

## Evaluation

All numbers are agreement with the label of the top option (teacher or human label), not human-verified correctness. The validation templates are not in the training set.

### This checkpoint (seed 42)

| Metric | Value |
|---|---|
| Validation accuracy | 0.468 (majority baseline 0.404) |
| Validation balanced accuracy | 0.455 |
| ECE raw / after temperature scaling | 0.229 / 0.047 |
| CPU latency, one question with 4 options | 52 ms median (6 threads) |

| Domain | Accuracy | Balanced accuracy | ECE |
|---|---|---|---|
| generic | 0.717 | 0.654 | 0.191 |
| shopping | 0.677 | 0.650 | 0.204 |
| tool_risk | 0.504 | 0.398 | 0.058 |
| open_tasksource | 0.407 | 0.470 | 0.063 |
| routing | 0.367 | 0.460 | 0.152 |
| support | 0.215 | 0.254 | 0.276 |

| Question type | Accuracy |
|---|---|
| noul | 0.683 |
| choice | 0.436 |
| score | 0.312 |

### Held-out domain test (same backbone, without open data)

This checkpoint has no held-out domain. A run with the same backbone and settings, but without the open data and with travel held out, gave travel accuracy 0.566, balanced accuracy 0.588 and ECE 0.098. That run had routing accuracy 0.450 and shopping accuracy 0.733. With the open data, routing and shopping are lower (one seed each).

## Limitations and bias

- Routing and support are weak (0.37 and 0.22 accuracy). Use the probabilities and a fallback when the top probability is low.
- Score questions are weak (0.31 accuracy).
- The open data lowered the accuracy on routing and shopping compared with a run without it.
- One seed only; differences of a few points can be noise.
- The confidence value measures concentration. It does not guarantee a correct answer.

## Licenses

| Part | License |
|---|---|
| This model | Apache 2.0 |
| Encoder tasksource/ModernBERT-base-nli (from answerdotai/ModernBERT-base) | Apache 2.0 |
| Teacher Qwen3-235B | Apache 2.0 |
| Open training data | Apache 2.0, MIT, CC BY 4.0, CC BY 3.0 (see Training data; CC BY sources are attributed above) |

## Citation

Cite the blog post that defines the decision model type: Jev, Laya and the emerging role of decision models, <https://runware.ai/blog/jev-laya-and-the-emerging-role-of-decision-models>.

	---
	license: apache-2.0
	base_model: tasksource/ModernBERT-base-nli
	library_name: transformers
	pipeline_tag: text-classification
	language: en
	tags:
	- decision-model
	- calibration
	- mcp
	- distillation
	---
	# decision-model v2

	Model version: `decision-model 0.1.0`. Export format version: 1. The encoder is [tasksource/ModernBERT-base-nli](https://huggingface.co/tasksource/ModernBERT-base-nli), a ModernBERT-base fine-tuned on NLI, with 149,015,041 (149.0M) parameters. This card lives in README.md of the export folder.

	## Summary

	An open decision model of the Jev / Laya type. The model reads a state, optional user preferences and one or more questions. The model returns calibrated probabilities for every option and does not generate text. A linear head maps the mean-pooled vector of each option span to one logit, so one model works with any option count. The attribute `temperature` calibrates the probabilities. An MCP server serves the model. The source code is not public yet.

	## Intended use

	The model serves decisions with calibrated probabilities over explicit options:

	- Model routing: pick haiku, sonnet or opus for one request.
	- Tool risk: decide if a tool call needs approval.
	- Shopping and support: pick one option, or score difficulty levels.
	- Score answers return the expected level; noul answers return p_yes.

	The consumers are routers, MCP agents and policy checks. The probabilities are calibrated on the training domains with temperature scaling.

	## NOT intended use

	- Do not use the model to generate text. The model returns probabilities only.
	- Do not expect high accuracy in domains that are not in the training data. In a held-out domain test, the accuracy was 0.57; see Evaluation.
	- Do not use the model as the only check in a safety critical flow. The model is a probability estimate, not an approval.
	- Do not use the model for medical, legal or financial decisions with no human in the loop.
	- Do not trust the confidence value as a correctness guarantee. It measures the concentration of the distribution only.

	## Input and output format

	A request holds one `state`, optional `preferences`, and a list of `questions`. Each question has `id`, `type` (`choice`, `score` or `noul`), `instructions`, and optional `criteria`. This shape is the one format for the whole repo.

	```json
	{
	"state": "Refactor the auth module and add tests. The repo has 40 files.",
	"preferences": {"cost": "low cost before speed"},
	"questions": [
	{"id": "model", "type": "choice", "instructions": "Which model must handle this request?",
	"criteria": {"haiku": "Short and simple tasks", "sonnet": "Normal coding tasks", "opus": "Long or difficult tasks"}},
	{"id": "difficulty", "type": "score", "instructions": "How difficult is this task?",
	"criteria": ["trivial", "easy", "medium", "hard", "very hard"]},
	{"id": "risky", "type": "noul", "instructions": "Can this task delete user data?"}
	]
	}
	```

	The response holds `answers` with one entry per question. `probs` holds one probability per option label. `confidence` is 1 minus the normalized entropy of the probability row. A `score` answer adds `expected_level` (0-based) and a `legend`. A `noul` answer adds `p_yes`.

	```json
	{
	"decision_id": "3f9d2b1e-4c8a-4b6f-9e21-a7d5c0b8e1f2",
	"model_version": "decision-model 0.1.0",
	"answers": [
	{"id": "model", "type": "choice",
	"probs": {"haiku": 0.62, "sonnet": 0.31, "opus": 0.07}, "confidence": 0.51},
	{"id": "difficulty", "type": "score",
	"probs": {"trivial": 0.05, "easy": 0.35, "medium": 0.40, "hard": 0.15, "very hard": 0.05},
	"confidence": 0.42, "expected_level": 1.8, "legend": {"0": "trivial", "1": "easy", "2": "medium", "3": "hard", "4": "very hard"}},
	{"id": "risky", "type": "noul",
	"probs": {"yes": 0.09, "no": 0.91}, "confidence": 0.82, "p_yes": 0.09}
	]
	}
	```

	Questions in one request are evaluated independently. The encoder reads one sequence: `[preferences] [SEP] [state] [SEP] [instructions] [SEP] [option 1] [SEP] ...`. For the exact option order rules read `decision_config.json`.

	## How to use

	The source code is not public yet. This card ships the model weights and the files of this folder.

	1. Load the export folder and predict:

	```python
	from decision_model.model import DecisionModel

	model = DecisionModel.load("export/NAME")
	result = model.predict({
	"state": "Refactor the auth module and add tests.",
	"questions": [{
	"id": "model",
	"type": "choice",
	"instructions": "Which model must handle this request?",
	"criteria": {"haiku": "Short and simple tasks", "sonnet": "Normal coding tasks", "opus": "Long or difficult tasks"},
	}],
	})
	print(result["answers"][0]["probs"])
	```

	2. Run `example.py` in this folder for the same flow.

	3. Serve the model with the MCP server:

	```bash
	DECISION_MODEL_PATH=/absolute/path/to/this/folder claude mcp add decision -- uv run --directory <repo> decision-mcp
	```

	4. Serve the model over HTTP (decision-http):

	```bash
	DECISION_MODEL_PATH=/absolute/path/to/this/folder python -m decision_model.http_server
	```

	## Training data

	- Teacher-labelled synthetic tasks (v2, 185,131 records). Template generators for model routing, tool call risk, shopping, support and travel, plus about 30,000 generic tasks, paraphrases and turned-around questions. The template domains are balanced.
	- The teacher is Qwen3-235B FP8 (run `q235fp8`) served through the vLLM OpenAI API. Soft labels come from the top-20 log-probabilities of the first answer token. Each task is labelled with two option orders, and the mean removes position bias. Pairs that disagree by more than 0.5 total variation distance are removed.
	- Open data with human labels (109,767 records). Only datasets with a clear commercial-use license are used:

	\| Source \| Records \| License \| Use \|
	\|---\|---\|---\|---\|
	\| tasksource/bigbench \| 44,427 \| Apache 2.0 \| choice questions \|
	\| tasksource/mmlu \| 15,573 \| MIT \| choice questions \|
	\| nvidia/HelpSteer2 \| 21,362 \| CC BY 4.0 \| rating questions \|
	\| PolyAI/banking77 \| 16,430 \| CC BY 4.0 \| support intents \|
	\| clinc/clinc_oos \| 11,975 \| CC BY 3.0 \| support intents \|

	## Training setup

	\| Item \| Value \|
	\|---\|---\|
	\| Base encoder \| tasksource/ModernBERT-base-nli \|
	\| Loss \| KL divergence to the label probabilities, plus 0.1 x cross-entropy to the top option \|
	\| Optimizer \| AdamW, LR 3e-5, batch 16, 3 epochs, 48,174 steps \|
	\| Max length \| 1024 tokens; a long state is trimmed from the start, the options stay \|
	\| Split \| by template: validation templates are not in the training set \|
	\| Weights \| class weights; domain weights generic=3, open_tasksource=2 \|
	\| Seed \| 42 (one run) \|
	\| Hardware \| one B300 GPU, about 47 minutes \|
	\| Calibration \| temperature scaling on the validation split; temperature 3.9863 \|

	## Evaluation

	All numbers are agreement with the label of the top option (teacher or human label), not human-verified correctness. The validation templates are not in the training set.

	### This checkpoint (seed 42)

	\| Metric \| Value \|
	\|---\|---\|
	\| Validation accuracy \| 0.468 (majority baseline 0.404) \|
	\| Validation balanced accuracy \| 0.455 \|
	\| ECE raw / after temperature scaling \| 0.229 / 0.047 \|
	\| CPU latency, one question with 4 options \| 52 ms median (6 threads) \|

	\| Domain \| Accuracy \| Balanced accuracy \| ECE \|
	\|---\|---\|---\|---\|
	\| generic \| 0.717 \| 0.654 \| 0.191 \|
	\| shopping \| 0.677 \| 0.650 \| 0.204 \|
	\| tool_risk \| 0.504 \| 0.398 \| 0.058 \|
	\| open_tasksource \| 0.407 \| 0.470 \| 0.063 \|
	\| routing \| 0.367 \| 0.460 \| 0.152 \|
	\| support \| 0.215 \| 0.254 \| 0.276 \|

	\| Question type \| Accuracy \|
	\|---\|---\|
	\| noul \| 0.683 \|
	\| choice \| 0.436 \|
	\| score \| 0.312 \|

	### Held-out domain test (same backbone, without open data)

	This checkpoint has no held-out domain. A run with the same backbone and settings, but without the open data and with travel held out, gave travel accuracy 0.566, balanced accuracy 0.588 and ECE 0.098. That run had routing accuracy 0.450 and shopping accuracy 0.733. With the open data, routing and shopping are lower (one seed each).

	## Limitations and bias

	- Routing and support are weak (0.37 and 0.22 accuracy). Use the probabilities and a fallback when the top probability is low.
	- Score questions are weak (0.31 accuracy).
	- The open data lowered the accuracy on routing and shopping compared with a run without it.
	- One seed only; differences of a few points can be noise.
	- The confidence value measures concentration. It does not guarantee a correct answer.

	## Licenses

	\| Part \| License \|
	\|---\|---\|
	\| This model \| Apache 2.0 \|
	\| Encoder tasksource/ModernBERT-base-nli (from answerdotai/ModernBERT-base) \| Apache 2.0 \|
	\| Teacher Qwen3-235B \| Apache 2.0 \|
	\| Open training data \| Apache 2.0, MIT, CC BY 4.0, CC BY 3.0 (see Training data; CC BY sources are attributed above) \|

	## Citation

	Cite the blog post that defines the decision model type: Jev, Laya and the emerging role of decision models, <https://runware.ai/blog/jev-laya-and-the-emerging-role-of-decision-models>.