decider-75M

Introduction

decider-75M makes the typed decisions of the Typed Decisions benchmark. Given the workflow name and a case's state, it returns a probability for every option of all five of that workflow's questions, in one forward pass. It covers the benchmark's three question types:

• noul: a yes/no statement, answered as the probability that it is true.
• choice: one of several named options, such as a disposition or a next action.
• score: a point on an ordinal scale, such as severity or urgency from 0 to 3.

It is a specialist, finetuned on the benchmark's four workflows (agent-trace review, customer service, invoice processing, security incidents). For 0.7 points more accuracy at 2.3x the size, see decider-175M.

Evaluation

The benchmark's test split: 400 cases, 2,000 decisions. decider-75M and decider-175M were scored through their own released folders; the other rows are as reported on the benchmark's leaderboard.

Model	Parameters	Accuracy	Decisions correct	KL from gold	Brier	ECE
decider-175M	175.6M	0.8120	1,624 / 2,000	0.077	0.042	0.183
decider-75M	74.8M	0.8050	1,610 / 2,000	0.080	0.045	0.176
sysone-typed-decisions-0.4B	400M	0.8025	1,605 / 2,000	0.080	0.044	0.174
OpenDecider-nano	400M	0.796	1,592 / 2,000	0.079	0.043	–
Phocinae-Largha-150M-v1	144.3M	0.791	1,582 / 2,000	–	–	–
SafeDecide-400M	400M	0.787	1,574 / 2,000	0.083	0.045	0.171
SafeDecide-140M	140M	0.754	1,508 / 2,000	0.126	0.069	0.147

decider-75M or decider-175M both sit on the accuracy-for-size frontier, and the next point on it is at about 23x decider-175M's size.

An accuracy on 2,000 decisions has a standard error of about 0.009.

Its confidence is informative: answering only the decisions it is most sure of raises accuracy to 0.832 at 90% of decisions, 0.859 at 80% and 0.885 at 70%, so the least certain ones can be routed elsewhere.

By workflow

Workflow	Accuracy	Decisions correct
Invoice processing	85.4%	427 / 500
Customer service	80.6%	403 / 500
Security incidents	79.6%	398 / 500
Agent-trace observability	76.4%	382 / 500

By question type

Question type	Accuracy	Decisions correct
noul (yes/no)	85.3%	512 / 600
score (ordinal)	78.9%	631 / 800
choice (categorical)	77.8%	467 / 600

Model Details

Property	Value
Backbone	mmBERT-small: 22 encoder layers, hidden size 384, 6 attention heads
Decision heads	Mean-pooled encoder output into 20 linear heads, one per question
Parameters	74,814,407 (heads: 27,335)
Vocabulary	84,889 tokens, mmBERT's vocabulary cut to the tokens English uses; byte fallback still encodes any text
Input length	1,024 tokens, longer inputs truncated (the longest test case is 534)
Weights	float32, 299 MB; bfloat16 autocast on GPU, float32 on CPU
Latency	~22 ms per case, all five decisions (RTX PRO 4000 Blackwell, batch 1)
GPU memory	383 MiB peak while scoring test
License	CC BY 4.0

Training

All 22 layers fine-tuned for 12 epochs with soft cross-entropy against full probability distributions, on the benchmark's train split and the synthetic cases in decider-data.

The synthetic cases add 2.7 points: with the same settings on train alone, the model scores 0.7765 on test (KL from gold 0.114), against 0.8035 (KL 0.081) with them, both scored at the end of training.

Quickstart

1. Python inference

The folder ships its own loader, typed_decisions.py; it needs torch, transformers and safetensors.

import sys
from huggingface_hub import snapshot_download

folder = snapshot_download("semgram/decider-75M")
sys.path.insert(0, folder)
from typed_decisions import TypedDecisions

model = TypedDecisions.load(folder)

state = {
    "agent": {"autonomy": "checkpointed", "model": "internal-agent-v1"},
    "constraints": ["Do not exceed a $50 spend on cloud resources"],
    "task": "Rotate the expired TLS certificate on the staging load balancer.",
    "trace_summary": {"constraint_violations": 0, "duration_s": 32.5, "irreversible_actions": 0,
                      "steps": 11, "tool_errors": 0},
}
for question, dist in model.predict("agent_trace_observability", state).items():
    print(question, {k: round(v, 2) for k, v in dist.items()})
# action        {'continue': 0.52, 'human_review': 0.13, 'observe': 0.34, 'stop': 0.02}
# needs_review  {'false': 0.84, 'true': 0.16}
# outcome       {'failure': 0.06, 'harmful': 0.02, 'partial': 0.2, 'success': 0.72}
# risk          {'0': 0.25, '1': 0.6, '2': 0.12, '3': 0.03}
# urgency       {'0': 0.52, '1': 0.35, '2': 0.1, '3': 0.02}

The state can be a dict or a JSON string, as in the benchmark's state column. The workflows, questions and options are listed in typed_decisions.json.

2. Reproducing the evaluation

import json
from datasets import load_dataset

test = load_dataset("LocalLLaMA/typed-decisions", "all", split="test")
right = total = 0
for case in test:
    gold = json.loads(case["gold"])
    for question, dist in model.predict(case["workflow"], case["state"]).items():
        g = gold[question]["probabilities"]
        right += max(dist, key=dist.get) == max(g, key=g.get)
        total += 1
print(right / total)   # 0.8050

Limitations & Intended Use

• Intended use: fast, calibrated decisions inside the benchmark's four workflows, and a confidence signal for sending uncertain decisions to a person or a larger model.
• Scope: only the benchmark's four workflows. English only.

Citation

@misc{semgram2026decider75m,
  author       = {semgram},
  title        = {decider-75M: an encoder for typed probabilistic decisions},
  howpublished = {Hugging Face},
  year         = {2026},
  url          = {https://huggingface.co/semgram/decider-75M}
}