decider-75M: weights, tokenizer, typed_decisions loader and card (CC BY 4.0)

d114dcc verified 4 days ago

7.03 kB

---
license: cc-by-4.0
language:
- en
base_model: jhu-clsp/mmBERT-small
datasets:
- LocalLLaMA/typed-decisions
pipeline_tag: text-classification
tags:
- typed-decisions
- decision-model
---
# decider-75M

## Introduction

decider-75M makes the typed decisions of the
[Typed Decisions](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) benchmark. Given the workflow name and a
case's state, it returns a probability for every option of all five of that workflow's questions, in one forward pass.
It covers the benchmark's three question types:

- `noul`: a yes/no statement, answered as the probability that it is true.
- `choice`: one of several named options, such as a disposition or a next action.
- `score`: a point on an ordinal scale, such as severity or urgency from 0 to 3.

It is a specialist, finetuned on the benchmark's four workflows (agent-trace review, customer
service, invoice processing, security incidents). For 0.7 points more accuracy at 2.3x the
size, see [decider-175M](https://huggingface.co/semgram/decider-175M).

## Evaluation

The benchmark's `test` split: 400 cases, 2,000 decisions. decider-75M and decider-175M were scored through their own
released folders; the other rows are as reported on the benchmark's leaderboard.

| Model | Parameters | Accuracy | Decisions correct | KL from gold | Brier | ECE |
|---|---|---|---|---|---|---|
| [decider-175M](https://huggingface.co/semgram/decider-175M) | 175.6M | 0.8120 | 1,624 / 2,000 | 0.077 | 0.042 | 0.183 |
| **decider-75M** | **74.8M** | **0.8050** | **1,610 / 2,000** | **0.080** | **0.045** | **0.176** |
| [sysone-typed-decisions-0.4B](https://huggingface.co/forward-metric/sysone-typed-decisions-0.4B) | 400M | 0.8025 | 1,605 / 2,000 | 0.080 | 0.044 | 0.174 |
| [OpenDecider-nano](https://huggingface.co/manjunathshiva/opendecider-nano) | 400M | 0.796 | 1,592 / 2,000 | 0.079 | 0.043 | – |
| [Phocinae-Largha-150M-v1](https://huggingface.co/Phocinae/Phocinae-Largha-150M-v1) | 144.3M | 0.791 | 1,582 / 2,000 | – | – | – |
| [SafeDecide-400M](https://huggingface.co/yotambr/safedecide-400m) | 400M | 0.787 | 1,574 / 2,000 | 0.083 | 0.045 | 0.171 |
| [SafeDecide-140M](https://huggingface.co/yotambr/safedecide-140m) | 140M | 0.754 | 1,508 / 2,000 | 0.126 | 0.069 | 0.147 |

![Accuracy on test against parameter count, for every leaderboard model with a published size](https://huggingface.co/semgram/decider-75M/resolve/main/accuracy_vs_size.png)

decider-75M or decider-175M both sit on the accuracy-for-size frontier, and the next point on it is at about 23x decider-175M's size.

An accuracy on 2,000 decisions has a standard error of about 0.009.

Its confidence is informative: answering only the decisions it is most sure of raises accuracy to 0.832 at 90% of
decisions, 0.859 at 80% and 0.885 at 70%, so the least certain ones can be routed elsewhere.

### By workflow

| Workflow | Accuracy | Decisions correct |
|---|---|---|
| Invoice processing | 85.4% | 427 / 500 |
| Customer service | 80.6% | 403 / 500 |
| Security incidents | 79.6% | 398 / 500 |
| Agent-trace observability | 76.4% | 382 / 500 |

### By question type

| Question type | Accuracy | Decisions correct |
|---|---|---|
| `noul` (yes/no) | 85.3% | 512 / 600 |
| `score` (ordinal) | 78.9% | 631 / 800 |
| `choice` (categorical) | 77.8% | 467 / 600 |

## Model Details

| Property | Value |
|---|---|
| Backbone | [mmBERT-small](https://huggingface.co/jhu-clsp/mmBERT-small): 22 encoder layers, hidden size 384, 6 attention heads |
| Decision heads | Mean-pooled encoder output into 20 linear heads, one per question |
| Parameters | 74,814,407 (heads: 27,335) |
| Vocabulary | 84,889 tokens, mmBERT's vocabulary cut to the tokens English uses; byte fallback still encodes any text |
| Input length | 1,024 tokens, longer inputs truncated (the longest `test` case is 534) |
| Weights | float32, 299 MB; bfloat16 autocast on GPU, float32 on CPU |
| Latency | ~22 ms per case, all five decisions (RTX PRO 4000 Blackwell, batch 1) |
| GPU memory | 383 MiB peak while scoring `test` |
| License | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |

### Training

All 22 layers fine-tuned for 12 epochs with soft cross-entropy against full probability distributions, on the
benchmark's `train` split and the synthetic cases in [decider-data](https://huggingface.co/datasets/semgram/decider-data).

The synthetic cases add 2.7 points: with the same settings on `train` alone, the model scores 0.7765 on `test` (KL from
gold 0.114), against 0.8035 (KL 0.081) with them, both scored at the end of training.

## Quickstart

### 1. Python inference

The folder ships its own loader, `typed_decisions.py`; it needs `torch`, `transformers` and `safetensors`.

```python
import sys
from huggingface_hub import snapshot_download
folder = snapshot_download("semgram/decider-75M")
sys.path.insert(0, folder)
from typed_decisions import TypedDecisions
model = TypedDecisions.load(folder)
state = {
    "agent": {"autonomy": "checkpointed", "model": "internal-agent-v1"},
    "constraints": ["Do not exceed a $50 spend on cloud resources"],
    "task": "Rotate the expired TLS certificate on the staging load balancer.",
    "trace_summary": {"constraint_violations": 0, "duration_s": 32.5, "irreversible_actions": 0,
                      "steps": 11, "tool_errors": 0},
}
for question, dist in model.predict("agent_trace_observability", state).items():
    print(question, {k: round(v, 2) for k, v in dist.items()})
# action        {'continue': 0.52, 'human_review': 0.13, 'observe': 0.34, 'stop': 0.02}
# needs_review  {'false': 0.84, 'true': 0.16}
# outcome       {'failure': 0.06, 'harmful': 0.02, 'partial': 0.2, 'success': 0.72}
# risk          {'0': 0.25, '1': 0.6, '2': 0.12, '3': 0.03}
# urgency       {'0': 0.52, '1': 0.35, '2': 0.1, '3': 0.02}
```

The state can be a dict or a JSON string, as in the benchmark's `state` column. The workflows, questions and options
are listed in `typed_decisions.json`.

### 2. Reproducing the evaluation

```python
import json
from datasets import load_dataset
test = load_dataset("LocalLLaMA/typed-decisions", "all", split="test")
right = total = 0
for case in test:
    gold = json.loads(case["gold"])
    for question, dist in model.predict(case["workflow"], case["state"]).items():
        g = gold[question]["probabilities"]
        right += max(dist, key=dist.get) == max(g, key=g.get)
        total += 1
print(right / total)   # 0.8050
```

## Limitations & Intended Use

- **Intended use:** fast, calibrated decisions inside the benchmark's four workflows, and a confidence signal for
  sending uncertain decisions to a person or a larger model.
- **Scope:** only the benchmark's four workflows. English only.

## Citation

```bibtex
@misc{semgram2026decider75m,
  author       = {semgram},
  title        = {decider-75M: an encoder for typed probabilistic decisions},
  howpublished = {Hugging Face},
  year         = {2026},
  url          = {https://huggingface.co/semgram/decider-75M}
}
```

	---
	license: cc-by-4.0
	language:
	- en
	base_model: jhu-clsp/mmBERT-small
	datasets:
	- LocalLLaMA/typed-decisions
	pipeline_tag: text-classification
	tags:
	- typed-decisions
	- decision-model
	---

	# decider-75M

	## Introduction

	decider-75M makes the typed decisions of the
	[Typed Decisions](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) benchmark. Given the workflow name and a
	case's state, it returns a probability for every option of all five of that workflow's questions, in one forward pass.
	It covers the benchmark's three question types:

	- `noul`: a yes/no statement, answered as the probability that it is true.
	- `choice`: one of several named options, such as a disposition or a next action.
	- `score`: a point on an ordinal scale, such as severity or urgency from 0 to 3.

	It is a specialist, finetuned on the benchmark's four workflows (agent-trace review, customer
	service, invoice processing, security incidents). For 0.7 points more accuracy at 2.3x the
	size, see [decider-175M](https://huggingface.co/semgram/decider-175M).

	## Evaluation

	The benchmark's `test` split: 400 cases, 2,000 decisions. decider-75M and decider-175M were scored through their own
	released folders; the other rows are as reported on the benchmark's leaderboard.

	\| Model \| Parameters \| Accuracy \| Decisions correct \| KL from gold \| Brier \| ECE \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| [decider-175M](https://huggingface.co/semgram/decider-175M) \| 175.6M \| 0.8120 \| 1,624 / 2,000 \| 0.077 \| 0.042 \| 0.183 \|
	\| decider-75M \| 74.8M \| 0.8050 \| 1,610 / 2,000 \| 0.080 \| 0.045 \| 0.176 \|
	\| [sysone-typed-decisions-0.4B](https://huggingface.co/forward-metric/sysone-typed-decisions-0.4B) \| 400M \| 0.8025 \| 1,605 / 2,000 \| 0.080 \| 0.044 \| 0.174 \|
	\| [OpenDecider-nano](https://huggingface.co/manjunathshiva/opendecider-nano) \| 400M \| 0.796 \| 1,592 / 2,000 \| 0.079 \| 0.043 \| – \|
	\| [Phocinae-Largha-150M-v1](https://huggingface.co/Phocinae/Phocinae-Largha-150M-v1) \| 144.3M \| 0.791 \| 1,582 / 2,000 \| – \| – \| – \|
	\| [SafeDecide-400M](https://huggingface.co/yotambr/safedecide-400m) \| 400M \| 0.787 \| 1,574 / 2,000 \| 0.083 \| 0.045 \| 0.171 \|
	\| [SafeDecide-140M](https://huggingface.co/yotambr/safedecide-140m) \| 140M \| 0.754 \| 1,508 / 2,000 \| 0.126 \| 0.069 \| 0.147 \|

	![Accuracy on test against parameter count, for every leaderboard model with a published size](https://huggingface.co/semgram/decider-75M/resolve/main/accuracy_vs_size.png)

	decider-75M or decider-175M both sit on the accuracy-for-size frontier, and the next point on it is at about 23x decider-175M's size.

	An accuracy on 2,000 decisions has a standard error of about 0.009.

	Its confidence is informative: answering only the decisions it is most sure of raises accuracy to 0.832 at 90% of
	decisions, 0.859 at 80% and 0.885 at 70%, so the least certain ones can be routed elsewhere.

	### By workflow

	\| Workflow \| Accuracy \| Decisions correct \|
	\|---\|---\|---\|
	\| Invoice processing \| 85.4% \| 427 / 500 \|
	\| Customer service \| 80.6% \| 403 / 500 \|
	\| Security incidents \| 79.6% \| 398 / 500 \|
	\| Agent-trace observability \| 76.4% \| 382 / 500 \|

	### By question type

	\| Question type \| Accuracy \| Decisions correct \|
	\|---\|---\|---\|
	\| `noul` (yes/no) \| 85.3% \| 512 / 600 \|
	\| `score` (ordinal) \| 78.9% \| 631 / 800 \|
	\| `choice` (categorical) \| 77.8% \| 467 / 600 \|

	## Model Details

	\| Property \| Value \|
	\|---\|---\|
	\| Backbone \| [mmBERT-small](https://huggingface.co/jhu-clsp/mmBERT-small): 22 encoder layers, hidden size 384, 6 attention heads \|
	\| Decision heads \| Mean-pooled encoder output into 20 linear heads, one per question \|
	\| Parameters \| 74,814,407 (heads: 27,335) \|
	\| Vocabulary \| 84,889 tokens, mmBERT's vocabulary cut to the tokens English uses; byte fallback still encodes any text \|
	\| Input length \| 1,024 tokens, longer inputs truncated (the longest `test` case is 534) \|
	\| Weights \| float32, 299 MB; bfloat16 autocast on GPU, float32 on CPU \|
	\| Latency \| ~22 ms per case, all five decisions (RTX PRO 4000 Blackwell, batch 1) \|
	\| GPU memory \| 383 MiB peak while scoring `test` \|
	\| License \| [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) \|

	### Training

	All 22 layers fine-tuned for 12 epochs with soft cross-entropy against full probability distributions, on the
	benchmark's `train` split and the synthetic cases in [decider-data](https://huggingface.co/datasets/semgram/decider-data).

	The synthetic cases add 2.7 points: with the same settings on `train` alone, the model scores 0.7765 on `test` (KL from
	gold 0.114), against 0.8035 (KL 0.081) with them, both scored at the end of training.

	## Quickstart

	### 1. Python inference

	The folder ships its own loader, `typed_decisions.py`; it needs `torch`, `transformers` and `safetensors`.

	```python
	import sys
	from huggingface_hub import snapshot_download

	folder = snapshot_download("semgram/decider-75M")
	sys.path.insert(0, folder)
	from typed_decisions import TypedDecisions

	model = TypedDecisions.load(folder)

	state = {
	"agent": {"autonomy": "checkpointed", "model": "internal-agent-v1"},
	"constraints": ["Do not exceed a $50 spend on cloud resources"],
	"task": "Rotate the expired TLS certificate on the staging load balancer.",
	"trace_summary": {"constraint_violations": 0, "duration_s": 32.5, "irreversible_actions": 0,
	"steps": 11, "tool_errors": 0},
	}
	for question, dist in model.predict("agent_trace_observability", state).items():
	print(question, {k: round(v, 2) for k, v in dist.items()})
	# action {'continue': 0.52, 'human_review': 0.13, 'observe': 0.34, 'stop': 0.02}
	# needs_review {'false': 0.84, 'true': 0.16}
	# outcome {'failure': 0.06, 'harmful': 0.02, 'partial': 0.2, 'success': 0.72}
	# risk {'0': 0.25, '1': 0.6, '2': 0.12, '3': 0.03}
	# urgency {'0': 0.52, '1': 0.35, '2': 0.1, '3': 0.02}
	```

	The state can be a dict or a JSON string, as in the benchmark's `state` column. The workflows, questions and options
	are listed in `typed_decisions.json`.

	### 2. Reproducing the evaluation

	```python
	import json
	from datasets import load_dataset

	test = load_dataset("LocalLLaMA/typed-decisions", "all", split="test")
	right = total = 0
	for case in test:
	gold = json.loads(case["gold"])
	for question, dist in model.predict(case["workflow"], case["state"]).items():
	g = gold[question]["probabilities"]
	right += max(dist, key=dist.get) == max(g, key=g.get)
	total += 1
	print(right / total) # 0.8050
	```

	## Limitations & Intended Use

	- Intended use: fast, calibrated decisions inside the benchmark's four workflows, and a confidence signal for
	sending uncertain decisions to a person or a larger model.
	- Scope: only the benchmark's four workflows. English only.

	## Citation

	```bibtex
	@misc{semgram2026decider75m,
	author = {semgram},
	title = {decider-75M: an encoder for typed probabilistic decisions},
	howpublished = {Hugging Face},
	year = {2026},
	url = {https://huggingface.co/semgram/decider-75M}
	}
	```