File size: 11,140 Bytes

---
license: apache-2.0
base_model: Qwen/Qwen3-4B-Base
library_name: pytorch
tags:
  - decision-model
  - system-one
  - calibrated-probabilities
  - typed-decisions
  - jev-like
pipeline_tag: text-classification
datasets:
  - stanfordnlp/snli
  - nyu-mll/multi_nli
  - facebook/anli
  - google/boolq
  - cais/mmlu
  - allenai/ai2_arc
  - allenai/openbookqa
  - tau/commonsense_qa
  - allenai/sciq
  - allenai/qasc
  - jeggers/logiqa2_formatted
  - deepmind/aqua_rat
  - openlifescienceai/medmcqa
  - cua-ai/cua-s1-forms
  - dwidlee/systemone-lite-general
  - MagaBitmex/jev-4b-distill-data
  - Zhengping/UNLI
  - metaeval/ambient
---

# typical-medium

**Typical is an open System-One decision model family, built by OzLabs.**

Give it a state — a support ticket, a document, a form, a log line, any text — a question with its
rubric, and a label set defined at runtime, and it returns a calibrated probability over those
labels plus an explicit abstain option. One forward pass, no generated text to parse, about 57 ms
per decision on an H100. Every decision is one of three typed primitives: **Choice** (categorical),
**Score** (ordinal), **Noul** (yes/no, exactly order-invariant).

`typical-medium` is **Release 1 at 4B** (checkpoint `tm1b`): the same recipe as
[`typical-small`](https://huggingface.co/OzLabs/typical-small) — dedicated Score and Noul heads, the
hard curriculum, 1,024-token states — on a larger backbone. It is the knee of capability per
millisecond in the release ladder: +11 points of JevBench standard and +11 points of MMLU-Pro
among-K over `typical-small`, at 1.25x the per-decision latency.

## Version 2 (2026-09-23) — new backbone, and what changed

The weights here were replaced on 2026-09-23, and **the backbone changed from `Qwen/Qwen3-4B-Base` to
`Qwen/Qwen3.5-4B-Base`** (both Apache-2.0). The previous checkpoint was trained on a corpus whose long
states rendered the case facts *after* the policy body and had learned to expect that ordering: on long
documents it lost 25 points when callers put the facts first, falling to the majority-class floor on
yes/no decisions. This version is trained on the corrected corpus and has no such bias.

Against the previous checkpoint:

| | v1 (Qwen3-4B) | **v2 (Qwen3.5-4B)** | |
|---|---:|---:|---|
| long states, facts-first (605 held-out items) | .612 | **.950** | **+33.8** |
| JevBench hard (public subset) | .423 | **.495** | +7.2 |
| TREC-fine | .420 | .486 | +6.6 |
| JevBench standard (public subset) | .806 | .861 | +5.6 |
| held-out Score | .528 | .586 | +5.8 |
| held-out Noul (yes/no) | .811 | .858 | +4.7 |
| uncertainty (u_real, held out) | .658 | .694 | +3.6 |
| CLINC-150 | .850 | **.795** | **−5.5** |
| HWU64 | .773 | **.720** | **−5.3** |

Broadly a clear improvement, with one real regression: **fine-grained intent classification over large
fixed label sets** (CLINC-150, HWU64) is 5 points worse. If that is your workload, v1's weights remain in
this repository's git history.

**Update your inference code.** The backbone change means this checkpoint will not load with an
`inference/` copy pulled before 2026-09-23: Qwen3.5's hybrid stack needs the VL-wrapper unwrap and the
Gated-DeltaNet LoRA targets, and 60 of this checkpoint's 124 LoRA tensors live on modules the older
`backbone.py` does not know about. Re-pull `inference/` from this repo.

## Why it's different

- **The decision is read directly from the model's own candidate-conditioned computation.** The
  state and candidates are encoded once; the answer is read off that computation instead of being
  sampled one token at a time.
- **The state is KV-cached**, so asking many questions about one state is cheap — each additional
  question is a short suffix against a cache that's already built.
- **Latency is flat up to ~64 options** (57 ms at K = 2, 58 ms at K = 32); 96 ms at K = 256 — the
  state dominates until the rendered option list does.
- **Abstention is a first-class output**, not a parsed refusal: every decision carries a calibrated
  P(∅) alongside the label probabilities.
- **Probabilities, not verbalized confidence.** No "I'm 80% sure" string to parse — an actual
  calibrated number.

## Model family

| | [`typical-small-preview`](https://huggingface.co/OzLabs/typical-small-preview) | [`typical-small`](https://huggingface.co/OzLabs/typical-small) | **`typical-medium`** |
|---|---|---|---|
| Backbone | Qwen3-1.7B-Base | Qwen3-1.7B-Base | Qwen3-4B-Base |
| Tap layer | 20 / 28 | 20 / 28 | 26 / 36 |
| Per-decision latency, K=2 (one H100) | 45 ms | 45 ms | 57 ms |
| Peak memory, K=2 | 6.4 GB | 6.4 GB | 14.6 GB |
| JevBench standard / easy / hard | .750 / 1.00 / .387 | .694 / 1.00 / .432 | .806 / 1.00 / .423 |
| MMLU-Pro among-K | .330 | .343 | .458 |
| Held-out workflow noul / score | .699 / .498 | .715 / .520 | .811 / .528 |
| Calibration (typed-decisions NLL) | 2.06 | 1.25 | **1.18** |

Full ablations and matched-baseline tables: REPORT.md §3af.

## Results

| | `typical-medium` (`tm1b`) |
|---|---|
| SNLI / MNLI / BoolQ / ANLI | .909 / .861 / .843 / .544 |
| CLINC-150 / TREC-fine / HWU64 / 20NG | .847 / .414 / .769 / .588 |
| Held-out workflow: noul / score / style | .811 / .528 / .859 |
| External floor: PagerDuty (floor .792) | **.838** |
| JevBench standard / easy / hard (Brier std / hard; ECE std) | .806 / 1.00 / .423 (.30 / .77; **.09**) |

**JevBench standard, same 231 public ids, next to the open field:**

| jeff (GLiFormer 400M) | Laya (ModernBERT) | **typical-medium** | open-alternative-jev (Qwen3.5-4B) | system-one-open (Gemma E2B LoRA) |
|---|---|---|---|---|
| .750 | .694 | **.806** | .833 | .931 |

Public-subset per-item comparison on the 231 public ids; not a ranked leaderboard entry.

## Quickstart

```bash
pip install typical-ai
```

```python
from typical_ai import Typical

m = Typical.from_pretrained("OzLabs/typical-medium", device="auto")

ticket = ("Customer reports repeated 500 errors on checkout after the last deploy; "
          "asking for a refund.")
m.choice(ticket, "Which team should handle this?", ["billing", "infra", "support", "security"])
m.score(ticket, "How severe is this ticket?", ["1", "2", "3", "4"])
m.noul(ticket, "Does this ticket violate the refund-abuse policy (3+ refunds in 30 days)?")
```

Verified: this exact sequence (download, install, run) works on CPU (checked against `typical-small`'s
identical inference package; `typical-medium` uses the same code path on a larger backbone).

`choice`/`score` return `{label: probability, ...}` plus `p_null`; `noul` returns `P(yes)`. `decide`
mirrors the internal `PCDMDecider.decide` exactly, including the runtime block (latency, `p_null`,
truncation flags) — see `inference/README.md` for the full API and `inference/example.py` for a
runnable script.

## Training recipe

- **Data mix** E .40 / K .15 / W .35 / U .10 of every batch (evidence, knowledge-MCQ, rubric-
  conditioned workflow, soft-target uncertainty), plus 15–20% null augmentation on W rows
  (`--null_aug W:0.20`) so the model learns to abstain, not just choose.
- **1,024-token decision states**, so policy documents and multi-hop states are in-distribution.
- **Typed heads:** Score is K-way Choice trained with ordinal-smoothed targets (τ = 0.7, zero head
  code); Noul is a dedicated Bernoulli head, routed per row — only rows whose candidates are exactly
  `{yes, no}` use it, every other 2-way set scores as ordinary Choice.
- **12,000 steps, effective batch 64** (`bs 64`, `grad_accum 4`), LoRA rank 16 on the top 8 layers of
  a 26-of-36-layer truncated backbone.
- **Held-out by construction:** one rubric style and three whole domains of the hard curriculum
  (`data_wh`), plus temporal/numeric/probability/trade-off composition (level 7), are eval-only —
  never trained on.

**Data sources and licenses:**

| Source | Role | License |
|---|---|---|
| SNLI, MultiNLI, ANLI, BoolQ, MMLU-aux, ARC, OpenBookQA, CommonsenseQA, SciQ, QASC, LogiQA2, AQuA-RAT, MedMCQA | evidence + knowledge-MCQ | permissive research licenses (CC-BY-SA / MIT / Apache-2.0 family) |
| `cua-ai/cua-s1-forms` | workflow (GUI-form synth) | MIT |
| `dwidlee/systemone-lite-general` | workflow (rule-labeled typed decisions) | MIT |
| `MagaBitmex/jev-4b-distill-data` | workflow (programmatic gold) | Apache-2.0 |
| in-repo rubric-conditioned generator (`data_wf`, `data_wf_long`) | workflow | generated, no external license |
| in-repo rule-engine generator (`data_wh`, DecisionMix v2) | hard curriculum | generated, no external license |
| `Zhengping/UNLI` (validation split) | soft-target uncertainty | MIT |
| `metaeval/ambient` (ambiguous rows, trained on) | soft-target uncertainty | **license undeclared on its HF card — flagged, not asserted** |
| `metaeval/chaos-mnli-ambiguity` (eval-only, never trained on) | soft-target uncertainty | **license undeclared on its HF card — flagged, not asserted** |

Evaluation-only, never trained on: JevBench (public subset), `typed-decisions`, PagerDuty, tree-choice,
jevlogs (Loghub research license, caveated), Mind2Web.

## Limitations

- Hard tier is .423 (chance .336, SE ≈ .045 at n = 111) — the least reliable tier; the open field's
  best is .73. The long-policy slice specifically is .21 — the weakest single family.
- TREC-fine regresses 10 points vs. `ladder_4b`, the same backbone trained without typed heads or
  DecisionMix v2 (K = 50 fine-grained topics) — the one clear regression at this size.
- Held-out rubric styles and Noul accuracy give back a few points vs. that same untyped 4B baseline,
  in exchange for the calibration and external-floor gains above.
- Long-document state ordering is **fixed in v2**: .950 with the case facts first and .879 with
  them last on the 605-item held-out set, against .612 / .866 for v1. Either ordering works now.
- Inference code is included in this repo; training code release to follow.

## License

Apache-2.0 (backbone `Qwen/Qwen3.5-4B-Base` is Apache-2.0; this release's weights are a LoRA adapter
plus small Choice/Score/Noul heads over that backbone). Training-data licenses above; JevBench
numbers are a public-subset run against `fstandhartinger/jevbench` v1.2.1, not a submitted or ranked
leaderboard entry.

## Files in this repository

- `best.pt` — this checkpoint (`tm1b`).
- `results.json` / `eval_wf_full.json` / `probe_results.json` / `jevbench_summary.json` — full eval
  suite, post-hoc held-out/external eval, probe results, JevBench run summary.
- `inference/` — the self-contained `Typical` inference package (see Quickstart above).
- `MANIFEST.json` — checksum + provenance record for this release.

## Lineage

[`typical-small-preview`](https://huggingface.co/OzLabs/typical-small-preview) (Release 0) →
[`typical-small`](https://huggingface.co/OzLabs/typical-small) (Release 1 at 1.7B) →
**`typical-medium`** (Release 1 at 4B, this repo).

Coming: `typical-large` — a 14B ladder checkpoint scores JevBench .875/.468 (standard/hard), but
probability calibration at that size is not yet fixed (§3ab). Trained, not released.

Typical is built by [OzLabs](https://huggingface.co/OzLabs).