| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3-4B-Base |
| library_name: pytorch |
| tags: |
| - decision-model |
| - system-one |
| - calibrated-probabilities |
| - typed-decisions |
| - jev-like |
| pipeline_tag: text-classification |
| datasets: |
| - stanfordnlp/snli |
| - nyu-mll/multi_nli |
| - facebook/anli |
| - google/boolq |
| - cais/mmlu |
| - allenai/ai2_arc |
| - allenai/openbookqa |
| - tau/commonsense_qa |
| - allenai/sciq |
| - allenai/qasc |
| - jeggers/logiqa2_formatted |
| - deepmind/aqua_rat |
| - openlifescienceai/medmcqa |
| - cua-ai/cua-s1-forms |
| - dwidlee/systemone-lite-general |
| - MagaBitmex/jev-4b-distill-data |
| - Zhengping/UNLI |
| - metaeval/ambient |
| --- |
| |
| # typical-medium |
|
|
| **Typical is an open System-One decision model family, built by OzLabs.** |
|
|
| Give it a state β a support ticket, a document, a form, a log line, any text β a question with its |
| rubric, and a label set defined at runtime, and it returns a calibrated probability over those |
| labels plus an explicit abstain option. One forward pass, no generated text to parse, about 57 ms |
| per decision on an H100. Every decision is one of three typed primitives: **Choice** (categorical), |
| **Score** (ordinal), **Noul** (yes/no, exactly order-invariant). |
|
|
| `typical-medium` is **Release 1 at 4B** (checkpoint `tm1b`): the same recipe as |
| [`typical-small`](https://huggingface.co/OzLabs/typical-small) β dedicated Score and Noul heads, the |
| hard curriculum, 1,024-token states β on a larger backbone. It is the knee of capability per |
| millisecond in the release ladder: +11 points of JevBench standard and +11 points of MMLU-Pro |
| among-K over `typical-small`, at 1.25x the per-decision latency. |
|
|
| ## Version 2 (2026-09-23) β new backbone, and what changed |
|
|
| The weights here were replaced on 2026-09-23, and **the backbone changed from `Qwen/Qwen3-4B-Base` to |
| `Qwen/Qwen3.5-4B-Base`** (both Apache-2.0). The previous checkpoint was trained on a corpus whose long |
| states rendered the case facts *after* the policy body and had learned to expect that ordering: on long |
| documents it lost 25 points when callers put the facts first, falling to the majority-class floor on |
| yes/no decisions. This version is trained on the corrected corpus and has no such bias. |
|
|
| Against the previous checkpoint: |
|
|
| | | v1 (Qwen3-4B) | **v2 (Qwen3.5-4B)** | | |
| |---|---:|---:|---| |
| | long states, facts-first (605 held-out items) | .612 | **.950** | **+33.8** | |
| | JevBench hard (public subset) | .423 | **.495** | +7.2 | |
| | TREC-fine | .420 | .486 | +6.6 | |
| | JevBench standard (public subset) | .806 | .861 | +5.6 | |
| | held-out Score | .528 | .586 | +5.8 | |
| | held-out Noul (yes/no) | .811 | .858 | +4.7 | |
| | uncertainty (u_real, held out) | .658 | .694 | +3.6 | |
| | CLINC-150 | .850 | **.795** | **β5.5** | |
| | HWU64 | .773 | **.720** | **β5.3** | |
| |
| Broadly a clear improvement, with one real regression: **fine-grained intent classification over large |
| fixed label sets** (CLINC-150, HWU64) is 5 points worse. If that is your workload, v1's weights remain in |
| this repository's git history. |
| |
| **Update your inference code.** The backbone change means this checkpoint will not load with an |
| `inference/` copy pulled before 2026-09-23: Qwen3.5's hybrid stack needs the VL-wrapper unwrap and the |
| Gated-DeltaNet LoRA targets, and 60 of this checkpoint's 124 LoRA tensors live on modules the older |
| `backbone.py` does not know about. Re-pull `inference/` from this repo. |
| |
| ## Why it's different |
| |
| - **The decision is read directly from the model's own candidate-conditioned computation.** The |
| state and candidates are encoded once; the answer is read off that computation instead of being |
| sampled one token at a time. |
| - **The state is KV-cached**, so asking many questions about one state is cheap β each additional |
| question is a short suffix against a cache that's already built. |
| - **Latency is flat up to ~64 options** (57 ms at K = 2, 58 ms at K = 32); 96 ms at K = 256 β the |
| state dominates until the rendered option list does. |
| - **Abstention is a first-class output**, not a parsed refusal: every decision carries a calibrated |
| P(β
) alongside the label probabilities. |
| - **Probabilities, not verbalized confidence.** No "I'm 80% sure" string to parse β an actual |
| calibrated number. |
| |
| ## Model family |
| |
| | | [`typical-small-preview`](https://huggingface.co/OzLabs/typical-small-preview) | [`typical-small`](https://huggingface.co/OzLabs/typical-small) | **`typical-medium`** | |
| |---|---|---|---| |
| | Backbone | Qwen3-1.7B-Base | Qwen3-1.7B-Base | Qwen3-4B-Base | |
| | Tap layer | 20 / 28 | 20 / 28 | 26 / 36 | |
| | Per-decision latency, K=2 (one H100) | 45 ms | 45 ms | 57 ms | |
| | Peak memory, K=2 | 6.4 GB | 6.4 GB | 14.6 GB | |
| | JevBench standard / easy / hard | .750 / 1.00 / .387 | .694 / 1.00 / .432 | .806 / 1.00 / .423 | |
| | MMLU-Pro among-K | .330 | .343 | .458 | |
| | Held-out workflow noul / score | .699 / .498 | .715 / .520 | .811 / .528 | |
| | Calibration (typed-decisions NLL) | 2.06 | 1.25 | **1.18** | |
| |
| Full ablations and matched-baseline tables: REPORT.md Β§3af. |
| |
| ## Results |
| |
| | | `typical-medium` (`tm1b`) | |
| |---|---| |
| | SNLI / MNLI / BoolQ / ANLI | .909 / .861 / .843 / .544 | |
| | CLINC-150 / TREC-fine / HWU64 / 20NG | .847 / .414 / .769 / .588 | |
| | Held-out workflow: noul / score / style | .811 / .528 / .859 | |
| | External floor: PagerDuty (floor .792) | **.838** | |
| | JevBench standard / easy / hard (Brier std / hard; ECE std) | .806 / 1.00 / .423 (.30 / .77; **.09**) | |
| |
| **JevBench standard, same 231 public ids, next to the open field:** |
| |
| | jeff (GLiFormer 400M) | Laya (ModernBERT) | **typical-medium** | open-alternative-jev (Qwen3.5-4B) | system-one-open (Gemma E2B LoRA) | |
| |---|---|---|---|---| |
| | .750 | .694 | **.806** | .833 | .931 | |
| |
| Public-subset per-item comparison on the 231 public ids; not a ranked leaderboard entry. |
| |
| ## Quickstart |
| |
| ```bash |
| pip install typical-ai |
| ``` |
| |
| ```python |
| from typical_ai import Typical |
|
|
| m = Typical.from_pretrained("OzLabs/typical-medium", device="auto") |
| |
| ticket = ("Customer reports repeated 500 errors on checkout after the last deploy; " |
| "asking for a refund.") |
| m.choice(ticket, "Which team should handle this?", ["billing", "infra", "support", "security"]) |
| m.score(ticket, "How severe is this ticket?", ["1", "2", "3", "4"]) |
| m.noul(ticket, "Does this ticket violate the refund-abuse policy (3+ refunds in 30 days)?") |
| ``` |
| |
| Verified: this exact sequence (download, install, run) works on CPU (checked against `typical-small`'s |
| identical inference package; `typical-medium` uses the same code path on a larger backbone). |
| |
| `choice`/`score` return `{label: probability, ...}` plus `p_null`; `noul` returns `P(yes)`. `decide` |
| mirrors the internal `PCDMDecider.decide` exactly, including the runtime block (latency, `p_null`, |
| truncation flags) β see `inference/README.md` for the full API and `inference/example.py` for a |
| runnable script. |
|
|
| ## Training recipe |
|
|
| - **Data mix** E .40 / K .15 / W .35 / U .10 of every batch (evidence, knowledge-MCQ, rubric- |
| conditioned workflow, soft-target uncertainty), plus 15β20% null augmentation on W rows |
| (`--null_aug W:0.20`) so the model learns to abstain, not just choose. |
| - **1,024-token decision states**, so policy documents and multi-hop states are in-distribution. |
| - **Typed heads:** Score is K-way Choice trained with ordinal-smoothed targets (Ο = 0.7, zero head |
| code); Noul is a dedicated Bernoulli head, routed per row β only rows whose candidates are exactly |
| `{yes, no}` use it, every other 2-way set scores as ordinary Choice. |
| - **12,000 steps, effective batch 64** (`bs 64`, `grad_accum 4`), LoRA rank 16 on the top 8 layers of |
| a 26-of-36-layer truncated backbone. |
| - **Held-out by construction:** one rubric style and three whole domains of the hard curriculum |
| (`data_wh`), plus temporal/numeric/probability/trade-off composition (level 7), are eval-only β |
| never trained on. |
|
|
| **Data sources and licenses:** |
|
|
| | Source | Role | License | |
| |---|---|---| |
| | SNLI, MultiNLI, ANLI, BoolQ, MMLU-aux, ARC, OpenBookQA, CommonsenseQA, SciQ, QASC, LogiQA2, AQuA-RAT, MedMCQA | evidence + knowledge-MCQ | permissive research licenses (CC-BY-SA / MIT / Apache-2.0 family) | |
| | `cua-ai/cua-s1-forms` | workflow (GUI-form synth) | MIT | |
| | `dwidlee/systemone-lite-general` | workflow (rule-labeled typed decisions) | MIT | |
| | `MagaBitmex/jev-4b-distill-data` | workflow (programmatic gold) | Apache-2.0 | |
| | in-repo rubric-conditioned generator (`data_wf`, `data_wf_long`) | workflow | generated, no external license | |
| | in-repo rule-engine generator (`data_wh`, DecisionMix v2) | hard curriculum | generated, no external license | |
| | `Zhengping/UNLI` (validation split) | soft-target uncertainty | MIT | |
| | `metaeval/ambient` (ambiguous rows, trained on) | soft-target uncertainty | **license undeclared on its HF card β flagged, not asserted** | |
| | `metaeval/chaos-mnli-ambiguity` (eval-only, never trained on) | soft-target uncertainty | **license undeclared on its HF card β flagged, not asserted** | |
|
|
| Evaluation-only, never trained on: JevBench (public subset), `typed-decisions`, PagerDuty, tree-choice, |
| jevlogs (Loghub research license, caveated), Mind2Web. |
|
|
| ## Limitations |
|
|
| - Hard tier is .423 (chance .336, SE β .045 at n = 111) β the least reliable tier; the open field's |
| best is .73. The long-policy slice specifically is .21 β the weakest single family. |
| - TREC-fine regresses 10 points vs. `ladder_4b`, the same backbone trained without typed heads or |
| DecisionMix v2 (K = 50 fine-grained topics) β the one clear regression at this size. |
| - Held-out rubric styles and Noul accuracy give back a few points vs. that same untyped 4B baseline, |
| in exchange for the calibration and external-floor gains above. |
| - Long-document state ordering is **fixed in v2**: .950 with the case facts first and .879 with |
| them last on the 605-item held-out set, against .612 / .866 for v1. Either ordering works now. |
| - Inference code is included in this repo; training code release to follow. |
|
|
| ## License |
|
|
| Apache-2.0 (backbone `Qwen/Qwen3.5-4B-Base` is Apache-2.0; this release's weights are a LoRA adapter |
| plus small Choice/Score/Noul heads over that backbone). Training-data licenses above; JevBench |
| numbers are a public-subset run against `fstandhartinger/jevbench` v1.2.1, not a submitted or ranked |
| leaderboard entry. |
|
|
| ## Files in this repository |
|
|
| - `best.pt` β this checkpoint (`tm1b`). |
| - `results.json` / `eval_wf_full.json` / `probe_results.json` / `jevbench_summary.json` β full eval |
| suite, post-hoc held-out/external eval, probe results, JevBench run summary. |
| - `inference/` β the self-contained `Typical` inference package (see Quickstart above). |
| - `MANIFEST.json` β checksum + provenance record for this release. |
|
|
| ## Lineage |
|
|
| [`typical-small-preview`](https://huggingface.co/OzLabs/typical-small-preview) (Release 0) β |
| [`typical-small`](https://huggingface.co/OzLabs/typical-small) (Release 1 at 1.7B) β |
| **`typical-medium`** (Release 1 at 4B, this repo). |
|
|
| Coming: `typical-large` β a 14B ladder checkpoint scores JevBench .875/.468 (standard/hard), but |
| probability calibration at that size is not yet fixed (Β§3ab). Trained, not released. |
|
|
| Typical is built by [OzLabs](https://huggingface.co/OzLabs). |
|
|
