Publish release artefacts + model card + MANIFEST

MANIFEST.json

@@ -0,0 +1,22 @@
+{
+  "release": "typical-small",
+  "public_repo": "OzLabs/typical-small",
+  "public_url": "https://huggingface.co/OzLabs/typical-small",
+  "date": "2026-09-21",
+  "publish_date": "2026-09-21",
+  "source_git_commit": "1ff61f4c40b84ba28983d2258e96ffc759bda162",
+  "source_checkpoint": "ts1b",
+  "parent": "typical-small-preview",
+  "parent_public_repo": "OzLabs/typical-small-preview",
+  "lineage": "typical-small-preview -> typical-small",
+  "source_hf_repo": "guychuk/pcdm-runs",
+  "source_hf_path": "typical-small/",
+  "files": {
+    "best.pt": "sha256:f62d6e083ea80a08716d901f8409510b6a97cfe00bc65db25a37691ed555c46e",
+    "results.json": null,
+    "eval_wf_full.json": null,
+    "probe_results.json": null,
+    "jevbench_summary.json": null
+  },
+  "notes": "Public mirror of guychuk/pcdm-runs/typical-small/{best.pt,results.json,eval_wf_full.json,probe_results.json,jevbench_summary.json}, downloaded and re-uploaded byte-identical (best.pt sha256 verified against releases/MANIFEST.json before and after upload). results_nomm.json (the pre-eval-only-rerun results file) present in the private source repo was not published here. README.md is a newly authored public model card, not copied from the source repo."
+}

README.md

@@ -0,0 +1,274 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen3-1.7B-Base
+library_name: pytorch
+tags:
+  - decision-model
+  - system-one
+  - calibrated-probabilities
+  - typed-decisions
+  - jev-like
+pipeline_tag: text-classification
+datasets:
+  - stanfordnlp/snli
+  - nyu-mll/multi_nli
+  - facebook/anli
+  - google/boolq
+  - cais/mmlu
+  - allenai/ai2_arc
+  - allenai/openbookqa
+  - tau/commonsense_qa
+  - allenai/sciq
+  - allenai/qasc
+  - jeggers/logiqa2_formatted
+  - deepmind/aqua_rat
+  - openlifescienceai/medmcqa
+  - cua-ai/cua-s1-forms
+  - dwidlee/systemone-lite-general
+  - MagaBitmex/jev-4b-distill-data
+  - Zhengping/UNLI
+  - metaeval/ambient
+---
+
+# typical-small
+
+**Credit: OzLabs.**
+
+## What this is
+
+`typical-small` is a Jev-style System-One decision model: given a **state** (context), a
+**question** (instructions + rubric/criteria), and a **runtime-defined label set** (not a fixed
+vocabulary — the candidates are passed in at inference time), it returns a calibrated probability
+distribution over those labels plus an explicit abstain (∅) option. It implements three typed
+decision primitives directly: **Choice** (categorical, N-way, native contextual readout),
+**Score** (ordinal, K-way Choice trained with ordinal-smoothed targets), and **Noul** (yes/no, a
+dedicated Bernoulli head routed per row). The model does not generate text; it produces a
+probability vector (plus abstain mass) over whatever options the caller supplies for that decision.
+
+Release 1 (PLAN7 Tracks A–D): a clean retrain of the 1.7B checkpoint, `ts1b`, uploaded verbatim to
+`guychuk/pcdm-runs` under `typical-small/`. This is the frozen successor to `typical-small-preview`
+(`nc_v3_tap20_wf`) — REPORT.md §3ae's verdict, unchanged. Parent release: `typical-small-preview`.
+
+## What this model is (architecture)
+
+- **Backbone:** `Qwen/Qwen3-1.7B-Base`, truncated at layer 20 of 28 — same tap as the preview.
+- **Adaptation:** LoRA, rank 16, on the top 8 layers (13–20) of the truncated backbone.
+- **Readout:** native contextual decision head (`nc_head=n3`), `letters_nonull` rendering, *factored*
+  null (∅ is a separate head decision, not a rendered candidate) — identical shape to the preview.
+- **Typed primitives** (PLAN7 Track C, new in this release):
+  - **Choice** — the N3 contextual readout above, unchanged.
+  - **Noul** (yes/no) — a Bernoulli head, `P(yes) = σ(w·h_D)`, on the decision state with a
+    query-only suffix (no rendered candidates). Routed **per row** (commit `7b9d520`): a row only
+    goes through the Bernoulli head if its candidate set is exactly `{yes, no}`; every other 2-way
+    label set (e.g. `{true, false}`, `{approve, reject}`) is scored as ordinary Choice. The first
+    `ts1` run collapsed to chance because this routing was global instead of per-row (REPORT.md
+    §3ae) — `ts1b` is the fix, gated by 140 tests.
+  - **Score** — K-way Choice trained with ordinal-smoothed targets (τ = 0.7, §3ac's adopted recipe).
+    Note: `results.json["args"]` was regenerated by an `--eval_only` rerun (to restore missing MMLU
+    sets) and therefore records the *eval* command's flags (`ordinal_smooth: 0.0`); the training
+    args stored inside `best.pt` are authoritative: `ordinal_smooth 0.7`, `noul_head bern`,
+    `family_weights E:0.40,K:0.15,W:0.35,U:0.10`, `null_aug W:0.20`, `max_state 1024`, `bs 64`,
+    `grad_accum 4`, `steps 12000`. The original results file is preserved as `results_nomm.json`
+    (not included in this public release).
+
+## Training args (from `results.json["args"]` of the eval-only rerun — see the Score note above; `best.pt` holds the training args)
+
+```
+backbone: Qwen/Qwen3-1.7B-Base
+lora_layers: 8            lora_r: 16            lora_lr: 0.0001
+tap_layer: 20             zscore: false
+readout: native           nc_head: n3            nc_render: letters_nonull
+null: factored            noul_head: bern        score_head: choice     ordinal_smooth: 0.0   # eval-only rerun flag; training used 0.7 (see above)
+steps: 12000              bs: 64                 grad_accum: 1
+eval_every: 4000          val_every: 1000        ckpt_every: 2000
+data: data_v5             extra_data: data_kb,data_wf,data_wf_hf,data_wf_long,data_wh,data_u
+family_weights: E:0.40,K:0.15,W:0.35,U:0.10
+bucket_map: data_wf_long=W,data_wh=W,data_u=U
+null_aug: W:0.20          max_state: 1024
+tower_d: 512              tower_layers: 2        tower_heads: 8
+seed: 0
+```
+
+Final val NLL 0.376, temperature T = 1.124, null_offset 0.0 (fit post-hoc on the `val` split, applied
+at eval).
+
+## Data
+
+Family-balanced sampler, E .40 / K .15 / W .35 / U .10 of every training batch, plus `--null_aug
+W:0.20` (15–20% of W rows get their gold candidate removed so the target is ∅) and `--max_state 1024`
+(vs. the preview's 256):
+
+- **`data_v5`** — evidence tasks `stanfordnlp/snli`, `nyu-mll/multi_nli`, `facebook/anli`,
+  `google/boolq` plus a wide span of classification label vocabularies at low rows-per-source, so
+  the model can't memorize a fixed set of label vocabularies.
+- **`data_kb`** — closed-book knowledge-MCQ distillation from `cais/mmlu` (auxiliary_train),
+  `allenai/ai2_arc`, `allenai/openbookqa`, `tau/commonsense_qa`, `allenai/sciq`, `allenai/qasc`,
+  `jeggers/logiqa2_formatted`, `deepmind/aqua_rat`, `openlifescienceai/medmcqa`.
+- **`data_wf` / `data_wf_hf`** — unchanged from the preview: our own rubric-conditioned
+  typed-workflow corpus (Choice/Noul/Score, held-out rubric styles), plus `cua-ai/cua-s1-forms`
+  (60k, MIT), `dwidlee/systemone-lite-general` (20k, MIT), `MagaBitmex/jev-4b-distill-data` (6.6k,
+  Apache-2.0). See `releases/typical-small-preview.md` (this org's preview release card) for the
+  per-source breakdown.
+- **`data_wf_long`** — 1,024-token-state variant of the workflow corpus (PLAN7 §3z/§3aa): the same
+  in-repo rubric-conditioned generator at longer state length, added because long-state families
+  (policy documents, multi-hop) were untrained at 256 tokens. No external license constraints.
+- **`data_wh`** (`scripts/decisionmix_v2.py`, DecisionMix v2, PLAN7 Track D): 60,605 train / 3,000
+  val / 6 eval files, a programmatic rule engine (12 domains × 6 boolean conditions, levels 1–7),
+  100% of train rows in counterfactual rubric groups (same state+candidates, ≥2 distinct golds).
+  Generated in-repo, no external license constraints. Held out: 1 domain/level
+  (`wh_heldout_grammar`), 3 whole domains (`wh_heldout_family`), 2 rubric styles
+  (`wh_heldout_style`), and level 7 — temporal/numeric/probability/trade-off composition — which is
+  eval-only, never trained on. 0/231 JevBench leak hits.
+- **`data_u`** (same script, DecisionMix v2): 31,029 train / 2,000 val / 3 eval files, 100%
+  soft-target rows. Real sources: `Zhengping/UNLI`'s *validation* split (train/test already
+  consumed elsewhere) and `metaeval/ambient`'s ambiguous rows (uniform target over listed labels).
+  `metaeval/chaos-mnli-ambiguity` is **eval-only** (MNLI-only mirror of `chaos-nli`, which 404s),
+  never trained on. The remaining 92% is four synthetic generators with exact closed-form targets
+  (partial evidence, noisy-sensor Bayes posterior, ordinal confusion, conflicting-sources log-odds),
+  each with a held-out parameter range, generated in-repo. 0/231 JevBench leak hits.
+
+See PLAN7.md's "Track D — DecisionMix v2 build notes" for the full corpus notes, and REPORT.md
+§3z–§3ae for the mixture-sweep and Release-1-candidate history.
+
+## Results
+
+### §3ae — Release-1 candidate at 1.7B (`ts1b`) vs the preview and the intermediate `r1_dmv2`
+
+| | `typical-small-preview` (§3t/§3w) | `r1_dmv2` (§3ad) | **`ts1b`** | 1.7B base |
+|---|---|---|---|---|
+| CLINC-150 / TREC-fine / HWU64 / 20NG | .734 / .372 / .735 / .522 | .820 / .474 / .726 / .501 | .804 / **.508** / .761 / .540 | .845 / .468 / .757 / .515 |
+| SNLI / MNLI / BoolQ / ANLI | .894 / .849 / .831 / .481 | .894 / .852 / .834 / .501 | .894 / .859 / .824 / .497 | .904 / .865 / .834 / .507 |
+| MMLU-Pro among-K / Δ_q_sh / false-abstain | .330 / .119 / .006 | .333 / .110 / – | .343 / .127 / .005 | .353 / .122 / .003 |
+| held-out noul / score / style / flip both-correct (full rows) | .699 / .498 / .901 / .565 | .658 / .501 / .876 / .519 | **.715** / **.520** / .859 / **.581** | – |
+| held-out score NLL / typed-decisions NLL (full rows) | 2.03 / 2.06 | 1.56 / 1.71 | **1.01 / 1.25** | – / 1.22 |
+| wh held-out family / grammar / style / level 7 / flip | – | .827 / .881 / .888 / .498 / .710 | **.836 / .898 / .896** / .495 / **.718** | – |
+| u ChaosNLI / real held-out / synthetic (acc, NLL) | – | .537 (1.00) / .623 (.67) / .909 (.67) | .547 (1.00) / **.649** (.66) / .916 (.67) | – |
+| external: PagerDuty (floor .792) / jevlogs (.697) / Mind2Web (.427) / tree-choice / typed-decisions acc | .779 / .522 / .427 / .539 / .457 | .651† / .500† / .346† / – / .421† | **.817 / .710** / .357 / .474 / **.510** | .776 / .697 / .300 / .506 / .487 |
+| JevBench std / easy / hard (Brier std / hard; ECE std) | .750 / 1.00 / .387 (.40 / .88; .15) | .750 / 1.00 / .441 (.42 / .79) | .694 / 1.00 / .432 (.40 / .79; **.11**) | .694 / 1.00 / .378 (.47 / .74) |
+| Noul reversed-label |ΔP(yes)| max | up to .55 | – | **0** (10/11 sets; .009 on one) | – |
+
+† `r1_dmv2`'s external numbers are a train-time 1,500-row/256-token pass, not re-run at full length.
+
+**Read (REPORT.md §3ae).** Against the frozen preview, `ts1b` keeps E and K inside budget (CLINC −4 vs
+the untrained base is the one edge; TREC/20NG land above base), raises held-out noul/score/flip,
+halves the soft-target NLLs (score 2.03 → 1.01, typed-decisions 2.06 → 1.25), beats or ties `r1_dmv2`
+on every curriculum and uncertainty set, is exactly order-invariant on Noul, and is the first 1.7B
+model above the constant-prediction floor on PagerDuty (.817) and jevlogs (.710, marginal). It gives
+back held-out styles (−4), Mind2Web (below floor again), tree-choice (−6), and JevBench standard
+(.750 → .694, ~1 SE at n_eff = 36; extraction 12 → 9 of 12) while JevBench hard rises (.387 → .432,
+Brier .88 → .79). **Decision: freeze `ts1b` as `typical-small` (Release 1 at 1.7B)**, with the preview
+kept as the reference.
+
+### JevBench disclosures (REPORT.md §3q — apply to the numbers above)
+
+This is a **public-subset run, not a ranked entry** (JevBench's own leaderboard requires ≥95%
+coverage including the 146 non-public judge items; we ran the 231 public ids only). Reported
+probabilities are the head's softmax **conditioned on non-∅** (P(∅) is dropped and the rest
+renormalized; mean p_null ≈ .03, native). Checkpoints trained at up to 1,024-token states / 64-token
+queries are run at up to 4096 / 256 tokens (out-of-training-length but not truncated). Latency is
+in-process on one H100, one decision at a time, model load excluded. Cost is null (no tariff — this
+is not a hosted-API run). Option order is the harness's own label order; a reversed-order control on
+a related checkpoint swung standard accuracy ±4 points, larger than the differences between our own
+model variants, so no fine ranking among our own checkpoints is supported by these numbers alone.
+Majority/chance baselines on this split are .311 standard / .284 easy / .336 hard (n=72 standard,
+SE ≈ .058; n_eff = 36) — `ts1b`'s hard tier (.432) is within 1 SE of chance, standard (.694) is ~1 SE
+below the preview.
+
+## Latency (REPORT.md §3ab, apples-to-apples ladder, one pod, one torch build, L_s = 256)
+
+| 1.7B | single decision, K = 2 / 32 / 256 (ms) | marginal per query, M = 32, K = 2 / 32 / 256 (ms) | peak memory, K = 2 → 256 (GB) |
+|---|---|---|---|
+| `ts1b`-shape recipe | 45 / 46 / 106 | 2.7 / 3.7 / 28.0 | 6.4 → 9.3 |
+
+A single decision costs 45 ms regardless of most of K (the KV-cached state and a short suffix
+dominate); batched marginal cost is 2.7 ms/query at K = 2, rising to 28 ms/query at K = 256. Capability
+per millisecond (JevBench standard / single-decision ms) puts the 4B at the knee (.833/56 vs 1.7B's
+.750/45 vs 14B's .875/60) — this checkpoint is the cheapest point on that ladder, not the highest-
+capability one.
+
+## Known limitations
+
+- JevBench standard is ~1 SE below the preview (.694 vs .750, n_eff = 36) — a real trade, not noise
+  in the other direction: `ts1b` buys held-out noul/score/flip and both external floors (PagerDuty,
+  jevlogs) at that cost (§3ae).
+- Mind2Web stays below its constant-prediction floor (.357 vs floor .427) — general web-agent
+  candidate sets remain outside what this training distribution covers.
+- Level-7 composition (temporal / unit / expected-value / trade-off reasoning, `wh_level7`) sits at
+  ~.50 — chance — same as every other checkpoint on the ladder (§3ad); nothing trained so far moves
+  it, because the generator never produces this composition.
+- Hard tier is .432 with Brier .79 — within 1 SE of the .336 chance baseline; probability quality on
+  soft/ordinal gold is still the clearest open defect at this size (§3aa, §3ae).
+- Held-out rubric styles regress −4 vs the preview (.859 vs .901).
+- Very large candidate sets (K in the hundreds to thousands) still need the energy → top-r → native
+  path, not this native head directly (unchanged from the preview).
+
+## Licensing of training data
+
+- Backbone (`Qwen/Qwen3-1.7B-Base`): Apache-2.0. This release's weights are a LoRA adapter plus
+  small Choice/Score/Noul heads over that backbone; the combined artifact is released under
+  Apache-2.0.
+- `stanfordnlp/snli`, `nyu-mll/multi_nli`, `facebook/anli`, `google/boolq`, `cais/mmlu`,
+  `allenai/ai2_arc`, `allenai/openbookqa`, `tau/commonsense_qa`, `allenai/sciq`, `allenai/qasc`,
+  `jeggers/logiqa2_formatted`, `deepmind/aqua_rat`, `openlifescienceai/medmcqa`: per-source licenses
+  as triaged in PLAN6.md's dataset-survey table (standard open NLI/QA/knowledge-MCQ benchmark
+  licenses, permissive for research and redistribution of derived model weights).
+- `cua-ai/cua-s1-forms`: MIT. `dwidlee/systemone-lite-general`: MIT. `MagaBitmex/jev-4b-distill-data`:
+  Apache-2.0.
+- New in this release (PLAN7 Track D, `data_wh` / `data_u`): `data_wh` is a programmatic rule-engine
+  corpus generated in-repo (`scripts/decisionmix_v2.py`), no external license constraints.
+  `Zhengping/UNLI`'s validation split (trained on) is MIT. **`metaeval/ambient`** (trained on, its
+  ambiguous rows) and **`metaeval/chaos-mnli-ambiguity`** (eval-only, never trained on) **do not
+  declare a license on their HF cards** — flagged here, not asserted; treat both as unlicensed for
+  redistribution/derivative-rights purposes even though `ambient` contributed rows to training.
+- **Evaluation-only, never trained on:** JevBench (`fstandhartinger/jevbench`, public subset),
+  `LocalLLaMA/typed-decisions`, `reachjalil/jev-luna-pagerduty-trigger` (PagerDuty), `reachjalil/
+  jev-tree-choice-cap` (tree-choice), `reachjalil/jevlogs-log-triage-benchmark` (jevlogs — Loghub
+  research license, caveated, block-level labels, never used in commercial claims),
+  `AndeyTait/JevForge-Mind2Web` (Mind2Web), and `metaeval/chaos-mnli-ambiguity` (ChaosNLI, MNLI-only
+  mirror).
+- JevBench numbers in this card are a **public-subset run** against `fstandhartinger/jevbench` v1.2.1
+  (72 standard / 48 easy / 111 hard public ids) — not a submitted or ranked leaderboard entry.
+
+## How to run
+
+Inference currently requires this project's code
+(`pcdm_jev.decider.PCDMDecider`, `native.py`), which is **not yet public — code release to follow.**
+The snippet below documents the intended call shape, not something runnable from this repo alone:
+
+```python
+from pcdm_jev.decider import PCDMDecider
+
+decider = PCDMDecider("runs/ts1b", mode="native")
+question = {"instructions": "...", "type": "choice", "criteria": {"a": "option a", "b": "option b"}}
+probs, runtime = decider.decide(state="...", question=question, labels=["a", "b"])
+# probs: dict label -> probability; runtime["p_null"] is the abstention mass
+```
+
+(`PCDMDecider.__init__(run_dir=None, device="auto", mode="energy", energy_run=None, max_state=4096,
+max_query=256, backbone=None, tap_layer=0, shots=0)` — pass `mode="native"` and the checkpoint dir.)
+
+Held-out workflow / external eval reproduction (post-hoc, full state length; also requires the
+project code):
+
+```bash
+uv run --no-sync python scripts/eval_wf.py --run runs/ts1b --mode native \
+    --files data_wf/eval/*.jsonl data_wf_hf/eval/*.jsonl data_wh/eval/*.jsonl data_u/eval/*.jsonl \
+    --limit 0 --out runs/ts1b/eval_wf_full.json
+```
+
+JevBench: `scripts/jevbench_run.py` against the checkpoint in `mode="native"` (see `pcdm_jev/` for the
+harness adapter and REPORT.md §3q for the exact protocol and disclosures reproduced above).
+
+## Files in this repository
+
+- `best.pt` — this checkpoint (`ts1b`).
+- `results.json` — training args + full eval suite (train-time pass).
+- `eval_wf_full.json` — post-hoc held-out workflow / external eval (full state length, `scripts/eval_wf.py`).
+- `probe_results.json` — Δ_q probe results.
+- `jevbench_summary.json` — JevBench public-subset run summary.
+- `MANIFEST.json` — checksum + provenance record for this release.
+
+## Public release
+
+- https://huggingface.co/OzLabs/typical-small
+- Parent release: https://huggingface.co/OzLabs/typical-small-preview

best.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f62d6e083ea80a08716d901f8409510b6a97cfe00bc65db25a37691ed555c46e
+size 33945265

eval_wf_full.json

@@ -0,0 +1,3292 @@
+{
+ "run": "runs/ts1b",
+ "mode": "native",
+ "max_state": 4096,
+ "limit": 0,
+ "eval": {
+  "u_chaosnli": {
+   "n": 1594,
+   "n_total": 1594,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 0.9986934642201571,
+    "brier": 0.15597729602040825,
+    "jsd": 0.06276268334293099,
+    "acc": 0.5470514429109159,
+    "acc_k": 0.5470514429109159,
+    "ece": 0.05734760788184035,
+    "sel_acc80": 0.5626959247648903,
+    "coverage": 1.0,
+    "false_abstain": 0.0,
+    "null_recall": "n/a"
+   },
+   "by_family": {
+    "": {
+     "nll": 0.9986934642201571,
+     "brier": 0.15597729602040825,
+     "jsd": 0.06276268334293099,
+     "acc": 0.5470514429109159,
+     "acc_k": 0.5470514429109159,
+     "ece": 0.05734760788184035,
+     "sel_acc80": 0.5626959247648903,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 1594
+    }
+   },
+   "by_qtype": {},
+   "by_label_class": {
+    "answer": {
+     "nll": 0.9986934642201571,
+     "brier": 0.15597729602040825,
+     "jsd": 0.06276268334293099,
+     "acc": 0.5470514429109159,
+     "acc_k": 0.5470514429109159,
+     "ece": 0.05734760788184035,
+     "sel_acc80": 0.5626959247648903,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 1594
+    }
+   },
+   "baselines": {
+    "majority_class": 0,
+    "majority_acc": 0.46361355081555833
+   }
+  },
+  "u_real_heldout": {
+   "n": 562,
+   "n_total": 562,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 0.6608096977843964,
+    "brier": 0.13465855536230573,
+    "jsd": 0.05983032012095985,
+    "acc": 0.6494661921708185,
+    "acc_k": 0.6494661921708185,
+    "ece": 0.061910581334205725,
+    "sel_acc80": 0.6911111111111111,
+    "coverage": 1.0,
+    "false_abstain": 0.0,
+    "null_recall": "n/a"
+   },
+   "by_family": {
+    "": {
+     "nll": 0.6608096977843964,
+     "brier": 0.13465855536230573,
+     "jsd": 0.05983032012095985,
+     "acc": 0.6494661921708185,
+     "acc_k": 0.6494661921708185,
+     "ece": 0.061910581334205725,
+     "sel_acc80": 0.6911111111111111,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 562
+    }
+   },
+   "by_qtype": {},
+   "by_label_class": {
+    "answer": {
+     "nll": 0.6608096977843964,
+     "brier": 0.13465855536230573,
+     "jsd": 0.05983032012095985,
+     "acc": 0.6494661921708185,
+     "acc_k": 0.6494661921708185,
+     "ece": 0.061910581334205725,
+     "sel_acc80": 0.6911111111111111,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 562
+    }
+   },
+   "baselines": {
+    "majority_class": 0,
+    "majority_acc": 0.5071174377224199
+   }
+  },
+  "u_synthetic_heldout": {
+   "n": 475,
+   "n_total": 475,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 0.6666821294215001,
+    "brier": 0.031944849397233334,
+    "jsd": 0.01431239917904498,
+    "acc": 0.9157894736842105,
+    "acc_k": 0.9157894736842105,
+    "ece": 0.16440801394613166,
+    "sel_acc80": 0.968421052631579,
+    "coverage": 1.0,
+    "false_abstain": 0.0,
+    "null_recall": "n/a"
+   },
+   "by_family": {
+    "": {
+     "nll": 0.6666821294215001,
+     "brier": 0.031944849397233334,
+     "jsd": 0.01431239917904498,
+     "acc": 0.9157894736842105,
+     "acc_k": 0.9157894736842105,
+     "ece": 0.16440801394613166,
+     "sel_acc80": 0.968421052631579,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 475
+    }
+   },
+   "by_qtype": {},
+   "by_label_class": {
+    "answer": {
+     "nll": 0.6666821294215001,
+     "brier": 0.031944849397233334,
+     "jsd": 0.01431239917904498,
+     "acc": 0.9157894736842105,
+     "acc_k": 0.9157894736842105,
+     "ece": 0.16440801394613166,
+     "sel_acc80": 0.968421052631579,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 475
+    }
+   },
+   "baselines": {
+    "majority_class": 1,
+    "majority_acc": 0.5747368421052632
+   }
+  },
+  "wf_heldout_choice": {
+   "n": 1500,
+   "n_total": 1500,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 0.02220144267887235,
+    "brier": 0.009909953890114582,
+    "jsd": 0.0069410462861019965,
+    "acc": 0.994,
+    "acc_k": 0.9950460014154282,
+    "ece": 0.010105259974797547,
+    "sel_acc80": 1.0,
+    "conf_wrong": 0.0,
+    "coverage": 0.9433333333333334,
+    "false_abstain": 0.0,
+    "null_recall": 0.9770114942528736,
+    "auroc_null": 1.0,
+    "auroc_conf": 0.999487517387803
+   },
+   "by_family": {
+    "tool_select": {
+     "nll": 0.02220144267887235,
+     "brier": 0.009909953890114582,
+     "jsd": 0.0069410462861019965,
+     "acc": 0.994,
+     "acc_k": 0.9950460014154282,
+     "ece": 0.010105259974797547,
+     "sel_acc80": 1.0,
+     "conf_wrong": 0.0,
+     "coverage": 0.9433333333333334,
+     "false_abstain": 0.0,
+     "null_recall": 0.9770114942528736,
+     "auroc_null": 1.0,
+     "auroc_conf": 0.999487517387803,
+     "n": 1500
+    }
+   },
+   "by_qtype": {
+    "choice": {
+     "nll": 0.02220144267887235,
+     "brier": 0.009909953890114582,
+     "jsd": 0.0069410462861019965,
+     "acc": 0.994,
+     "acc_k": 0.9950460014154282,
+     "ece": 0.010105259974797547,
+     "sel_acc80": 1.0,
+     "conf_wrong": 0.0,
+     "coverage": 0.9433333333333334,
+     "false_abstain": 0.0,
+     "null_recall": 0.9770114942528736,
+     "auroc_null": 1.0,
+     "auroc_conf": 0.999487517387803,
+     "n": 1500
+    }
+   },
+   "by_label_class": {
+    "answer": {
+     "nll": 0.020595754550765025,
+     "brier": 0.008593300755277252,
+     "jsd": 0.006623562877006804,
+     "acc": 0.9950460014154282,
+     "acc_k": 0.9950460014154282,
+     "ece": 0.011593746025344638,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 1413
+    },
+    "null": {
+     "nll": 0.04828003262158096,
+     "brier": 0.03129421687316223,
+     "jsd": 0.012097414757958358,
+     "acc": 0.9770114942528736,
+     "ece": 0.018105437015664985,
+     "sel_acc80": 1.0,
+     "conf_wrong": 0.0,
+     "coverage": 0.02298850574712641,
+     "false_abstain": "n/a",
+     "null_recall": 0.9770114942528736,
+     "n": 87
+    }
+   },
+   "baselines": {
+    "majority_class": 2,
+    "majority_acc": 0.21333333333333335
+   }
+  },
+  "wf_heldout_noul": {
+   "n": 1500,
+   "n_total": 1500,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 0.6509143929801903,
+    "brier": 0.4111122025864996,
+    "jsd": 0.16130160222798604,
+    "acc": 0.7146666666666667,
+    "acc_k": 0.7146666666666667,
+    "ece": 0.11673003478844962,
+    "sel_acc80": 0.7566666666666667,
+    "coverage": 1.0,
+    "false_abstain": 0.0,
+    "null_recall": "n/a"
+   },
+   "by_family": {
+    "eligibility": {
+     "nll": 0.6509143929801903,
+     "brier": 0.4111122025864996,
+     "jsd": 0.16130160222798604,
+     "acc": 0.7146666666666667,
+     "acc_k": 0.7146666666666667,
+     "ece": 0.11673003478844962,
+     "sel_acc80": 0.7566666666666667,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 1500
+    }
+   },
+   "by_qtype": {
+    "noul": {
+     "nll": 0.6509143929801903,
+     "brier": 0.4111122025864996,
+     "jsd": 0.16130160222798604,
+     "acc": 0.7146666666666667,
+     "acc_k": 0.7146666666666667,
+     "ece": 0.11673003478844962,
+     "sel_acc80": 0.7566666666666667,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 1500
+    }
+   },
+   "by_label_class": {
+    "answer": {
+     "nll": 0.6509143929801903,
+     "brier": 0.4111122025864996,
+     "jsd": 0.16130160222798604,
+     "acc": 0.7146666666666667,
+     "acc_k": 0.7146666666666667,
+     "ece": 0.11673003478844962,
+     "sel_acc80": 0.7566666666666667,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 1500
+    }
+   },
+   "baselines": {
+    "majority_class": 0,
+    "majority_acc": 0.5833333333333334,
+    "always_positive_acc": 0.4166666666666667
+   }
+  },
+  "wf_heldout_score": {
+   "n": 1502,
+   "n_total": 1502,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 1.007726984380891,
+    "brier": 0.5732102322665574,
+    "jsd": 0.2761964760754948,
+    "acc": 0.5199733688415447,
+    "acc_k": 0.5206391478029294,
+    "ece": 0.04527540038651061,
+    "sel_acc80": 0.5740432612312812,
+    "coverage": 0.9993342210386151,
+    "false_abstain": 0.0006657789613848203,
+    "null_recall": "n/a"
+   },
+   "by_family": {
+    "urgency": {
+     "nll": 1.007726984380891,
+     "brier": 0.5732102322665574,
+     "jsd": 0.2761964760754948,
+     "acc": 0.5199733688415447,
+     "acc_k": 0.5206391478029294,
+     "ece": 0.04527540038651061,
+     "sel_acc80": 0.5740432612312812,
+     "coverage": 0.9993342210386151,
+     "false_abstain": 0.0006657789613848203,
+     "null_recall": "n/a",
+     "n": 1502
+    }
+   },
+   "by_qtype": {
+    "score": {
+     "nll": 1.007726984380891,
+     "brier": 0.5732102322665574,
+     "jsd": 0.2761964760754948,
+     "acc": 0.5199733688415447,
+     "acc_k": 0.5206391478029294,
+     "ece": 0.04527540038651061,
+     "sel_acc80": 0.5740432612312812,
+     "coverage": 0.9993342210386151,
+     "false_abstain": 0.0006657789613848203,
+     "null_recall": "n/a",
+     "n": 1502
+    }
+   },
+   "by_label_class": {
+    "answer": {
+     "nll": 1.007726984380891,
+     "brier": 0.5732102322665574,
+     "jsd": 0.2761964760754948,
+     "acc": 0.5199733688415447,
+     "acc_k": 0.5206391478029294,
+     "ece": 0.04527540038651061,
+     "sel_acc80": 0.5740432612312812,
+     "coverage": 0.9993342210386151,
+     "false_abstain": 0.0006657789613848203,
+     "null_recall": "n/a",
+     "n": 1502
+    }
+   },
+   "baselines": {
+    "majority_class": 2,
+    "majority_acc": 0.3548601864181092
+   }
+  },
+  "wf_heldout_style": {
+   "n": 1627,
+   "n_total": 1627,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 0.3835512182973631,
+    "brier": 0.21472157101975364,
+    "jsd": 0.1094748689100086,
+    "acc": 0.8586355255070682,
+    "acc_k": 0.8564294631710362,
+    "ece": 0.0744189031664843,
+    "sel_acc80": 0.9362519201228878,
+    "conf_wrong": 0.0,
+    "coverage": 0.9846342962507683,
+    "false_abstain": 0.0,
+    "null_recall": 1.0,
+    "auroc_null": 1.0,
+    "auroc_conf": 1.0
+   },
+   "by_family": {
+    "relevance": {
+     "nll": 0.649669278943508,
+     "brier": 0.37928978253549256,
+     "jsd": 0.20030994327991516,
+     "acc": 0.7733333333333333,
+     "acc_k": 0.7733333333333333,
+     "ece": 0.21542641083399458,
+     "sel_acc80": 0.8416666666666667,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 150
+    },
+    "support": {
+     "nll": 0.19559202627065286,
+     "brier": 0.1084407534057103,
+     "jsd": 0.05433300069657259,
+     "acc": 0.9328859060402684,
+     "acc_k": 0.9328859060402684,
+     "ece": 0.05307854301977477,
+     "sel_acc80": 0.9666666666666667,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 149
+    },
+    "adequacy": {
+     "nll": 0.31774727529596786,
+     "brier": 0.18597336232659226,
+     "jsd": 0.07811762183724544,
+     "acc": 0.88,
+     "acc_k": 0.88,
+     "ece": 0.06820252060890195,
+     "sel_acc80": 0.9416666666666667,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 150
+    },
+    "action_select": {
+     "nll": 0.33021187045608336,
+     "brier": 0.1823344595149311,
+     "jsd": 0.0826433025603459,
+     "acc": 0.8733333333333333,
+     "acc_k": 0.8633093525179856,
+     "ece": 0.04021375119686121,
+     "sel_acc80": 0.9416666666666667,
+     "conf_wrong": 0.0,
+     "coverage": 0.9266666666666666,
+     "false_abstain": 0.0,
+     "null_recall": 1.0,
+     "auroc_null": 1.0,
+     "auroc_conf": 1.0,
+     "n": 150
+    },
+    "completeness": {
+     "nll": 0.6991468263927095,
+     "brier": 0.39821441371968347,
+     "jsd": 0.21188899644050688,
+     "acc": 0.7449664429530202,
+     "acc_k": 0.7449664429530202,
+     "ece": 0.18803769630073702,
+     "sel_acc80": 0.8083333333333333,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 149
+    },
+    "quality": {
+     "nll": 0.8470839023804404,
+     "brier": 0.4714291525682993,
+     "jsd": 0.24669086204119084,
+     "acc": 0.6030534351145038,
+     "acc_k": 0.6030534351145038,
+     "ece": 0.14125171849745824,
+     "sel_acc80": 0.6952380952380952,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 131
+    },
+    "enum_extract": {
+     "nll": 0.04643218567658875,
+     "brier": 0.01990835242030093,
+     "jsd": 0.01334930443633335,
+     "acc": 0.9933333333333333,
+     "acc_k": 0.993103448275862,
+     "ece": 0.01974968155225116,
+     "sel_acc80": 1.0,
+     "conf_wrong": 0.0,
+     "coverage": 0.9666666666666667,
+     "false_abstain": 0.0,
+     "null_recall": 1.0,
+     "auroc_null": 1.0,
+     "auroc_conf": 1.0,
+     "n": 150
+    },
+    "categorical": {
+     "nll": 0.2844007555346479,
+     "brier": 0.15958706208754614,
+     "jsd": 0.08558952069896848,
+     "acc": 0.8976377952755905,
+     "acc_k": 0.8951612903225806,
+     "ece": 0.07876657641778778,
+     "sel_acc80": 0.9705882352941176,
+     "conf_wrong": 0.0,
+     "coverage": 0.9763779527559056,
+     "false_abstain": 0.0,
+     "null_recall": 1.0,
+     "auroc_null": 1.0,
+     "auroc_conf": 1.0,
+     "n": 127
+    },
+    "routing": {
+     "nll": 0.2783524443014919,
+     "brier": 0.14330485597109158,
+     "jsd": 0.06294130823518836,
+     "acc": 0.8939393939393939,
+     "acc_k": 0.8888888888888888,
+     "ece": 0.08102671508536198,
+     "sel_acc80": 0.9716981132075472,
+     "conf_wrong": 0.0,
+     "coverage": 0.9545454545454546,
+     "false_abstain": 0.0,
+     "null_recall": 1.0,
+     "auroc_null": 1.0,
+     "auroc_conf": 1.0,
+     "n": 132
+    },
+    "severity": {
+     "nll": 0.5977246344600028,
+     "brier": 0.3145210200360813,
+     "jsd": 0.1824036601057107,
+     "acc": 0.8734177215189873,
+     "acc_k": 0.8734177215189873,
+     "ece": 0.2759175228921673,
+     "sel_acc80": 0.9375,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 79
+    },
+    "policy_permit": {
+     "nll": 0.15034539574002462,
+     "brier": 0.07961559213419059,
+     "jsd": 0.04591845815543799,
+     "acc": 0.9463087248322147,
+     "acc_k": 0.9463087248322147,
+     "ece": 0.05530904523478257,
+     "sel_acc80": 0.9833333333333333,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 149
+    },
+    "fact": {
+     "nll": 0.3212750600631328,
+     "brier": 0.189942179795167,
+     "jsd": 0.08593823576001701,
+     "acc": 0.8828828828828829,
+     "acc_k": 0.8828828828828829,
+     "ece": 0.08032987729923147,
+     "sel_acc80": 0.9213483146067416,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 111
+    }
+   },
+   "by_qtype": {
+    "score": {
+     "nll": 0.70689883361143,
+     "brier": 0.39849075558714014,
+     "jsd": 0.21285725744080541,
+     "acc": 0.7367387033398821,
+     "acc_k": 0.7367387033398821,
+     "ece": 0.19415499583208723,
+     "sel_acc80": 0.821078431372549,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 509
+    },
+    "noul": {
+     "nll": 0.24126708200535577,
+     "brier": 0.1377459423822855,
+     "jsd": 0.06474821970285918,
+     "acc": 0.9123434704830053,
+     "acc_k": 0.9123434704830053,
+     "ece": 0.03644280175616786,
+     "sel_acc80": 0.9553571428571429,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 559
+    },
+    "choice": {
+     "nll": 0.2314097084447192,
+     "brier": 0.12436533034630996,
+     "jsd": 0.06006621263920528,
+     "acc": 0.9159212880143113,
+     "acc_k": 0.9119850187265918,
+     "ece": 0.027254086807503464,
+     "sel_acc80": 0.9776785714285714,
+     "conf_wrong": 0.0,
+     "coverage": 0.9552772808586762,
+     "false_abstain": 0.0,
+     "null_recall": 1.0,
+     "auroc_null": 1.0,
+     "auroc_conf": 1.0,
+     "n": 559
+    }
+   },
+   "by_label_class": {
+    "answer": {
+     "nll": 0.3895064652358634,
+     "brier": 0.21807185170591808,
+     "jsd": 0.1111728077141086,
+     "acc": 0.8564294631710362,
+     "acc_k": 0.8564294631710362,
+     "ece": 0.07561033562328037,
+     "sel_acc80": 0.9321372854914196,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 1602
+    },
+    "null": {
+     "nll": 0.0019389944782613345,
+     "brier": 3.5584650339346533e-05,
+     "jsd": 0.0006709503432798227,
+     "acc": 1.0,
+     "ece": 0.0019280886650085982,
+     "sel_acc80": 1.0,
+     "conf_wrong": 0.0,
+     "coverage": 0.0,
+     "false_abstain": "n/a",
+     "null_recall": 1.0,
+     "n": 25
+    }
+   },
+   "baselines": {
+    "majority_class": 1,
+    "majority_acc": 0.36447449293177625
+   }
+  },
+  "wf_rubric_flip": {
+   "n": 2062,
+   "n_total": 2062,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 0.559445285535001,
+    "brier": 0.3272260198998571,
+    "jsd": 0.14724147310065305,
+    "acc": 0.7516973811833172,
+    "acc_k": 0.7516973811833172,
+    "ece": 0.0431027509250419,
+    "sel_acc80": 0.8351515151515152,
+    "coverage": 1.0,
+    "false_abstain": 0.0,
+    "null_recall": "n/a"
+   },
+   "by_family": {
+    "urgency": {
+     "nll": 1.00970911915973,
+     "brier": 0.5721670188295802,
+     "jsd": 0.276942053921465,
+     "acc": 0.5357142857142857,
+     "acc_k": 0.5357142857142857,
+     "ece": 0.04788954292024886,
+     "sel_acc80": 0.6,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 700
+    },
+    "eligibility": {
+     "nll": 0.669296858496893,
+     "brier": 0.4143906038771635,
+     "jsd": 0.1633805475660813,
+     "acc": 0.7149390243902439,
+     "acc_k": 0.7149390243902439,
+     "ece": 0.10991631802625769,
+     "sel_acc80": 0.7561904761904762,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 656
+    },
+    "tool_select": {
+     "nll": 0.010936340208780766,
+     "brier": 0.0033752177186684626,
+     "jsd": 0.0036470829818297796,
+     "acc": 1.0,
+     "acc_k": 1.0,
+     "ece": 0.009704953043724294,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 706
+    }
+   },
+   "by_qtype": {
+    "score": {
+     "nll": 1.00970911915973,
+     "brier": 0.5721670188295802,
+     "jsd": 0.276942053921465,
+     "acc": 0.5357142857142857,
+     "acc_k": 0.5357142857142857,
+     "ece": 0.04788954292024886,
+     "sel_acc80": 0.6,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 700
+    },
+    "noul": {
+     "nll": 0.669296858496893,
+     "brier": 0.4143906038771635,
+     "jsd": 0.1633805475660813,
+     "acc": 0.7149390243902439,
+     "acc_k": 0.7149390243902439,
+     "ece": 0.10991631802625769,
+     "sel_acc80": 0.7561904761904762,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 656
+    },
+    "choice": {
+     "nll": 0.010936340208780766,
+     "brier": 0.0033752177186684626,
+     "jsd": 0.0036470829818297796,
+     "acc": 1.0,
+     "acc_k": 1.0,
+     "ece": 0.009704953043724294,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 706
+    }
+   },
+   "by_label_class": {
+    "answer": {
+     "nll": 0.559445285535001,
+     "brier": 0.3272260198998571,
+     "jsd": 0.14724147310065305,
+     "acc": 0.7516973811833172,
+     "acc_k": 0.7516973811833172,
+     "ece": 0.0431027509250419,
+     "sel_acc80": 0.8351515151515152,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 2062
+    }
+   },
+   "baselines": {
+    "majority_class": 0,
+    "majority_acc": 0.32104752667313285
+   },
+   "flip": {
+    "n_pairs": 1031,
+    "flip_rate": 0.7487875848690592,
+    "both_correct": 0.5809893307468477,
+    "acc": 0.7516973811833172
+   },
+   "flip_by_family": {
+    "eligibility": {
+     "n_pairs": 328,
+     "flip_rate": 0.5091463414634146,
+     "both_correct": 0.4695121951219512,
+     "acc": 0.7149390243902439
+    },
+    "tool_select": {
+     "n_pairs": 353,
+     "flip_rate": 1.0,
+     "both_correct": 1.0,
+     "acc": 1.0
+    },
+    "urgency": {
+     "n_pairs": 350,
+     "flip_rate": 0.72,
+     "both_correct": 0.26285714285714284,
+     "acc": 0.5357142857142857
+    }
+   }
+  },
+  "wf_rubric_shuffled": {
+   "n": 3775,
+   "n_total": 3775,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 2.698697412970932,
+    "brier": 0.938708681237318,
+    "jsd": 0.3677264317143067,
+    "acc": 0.3748344370860927,
+    "acc_k": 0.37509933774834436,
+    "ece": 0.36605128677080795,
+    "sel_acc80": 0.39006622516556294,
+    "coverage": 0.9997350993377483,
+    "false_abstain": 0.00026490066225165563,
+    "null_recall": "n/a"
+   },
+   "by_family": {
+    "tool_select": {
+     "nll": 6.7661947710653765,
+     "brier": 1.5175450979035083,
+     "jsd": 0.5510631818062146,
+     "acc": 0.17016574585635358,
+     "acc_k": 0.17016574585635358,
+     "ece": 0.7154725696335841,
+     "sel_acc80": 0.143646408839779,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 905
+    },
+    "eligibility": {
+     "nll": 1.2318526208591865,
+     "brier": 0.7084869470501419,
+     "jsd": 0.2544289537720909,
+     "acc": 0.5304835318850736,
+     "acc_k": 0.5304835318850736,
+     "ece": 0.31344534871289176,
+     "sel_acc80": 0.5385288966725044,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 1427
+    },
+    "urgency": {
+     "nll": 1.5982832821795177,
+     "brier": 0.80335140999837,
+     "jsd": 0.3647851581109563,
+     "acc": 0.3492723492723493,
+     "acc_k": 0.34996534996534995,
+     "ece": 0.2045487121873156,
+     "sel_acc80": 0.3575757575757576,
+     "coverage": 0.9993069993069993,
+     "false_abstain": 0.000693000693000693,
+     "null_recall": "n/a",
+     "n": 1443
+    }
+   },
+   "by_qtype": {
+    "choice": {
+     "nll": 6.7661947710653765,
+     "brier": 1.5175450979035083,
+     "jsd": 0.5510631818062146,
+     "acc": 0.17016574585635358,
+     "acc_k": 0.17016574585635358,
+     "ece": 0.7154725696335841,
+     "sel_acc80": 0.143646408839779,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 905
+    },
+    "noul": {
+     "nll": 1.2318526208591865,
+     "brier": 0.7084869470501419,
+     "jsd": 0.2544289537720909,
+     "acc": 0.5304835318850736,
+     "acc_k": 0.5304835318850736,
+     "ece": 0.31344534871289176,
+     "sel_acc80": 0.5385288966725044,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 1427
+    },
+    "score": {
+     "nll": 1.5982832821795177,
+     "brier": 0.80335140999837,
+     "jsd": 0.3647851581109563,
+     "acc": 0.3492723492723493,
+     "acc_k": 0.34996534996534995,
+     "ece": 0.2045487121873156,
+     "sel_acc80": 0.3575757575757576,
+     "coverage": 0.9993069993069993,
+     "false_abstain": 0.000693000693000693,
+     "null_recall": "n/a",
+     "n": 1443
+    }
+   },
+   "by_label_class": {
+    "answer": {
+     "nll": 2.698697412970932,
+     "brier": 0.938708681237318,
+     "jsd": 0.3677264317143067,
+     "acc": 0.3748344370860927,
+     "acc_k": 0.37509933774834436,
+     "ece": 0.36605128677080795,
+     "sel_acc80": 0.39006622516556294,
+     "coverage": 0.9997350993377483,
+     "false_abstain": 0.00026490066225165563,
+     "null_recall": "n/a",
+     "n": 3775
+    }
+   },
+   "baselines": {
+    "majority_class": 0,
+    "majority_acc": 0.36
+   }
+  },
+  "cua_s1_forms_demo": {
+   "n": 196,
+   "n_total": 196,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 0.0009298304114405211,
+    "brier": 1.7812480371703415e-05,
+    "jsd": 0.00032176846444351295,
+    "acc": 1.0,
+    "acc_k": 1.0,
+    "ece": 0.000924828101177555,
+    "sel_acc80": 1.0,
+    "coverage": 1.0,
+    "false_abstain": 0.0,
+    "null_recall": "n/a"
+   },
+   "by_family": {
+    "cua_s1_forms": {
+     "nll": 0.0009298304114405211,
+     "brier": 1.7812480371703415e-05,
+     "jsd": 0.00032176846444351295,
+     "acc": 1.0,
+     "acc_k": 1.0,
+     "ece": 0.000924828101177555,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 196
+    }
+   },
+   "by_qtype": {
+    "choice": {
+     "nll": 0.0009298304114405211,
+     "brier": 1.7812480371703415e-05,
+     "jsd": 0.00032176846444351295,
+     "acc": 1.0,
+     "acc_k": 1.0,
+     "ece": 0.000924828101177555,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 196
+    }
+   },
+   "by_label_class": {
+    "answer": {
+     "nll": 0.0009298304114405211,
+     "brier": 1.7812480371703415e-05,
+     "jsd": 0.00032176846444351295,
+     "acc": 1.0,
+     "acc_k": 1.0,
+     "ece": 0.000924828101177555,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 196
+    }
+   },
+   "baselines": {
+    "majority_class": 18,
+    "majority_acc": 0.27040816326530615
+   }
+  },
+  "cua_s1_forms_test": {
+   "n": 24370,
+   "n_total": 24370,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 0.031139153702024643,
+    "brier": 0.015968466328798284,
+    "jsd": 0.006616339278338574,
+    "acc": 0.9897004513746409,
+    "acc_k": 0.9897004513746409,
+    "ece": 0.0040463498602845566,
+    "sel_acc80": 1.0,
+    "coverage": 1.0,
+    "false_abstain": 0.0,
+    "null_recall": "n/a"
+   },
+   "by_family": {
+    "cua_s1_forms": {
+     "nll": 0.031139153702024643,
+     "brier": 0.015968466328798284,
+     "jsd": 0.006616339278338574,
+     "acc": 0.9897004513746409,
+     "acc_k": 0.9897004513746409,
+     "ece": 0.0040463498602845566,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 24370
+    }
+   },
+   "by_qtype": {
+    "choice": {
+     "nll": 0.031139153702024643,
+     "brier": 0.015968466328798284,
+     "jsd": 0.006616339278338574,
+     "acc": 0.9897004513746409,
+     "acc_k": 0.9897004513746409,
+     "ece": 0.0040463498602845566,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 24370
+    }
+   },
+   "by_label_class": {
+    "answer": {
+     "nll": 0.031139153702024643,
+     "brier": 0.015968466328798284,
+     "jsd": 0.006616339278338574,
+     "acc": 0.9897004513746409,
+     "acc_k": 0.9897004513746409,
+     "ece": 0.0040463498602845566,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 24370
+    }
+   },
+   "baselines": {
+    "majority_class": 16,
+    "majority_acc": 0.07033237587197373
+   }
+  },
+  "jev4b_adversarial": {
+   "n": 600,
+   "n_total": 600,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 0.27162374405442014,
+    "brier": 0.13008828636077177,
+    "jsd": 0.06711174695091494,
+    "acc": 0.935,
+    "acc_k": 0.935,
+    "ece": 0.06803944091002141,
+    "sel_acc80": 0.9645833333333333,
+    "coverage": 1.0,
+    "false_abstain": 0.0,
+    "null_recall": "n/a"
+   },
+   "by_family": {
+    "jev4b_distill": {
+     "nll": 0.27162374405442014,
+     "brier": 0.13008828636077177,
+     "jsd": 0.06711174695091494,
+     "acc": 0.935,
+     "acc_k": 0.935,
+     "ece": 0.06803944091002141,
+     "sel_acc80": 0.9645833333333333,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 600
+    }
+   },
+   "by_qtype": {
+    "choice": {
+     "nll": 0.2368204577035528,
+     "brier": 0.09787175781239878,
+     "jsd": 0.045669968907413504,
+     "acc": 0.9555555555555556,
+     "acc_k": 0.9555555555555556,
+     "ece": 0.05371427064140639,
+     "sel_acc80": 0.96875,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 360
+    },
+    "score": {
+     "nll": 0.5946715993673614,
+     "brier": 0.3368385842307513,
+     "jsd": 0.1811635778371567,
+     "acc": 0.8166666666666667,
+     "acc_k": 0.8166666666666667,
+     "ece": 0.19491817976037662,
+     "sel_acc80": 0.8854166666666666,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 120
+    },
+    "noul": {
+     "nll": 0.05298574779408091,
+     "brier": 0.019987574135911266,
+     "jsd": 0.01738525019517747,
+     "acc": 0.9916666666666667,
+     "acc_k": 0.9916666666666667,
+     "ece": 0.033880017697811134,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 120
+    }
+   },
+   "by_label_class": {
+    "answer": {
+     "nll": 0.27162374405442014,
+     "brier": 0.13008828636077177,
+     "jsd": 0.06711174695091494,
+     "acc": 0.935,
+     "acc_k": 0.935,
+     "ece": 0.06803944091002141,
+     "sel_acc80": 0.9645833333333333,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 600
+    }
+   },
+   "baselines": {
+    "majority_class": 0,
+    "majority_acc": 0.42833333333333334
+   }
+  },
+  "jevlogs_triage": {
+   "n": 5080,
+   "n_total": 5080,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 0.6941952121477953,
+    "brier": 0.4479620663111127,
+    "jsd": 0.1644331696428951,
+    "acc": 0.7104330708661417,
+    "acc_k": 0.7104330708661417,
+    "ece": 0.20131792304084056,
+    "sel_acc80": 0.8058562992125984,
+    "coverage": 1.0,
+    "false_abstain": 0.0,
+    "null_recall": "n/a"
+   },
+   "by_family": {
+    "log_hdfs": {
+     "nll": 0.8571451724027355,
+     "brier": 0.5156481325333623,
+     "jsd": 0.18672513520111952,
+     "acc": 0.7,
+     "acc_k": 0.7,
+     "ece": 0.21816099820137022,
+     "sel_acc80": 0.716,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 2500
+    },
+    "log_bgl": {
+     "nll": 0.519794702051961,
+     "brier": 0.3722576889026077,
+     "jsd": 0.1394862307236892,
+     "acc": 0.7256,
+     "acc_k": 0.7256,
+     "ece": 0.22598801379203798,
+     "sel_acc80": 0.793,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 2500
+    },
+    "log_adversarial": {
+     "nll": 1.052024894675745,
+     "brier": 0.6985342908815958,
+     "jsd": 0.24740108717356427,
+     "acc": 0.5625,
+     "acc_k": 0.5625,
+     "ece": 0.3473062872886658,
+     "sel_acc80": 0.53125,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 80
+    }
+   },
+   "by_qtype": {
+    "noul": {
+     "nll": 0.6941952121477953,
+     "brier": 0.4479620663111127,
+     "jsd": 0.1644331696428951,
+     "acc": 0.7104330708661417,
+     "acc_k": 0.7104330708661417,
+     "ece": 0.20131792304084056,
+     "sel_acc80": 0.8058562992125984,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 5080
+    }
+   },
+   "by_label_class": {
+    "answer": {
+     "nll": 0.6941952121477953,
+     "brier": 0.4479620663111127,
+     "jsd": 0.1644331696428951,
+     "acc": 0.7104330708661417,
+     "acc_k": 0.7104330708661417,
+     "ece": 0.20131792304084056,
+     "sel_acc80": 0.8058562992125984,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 5080
+    }
+   },
+   "baselines": {
+    "majority_class": 0,
+    "majority_acc": 0.6968503937007874,
+    "always_positive_acc": 0.3031496062992126
+   }
+  },
+  "mind2web_choice": {
+   "n": 1600,
+   "n_total": 1600,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 1.5136479569939831,
+    "brier": 0.7428329076657673,
+    "jsd": 0.34671858092870367,
+    "acc": 0.3575,
+    "acc_k": 0.36,
+    "ece": 0.20080654334276918,
+    "sel_acc80": 0.38828125,
+    "coverage": 0.995625,
+    "false_abstain": 0.004375,
+    "null_recall": "n/a"
+   },
+   "by_family": {
+    "mind2web": {
+     "nll": 1.5136479569939831,
+     "brier": 0.7428329076657673,
+     "jsd": 0.34671858092870367,
+     "acc": 0.3575,
+     "acc_k": 0.36,
+     "ece": 0.20080654334276918,
+     "sel_acc80": 0.38828125,
+     "coverage": 0.995625,
+     "false_abstain": 0.004375,
+     "null_recall": "n/a",
+     "n": 1600
+    }
+   },
+   "by_qtype": {
+    "choice": {
+     "nll": 2.0931226888545518,
+     "brier": 0.7784709109053702,
+     "jsd": 0.42772685578683883,
+     "acc": 0.37,
+     "acc_k": 0.375,
+     "ece": 0.06646796382963659,
+     "sel_acc80": 0.4109375,
+     "coverage": 0.99125,
+     "false_abstain": 0.00875,
+     "null_recall": "n/a",
+     "n": 800
+    },
+    "noul": {
+     "nll": 0.9341732251334145,
+     "brier": 0.7071949044261643,
+     "jsd": 0.2657103060705684,
+     "acc": 0.345,
+     "acc_k": 0.345,
+     "ece": 0.33514512285590164,
+     "sel_acc80": 0.3234375,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 800
+    }
+   },
+   "by_label_class": {
+    "answer": {
+     "nll": 1.5136479569939831,
+     "brier": 0.7428329076657673,
+     "jsd": 0.34671858092870367,
+     "acc": 0.3575,
+     "acc_k": 0.36,
+     "ece": 0.20080654334276918,
+     "sel_acc80": 0.38828125,
+     "coverage": 0.995625,
+     "false_abstain": 0.004375,
+     "null_recall": "n/a",
+     "n": 1600
+    }
+   },
+   "baselines": {
+    "majority_class": 0,
+    "majority_acc": 0.426875
+   }
+  },
+  "pagerduty_trigger": {
+   "n": 6000,
+   "n_total": 6000,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 0.45057378800764875,
+    "brier": 0.2645510776299787,
+    "jsd": 0.12898428946204524,
+    "acc": 0.8171666666666667,
+    "acc_k": 0.8171666666666667,
+    "ece": 0.08175959949195385,
+    "sel_acc80": 0.8864583333333333,
+    "coverage": 1.0,
+    "false_abstain": 0.0,
+    "null_recall": "n/a"
+   },
+   "by_family": {
+    "replica_lag_data_risk": {
+     "nll": 0.6898488965338108,
+     "brier": 0.4642770455099506,
+     "jsd": 0.21330152188587068,
+     "acc": 0.5789473684210527,
+     "acc_k": 0.5789473684210527,
+     "ece": 0.1760190186793344,
+     "sel_acc80": 0.5869565217391305,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 114
+    },
+    "tls_hours_left_serving": {
+     "nll": 0.9503486029544026,
+     "brier": 0.705578046531571,
+     "jsd": 0.24360104904451949,
+     "acc": 0.5,
+     "acc_k": 0.5,
+     "ece": 0.4599647521972656,
+     "sel_acc80": 0.620253164556962,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 98
+    },
+    "disk_full_writes_failing": {
+     "nll": 0.9617892499408918,
+     "brier": 0.7153070589979267,
+     "jsd": 0.24323647291422343,
+     "acc": 0.5,
+     "acc_k": 0.5,
+     "ece": 0.4525816380977631,
+     "sel_acc80": 0.625,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 110
+    },
+    "revoked_key_used": {
+     "nll": 1.6670805035219922,
+     "brier": 1.0476412905344457,
+     "jsd": 0.38638081928023327,
+     "acc": 0.5,
+     "acc_k": 0.5,
+     "ece": 0.7080944776535034,
+     "sel_acc80": 0.3780487804878049,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 102
+    },
+    "checksum_corruption": {
+     "nll": 0.5221658499392701,
+     "brier": 0.3021209402517869,
+     "jsd": 0.1668244358824877,
+     "acc": 1.0,
+     "acc_k": 1.0,
+     "ece": 0.4045400937398274,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 90
+    },
+    "checkout_slo_burn": {
+     "nll": 0.6211973900784526,
+     "brier": 0.47606719857859514,
+     "jsd": 0.1802507371509747,
+     "acc": 0.5,
+     "acc_k": 0.5,
+     "ece": 0.38283807039260864,
+     "sel_acc80": 0.625,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 100
+    },
+    "payments_processor_down": {
+     "nll": 0.9761891304656726,
+     "brier": 0.7124004465279202,
+     "jsd": 0.25978390316724703,
+     "acc": 0.5,
+     "acc_k": 0.5,
+     "ece": 0.3435124742223862,
+     "sel_acc80": 0.3815789473684211,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 94
+    },
+    "crash_loop_user_impact": {
+     "nll": 0.9124509739053913,
+     "brier": 0.6743519260962172,
+     "jsd": 0.2428845282176394,
+     "acc": 0.5,
+     "acc_k": 0.5,
+     "ece": 0.3444297431657712,
+     "sel_acc80": 0.6233766233766234,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 96
+    },
+    "credential_stuffing": {
+     "nll": 0.9519934639477086,
+     "brier": 0.6977005836612166,
+     "jsd": 0.2499338675952664,
+     "acc": 0.5,
+     "acc_k": 0.5,
+     "ece": 0.33941899364193284,
+     "sel_acc80": 0.6233766233766234,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 96
+    },
+    "queue_backup_checkout": {
+     "nll": 0.9013995609276422,
+     "brier": 0.6699464517470315,
+     "jsd": 0.24000688623489375,
+     "acc": 0.5,
+     "acc_k": 0.5,
+     "ece": 0.47097641229629517,
+     "sel_acc80": 0.625,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 100
+    },
+    "queue_slow_growth": {
+     "nll": 1.3707216103698314,
+     "brier": 0.658629060210223,
+     "jsd": 0.301554799426009,
+     "acc": 0.5,
+     "acc_k": 0.5,
+     "ece": 0.2951798737049103,
+     "sel_acc80": 0.6222222222222222,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 56
+    },
+    "cert_30d": {
+     "nll": 0.1727142536714221,
+     "brier": 0.059320063795585645,
+     "jsd": 0.05738444318715433,
+     "acc": 1.0,
+     "acc_k": 1.0,
+     "ece": 0.1501917839050293,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 58
+    },
+    "disk_80pct_48h": {
+     "nll": 0.9509672822978512,
+     "brier": 0.5419380620634318,
+     "jsd": 0.24201387380805145,
+     "acc": 0.5,
+     "acc_k": 0.5,
+     "ece": 0.28837817162275314,
+     "sel_acc80": 0.6153846153846154,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 64
+    },
+    "cpu_high_no_impact": {
+     "nll": 0.8010921147001497,
+     "brier": 0.46771347158423976,
+     "jsd": 0.21917217540903014,
+     "acc": 0.5,
+     "acc_k": 0.5,
+     "ece": 0.28503292202949526,
+     "sel_acc80": 0.625,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 50
+    },
+    "single_pod_oom": {
+     "nll": 1.314429901386395,
+     "brier": 0.7101412733644461,
+     "jsd": 0.28754181367618376,
+     "acc": 0.5,
+     "acc_k": 0.5,
+     "ece": 0.38835878555591286,
+     "sel_acc80": 0.6190476190476191,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 52
+    },
+    "deprecation": {
+     "nll": 1.162277843746267,
+     "brier": 0.7164361718413621,
+     "jsd": 0.2679758230576417,
+     "acc": 0.5,
+     "acc_k": 0.5,
+     "ece": 0.4070015847682953,
+     "sel_acc80": 0.625,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 60
+    },
+    "canary_within_budget": {
+     "nll": 0.997527949570193,
+     "brier": 0.6500690785390699,
+     "jsd": 0.24742592715354586,
+     "acc": 0.5,
+     "acc_k": 0.5,
+     "ece": 0.39401811361312866,
+     "sel_acc80": 0.6153846153846154,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 48
+    },
+    "replica_lag_seconds": {
+     "nll": 0.9901761862463576,
+     "brier": 0.5515541514998862,
+     "jsd": 0.2506408336180253,
+     "acc": 0.5,
+     "acc_k": 0.5,
+     "ece": 0.2882567197084427,
+     "sel_acc80": 0.625,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 50
+    },
+    "tls_expires_7d": {
+     "nll": 1.1567732954045722,
+     "brier": 0.7294461207505658,
+     "jsd": 0.2665059357272688,
+     "acc": 0.5,
+     "acc_k": 0.5,
+     "ece": 0.41363009810447693,
+     "sel_acc80": 0.6190476190476191,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 26
+    },
+    "slow_slo_burn": {
+     "nll": 0.6522867336147589,
+     "brier": 0.5114718599657583,
+     "jsd": 0.18362021573543594,
+     "acc": 0.5,
+     "acc_k": 0.5,
+     "ece": 0.36724624037742615,
+     "sel_acc80": 0.6206896551724138,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 36
+    },
+    "injection_noise": {
+     "nll": 0.5037106019010942,
+     "brier": 0.37437867309218403,
+     "jsd": 0.151060977826625,
+     "acc": 0.5,
+     "acc_k": 0.5,
+     "ece": 0.32909492561894077,
+     "sel_acc80": 0.625,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 310
+    },
+    "health_ok": {
+     "nll": 0.18152995080281234,
+     "brier": 0.047734969981545416,
+     "jsd": 0.061126679924072204,
+     "acc": 1.0,
+     "acc_k": 1.0,
+     "ece": 0.16462426345332776,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 306
+    },
+    "kafka_caught_up": {
+     "nll": 0.5039227087644147,
+     "brier": 0.3316987967829466,
+     "jsd": 0.151070396937696,
+     "acc": 0.6654676258992805,
+     "acc_k": 0.6654676258992805,
+     "ece": 0.21027986587380335,
+     "sel_acc80": 0.672645739910314,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 278
+    },
+    "invalid_coupon_error": {
+     "nll": 0.25784577121054475,
+     "brier": 0.1019467076656268,
+     "jsd": 0.0844192917311665,
+     "acc": 1.0,
+     "acc_k": 1.0,
+     "ece": 0.21481417610038142,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 322
+    },
+    "stack_trace_handled": {
+     "nll": 0.30060535025925805,
+     "brier": 0.13121770374213282,
+     "jsd": 0.09815941120194528,
+     "acc": 1.0,
+     "acc_k": 1.0,
+     "ece": 0.24862703573272885,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 332
+    },
+    "retry_then_ok": {
+     "nll": 0.2279076528120555,
+     "brier": 0.08093679155813602,
+     "jsd": 0.0753719006565446,
+     "acc": 1.0,
+     "acc_k": 1.0,
+     "ece": 0.1956012144055165,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 284
+    },
+    "feature_flag": {
+     "nll": 0.13855516551397323,
+     "brier": 0.02981594098678688,
+     "jsd": 0.04691808660224193,
+     "acc": 1.0,
+     "acc_k": 1.0,
+     "ece": 0.12797431425845368,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 282
+    },
+    "cache_hit": {
+     "nll": 0.16067786504109507,
+     "brier": 0.05175275250111869,
+     "jsd": 0.05360337986685591,
+     "acc": 1.0,
+     "acc_k": 1.0,
+     "ece": 0.14153279624618847,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 286
+    },
+    "metrics_scrape": {
+     "nll": 0.5509275726093205,
+     "brier": 0.37247962744689045,
+     "jsd": 0.1658098683017907,
+     "acc": 0.5,
+     "acc_k": 0.5,
+     "ece": 0.3461012159724062,
+     "sel_acc80": 0.623574144486692,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 328
+    },
+    "cron_ok": {
+     "nll": 0.25100095482382306,
+     "brier": 0.11987021871014797,
+     "jsd": 0.08118105449703346,
+     "acc": 1.0,
+     "acc_k": 1.0,
+     "ece": 0.20225984857928364,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 310
+    },
+    "rate_limit_bot": {
+     "nll": 0.3944067554874347,
+     "brier": 0.18381001382027407,
+     "jsd": 0.12796267513555565,
+     "acc": 1.0,
+     "acc_k": 1.0,
+     "ece": 0.31977638020180166,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 256
+    },
+    "http_404": {
+     "nll": 0.11865384131473723,
+     "brier": 0.023577944218449064,
+     "jsd": 0.04024055657795258,
+     "acc": 1.0,
+     "acc_k": 1.0,
+     "ece": 0.11015801009891918,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 318
+    },
+    "deploy_success": {
+     "nll": 0.0665660530378659,
+     "brier": 0.00769735271439915,
+     "jsd": 0.022807886905452414,
+     "acc": 1.0,
+     "acc_k": 1.0,
+     "ece": 0.06395390971762233,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 244
+    },
+    "user_not_found": {
+     "nll": 0.4304388910037287,
+     "brier": 0.24248037606659678,
+     "jsd": 0.13315740549304775,
+     "acc": 0.9387755102040817,
+     "acc_k": 0.9387755102040817,
+     "ece": 0.24541204691338703,
+     "sel_acc80": 0.9915254237288136,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 294
+    },
+    "request_ok": {
+     "nll": 0.24205577862251662,
+     "brier": 0.09208789387581466,
+     "jsd": 0.08013562084859333,
+     "acc": 1.0,
+     "acc_k": 1.0,
+     "ece": 0.20847708940505985,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 350
+    }
+   },
+   "by_qtype": {
+    "choice": {
+     "nll": 0.5648469629367534,
+     "brier": 0.3173471457305583,
+     "jsd": 0.16576450460353345,
+     "acc": 0.7796666666666666,
+     "acc_k": 0.7796666666666666,
+     "ece": 0.08924249602357544,
+     "sel_acc80": 0.8258333333333333,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 3000
+    },
+    "noul": {
+     "nll": 0.336300613078544,
+     "brier": 0.21175500952939919,
+     "jsd": 0.092204074320557,
+     "acc": 0.8546666666666667,
+     "acc_k": 0.8546666666666667,
+     "ece": 0.1531165913939476,
+     "sel_acc80": 0.9783333333333334,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 3000
+    }
+   },
+   "by_label_class": {
+    "answer": {
+     "nll": 0.45057378800764875,
+     "brier": 0.2645510776299787,
+     "jsd": 0.12898428946204524,
+     "acc": 0.8171666666666667,
+     "acc_k": 0.8171666666666667,
+     "ece": 0.08175959949195385,
+     "sel_acc80": 0.8864583333333333,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 6000
+    }
+   },
+   "baselines": {
+    "majority_class": 0,
+    "majority_acc": 0.7916666666666666
+   }
+  },
+  "systemone_lite_hard": {
+   "n": 5400,
+   "n_total": 5400,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 0.47733330766959625,
+    "brier": 0.293311500019393,
+    "jsd": 0.1385319090950344,
+    "acc": 0.7833333333333333,
+    "acc_k": 0.7837037037037037,
+    "ece": 0.03959470673843665,
+    "sel_acc80": 0.8453703703703703,
+    "coverage": 0.9996296296296296,
+    "false_abstain": 0.00037037037037037035,
+    "null_recall": "n/a"
+   },
+   "by_family": {
+    "systemone_lite": {
+     "nll": 0.47733330766959625,
+     "brier": 0.293311500019393,
+     "jsd": 0.1385319090950344,
+     "acc": 0.7833333333333333,
+     "acc_k": 0.7837037037037037,
+     "ece": 0.03959470673843665,
+     "sel_acc80": 0.8453703703703703,
+     "coverage": 0.9996296296296296,
+     "false_abstain": 0.00037037037037037035,
+     "null_recall": "n/a",
+     "n": 5400
+    }
+   },
+   "by_qtype": {
+    "choice": {
+     "nll": 0.24729003141422154,
+     "brier": 0.1407608173373836,
+     "jsd": 0.060118104806292254,
+     "acc": 0.8994444444444445,
+     "acc_k": 0.9005555555555556,
+     "ece": 0.02308347546392016,
+     "sel_acc80": 0.96875,
+     "coverage": 0.9988888888888889,
+     "false_abstain": 0.0011111111111111111,
+     "null_recall": "n/a",
+     "n": 1800
+    },
+    "noul": {
+     "nll": 0.4667849208576605,
+     "brier": 0.3108541755307743,
+     "jsd": 0.1404008352752232,
+     "acc": 0.7583333333333333,
+     "acc_k": 0.7583333333333333,
+     "ece": 0.026432335111829974,
+     "sel_acc80": 0.8138888888888889,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 1800
+    },
+    "score": {
+     "nll": 0.7179249707369069,
+     "brier": 0.42831950719002104,
+     "jsd": 0.21507678720358783,
+     "acc": 0.6922222222222222,
+     "acc_k": 0.6922222222222222,
+     "ece": 0.12505331529511346,
+     "sel_acc80": 0.7430555555555556,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 1800
+    }
+   },
+   "by_label_class": {
+    "answer": {
+     "nll": 0.47733330766959625,
+     "brier": 0.293311500019393,
+     "jsd": 0.1385319090950344,
+     "acc": 0.7833333333333333,
+     "acc_k": 0.7837037037037037,
+     "ece": 0.03959470673843665,
+     "sel_acc80": 0.8453703703703703,
+     "coverage": 0.9996296296296296,
+     "false_abstain": 0.00037037037037037035,
+     "null_recall": "n/a",
+     "n": 5400
+    }
+   },
+   "baselines": {
+    "majority_class": 1,
+    "majority_acc": 0.44166666666666665
+   }
+  },
+  "tree_choice_cap": {
+   "n": 720,
+   "n_total": 720,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 1.9904785368215312,
+    "brier": 0.776743891433536,
+    "jsd": 0.35171423237051735,
+    "acc": 0.4736111111111111,
+    "acc_k": 0.7069444444444445,
+    "ece": 0.2539656482378228,
+    "sel_acc80": 0.5295138888888888,
+    "coverage": 0.6055555555555556,
+    "false_abstain": 0.39444444444444443,
+    "null_recall": "n/a"
+   },
+   "by_family": {
+    "incident_catalog": {
+     "nll": 1.9904785368215312,
+     "brier": 0.776743891433536,
+     "jsd": 0.35171423237051735,
+     "acc": 0.4736111111111111,
+     "acc_k": 0.7069444444444445,
+     "ece": 0.2539656482378228,
+     "sel_acc80": 0.5295138888888888,
+     "coverage": 0.6055555555555556,
+     "false_abstain": 0.39444444444444443,
+     "null_recall": "n/a",
+     "n": 720
+    }
+   },
+   "by_qtype": {
+    "choice": {
+     "nll": 1.9904785368215312,
+     "brier": 0.776743891433536,
+     "jsd": 0.35171423237051735,
+     "acc": 0.4736111111111111,
+     "acc_k": 0.7069444444444445,
+     "ece": 0.2539656482378228,
+     "sel_acc80": 0.5295138888888888,
+     "coverage": 0.6055555555555556,
+     "false_abstain": 0.39444444444444443,
+     "null_recall": "n/a",
+     "n": 720
+    }
+   },
+   "by_label_class": {
+    "answer": {
+     "nll": 1.9904785368215312,
+     "brier": 0.776743891433536,
+     "jsd": 0.35171423237051735,
+     "acc": 0.4736111111111111,
+     "acc_k": 0.7069444444444445,
+     "ece": 0.2539656482378228,
+     "sel_acc80": 0.5295138888888888,
+     "coverage": 0.6055555555555556,
+     "false_abstain": 0.39444444444444443,
+     "null_recall": "n/a",
+     "n": 720
+    }
+   },
+   "baselines": {
+    "majority_class": 3,
+    "majority_acc": 0.11388888888888889
+   }
+  },
+  "typed_decisions_test": {
+   "n": 2000,
+   "n_total": 2000,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 1.2466883275443574,
+    "brier": 0.227664179143571,
+    "jsd": 0.10287304051205891,
+    "acc": 0.5105,
+    "acc_k": 0.5105,
+    "ece": 0.08129057371616365,
+    "sel_acc80": 0.538125,
+    "coverage": 1.0,
+    "false_abstain": 0.0,
+    "null_recall": "n/a"
+   },
+   "by_family": {
+    "agent_trace_observability": {
+     "nll": 1.3732859886964037,
+     "brier": 0.24645547871103435,
+     "jsd": 0.11540308138807402,
+     "acc": 0.328,
+     "acc_k": 0.328,
+     "ece": 0.17124855417013168,
+     "sel_acc80": 0.3525,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 500
+    },
+    "customer_service": {
+     "nll": 1.3569411167874115,
+     "brier": 0.23403111645670874,
+     "jsd": 0.10776170713367438,
+     "acc": 0.588,
+     "acc_k": 0.588,
+     "ece": 0.14043137019872665,
+     "sel_acc80": 0.6125,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 500
+    },
+    "invoice_processing": {
+     "nll": 1.1945578194974424,
+     "brier": 0.32410382732931886,
+     "jsd": 0.14279780746854975,
+     "acc": 0.52,
+     "acc_k": 0.52,
+     "ece": 0.18908735823631287,
+     "sel_acc80": 0.525,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 500
+    },
+    "security_incidents": {
+     "nll": 1.0619683851961725,
+     "brier": 0.10606629407722211,
+     "jsd": 0.04552956605793744,
+     "acc": 0.606,
+     "acc_k": 0.606,
+     "ece": 0.12156618690490725,
+     "sel_acc80": 0.615,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 500
+    }
+   },
+   "by_qtype": {
+    "choice": {
+     "nll": 1.6514770692249574,
+     "brier": 0.3027703648728876,
+     "jsd": 0.13991168303102539,
+     "acc": 0.4716666666666667,
+     "acc_k": 0.4716666666666667,
+     "ece": 0.20745025321841243,
+     "sel_acc80": 0.45416666666666666,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 600
+    },
+    "noul": {
+     "nll": 0.7184778931000595,
+     "brier": 0.1950290479092196,
+     "jsd": 0.0656428058548165,
+     "acc": 0.6616666666666666,
+     "acc_k": 0.6616666666666666,
+     "ece": 0.09978362967570625,
+     "sel_acc80": 0.6979166666666666,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 600
+    },
+    "score": {
+     "nll": 1.3392545971171312,
+     "brier": 0.19581088827234716,
+     "jsd": 0.10301673461576585,
+     "acc": 0.42625,
+     "acc_k": 0.42625,
+     "ece": 0.04972978692501786,
+     "sel_acc80": 0.43125,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 800
+    }
+   },
+   "by_label_class": {
+    "answer": {
+     "nll": 1.2466883275443574,
+     "brier": 0.227664179143571,
+     "jsd": 0.10287304051205891,
+     "acc": 0.5105,
+     "acc_k": 0.5105,
+     "ece": 0.08129057371616365,
+     "sel_acc80": 0.538125,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 2000
+    }
+   },
+   "baselines": {
+    "majority_class": 1,
+    "majority_acc": 0.2905
+   }
+  },
+  "typed_decisions_train": {
+   "n": 6000,
+   "n_total": 6000,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 1.2344590285798438,
+    "brier": 0.23647349698787676,
+    "jsd": 0.10473287517672107,
+    "acc": 0.49433333333333335,
+    "acc_k": 0.4945,
+    "ece": 0.09849566911160947,
+    "sel_acc80": 0.515625,
+    "coverage": 0.9995,
+    "false_abstain": 0.0005,
+    "null_recall": "n/a"
+   },
+   "by_family": {
+    "agent_trace_observability": {
+     "nll": 1.3073141496670826,
+     "brier": 0.21427903025345926,
+     "jsd": 0.09931973903182195,
+     "acc": 0.35933333333333334,
+     "acc_k": 0.35933333333333334,
+     "ece": 0.14304505523045857,
+     "sel_acc80": 0.37666666666666665,
+     "coverage": 0.9993333333333333,
+     "false_abstain": 0.0006666666666666666,
+     "null_recall": "n/a",
+     "n": 1500
+    },
+    "customer_service": {
+     "nll": 1.3559591669399014,
+     "brier": 0.25086907210466536,
+     "jsd": 0.11360149087906343,
+     "acc": 0.564,
+     "acc_k": 0.5646666666666667,
+     "ece": 0.14432381959756216,
+     "sel_acc80": 0.5766666666666667,
+     "coverage": 0.9986666666666667,
+     "false_abstain": 0.0013333333333333333,
+     "null_recall": "n/a",
+     "n": 1500
+    },
+    "invoice_processing": {
+     "nll": 1.1943725988328637,
+     "brier": 0.33419169017658246,
+     "jsd": 0.14631432442839343,
+     "acc": 0.5133333333333333,
+     "acc_k": 0.5133333333333333,
+     "ece": 0.16757222191492716,
+     "sel_acc80": 0.5283333333333333,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 1500
+    },
+    "security_incidents": {
+     "nll": 1.080190198879528,
+     "brier": 0.1465541954168001,
+     "jsd": 0.05969594636760551,
+     "acc": 0.5406666666666666,
+     "acc_k": 0.5406666666666666,
+     "ece": 0.08952901397148767,
+     "sel_acc80": 0.5558333333333333,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 1500
+    }
+   },
+   "by_qtype": {
+    "choice": {
+     "nll": 1.617389415082502,
+     "brier": 0.3021717268631206,
+     "jsd": 0.1377373031403558,
+     "acc": 0.4777777777777778,
+     "acc_k": 0.4777777777777778,
+     "ece": 0.2272327849931187,
+     "sel_acc80": 0.4625,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 1800
+    },
+    "noul": {
+     "nll": 0.7349958482352102,
+     "brier": 0.21425469853856416,
+     "jsd": 0.07184645864360849,
+     "acc": 0.6216666666666667,
+     "acc_k": 0.6216666666666667,
+     "ece": 0.09909297843774163,
+     "sel_acc80": 0.6618055555555555,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 1800
+    },
+    "score": {
+     "nll": 1.321858623961326,
+     "brier": 0.20386392341842843,
+     "jsd": 0.10464436660382949,
+     "acc": 0.41125,
+     "acc_k": 0.4116666666666667,
+     "ece": 0.05669290943692127,
+     "sel_acc80": 0.4119791666666667,
+     "coverage": 0.99875,
+     "false_abstain": 0.00125,
+     "null_recall": "n/a",
+     "n": 2400
+    }
+   },
+   "by_label_class": {
+    "answer": {
+     "nll": 1.2344590285798438,
+     "brier": 0.23647349698787676,
+     "jsd": 0.10473287517672107,
+     "acc": 0.49433333333333335,
+     "acc_k": 0.4945,
+     "ece": 0.09849566911160947,
+     "sel_acc80": 0.515625,
+     "coverage": 0.9995,
+     "false_abstain": 0.0005,
+     "null_recall": "n/a",
+     "n": 6000
+    }
+   },
+   "baselines": {
+    "majority_class": 1,
+    "majority_acc": 0.3105
+   }
+  },
+  "wh_heldout_family": {
+   "n": 422,
+   "n_total": 422,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 0.5457370583615766,
+    "brier": 0.251593830017657,
+    "jsd": 0.10264112252818929,
+    "acc": 0.8364928909952607,
+    "acc_k": 0.8647214854111406,
+    "ece": 0.07603205083670775,
+    "sel_acc80": 0.9112426035502958,
+    "conf_wrong": 0.0,
+    "coverage": 0.8767772511848342,
+    "false_abstain": 0.034482758620689655,
+    "null_recall": 0.8666666666666667,
+    "auroc_null": 0.9863837312113173,
+    "auroc_conf": 0.9761862658414382
+   },
+   "by_family": {
+    "procurement_request": {
+     "nll": 0.7397600080900862,
+     "brier": 0.32257210974221484,
+     "jsd": 0.11885485091987728,
+     "acc": 0.8028169014084507,
+     "acc_k": 0.828125,
+     "ece": 0.16723090347269892,
+     "sel_acc80": 0.8421052631578947,
+     "conf_wrong": 0.0,
+     "coverage": 0.9014084507042254,
+     "false_abstain": 0.0234375,
+     "null_recall": 0.7857142857142857,
+     "auroc_null": 0.9827008928571429,
+     "auroc_conf": 0.9754464285714286,
+     "n": 142
+    },
+    "research_ethics_review": {
+     "nll": 0.3750520999482153,
+     "brier": 0.19797667690325785,
+     "jsd": 0.0921636871204694,
+     "acc": 0.8549618320610687,
+     "acc_k": 0.8717948717948718,
+     "ece": 0.048931630285641604,
+     "sel_acc80": 0.9523809523809523,
+     "conf_wrong": 0.0,
+     "coverage": 0.8625954198473282,
+     "false_abstain": 0.03418803418803419,
+     "null_recall": 1.0,
+     "auroc_null": 0.9981684981684983,
+     "auroc_conf": 0.996947496947497,
+     "n": 131
+    },
+    "subscription_cancellation": {
+     "nll": 0.5108945797756836,
+     "brier": 0.2310900134881206,
+     "jsd": 0.09640081787578399,
+     "acc": 0.8523489932885906,
+     "acc_k": 0.8939393939393939,
+     "ece": 0.06457566934944,
+     "sel_acc80": 0.9083333333333333,
+     "conf_wrong": 0.0,
+     "coverage": 0.8657718120805369,
+     "false_abstain": 0.045454545454545456,
+     "null_recall": 0.8235294117647058,
+     "auroc_null": 0.9803921568627451,
+     "auroc_conf": 0.9670231729055259,
+     "n": 149
+    }
+   },
+   "by_qtype": {},
+   "by_label_class": {
+    "answer": {
+     "nll": 0.5753806494950396,
+     "brier": 0.26114281214808394,
+     "jsd": 0.10430497921088148,
+     "acc": 0.8328912466843501,
+     "acc_k": 0.8647214854111406,
+     "ece": 0.08095392489939215,
+     "sel_acc80": 0.9039735099337748,
+     "coverage": 0.9655172413793104,
+     "false_abstain": 0.034482758620689655,
+     "null_recall": "n/a",
+     "n": 377
+    },
+    "null": {
+     "nll": 0.29738963931011947,
+     "brier": 0.17159457972496925,
+     "jsd": 0.08870170098652361,
+     "acc": 0.8666666666666667,
+     "ece": 0.07395036551687453,
+     "sel_acc80": 0.9444444444444444,
+     "conf_wrong": 0.0,
+     "coverage": 0.1333333333333333,
+     "false_abstain": "n/a",
+     "null_recall": 0.8666666666666667,
+     "n": 45
+    }
+   },
+   "baselines": {
+    "majority_class": 0,
+    "majority_acc": 0.38388625592417064
+   }
+  },
+  "wh_heldout_grammar": {
+   "n": 816,
+   "n_total": 816,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 0.22640730243759352,
+    "brier": 0.1324652194635447,
+    "jsd": 0.06170653080121566,
+    "acc": 0.8982843137254902,
+    "acc_k": 0.9350828729281768,
+    "ece": 0.02578789211225277,
+    "sel_acc80": 0.9816232771822359,
+    "conf_wrong": 0.0,
+    "coverage": 0.8651960784313726,
+    "false_abstain": 0.04281767955801105,
+    "null_recall": 0.8586956521739131,
+    "auroc_null": 0.9852720393946673,
+    "auroc_conf": 0.9800024021138602
+   },
+   "by_family": {
+    "visa_extension": {
+     "nll": 0.3076947705638739,
+     "brier": 0.171614329447727,
+     "jsd": 0.08494472212807742,
+     "acc": 0.8794326241134752,
+     "acc_k": 0.9217391304347826,
+     "ece": 0.08212125449315875,
+     "sel_acc80": 0.9646017699115044,
+     "conf_wrong": 0.0,
+     "coverage": 0.7872340425531915,
+     "false_abstain": 0.06086956521739131,
+     "null_recall": 0.8846153846153846,
+     "auroc_null": 0.977257525083612,
+     "auroc_conf": 0.9782608695652173,
+     "n": 141
+    },
+    "loan_approval": {
+     "nll": 0.0009530163086001497,
+     "brier": 1.5960591192212375e-05,
+     "jsd": 0.0003298857214853352,
+     "acc": 1.0,
+     "acc_k": 1.0,
+     "ece": 0.0009488833211633851,
+     "sel_acc80": 1.0,
+     "conf_wrong": 0.0,
+     "coverage": 0.9708029197080292,
+     "false_abstain": 0.0,
+     "null_recall": 1.0,
+     "auroc_null": 1.0,
+     "auroc_conf": 1.0,
+     "n": 137
+    },
+    "content_moderation": {
+     "nll": 0.6091939270687743,
+     "brier": 0.35397883806023056,
+     "jsd": 0.16220081909887452,
+     "acc": 0.7109375,
+     "acc_k": 0.8144329896907216,
+     "ece": 0.07935253647156058,
+     "sel_acc80": 0.7961165048543689,
+     "conf_wrong": 0.0,
+     "coverage": 0.6796875,
+     "false_abstain": 0.17525773195876287,
+     "null_recall": 0.7741935483870968,
+     "auroc_null": 0.9165281010974393,
+     "auroc_conf": 0.8979048885932823,
+     "n": 128
+    },
+    "warranty_claim": {
+     "nll": 0.024673367126092534,
+     "brier": 0.015632475701378046,
+     "jsd": 0.007645685972270448,
+     "acc": 0.9788732394366197,
+     "acc_k": 0.9782608695652174,
+     "ece": 0.01797439724626676,
+     "sel_acc80": 1.0,
+     "conf_wrong": 0.0,
+     "coverage": 0.971830985915493,
+     "false_abstain": 0.0,
+     "null_recall": 1.0,
+     "auroc_null": 1.0,
+     "auroc_conf": 0.9999999999999999,
+     "n": 142
+    },
+    "shift_swap": {
+     "nll": 0.24592318700294757,
+     "brier": 0.1523044670340519,
+     "jsd": 0.06410268473755716,
+     "acc": 0.9,
+     "acc_k": 0.9327731092436975,
+     "ece": 0.07901871639948625,
+     "sel_acc80": 0.9711538461538461,
+     "conf_wrong": 0.0,
+     "coverage": 0.9153846153846154,
+     "false_abstain": 0.025210084033613446,
+     "null_recall": 0.7272727272727273,
+     "auroc_null": 0.988540870893812,
+     "auroc_conf": 0.9549274255156609,
+     "n": 130
+    },
+    "vendor_onboarding": {
+     "nll": 0.2013215782478294,
+     "brier": 0.12002267024201238,
+     "jsd": 0.05905353406731011,
+     "acc": 0.9057971014492754,
+     "acc_k": 0.9262295081967213,
+     "ece": 0.04172966316126395,
+     "sel_acc80": 0.9819819819819819,
+     "conf_wrong": 0.0,
+     "coverage": 0.855072463768116,
+     "false_abstain": 0.03278688524590164,
+     "null_recall": 1.0,
+     "auroc_null": 1.0,
+     "auroc_conf": 0.9994877049180327,
+     "n": 138
+    }
+   },
+   "by_qtype": {},
+   "by_label_class": {
+    "null": {
+     "nll": 0.31717554619495864,
+     "brier": 0.17949680628344114,
+     "jsd": 0.09801949919342964,
+     "acc": 0.8586956521739131,
+     "ece": 0.0898055029304131,
+     "sel_acc80": 0.972972972972973,
+     "conf_wrong": 0.0,
+     "coverage": 0.14130434782608692,
+     "false_abstain": "n/a",
+     "null_recall": 0.8586956521739131,
+     "n": 92
+    },
+    "answer": {
+     "nll": 0.21487321621428196,
+     "brier": 0.12648882997814348,
+     "jsd": 0.05709217570165255,
+     "acc": 0.9033149171270718,
+     "acc_k": 0.9350828729281768,
+     "ece": 0.03330249582206351,
+     "sel_acc80": 0.9896551724137931,
+     "coverage": 0.9571823204419889,
+     "false_abstain": 0.04281767955801105,
+     "null_recall": "n/a",
+     "n": 724
+    }
+   },
+   "baselines": {
+    "majority_class": 0,
+    "majority_acc": 0.38480392156862747
+   }
+  },
+  "wh_heldout_style": {
+   "n": 403,
+   "n_total": 403,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 0.2542938206897888,
+    "brier": 0.1392317138092667,
+    "jsd": 0.06525751858240728,
+    "acc": 0.8957816377171216,
+    "acc_k": 0.8957816377171216,
+    "ece": 0.034956357689116715,
+    "sel_acc80": 0.9845201238390093,
+    "coverage": 1.0,
+    "false_abstain": 0.0,
+    "null_recall": "n/a"
+   },
+   "by_family": {
+    "feature_flag_rollout": {
+     "nll": 0.23742768955897636,
+     "brier": 0.14074202037136146,
+     "jsd": 0.06479365092611951,
+     "acc": 0.8909090909090909,
+     "acc_k": 0.8909090909090909,
+     "ece": 0.05244000716642898,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 55
+    },
+    "shift_swap": {
+     "nll": 0.1414648972583102,
+     "brier": 0.08477401343554897,
+     "jsd": 0.04047649253786089,
+     "acc": 0.9393939393939394,
+     "acc_k": 0.9393939393939394,
+     "ece": 0.04961613859191087,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 66
+    },
+    "content_moderation": {
+     "nll": 0.48871335155887385,
+     "brier": 0.25520722330845164,
+     "jsd": 0.12672497899891835,
+     "acc": 0.813953488372093,
+     "acc_k": 0.813953488372093,
+     "ece": 0.09241127967834474,
+     "sel_acc80": 0.8857142857142857,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 43
+    },
+    "insurance_claim": {
+     "nll": 0.2962156035545307,
+     "brier": 0.15384219869919274,
+     "jsd": 0.06801950475356011,
+     "acc": 0.8529411764705882,
+     "acc_k": 0.8529411764705882,
+     "ece": 0.10579277136746572,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 34
+    },
+    "visa_extension": {
+     "nll": 0.0750532288436404,
+     "brier": 0.03625713740971259,
+     "jsd": 0.021720251237289188,
+     "acc": 0.972972972972973,
+     "acc_k": 0.972972972972973,
+     "ece": 0.04014645557145816,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 37
+    },
+    "warranty_claim": {
+     "nll": 0.35407836222932965,
+     "brier": 0.1812433311456853,
+     "jsd": 0.08922631164563992,
+     "acc": 0.864406779661017,
+     "acc_k": 0.864406779661017,
+     "ece": 0.07289651036262515,
+     "sel_acc80": 0.9583333333333334,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 59
+    },
+    "loan_approval": {
+     "nll": 0.23993441668876092,
+     "brier": 0.12858445334961865,
+     "jsd": 0.05228019078464824,
+     "acc": 0.9166666666666666,
+     "acc_k": 0.9166666666666666,
+     "ece": 0.08637669434150061,
+     "sel_acc80": 0.9487179487179487,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 48
+    },
+    "vendor_onboarding": {
+     "nll": 0.2264713789426459,
+     "brier": 0.13709860520312941,
+     "jsd": 0.06105561608466616,
+     "acc": 0.9016393442622951,
+     "acc_k": 0.9016393442622951,
+     "ece": 0.04174451700976639,
+     "sel_acc80": 0.9795918367346939,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 61
+    }
+   },
+   "by_qtype": {},
+   "by_label_class": {
+    "answer": {
+     "nll": 0.2542938206897888,
+     "brier": 0.1392317138092667,
+     "jsd": 0.06525751858240728,
+     "acc": 0.8957816377171216,
+     "acc_k": 0.8957816377171216,
+     "ece": 0.034956357689116715,
+     "sel_acc80": 0.9845201238390093,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 403
+    }
+   },
+   "baselines": {
+    "majority_class": 1,
+    "majority_acc": 0.4267990074441687
+   }
+  },
+  "wh_level7": {
+   "n": 880,
+   "n_total": 880,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 1.0933667634837307,
+    "brier": 0.6994650652421958,
+    "jsd": 0.27543897835324205,
+    "acc": 0.4954545454545455,
+    "acc_k": 0.4954545454545455,
+    "ece": 0.21106459528885108,
+    "sel_acc80": 0.5,
+    "coverage": 1.0,
+    "false_abstain": 0.0,
+    "null_recall": "n/a"
+   },
+   "by_family": {
+    "temporal": {
+     "nll": 1.3079168514120685,
+     "brier": 0.824014775523723,
+     "jsd": 0.2990870419496964,
+     "acc": 0.4409090909090909,
+     "acc_k": 0.4409090909090909,
+     "ece": 0.33636484200304206,
+     "sel_acc80": 0.4375,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 220
+    },
+    "probability": {
+     "nll": 0.9432978854839996,
+     "brier": 0.6099551660833942,
+     "jsd": 0.26288584878103655,
+     "acc": 0.5136363636363637,
+     "acc_k": 0.5136363636363637,
+     "ece": 0.11962269801985136,
+     "sel_acc80": 0.5454545454545454,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 220
+    },
+    "numeric": {
+     "nll": 1.1800030350316557,
+     "brier": 0.7649147915563862,
+     "jsd": 0.27636414067609133,
+     "acc": 0.4772727272727273,
+     "acc_k": 0.4772727272727273,
+     "ece": 0.3373937555334785,
+     "sel_acc80": 0.4602272727272727,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 220
+    },
+    "tradeoff": {
+     "nll": 0.9422492820071992,
+     "brier": 0.5989755278052799,
+     "jsd": 0.26341888200614394,
+     "acc": 0.55,
+     "acc_k": 0.55,
+     "ece": 0.1418665295297449,
+     "sel_acc80": 0.5909090909090909,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 220
+    }
+   },
+   "by_qtype": {},
+   "by_label_class": {
+    "answer": {
+     "nll": 1.0933667634837307,
+     "brier": 0.6994650652421958,
+     "jsd": 0.27543897835324205,
+     "acc": 0.4954545454545455,
+     "acc_k": 0.4954545454545455,
+     "ece": 0.21106459528885108,
+     "sel_acc80": 0.5,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 880
+    }
+   },
+   "baselines": {
+    "majority_class": 1,
+    "majority_acc": 0.4602272727272727
+   }
+  },
+  "wh_rubric_flip": {
+   "n": 2206,
+   "n_total": 2206,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 0.6494624590463446,
+    "brier": 0.3918518011011721,
+    "jsd": 0.15974617492224252,
+    "acc": 0.7180417044424298,
+    "acc_k": 0.7261847774054572,
+    "ece": 0.10136204418926378,
+    "sel_acc80": 0.7705382436260623,
+    "conf_wrong": 0.0,
+    "coverage": 0.9360834088848595,
+    "false_abstain": 0.019147917663954045,
+    "null_recall": 0.8632478632478633,
+    "auroc_null": 0.993150937143278,
+    "auroc_conf": 0.9631361670614902
+   },
+   "by_family": {
+    "loan_approval": {
+     "nll": 0.08023158175565527,
+     "brier": 0.04282249159650115,
+     "jsd": 0.017602264591185385,
+     "acc": 0.9722222222222222,
+     "acc_k": 0.9716312056737588,
+     "ece": 0.028174331204758714,
+     "sel_acc80": 1.0,
+     "conf_wrong": 0.0,
+     "coverage": 0.9791666666666666,
+     "false_abstain": 0.0,
+     "null_recall": 1.0,
+     "auroc_null": 1.0,
+     "auroc_conf": 1.0,
+     "n": 144
+    },
+    "shift_swap": {
+     "nll": 0.2465878122980097,
+     "brier": 0.15229776876027482,
+     "jsd": 0.06560878599344538,
+     "acc": 0.8975903614457831,
+     "acc_k": 0.9230769230769231,
+     "ece": 0.07610932674752656,
+     "sel_acc80": 0.9699248120300752,
+     "conf_wrong": 0.0,
+     "coverage": 0.9397590361445783,
+     "false_abstain": 0.019230769230769232,
+     "null_recall": 0.7,
+     "auroc_null": 0.9903846153846154,
+     "auroc_conf": 0.948076923076923,
+     "n": 166
+    },
+    "vendor_onboarding": {
+     "nll": 0.2647327007139645,
+     "brier": 0.16004266399217373,
+     "jsd": 0.07542805791158898,
+     "acc": 0.8782051282051282,
+     "acc_k": 0.8943661971830986,
+     "ece": 0.03141318280727436,
+     "sel_acc80": 0.96,
+     "conf_wrong": 0.0,
+     "coverage": 0.8846153846153846,
+     "false_abstain": 0.028169014084507043,
+     "null_recall": 1.0,
+     "auroc_null": 1.0,
+     "auroc_conf": 1.0,
+     "n": 156
+    },
+    "content_moderation": {
+     "nll": 0.582850606871788,
+     "brier": 0.33758376751430147,
+     "jsd": 0.15752963013387192,
+     "acc": 0.722972972972973,
+     "acc_k": 0.8083333333333333,
+     "ece": 0.0678231577212746,
+     "sel_acc80": 0.8067226890756303,
+     "conf_wrong": 0.0,
+     "coverage": 0.7567567567567568,
+     "false_abstain": 0.11666666666666667,
+     "null_recall": 0.7857142857142857,
+     "auroc_null": 0.949702380952381,
+     "auroc_conf": 0.9166666666666667,
+     "n": 148
+    },
+    "warranty_claim": {
+     "nll": 0.1483655041739517,
+     "brier": 0.07844069543217347,
+     "jsd": 0.037962196465545404,
+     "acc": 0.935064935064935,
+     "acc_k": 0.9342105263157895,
+     "ece": 0.037407628901593104,
+     "sel_acc80": 1.0,
+     "conf_wrong": 0.0,
+     "coverage": 0.987012987012987,
+     "false_abstain": 0.0,
+     "null_recall": 1.0,
+     "auroc_null": 1.0,
+     "auroc_conf": 1.0,
+     "n": 154
+    },
+    "visa_extension": {
+     "nll": 0.30290840119057044,
+     "brier": 0.1692011181936417,
+     "jsd": 0.08325356240443149,
+     "acc": 0.8835616438356164,
+     "acc_k": 0.9193548387096774,
+     "ece": 0.0858401405076458,
+     "sel_acc80": 0.9658119658119658,
+     "conf_wrong": 0.0,
+     "coverage": 0.815068493150685,
+     "false_abstain": 0.056451612903225805,
+     "null_recall": 0.9090909090909091,
+     "auroc_null": 0.9798387096774193,
+     "auroc_conf": 0.9791055718475072,
+     "n": 146
+    },
+    "procurement_request": {
+     "nll": 0.8189641494161496,
+     "brier": 0.37043124136320893,
+     "jsd": 0.1353190948728516,
+     "acc": 0.7727272727272727,
+     "acc_k": 0.797979797979798,
+     "ece": 0.19727202355861662,
+     "sel_acc80": 0.8068181818181818,
+     "conf_wrong": 0.0,
+     "coverage": 0.8909090909090909,
+     "false_abstain": 0.030303030303030304,
+     "null_recall": 0.8181818181818182,
+     "auroc_null": 0.9770431588613407,
+     "auroc_conf": 0.9669421487603305,
+     "n": 110
+    },
+    "research_ethics_review": {
+     "nll": 0.42143617293867885,
+     "brier": 0.22013918081838285,
+     "jsd": 0.10185807680534407,
+     "acc": 0.8425925925925926,
+     "acc_k": 0.8526315789473684,
+     "ece": 0.07520498804472113,
+     "sel_acc80": 0.9310344827586207,
+     "conf_wrong": 0.0,
+     "coverage": 0.8518518518518519,
+     "false_abstain": 0.031578947368421054,
+     "null_recall": 1.0,
+     "auroc_null": 1.0,
+     "auroc_conf": 1.0,
+     "n": 108
+    },
+    "subscription_cancellation": {
+     "nll": 0.5600637160483035,
+     "brier": 0.2487595991587257,
+     "jsd": 0.10345149368419938,
+     "acc": 0.847457627118644,
+     "acc_k": 0.9038461538461539,
+     "ece": 0.07515307351694268,
+     "sel_acc80": 0.8842105263157894,
+     "conf_wrong": 0.0,
+     "coverage": 0.8559322033898304,
+     "false_abstain": 0.057692307692307696,
+     "null_recall": 0.7857142857142857,
+     "auroc_null": 0.9732142857142857,
+     "auroc_conf": 0.9546703296703296,
+     "n": 118
+    },
+    "feature_flag_rollout": {
+     "nll": 0.25305771148082157,
+     "brier": 0.14540700448198826,
+     "jsd": 0.0686071941981727,
+     "acc": 0.8913043478260869,
+     "acc_k": 0.8913043478260869,
+     "ece": 0.04402980985848801,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 46
+    },
+    "insurance_claim": {
+     "nll": 0.33136397536790363,
+     "brier": 0.17315097095655338,
+     "jsd": 0.07520947753065718,
+     "acc": 0.8333333333333334,
+     "acc_k": 0.8333333333333334,
+     "ece": 0.11505284309387206,
+     "sel_acc80": 1.0,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 30
+    },
+    "temporal": {
+     "nll": 1.307820071546593,
+     "brier": 0.8239995465563967,
+     "jsd": 0.29908146201461855,
+     "acc": 0.4409090909090909,
+     "acc_k": 0.4409090909090909,
+     "ece": 0.336357594891028,
+     "sel_acc80": 0.4375,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 220
+    },
+    "numeric": {
+     "nll": 1.1808323022813243,
+     "brier": 0.764822347173263,
+     "jsd": 0.27637131819240957,
+     "acc": 0.4772727272727273,
+     "acc_k": 0.4772727272727273,
+     "ece": 0.3372702945362438,
+     "sel_acc80": 0.4602272727272727,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 220
+    },
+    "probability": {
+     "nll": 0.9432978854035944,
+     "brier": 0.6099551661446684,
+     "jsd": 0.26288584877310456,
+     "acc": 0.5136363636363637,
+     "acc_k": 0.5136363636363637,
+     "ece": 0.11962269707159565,
+     "sel_acc80": 0.5454545454545454,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 220
+    },
+    "tradeoff": {
+     "nll": 0.9422492821363462,
+     "brier": 0.5989755273230615,
+     "jsd": 0.2634188819148693,
+     "acc": 0.55,
+     "acc_k": 0.55,
+     "ece": 0.14186652952974493,
+     "sel_acc80": 0.5909090909090909,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 220
+    }
+   },
+   "by_qtype": {},
+   "by_label_class": {
+    "answer": {
+     "nll": 0.6678934681317353,
+     "brier": 0.40360123283903593,
+     "jsd": 0.16322706971212872,
+     "acc": 0.7099090473910962,
+     "acc_k": 0.7261847774054572,
+     "ece": 0.10932367604962376,
+     "sel_acc80": 0.7607655502392344,
+     "coverage": 0.9808520823360459,
+     "false_abstain": 0.019147917663954045,
+     "null_recall": "n/a",
+     "n": 2089
+    },
+    "null": {
+     "nll": 0.32038230537641876,
+     "brier": 0.18206921220888533,
+     "jsd": 0.09759583974213762,
+     "acc": 0.8632478632478633,
+     "ece": 0.10108689569000506,
+     "sel_acc80": 0.9574468085106383,
+     "conf_wrong": 0.0,
+     "coverage": 0.1367521367521367,
+     "false_abstain": "n/a",
+     "null_recall": 0.8632478632478633,
+     "n": 117
+    }
+   },
+   "baselines": {
+    "majority_class": 0,
+    "majority_acc": 0.39981867633726204
+   }
+  },
+  "wh_rubric_shuffled": {
+   "n": 2437,
+   "n_total": 2437,
+   "suffix_overflow": 0,
+   "raw": {
+    "nll": 2.8769261526719228,
+    "brier": 0.9536641216585189,
+    "jsd": 0.3499893495991487,
+    "acc": 0.4103405826836274,
+    "acc_k": 0.4380704041720991,
+    "ece": 0.4225114438809671,
+    "sel_acc80": 0.41435897435897434,
+    "conf_wrong": 0.1323529411764706,
+    "coverage": 0.9343455067706197,
+    "false_abstain": 0.04389395914819644,
+    "null_recall": 0.4338235294117647,
+    "auroc_null": 0.9573986374210701,
+    "auroc_conf": 0.7991522228187234
+   },
+   "by_family": {
+    "temporal": {
+     "nll": 0.810321112506112,
+     "brier": 0.5886056243893594,
+     "jsd": 0.21902761009641206,
+     "acc": 0.5363636363636364,
+     "acc_k": 0.5363636363636364,
+     "ece": 0.2581337186423215,
+     "sel_acc80": 0.5284090909090909,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 220
+    },
+    "shift_swap": {
+     "nll": 4.311645628513916,
+     "brier": 1.1784696578549476,
+     "jsd": 0.4179276671059122,
+     "acc": 0.3403141361256545,
+     "acc_k": 0.35,
+     "ece": 0.5781365601804244,
+     "sel_acc80": 0.33986928104575165,
+     "conf_wrong": 0.2727272727272727,
+     "coverage": 0.9424083769633508,
+     "false_abstain": 0.03888888888888889,
+     "null_recall": 0.36363636363636365,
+     "auroc_null": 0.9570707070707071,
+     "auroc_conf": 0.8712121212121213,
+     "n": 191
+    },
+    "vendor_onboarding": {
+     "nll": 3.8966259577761173,
+     "brier": 1.1348027236364968,
+     "jsd": 0.40071265182720534,
+     "acc": 0.34536082474226804,
+     "acc_k": 0.398876404494382,
+     "ece": 0.5599819146787998,
+     "sel_acc80": 0.358974358974359,
+     "conf_wrong": 0.1875,
+     "coverage": 0.9020618556701031,
+     "false_abstain": 0.07303370786516854,
+     "null_recall": 0.375,
+     "auroc_null": 0.9332865168539325,
+     "auroc_conf": 0.8381320224719101,
+     "n": 194
+    },
+    "loan_approval": {
+     "nll": 5.6109657929658034,
+     "brier": 1.2977122539746735,
+     "jsd": 0.4490296748646031,
+     "acc": 0.34269662921348315,
+     "acc_k": 0.3620689655172414,
+     "ece": 0.6546829131881843,
+     "sel_acc80": 0.35664335664335667,
+     "conf_wrong": 0.5,
+     "coverage": 0.9662921348314607,
+     "false_abstain": 0.022988505747126436,
+     "null_recall": 0.5,
+     "auroc_null": 0.9604885057471264,
+     "auroc_conf": 0.8153735632183908,
+     "n": 178
+    },
+    "probability": {
+     "nll": 1.1624105344814657,
+     "brier": 0.749513864357648,
+     "jsd": 0.30859526077870086,
+     "acc": 0.3409090909090909,
+     "acc_k": 0.3409090909090909,
+     "ece": 0.2819210297682068,
+     "sel_acc80": 0.36363636363636365,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 220
+    },
+    "feature_flag_rollout": {
+     "nll": 4.149504215160249,
+     "brier": 1.1719086356322357,
+     "jsd": 0.41258248701508426,
+     "acc": 0.3695652173913043,
+     "acc_k": 0.3695652173913043,
+     "ece": 0.5901150580333627,
+     "sel_acc80": 0.35135135135135137,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 46
+    },
+    "research_ethics_review": {
+     "nll": 3.6036260558065356,
+     "brier": 1.1577480248499434,
+     "jsd": 0.41970080669001747,
+     "acc": 0.30952380952380953,
+     "acc_k": 0.375,
+     "ece": 0.5508490079452122,
+     "sel_acc80": 0.3069306930693069,
+     "conf_wrong": 0.07142857142857142,
+     "coverage": 0.8571428571428572,
+     "false_abstain": 0.125,
+     "null_recall": 0.2857142857142857,
+     "auroc_null": 0.8801020408163265,
+     "auroc_conf": 0.7704081632653063,
+     "n": 126
+    },
+    "warranty_claim": {
+     "nll": 4.5780489648436005,
+     "brier": 1.1941381442153995,
+     "jsd": 0.4155502092598703,
+     "acc": 0.36507936507936506,
+     "acc_k": 0.372972972972973,
+     "ece": 0.599066392612205,
+     "sel_acc80": 0.3684210526315789,
+     "conf_wrong": 0.5,
+     "coverage": 0.9788359788359788,
+     "false_abstain": 0.010810810810810811,
+     "null_recall": 0.5,
+     "auroc_null": 0.9864864864864865,
+     "auroc_conf": 0.85,
+     "n": 189
+    },
+    "numeric": {
+     "nll": 0.6629021089805935,
+     "brier": 0.41855250364207175,
+     "jsd": 0.1702423570572618,
+     "acc": 0.7272727272727273,
+     "acc_k": 0.7272727272727273,
+     "ece": 0.1474715121767738,
+     "sel_acc80": 0.7670454545454546,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 220
+    },
+    "content_moderation": {
+     "nll": 2.264195490666992,
+     "brier": 0.9375361619969083,
+     "jsd": 0.3576627856018118,
+     "acc": 0.37341772151898733,
+     "acc_k": 0.4251968503937008,
+     "ece": 0.3937665090153489,
+     "sel_acc80": 0.36220472440944884,
+     "conf_wrong": 0.0,
+     "coverage": 0.7341772151898734,
+     "false_abstain": 0.1889763779527559,
+     "null_recall": 0.5806451612903226,
+     "auroc_null": 0.8219456438912878,
+     "auroc_conf": 0.8003556007112014,
+     "n": 158
+    },
+    "visa_extension": {
+     "nll": 3.391274110219176,
+     "brier": 1.221747900683501,
+     "jsd": 0.432846333134556,
+     "acc": 0.27325581395348836,
+     "acc_k": 0.3424657534246575,
+     "ece": 0.6095082028660663,
+     "sel_acc80": 0.2753623188405797,
+     "conf_wrong": 0.038461538461538464,
+     "coverage": 0.8313953488372093,
+     "false_abstain": 0.13013698630136986,
+     "null_recall": 0.38461538461538464,
+     "auroc_null": 0.8746048472075869,
+     "auroc_conf": 0.8253424657534246,
+     "n": 172
+    },
+    "subscription_cancellation": {
+     "nll": 3.274311082023101,
+     "brier": 1.0411595744094404,
+     "jsd": 0.37374148544328023,
+     "acc": 0.3829787234042553,
+     "acc_k": 0.44,
+     "ece": 0.5014726749971403,
+     "sel_acc80": 0.40707964601769914,
+     "conf_wrong": 0.1875,
+     "coverage": 0.8723404255319149,
+     "false_abstain": 0.08,
+     "null_recall": 0.5,
+     "auroc_null": 0.9175,
+     "auroc_conf": 0.8180000000000001,
+     "n": 141
+    },
+    "procurement_request": {
+     "nll": 4.339458520994687,
+     "brier": 1.139405576261341,
+     "jsd": 0.4016875272695481,
+     "acc": 0.37593984962406013,
+     "acc_k": 0.4369747899159664,
+     "ece": 0.557007525424312,
+     "sel_acc80": 0.3925233644859813,
+     "conf_wrong": 0.21428571428571427,
+     "coverage": 0.9022556390977443,
+     "false_abstain": 0.06722689075630252,
+     "null_recall": 0.35714285714285715,
+     "auroc_null": 0.9381752701080432,
+     "auroc_conf": 0.858343337334934,
+     "n": 133
+    },
+    "tradeoff": {
+     "nll": 1.0485382653766773,
+     "brier": 0.6644986709620295,
+     "jsd": 0.2847643375853819,
+     "acc": 0.4863636363636364,
+     "acc_k": 0.4863636363636364,
+     "ece": 0.18097281794656409,
+     "sel_acc80": 0.45454545454545453,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 220
+    },
+    "insurance_claim": {
+     "nll": 4.561252266824203,
+     "brier": 1.1413018112706845,
+     "jsd": 0.4059811712854699,
+     "acc": 0.41379310344827586,
+     "acc_k": 0.41379310344827586,
+     "ece": 0.5548728519472582,
+     "sel_acc80": 0.375,
+     "coverage": 1.0,
+     "false_abstain": 0.0,
+     "null_recall": "n/a",
+     "n": 29
+    }
+   },
+   "by_qtype": {},
+   "by_label_class": {
+    "answer": {
+     "nll": 2.961086866040626,
+     "brier": 0.9651978530800092,
+     "jsd": 0.35318397177542504,
+     "acc": 0.40895262929161236,
+     "acc_k": 0.4380704041720991,
+     "ece": 0.43056437424968913,
+     "sel_acc80": 0.41227593699076587,
+     "coverage": 0.9561060408518036,
+     "false_abstain": 0.04389395914819644,
+     "null_recall": "n/a",
+     "n": 2301
+    },
+    "null": {
+     "nll": 1.4530011419264246,
+     "brier": 0.7585235628287442,
+     "jsd": 0.2959391611608268,
+     "acc": 0.4338235294117647,
+     "ece": 0.29192931555649815,
+     "sel_acc80": 0.46788990825688076,
+     "conf_wrong": 0.1323529411764706,
+     "coverage": 0.5661764705882353,
+     "false_abstain": "n/a",
+     "null_recall": 0.4338235294117647,
+     "n": 136
+    }
+   },
+   "baselines": {
+    "majority_class": 0,
+    "majority_acc": 0.4095199015182602
+   }
+  }
+ }
+}

jevbench_summary.json

@@ -0,0 +1,2824 @@
+{
+  "easy": {
+    "accuracy": 1.0,
+    "brier_mean": 0.0047407252593300855,
+    "calibration_n": 48,
+    "complete": true,
+    "cost_basis": [
+      "local_gpu_no_provider_tariff"
+    ],
+    "coverage": 1.0,
+    "dataset_hash": "3cbd0e951fa8059741955ef2bd3273b0bb60aecae8f4a68b89872f55f7376b3d",
+    "ece": {
+      "bins": [
+        {
+          "accuracy": null,
+          "hi": 0.1,
+          "lo": 0.0,
+          "mean_confidence": null,
+          "n": 0
+        },
+        {
+          "accuracy": null,
+          "hi": 0.2,
+          "lo": 0.1,
+          "mean_confidence": null,
+          "n": 0
+        },
+        {
+          "accuracy": null,
+          "hi": 0.3,
+          "lo": 0.2,
+          "mean_confidence": null,
+          "n": 0
+        },
+        {
+          "accuracy": null,
+          "hi": 0.4,
+          "lo": 0.3,
+          "mean_confidence": null,
+          "n": 0
+        },
+        {
+          "accuracy": null,
+          "hi": 0.5,
+          "lo": 0.4,
+          "mean_confidence": null,
+          "n": 0
+        },
+        {
+          "accuracy": null,
+          "hi": 0.6,
+          "lo": 0.5,
+          "mean_confidence": null,
+          "n": 0
+        },
+        {
+          "accuracy": null,
+          "hi": 0.7,
+          "lo": 0.6,
+          "mean_confidence": null,
+          "n": 0
+        },
+        {
+          "accuracy": 1.0,
+          "hi": 0.8,
+          "lo": 0.7,
+          "mean_confidence": 0.7135202288627625,
+          "n": 1
+        },
+        {
+          "accuracy": 1.0,
+          "hi": 0.9,
+          "lo": 0.8,
+          "mean_confidence": 0.8538632392883301,
+          "n": 1
+        },
+        {
+          "accuracy": 1.0,
+          "hi": 1.0,
+          "lo": 0.9,
+          "mean_confidence": 0.9924926835557689,
+          "n": 46
+        }
+      ],
+      "ece": 0.016207356005907027,
+      "n": 48
+    },
+    "latency": {
+      "n": 48,
+      "p50_s": 0.06309161661192775,
+      "p95_s": 0.0954449666896835
+    },
+    "latency_failures": {
+      "n": 0,
+      "p50_s": null,
+      "p95_s": null
+    },
+    "ledger_charged_usd": null,
+    "macro_accuracy": 1.0,
+    "model_identities": [
+      "pcdm:native:runs/ts1b"
+    ],
+    "n_attempted": 48,
+    "n_correct": 48,
+    "n_planned": 48,
+    "n_renormalized": 0,
+    "n_scorable": 48,
+    "n_valid": 48,
+    "operational_success": 1.0,
+    "ordinal_mae": null,
+    "paraphrase_consistency": {
+      "agree": 0,
+      "agreement": null,
+      "both_correct_rate_all_pairs": null,
+      "both_valid": 0,
+      "pairs": 0
+    },
+    "per_family": {
+      "extraction": {
+        "accuracy": 1.0,
+        "brier_mean": 0.0006161597573289517,
+        "calibration_n": 12,
+        "cost_basis": [
+          "local_gpu_no_provider_tariff"
+        ],
+        "coverage": 1.0,
+        "ece": {
+          "bins": [
+            {
+              "accuracy": null,
+              "hi": 0.1,
+              "lo": 0.0,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.2,
+              "lo": 0.1,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.3,
+              "lo": 0.2,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.4,
+              "lo": 0.3,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.5,
+              "lo": 0.4,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.6,
+              "lo": 0.5,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.7,
+              "lo": 0.6,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.8,
+              "lo": 0.7,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.9,
+              "lo": 0.8,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 1.0,
+              "lo": 0.9,
+              "mean_confidence": 0.9888500918944677,
+              "n": 12
+            }
+          ],
+          "ece": 0.011149908105532291,
+          "n": 12
+        },
+        "latency": {
+          "n": 12,
+          "p50_s": 0.06396663328632712,
+          "p95_s": 0.08068648739717899
+        },
+        "latency_failures": {
+          "n": 0,
+          "p50_s": null,
+          "p95_s": null
+        },
+        "n_attempted": 12,
+        "n_correct": 12,
+        "n_planned": 12,
+        "n_renormalized": 0,
+        "n_scorable": 12,
+        "n_valid": 12,
+        "operational_success": 1.0,
+        "ordinal_mae": null,
+        "price_per_1000_decisions_usd": null,
+        "schema_validity": 1.0,
+        "schema_validity_strict": 1.0
+      },
+      "fact": {
+        "accuracy": 1.0,
+        "brier_mean": 0.018346439983612378,
+        "calibration_n": 12,
+        "cost_basis": [
+          "local_gpu_no_provider_tariff"
+        ],
+        "coverage": 1.0,
+        "ece": {
+          "bins": [
+            {
+              "accuracy": null,
+              "hi": 0.1,
+              "lo": 0.0,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.2,
+              "lo": 0.1,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.3,
+              "lo": 0.2,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.4,
+              "lo": 0.3,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.5,
+              "lo": 0.4,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.6,
+              "lo": 0.5,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.7,
+              "lo": 0.6,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.8,
+              "lo": 0.7,
+              "mean_confidence": 0.7135202288627625,
+              "n": 1
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.9,
+              "lo": 0.8,
+              "mean_confidence": 0.8538632392883301,
+              "n": 1
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 1.0,
+              "lo": 0.9,
+              "mean_confidence": 0.9790186047554016,
+              "n": 10
+            }
+          ],
+          "ece": 0.053535873691240965,
+          "n": 12
+        },
+        "latency": {
+          "n": 12,
+          "p50_s": 0.06284108175896108,
+          "p95_s": 0.09521680150646716
+        },
+        "latency_failures": {
+          "n": 0,
+          "p50_s": null,
+          "p95_s": null
+        },
+        "n_attempted": 12,
+        "n_correct": 12,
+        "n_planned": 12,
+        "n_renormalized": 0,
+        "n_scorable": 12,
+        "n_valid": 12,
+        "operational_success": 1.0,
+        "ordinal_mae": null,
+        "price_per_1000_decisions_usd": null,
+        "schema_validity": 1.0,
+        "schema_validity_strict": 1.0
+      },
+      "intent": {
+        "accuracy": 1.0,
+        "brier_mean": 3.0127070021926845e-07,
+        "calibration_n": 12,
+        "cost_basis": [
+          "local_gpu_no_provider_tariff"
+        ],
+        "coverage": 1.0,
+        "ece": {
+          "bins": [
+            {
+              "accuracy": null,
+              "hi": 0.1,
+              "lo": 0.0,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.2,
+              "lo": 0.1,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.3,
+              "lo": 0.2,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.4,
+              "lo": 0.3,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.5,
+              "lo": 0.4,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.6,
+              "lo": 0.5,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.7,
+              "lo": 0.6,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.8,
+              "lo": 0.7,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.9,
+              "lo": 0.8,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 1.0,
+              "lo": 0.9,
+              "mean_confidence": 0.9998585979143778,
+              "n": 12
+            }
+          ],
+          "ece": 0.0001414020856221887,
+          "n": 12
+        },
+        "latency": {
+          "n": 12,
+          "p50_s": 0.06062154611572623,
+          "p95_s": 0.12287021279335017
+        },
+        "latency_failures": {
+          "n": 0,
+          "p50_s": null,
+          "p95_s": null
+        },
+        "n_attempted": 12,
+        "n_correct": 12,
+        "n_planned": 12,
+        "n_renormalized": 0,
+        "n_scorable": 12,
+        "n_valid": 12,
+        "operational_success": 1.0,
+        "ordinal_mae": null,
+        "price_per_1000_decisions_usd": null,
+        "schema_validity": 1.0,
+        "schema_validity_strict": 1.0
+      },
+      "tool_selection": {
+        "accuracy": 1.0,
+        "brier_mean": 2.5678791016948264e-11,
+        "calibration_n": 12,
+        "cost_basis": [
+          "local_gpu_no_provider_tariff"
+        ],
+        "coverage": 1.0,
+        "ece": {
+          "bins": [
+            {
+              "accuracy": null,
+              "hi": 0.1,
+              "lo": 0.0,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.2,
+              "lo": 0.1,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.3,
+              "lo": 0.2,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.4,
+              "lo": 0.3,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.5,
+              "lo": 0.4,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.6,
+              "lo": 0.5,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.7,
+              "lo": 0.6,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.8,
+              "lo": 0.7,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.9,
+              "lo": 0.8,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 1.0,
+              "lo": 0.9,
+              "mean_confidence": 0.9999977598587672,
+              "n": 12
+            }
+          ],
+          "ece": 2.2401412328454384e-06,
+          "n": 12
+        },
+        "latency": {
+          "n": 12,
+          "p50_s": 0.06312976451590657,
+          "p95_s": 0.06712341390084475
+        },
+        "latency_failures": {
+          "n": 0,
+          "p50_s": null,
+          "p95_s": null
+        },
+        "n_attempted": 12,
+        "n_correct": 12,
+        "n_planned": 12,
+        "n_renormalized": 0,
+        "n_scorable": 12,
+        "n_valid": 12,
+        "operational_success": 1.0,
+        "ordinal_mae": null,
+        "price_per_1000_decisions_usd": null,
+        "schema_validity": 1.0,
+        "schema_validity_strict": 1.0
+      }
+    },
+    "price_per_1000_decisions_usd": null,
+    "probability_sources": [
+      "native"
+    ],
+    "schema_validity": 1.0,
+    "schema_validity_strict": 1.0,
+    "splits": {
+      "public": {
+        "accuracy": 1.0,
+        "brier_mean": 0.0047407252593300855,
+        "calibration_n": 48,
+        "cost_basis": [
+          "local_gpu_no_provider_tariff"
+        ],
+        "coverage": 1.0,
+        "ece": {
+          "bins": [
+            {
+              "accuracy": null,
+              "hi": 0.1,
+              "lo": 0.0,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.2,
+              "lo": 0.1,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.3,
+              "lo": 0.2,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.4,
+              "lo": 0.3,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.5,
+              "lo": 0.4,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.6,
+              "lo": 0.5,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.7,
+              "lo": 0.6,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.8,
+              "lo": 0.7,
+              "mean_confidence": 0.7135202288627625,
+              "n": 1
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.9,
+              "lo": 0.8,
+              "mean_confidence": 0.8538632392883301,
+              "n": 1
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 1.0,
+              "lo": 0.9,
+              "mean_confidence": 0.9924926835557689,
+              "n": 46
+            }
+          ],
+          "ece": 0.016207356005907027,
+          "n": 48
+        },
+        "latency": {
+          "n": 48,
+          "p50_s": 0.06309161661192775,
+          "p95_s": 0.0954449666896835
+        },
+        "latency_failures": {
+          "n": 0,
+          "p50_s": null,
+          "p95_s": null
+        },
+        "n_attempted": 48,
+        "n_correct": 48,
+        "n_planned": 48,
+        "n_renormalized": 0,
+        "n_scorable": 48,
+        "n_valid": 48,
+        "operational_success": 1.0,
+        "ordinal_mae": null,
+        "price_per_1000_decisions_usd": null,
+        "schema_validity": 1.0,
+        "schema_validity_strict": 1.0
+      }
+    },
+    "version": "jevbench-v1"
+  },
+  "hard": {
+    "accuracy": 0.43243243243243246,
+    "brier_mean": 0.7909445007358973,
+    "calibration_n": 111,
+    "complete": true,
+    "cost_basis": [
+      "local_gpu_no_provider_tariff"
+    ],
+    "coverage": 1.0,
+    "dataset_hash": "e816e106f562dec75a8da12e69b75e957a4bbd9f6300f1145fc71837327a3c61",
+    "ece": {
+      "bins": [
+        {
+          "accuracy": null,
+          "hi": 0.1,
+          "lo": 0.0,
+          "mean_confidence": null,
+          "n": 0
+        },
+        {
+          "accuracy": null,
+          "hi": 0.2,
+          "lo": 0.1,
+          "mean_confidence": null,
+          "n": 0
+        },
+        {
+          "accuracy": 0.42857142857142855,
+          "hi": 0.3,
+          "lo": 0.2,
+          "mean_confidence": 0.27341367091451374,
+          "n": 7
+        },
+        {
+          "accuracy": 0.3076923076923077,
+          "hi": 0.4,
+          "lo": 0.3,
+          "mean_confidence": 0.36204201441544753,
+          "n": 13
+        },
+        {
+          "accuracy": 0.4166666666666667,
+          "hi": 0.5,
+          "lo": 0.4,
+          "mean_confidence": 0.4574062079191208,
+          "n": 12
+        },
+        {
+          "accuracy": 0.2857142857142857,
+          "hi": 0.6,
+          "lo": 0.5,
+          "mean_confidence": 0.5474048810345786,
+          "n": 14
+        },
+        {
+          "accuracy": 0.5454545454545454,
+          "hi": 0.7,
+          "lo": 0.6,
+          "mean_confidence": 0.6413787711750377,
+          "n": 11
+        },
+        {
+          "accuracy": 0.5454545454545454,
+          "hi": 0.8,
+          "lo": 0.7,
+          "mean_confidence": 0.743556255644018,
+          "n": 11
+        },
+        {
+          "accuracy": 0.4117647058823529,
+          "hi": 0.9,
+          "lo": 0.8,
+          "mean_confidence": 0.8476672382915721,
+          "n": 17
+        },
+        {
+          "accuracy": 0.5,
+          "hi": 1.0,
+          "lo": 0.9,
+          "mean_confidence": 0.9562405920945681,
+          "n": 26
+        }
+      ],
+      "ece": 0.25632501856700796,
+      "n": 111
+    },
+    "latency": {
+      "n": 111,
+      "p50_s": 0.09745004074648023,
+      "p95_s": 0.11606183066032827
+    },
+    "latency_failures": {
+      "n": 0,
+      "p50_s": null,
+      "p95_s": null
+    },
+    "ledger_charged_usd": null,
+    "macro_accuracy": 0.4864605140301735,
+    "model_identities": [
+      "pcdm:native:runs/ts1b"
+    ],
+    "n_attempted": 111,
+    "n_correct": 48,
+    "n_planned": 111,
+    "n_renormalized": 0,
+    "n_scorable": 111,
+    "n_valid": 111,
+    "operational_success": 1.0,
+    "ordinal_mae": 0.682668519516786,
+    "paraphrase_consistency": {
+      "agree": 0,
+      "agreement": null,
+      "both_correct_rate_all_pairs": null,
+      "both_valid": 0,
+      "pairs": 0
+    },
+    "per_family": {
+      "adversarial": {
+        "accuracy": 0.3333333333333333,
+        "brier_mean": 0.8822798833915354,
+        "calibration_n": 6,
+        "cost_basis": [
+          "local_gpu_no_provider_tariff"
+        ],
+        "coverage": 1.0,
+        "ece": {
+          "bins": [
+            {
+              "accuracy": null,
+              "hi": 0.1,
+              "lo": 0.0,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.2,
+              "lo": 0.1,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.3,
+              "lo": 0.2,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 0.0,
+              "hi": 0.4,
+              "lo": 0.3,
+              "mean_confidence": 0.392834335565567,
+              "n": 1
+            },
+            {
+              "accuracy": null,
+              "hi": 0.5,
+              "lo": 0.4,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.6,
+              "lo": 0.5,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 0.0,
+              "hi": 0.7,
+              "lo": 0.6,
+              "mean_confidence": 0.6013781428337097,
+              "n": 1
+            },
+            {
+              "accuracy": null,
+              "hi": 0.8,
+              "lo": 0.7,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.9,
+              "lo": 0.8,
+              "mean_confidence": 0.8074021339416504,
+              "n": 1
+            },
+            {
+              "accuracy": 0.3333333333333333,
+              "hi": 1.0,
+              "lo": 0.9,
+              "mean_confidence": 0.9713454445203146,
+              "n": 3
+            }
+          ],
+          "ece": 0.5168077796697617,
+          "n": 6
+        },
+        "latency": {
+          "n": 6,
+          "p50_s": 0.09237208357080817,
+          "p95_s": 0.0955617759609595
+        },
+        "latency_failures": {
+          "n": 0,
+          "p50_s": null,
+          "p95_s": null
+        },
+        "n_attempted": 6,
+        "n_correct": 2,
+        "n_planned": 6,
+        "n_renormalized": 0,
+        "n_scorable": 6,
+        "n_valid": 6,
+        "operational_success": 1.0,
+        "ordinal_mae": 0.5170895606279373,
+        "price_per_1000_decisions_usd": null,
+        "schema_validity": 1.0,
+        "schema_validity_strict": 1.0
+      },
+      "ambiguous": {
+        "accuracy": 0.42857142857142855,
+        "brier_mean": 0.7734691553069819,
+        "calibration_n": 7,
+        "cost_basis": [
+          "local_gpu_no_provider_tariff"
+        ],
+        "coverage": 1.0,
+        "ece": {
+          "bins": [
+            {
+              "accuracy": null,
+              "hi": 0.1,
+              "lo": 0.0,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.2,
+              "lo": 0.1,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.3,
+              "lo": 0.2,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.4,
+              "lo": 0.3,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 0.0,
+              "hi": 0.5,
+              "lo": 0.4,
+              "mean_confidence": 0.43299877643585205,
+              "n": 1
+            },
+            {
+              "accuracy": null,
+              "hi": 0.6,
+              "lo": 0.5,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.7,
+              "lo": 0.6,
+              "mean_confidence": 0.6301537156105042,
+              "n": 1
+            },
+            {
+              "accuracy": 0.0,
+              "hi": 0.8,
+              "lo": 0.7,
+              "mean_confidence": 0.751984715461731,
+              "n": 1
+            },
+            {
+              "accuracy": 0.3333333333333333,
+              "hi": 0.9,
+              "lo": 0.8,
+              "mean_confidence": 0.8551861643791199,
+              "n": 3
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 1.0,
+              "lo": 0.9,
+              "mean_confidence": 0.9342254996299744,
+              "n": 1
+            }
+          ],
+          "ece": 0.4551661099706378,
+          "n": 7
+        },
+        "latency": {
+          "n": 7,
+          "p50_s": 0.0957035981118679,
+          "p95_s": 0.10261358409188688
+        },
+        "latency_failures": {
+          "n": 0,
+          "p50_s": null,
+          "p95_s": null
+        },
+        "n_attempted": 7,
+        "n_correct": 3,
+        "n_planned": 7,
+        "n_renormalized": 0,
+        "n_scorable": 7,
+        "n_valid": 7,
+        "operational_success": 1.0,
+        "ordinal_mae": null,
+        "price_per_1000_decisions_usd": null,
+        "schema_validity": 1.0,
+        "schema_validity_strict": 1.0
+      },
+      "judge_hard": {
+        "accuracy": 0.4117647058823529,
+        "brier_mean": 0.9550899406207743,
+        "calibration_n": 17,
+        "cost_basis": [
+          "local_gpu_no_provider_tariff"
+        ],
+        "coverage": 1.0,
+        "ece": {
+          "bins": [
+            {
+              "accuracy": null,
+              "hi": 0.1,
+              "lo": 0.0,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.2,
+              "lo": 0.1,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.3,
+              "lo": 0.2,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.4,
+              "lo": 0.3,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.5,
+              "lo": 0.4,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 0.0,
+              "hi": 0.6,
+              "lo": 0.5,
+              "mean_confidence": 0.5444756150245667,
+              "n": 1
+            },
+            {
+              "accuracy": null,
+              "hi": 0.7,
+              "lo": 0.6,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.8,
+              "lo": 0.7,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 0.3333333333333333,
+              "hi": 0.9,
+              "lo": 0.8,
+              "mean_confidence": 0.8754598399003347,
+              "n": 6
+            },
+            {
+              "accuracy": 0.5,
+              "hi": 1.0,
+              "lo": 0.9,
+              "mean_confidence": 0.9572476387023926,
+              "n": 10
+            }
+          ],
+          "ece": 0.49233594361473537,
+          "n": 17
+        },
+        "latency": {
+          "n": 17,
+          "p50_s": 0.09264156315475702,
+          "p95_s": 0.0934916635043919
+        },
+        "latency_failures": {
+          "n": 0,
+          "p50_s": null,
+          "p95_s": null
+        },
+        "n_attempted": 17,
+        "n_correct": 7,
+        "n_planned": 17,
+        "n_renormalized": 0,
+        "n_scorable": 17,
+        "n_valid": 17,
+        "operational_success": 1.0,
+        "ordinal_mae": null,
+        "price_per_1000_decisions_usd": null,
+        "schema_validity": 1.0,
+        "schema_validity_strict": 1.0
+      },
+      "long_policy": {
+        "accuracy": 0.2631578947368421,
+        "brier_mean": 0.8194034386815301,
+        "calibration_n": 19,
+        "cost_basis": [
+          "local_gpu_no_provider_tariff"
+        ],
+        "coverage": 1.0,
+        "ece": {
+          "bins": [
+            {
+              "accuracy": null,
+              "hi": 0.1,
+              "lo": 0.0,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.2,
+              "lo": 0.1,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 0.3333333333333333,
+              "hi": 0.3,
+              "lo": 0.2,
+              "mean_confidence": 0.2753242254257202,
+              "n": 3
+            },
+            {
+              "accuracy": 0.16666666666666666,
+              "hi": 0.4,
+              "lo": 0.3,
+              "mean_confidence": 0.3495459308226903,
+              "n": 6
+            },
+            {
+              "accuracy": 0.0,
+              "hi": 0.5,
+              "lo": 0.4,
+              "mean_confidence": 0.44854867458343506,
+              "n": 1
+            },
+            {
+              "accuracy": 0.3333333333333333,
+              "hi": 0.6,
+              "lo": 0.5,
+              "mean_confidence": 0.5452876488367716,
+              "n": 3
+            },
+            {
+              "accuracy": 0.0,
+              "hi": 0.7,
+              "lo": 0.6,
+              "mean_confidence": 0.6139329075813293,
+              "n": 1
+            },
+            {
+              "accuracy": 0.5,
+              "hi": 0.8,
+              "lo": 0.7,
+              "mean_confidence": 0.7318354845046997,
+              "n": 2
+            },
+            {
+              "accuracy": null,
+              "hi": 0.9,
+              "lo": 0.8,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 0.3333333333333333,
+              "hi": 1.0,
+              "lo": 0.9,
+              "mean_confidence": 0.9440843065579733,
+              "n": 3
+            }
+          ],
+          "ece": 0.27713533294828313,
+          "n": 19
+        },
+        "latency": {
+          "n": 19,
+          "p50_s": 0.11336228996515274,
+          "p95_s": 0.12370569789782163
+        },
+        "latency_failures": {
+          "n": 0,
+          "p50_s": null,
+          "p95_s": null
+        },
+        "n_attempted": 19,
+        "n_correct": 5,
+        "n_planned": 19,
+        "n_renormalized": 0,
+        "n_scorable": 19,
+        "n_valid": 19,
+        "operational_success": 1.0,
+        "ordinal_mae": 0.4551819935441017,
+        "price_per_1000_decisions_usd": null,
+        "schema_validity": 1.0,
+        "schema_validity_strict": 1.0
+      },
+      "multi_hop": {
+        "accuracy": 0.4444444444444444,
+        "brier_mean": 0.7247968278780245,
+        "calibration_n": 18,
+        "cost_basis": [
+          "local_gpu_no_provider_tariff"
+        ],
+        "coverage": 1.0,
+        "ece": {
+          "bins": [
+            {
+              "accuracy": null,
+              "hi": 0.1,
+              "lo": 0.0,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.2,
+              "lo": 0.1,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 0.5,
+              "hi": 0.3,
+              "lo": 0.2,
+              "mean_confidence": 0.2637081891298294,
+              "n": 2
+            },
+            {
+              "accuracy": 0.6666666666666666,
+              "hi": 0.4,
+              "lo": 0.3,
+              "mean_confidence": 0.3658106327056885,
+              "n": 3
+            },
+            {
+              "accuracy": 0.2,
+              "hi": 0.5,
+              "lo": 0.4,
+              "mean_confidence": 0.47336041927337646,
+              "n": 5
+            },
+            {
+              "accuracy": 0.0,
+              "hi": 0.6,
+              "lo": 0.5,
+              "mean_confidence": 0.5131076773007711,
+              "n": 3
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.7,
+              "lo": 0.6,
+              "mean_confidence": 0.6799950003623962,
+              "n": 1
+            },
+            {
+              "accuracy": 0.5,
+              "hi": 0.8,
+              "lo": 0.7,
+              "mean_confidence": 0.7172561585903168,
+              "n": 2
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.9,
+              "lo": 0.8,
+              "mean_confidence": 0.8304344415664673,
+              "n": 2
+            },
+            {
+              "accuracy": null,
+              "hi": 1.0,
+              "lo": 0.9,
+              "mean_confidence": null,
+              "n": 0
+            }
+          ],
+          "ece": 0.29860696030987627,
+          "n": 18
+        },
+        "latency": {
+          "n": 18,
+          "p50_s": 0.10305563476867974,
+          "p95_s": 0.11012194301001728
+        },
+        "latency_failures": {
+          "n": 0,
+          "p50_s": null,
+          "p95_s": null
+        },
+        "n_attempted": 18,
+        "n_correct": 8,
+        "n_planned": 18,
+        "n_renormalized": 0,
+        "n_scorable": 18,
+        "n_valid": 18,
+        "operational_success": 1.0,
+        "ordinal_mae": 0.16368193924427032,
+        "price_per_1000_decisions_usd": null,
+        "schema_validity": 1.0,
+        "schema_validity_strict": 1.0
+      },
+      "probability": {
+        "accuracy": 0.4,
+        "brier_mean": 0.9394113544796729,
+        "calibration_n": 10,
+        "cost_basis": [
+          "local_gpu_no_provider_tariff"
+        ],
+        "coverage": 1.0,
+        "ece": {
+          "bins": [
+            {
+              "accuracy": null,
+              "hi": 0.1,
+              "lo": 0.0,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.2,
+              "lo": 0.1,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.3,
+              "lo": 0.2,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.4,
+              "lo": 0.3,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.5,
+              "lo": 0.4,
+              "mean_confidence": 0.4608645737171173,
+              "n": 1
+            },
+            {
+              "accuracy": 0.5,
+              "hi": 0.6,
+              "lo": 0.5,
+              "mean_confidence": 0.5587047636508942,
+              "n": 2
+            },
+            {
+              "accuracy": 0.3333333333333333,
+              "hi": 0.7,
+              "lo": 0.6,
+              "mean_confidence": 0.6281190514564514,
+              "n": 3
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.8,
+              "lo": 0.7,
+              "mean_confidence": 0.708816409111023,
+              "n": 1
+            },
+            {
+              "accuracy": 0.0,
+              "hi": 0.9,
+              "lo": 0.8,
+              "mean_confidence": 0.8403750658035278,
+              "n": 2
+            },
+            {
+              "accuracy": 0.0,
+              "hi": 1.0,
+              "lo": 0.9,
+              "mean_confidence": 0.9059809446334839,
+              "n": 1
+            }
+          ],
+          "ece": 0.4418816775083542,
+          "n": 10
+        },
+        "latency": {
+          "n": 10,
+          "p50_s": 0.09962555137462914,
+          "p95_s": 0.10702485411893575
+        },
+        "latency_failures": {
+          "n": 0,
+          "p50_s": null,
+          "p95_s": null
+        },
+        "n_attempted": 10,
+        "n_correct": 4,
+        "n_planned": 10,
+        "n_renormalized": 0,
+        "n_scorable": 10,
+        "n_valid": 10,
+        "operational_success": 1.0,
+        "ordinal_mae": null,
+        "price_per_1000_decisions_usd": null,
+        "schema_validity": 1.0,
+        "schema_validity_strict": 1.0
+      },
+      "routing_hard": {
+        "accuracy": 1.0,
+        "brier_mean": 0.13808678499012011,
+        "calibration_n": 5,
+        "cost_basis": [
+          "local_gpu_no_provider_tariff"
+        ],
+        "coverage": 1.0,
+        "ece": {
+          "bins": [
+            {
+              "accuracy": null,
+              "hi": 0.1,
+              "lo": 0.0,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.2,
+              "lo": 0.1,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.3,
+              "lo": 0.2,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.4,
+              "lo": 0.3,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.5,
+              "lo": 0.4,
+              "mean_confidence": 0.43413904309272766,
+              "n": 1
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.6,
+              "lo": 0.5,
+              "mean_confidence": 0.5734199285507202,
+              "n": 1
+            },
+            {
+              "accuracy": null,
+              "hi": 0.7,
+              "lo": 0.6,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.8,
+              "lo": 0.7,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.9,
+              "lo": 0.8,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 1.0,
+              "lo": 0.9,
+              "mean_confidence": 0.9703327616055807,
+              "n": 3
+            }
+          ],
+          "ece": 0.21628854870796202,
+          "n": 5
+        },
+        "latency": {
+          "n": 5,
+          "p50_s": 0.09371192893013358,
+          "p95_s": 0.09542652918025851
+        },
+        "latency_failures": {
+          "n": 0,
+          "p50_s": null,
+          "p95_s": null
+        },
+        "n_attempted": 5,
+        "n_correct": 5,
+        "n_planned": 5,
+        "n_renormalized": 0,
+        "n_scorable": 5,
+        "n_valid": 5,
+        "operational_success": 1.0,
+        "ordinal_mae": null,
+        "price_per_1000_decisions_usd": null,
+        "schema_validity": 1.0,
+        "schema_validity_strict": 1.0
+      },
+      "temporal_numeric": {
+        "accuracy": 0.3333333333333333,
+        "brier_mean": 0.8541664497755332,
+        "calibration_n": 15,
+        "cost_basis": [
+          "local_gpu_no_provider_tariff"
+        ],
+        "coverage": 1.0,
+        "ece": {
+          "bins": [
+            {
+              "accuracy": null,
+              "hi": 0.1,
+              "lo": 0.0,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.2,
+              "lo": 0.1,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 0.5,
+              "hi": 0.3,
+              "lo": 0.2,
+              "mean_confidence": 0.2802533209323883,
+              "n": 2
+            },
+            {
+              "accuracy": 0.3333333333333333,
+              "hi": 0.4,
+              "lo": 0.3,
+              "mean_confidence": 0.37300145626068115,
+              "n": 3
+            },
+            {
+              "accuracy": 0.5,
+              "hi": 0.5,
+              "lo": 0.4,
+              "mean_confidence": 0.43682731688022614,
+              "n": 2
+            },
+            {
+              "accuracy": 0.0,
+              "hi": 0.6,
+              "lo": 0.5,
+              "mean_confidence": 0.5656636953353882,
+              "n": 3
+            },
+            {
+              "accuracy": 0.5,
+              "hi": 0.7,
+              "lo": 0.6,
+              "mean_confidence": 0.6869688034057617,
+              "n": 2
+            },
+            {
+              "accuracy": 0.0,
+              "hi": 0.8,
+              "lo": 0.7,
+              "mean_confidence": 0.7070299386978149,
+              "n": 1
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.9,
+              "lo": 0.8,
+              "mean_confidence": 0.8164684176445007,
+              "n": 1
+            },
+            {
+              "accuracy": 0.0,
+              "hi": 1.0,
+              "lo": 0.9,
+              "mean_confidence": 0.973138689994812,
+              "n": 1
+            }
+          ],
+          "ece": 0.3079647998015086,
+          "n": 15
+        },
+        "latency": {
+          "n": 15,
+          "p50_s": 0.09936860809102654,
+          "p95_s": 0.10479145389981567
+        },
+        "latency_failures": {
+          "n": 0,
+          "p50_s": null,
+          "p95_s": null
+        },
+        "n_attempted": 15,
+        "n_correct": 5,
+        "n_planned": 15,
+        "n_renormalized": 0,
+        "n_scorable": 15,
+        "n_valid": 15,
+        "operational_success": 1.0,
+        "ordinal_mae": 1.607450231909752,
+        "price_per_1000_decisions_usd": null,
+        "schema_validity": 1.0,
+        "schema_validity_strict": 1.0
+      },
+      "tradeoff": {
+        "accuracy": 0.5,
+        "brier_mean": 0.8777460761269166,
+        "calibration_n": 6,
+        "cost_basis": [
+          "local_gpu_no_provider_tariff"
+        ],
+        "coverage": 1.0,
+        "ece": {
+          "bins": [
+            {
+              "accuracy": null,
+              "hi": 0.1,
+              "lo": 0.0,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.2,
+              "lo": 0.1,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.3,
+              "lo": 0.2,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.4,
+              "lo": 0.3,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.5,
+              "lo": 0.4,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.6,
+              "lo": 0.5,
+              "mean_confidence": 0.5561861991882324,
+              "n": 1
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.7,
+              "lo": 0.6,
+              "mean_confidence": 0.646073043346405,
+              "n": 1
+            },
+            {
+              "accuracy": 0.5,
+              "hi": 0.8,
+              "lo": 0.7,
+              "mean_confidence": 0.7889719605445862,
+              "n": 2
+            },
+            {
+              "accuracy": 0.0,
+              "hi": 0.9,
+              "lo": 0.8,
+              "mean_confidence": 0.824085533618927,
+              "n": 1
+            },
+            {
+              "accuracy": 0.0,
+              "hi": 1.0,
+              "lo": 0.9,
+              "mean_confidence": 0.9255269169807434,
+              "n": 1
+            }
+          ],
+          "ece": 0.5208828548590341,
+          "n": 6
+        },
+        "latency": {
+          "n": 6,
+          "p50_s": 0.09490847331471741,
+          "p95_s": 0.10340702254325151
+        },
+        "latency_failures": {
+          "n": 0,
+          "p50_s": null,
+          "p95_s": null
+        },
+        "n_attempted": 6,
+        "n_correct": 3,
+        "n_planned": 6,
+        "n_renormalized": 0,
+        "n_scorable": 6,
+        "n_valid": 6,
+        "operational_success": 1.0,
+        "ordinal_mae": null,
+        "price_per_1000_decisions_usd": null,
+        "schema_validity": 1.0,
+        "schema_validity_strict": 1.0
+      },
+      "trap": {
+        "accuracy": 0.75,
+        "brier_mean": 0.5089772867172517,
+        "calibration_n": 8,
+        "cost_basis": [
+          "local_gpu_no_provider_tariff"
+        ],
+        "coverage": 1.0,
+        "ece": {
+          "bins": [
+            {
+              "accuracy": null,
+              "hi": 0.1,
+              "lo": 0.0,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.2,
+              "lo": 0.1,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.3,
+              "lo": 0.2,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.4,
+              "lo": 0.3,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.5,
+              "lo": 0.4,
+              "mean_confidence": 0.4718666970729828,
+              "n": 1
+            },
+            {
+              "accuracy": null,
+              "hi": 0.6,
+              "lo": 0.5,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.7,
+              "lo": 0.6,
+              "mean_confidence": 0.6253389120101929,
+              "n": 1
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.8,
+              "lo": 0.7,
+              "mean_confidence": 0.7675802707672119,
+              "n": 2
+            },
+            {
+              "accuracy": 0.0,
+              "hi": 0.9,
+              "lo": 0.8,
+              "mean_confidence": 0.80245041847229,
+              "n": 1
+            },
+            {
+              "accuracy": 0.6666666666666666,
+              "hi": 1.0,
+              "lo": 0.9,
+              "mean_confidence": 0.9645398060480753,
+              "n": 3
+            }
+          ],
+          "ece": 0.3829629607498646,
+          "n": 8
+        },
+        "latency": {
+          "n": 8,
+          "p50_s": 0.09288997249677777,
+          "p95_s": 0.09611767800524831
+        },
+        "latency_failures": {
+          "n": 0,
+          "p50_s": null,
+          "p95_s": null
+        },
+        "n_attempted": 8,
+        "n_correct": 6,
+        "n_planned": 8,
+        "n_renormalized": 0,
+        "n_scorable": 8,
+        "n_valid": 8,
+        "operational_success": 1.0,
+        "ordinal_mae": 0.8974253982305527,
+        "price_per_1000_decisions_usd": null,
+        "schema_validity": 1.0,
+        "schema_validity_strict": 1.0
+      }
+    },
+    "price_per_1000_decisions_usd": null,
+    "probability_sources": [
+      "native"
+    ],
+    "schema_validity": 1.0,
+    "schema_validity_strict": 1.0,
+    "splits": {
+      "public": {
+        "accuracy": 0.43243243243243246,
+        "brier_mean": 0.7909445007358973,
+        "calibration_n": 111,
+        "cost_basis": [
+          "local_gpu_no_provider_tariff"
+        ],
+        "coverage": 1.0,
+        "ece": {
+          "bins": [
+            {
+              "accuracy": null,
+              "hi": 0.1,
+              "lo": 0.0,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.2,
+              "lo": 0.1,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 0.42857142857142855,
+              "hi": 0.3,
+              "lo": 0.2,
+              "mean_confidence": 0.27341367091451374,
+              "n": 7
+            },
+            {
+              "accuracy": 0.3076923076923077,
+              "hi": 0.4,
+              "lo": 0.3,
+              "mean_confidence": 0.36204201441544753,
+              "n": 13
+            },
+            {
+              "accuracy": 0.4166666666666667,
+              "hi": 0.5,
+              "lo": 0.4,
+              "mean_confidence": 0.4574062079191208,
+              "n": 12
+            },
+            {
+              "accuracy": 0.2857142857142857,
+              "hi": 0.6,
+              "lo": 0.5,
+              "mean_confidence": 0.5474048810345786,
+              "n": 14
+            },
+            {
+              "accuracy": 0.5454545454545454,
+              "hi": 0.7,
+              "lo": 0.6,
+              "mean_confidence": 0.6413787711750377,
+              "n": 11
+            },
+            {
+              "accuracy": 0.5454545454545454,
+              "hi": 0.8,
+              "lo": 0.7,
+              "mean_confidence": 0.743556255644018,
+              "n": 11
+            },
+            {
+              "accuracy": 0.4117647058823529,
+              "hi": 0.9,
+              "lo": 0.8,
+              "mean_confidence": 0.8476672382915721,
+              "n": 17
+            },
+            {
+              "accuracy": 0.5,
+              "hi": 1.0,
+              "lo": 0.9,
+              "mean_confidence": 0.9562405920945681,
+              "n": 26
+            }
+          ],
+          "ece": 0.25632501856700796,
+          "n": 111
+        },
+        "latency": {
+          "n": 111,
+          "p50_s": 0.09745004074648023,
+          "p95_s": 0.11606183066032827
+        },
+        "latency_failures": {
+          "n": 0,
+          "p50_s": null,
+          "p95_s": null
+        },
+        "n_attempted": 111,
+        "n_correct": 48,
+        "n_planned": 111,
+        "n_renormalized": 0,
+        "n_scorable": 111,
+        "n_valid": 111,
+        "operational_success": 1.0,
+        "ordinal_mae": 0.682668519516786,
+        "price_per_1000_decisions_usd": null,
+        "schema_validity": 1.0,
+        "schema_validity_strict": 1.0
+      }
+    },
+    "version": "jevbench-v1"
+  },
+  "original": {
+    "accuracy": 0.6944444444444444,
+    "brier_mean": 0.39750463081241993,
+    "calibration_n": 72,
+    "complete": true,
+    "cost_basis": [
+      "local_gpu_no_provider_tariff"
+    ],
+    "coverage": 1.0,
+    "dataset_hash": "914276022fa5f0ea70e4231f2af98e68c67b274b61bea8c440b33890401cd2f9",
+    "ece": {
+      "bins": [
+        {
+          "accuracy": null,
+          "hi": 0.1,
+          "lo": 0.0,
+          "mean_confidence": null,
+          "n": 0
+        },
+        {
+          "accuracy": null,
+          "hi": 0.2,
+          "lo": 0.1,
+          "mean_confidence": null,
+          "n": 0
+        },
+        {
+          "accuracy": null,
+          "hi": 0.3,
+          "lo": 0.2,
+          "mean_confidence": null,
+          "n": 0
+        },
+        {
+          "accuracy": null,
+          "hi": 0.4,
+          "lo": 0.3,
+          "mean_confidence": null,
+          "n": 0
+        },
+        {
+          "accuracy": 0.4,
+          "hi": 0.5,
+          "lo": 0.4,
+          "mean_confidence": 0.45405682921409607,
+          "n": 5
+        },
+        {
+          "accuracy": 0.25,
+          "hi": 0.6,
+          "lo": 0.5,
+          "mean_confidence": 0.538318544626236,
+          "n": 8
+        },
+        {
+          "accuracy": 0.4,
+          "hi": 0.7,
+          "lo": 0.6,
+          "mean_confidence": 0.6290562272071838,
+          "n": 5
+        },
+        {
+          "accuracy": 0.7272727272727273,
+          "hi": 0.8,
+          "lo": 0.7,
+          "mean_confidence": 0.7492249011993408,
+          "n": 11
+        },
+        {
+          "accuracy": 0.7,
+          "hi": 0.9,
+          "lo": 0.8,
+          "mean_confidence": 0.8450712203979492,
+          "n": 10
+        },
+        {
+          "accuracy": 0.8787878787878788,
+          "hi": 1.0,
+          "lo": 0.9,
+          "mean_confidence": 0.9637182950973511,
+          "n": 33
+        }
+      ],
+      "ece": 0.11412504853473769,
+      "n": 72
+    },
+    "latency": {
+      "n": 72,
+      "p50_s": 0.06066564843058586,
+      "p95_s": 0.12478509345091895
+    },
+    "latency_failures": {
+      "n": 0,
+      "p50_s": null,
+      "p95_s": null
+    },
+    "ledger_charged_usd": null,
+    "macro_accuracy": 0.6944444444444445,
+    "model_identities": [
+      "pcdm:native:runs/ts1b"
+    ],
+    "n_attempted": 72,
+    "n_correct": 50,
+    "n_planned": 72,
+    "n_renormalized": 0,
+    "n_scorable": 72,
+    "n_valid": 72,
+    "operational_success": 1.0,
+    "ordinal_mae": 0.556312729138881,
+    "paraphrase_consistency": {
+      "agree": 26,
+      "agreement": 0.7222222222222222,
+      "both_correct_rate_all_pairs": 0.5555555555555556,
+      "both_valid": 36,
+      "pairs": 36
+    },
+    "per_family": {
+      "adequacy": {
+        "accuracy": 0.5833333333333334,
+        "brier_mean": 0.5449021935508932,
+        "calibration_n": 12,
+        "cost_basis": [
+          "local_gpu_no_provider_tariff"
+        ],
+        "coverage": 1.0,
+        "ece": {
+          "bins": [
+            {
+              "accuracy": null,
+              "hi": 0.1,
+              "lo": 0.0,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.2,
+              "lo": 0.1,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.3,
+              "lo": 0.2,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.4,
+              "lo": 0.3,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.5,
+              "lo": 0.4,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.6,
+              "lo": 0.5,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 0.0,
+              "hi": 0.7,
+              "lo": 0.6,
+              "mean_confidence": 0.6186434030532837,
+              "n": 1
+            },
+            {
+              "accuracy": 0.6666666666666666,
+              "hi": 0.8,
+              "lo": 0.7,
+              "mean_confidence": 0.749783436457316,
+              "n": 3
+            },
+            {
+              "accuracy": 0.25,
+              "hi": 0.9,
+              "lo": 0.8,
+              "mean_confidence": 0.8545016646385193,
+              "n": 4
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 1.0,
+              "lo": 0.9,
+              "mean_confidence": 0.9632230699062347,
+              "n": 4
+            }
+          ],
+          "ece": 0.2860923409461975,
+          "n": 12
+        },
+        "latency": {
+          "n": 12,
+          "p50_s": 0.06046207994222641,
+          "p95_s": 0.07607227428816257
+        },
+        "latency_failures": {
+          "n": 0,
+          "p50_s": null,
+          "p95_s": null
+        },
+        "n_attempted": 12,
+        "n_correct": 7,
+        "n_planned": 12,
+        "n_renormalized": 0,
+        "n_scorable": 12,
+        "n_valid": 12,
+        "operational_success": 1.0,
+        "ordinal_mae": null,
+        "price_per_1000_decisions_usd": null,
+        "schema_validity": 1.0,
+        "schema_validity_strict": 1.0
+      },
+      "extraction": {
+        "accuracy": 0.75,
+        "brier_mean": 0.22976573909523312,
+        "calibration_n": 12,
+        "cost_basis": [
+          "local_gpu_no_provider_tariff"
+        ],
+        "coverage": 1.0,
+        "ece": {
+          "bins": [
+            {
+              "accuracy": null,
+              "hi": 0.1,
+              "lo": 0.0,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.2,
+              "lo": 0.1,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.3,
+              "lo": 0.2,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.4,
+              "lo": 0.3,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.5,
+              "lo": 0.4,
+              "mean_confidence": 0.46592795848846436,
+              "n": 1
+            },
+            {
+              "accuracy": 0.0,
+              "hi": 0.6,
+              "lo": 0.5,
+              "mean_confidence": 0.5530417263507843,
+              "n": 2
+            },
+            {
+              "accuracy": 0.0,
+              "hi": 0.7,
+              "lo": 0.6,
+              "mean_confidence": 0.6272711157798767,
+              "n": 1
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.8,
+              "lo": 0.7,
+              "mean_confidence": 0.7457770109176636,
+              "n": 1
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.9,
+              "lo": 0.8,
+              "mean_confidence": 0.8463155627250671,
+              "n": 1
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 1.0,
+              "lo": 0.9,
+              "mean_confidence": 0.94781161348025,
+              "n": 6
+            }
+          ],
+          "ece": 0.24903869628906247,
+          "n": 12
+        },
+        "latency": {
+          "n": 12,
+          "p50_s": 0.045599582605063915,
+          "p95_s": 0.0653196045430377
+        },
+        "latency_failures": {
+          "n": 0,
+          "p50_s": null,
+          "p95_s": null
+        },
+        "n_attempted": 12,
+        "n_correct": 9,
+        "n_planned": 12,
+        "n_renormalized": 0,
+        "n_scorable": 12,
+        "n_valid": 12,
+        "operational_success": 1.0,
+        "ordinal_mae": null,
+        "price_per_1000_decisions_usd": null,
+        "schema_validity": 1.0,
+        "schema_validity_strict": 1.0
+      },
+      "intent": {
+        "accuracy": 0.75,
+        "brier_mean": 0.37706546705662053,
+        "calibration_n": 12,
+        "cost_basis": [
+          "local_gpu_no_provider_tariff"
+        ],
+        "coverage": 1.0,
+        "ece": {
+          "bins": [
+            {
+              "accuracy": null,
+              "hi": 0.1,
+              "lo": 0.0,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.2,
+              "lo": 0.1,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.3,
+              "lo": 0.2,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.4,
+              "lo": 0.3,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.5,
+              "lo": 0.4,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.6,
+              "lo": 0.5,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 0.0,
+              "hi": 0.7,
+              "lo": 0.6,
+              "mean_confidence": 0.6453130841255188,
+              "n": 1
+            },
+            {
+              "accuracy": null,
+              "hi": 0.8,
+              "lo": 0.7,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.9,
+              "lo": 0.8,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 0.8181818181818182,
+              "hi": 1.0,
+              "lo": 0.9,
+              "mean_confidence": 0.9673931002616882,
+              "n": 11
+            }
+          ],
+          "ece": 0.1905530989170074,
+          "n": 12
+        },
+        "latency": {
+          "n": 12,
+          "p50_s": 0.09129977598786354,
+          "p95_s": 0.16712851964402942
+        },
+        "latency_failures": {
+          "n": 0,
+          "p50_s": null,
+          "p95_s": null
+        },
+        "n_attempted": 12,
+        "n_correct": 9,
+        "n_planned": 12,
+        "n_renormalized": 0,
+        "n_scorable": 12,
+        "n_valid": 12,
+        "operational_success": 1.0,
+        "ordinal_mae": null,
+        "price_per_1000_decisions_usd": null,
+        "schema_validity": 1.0,
+        "schema_validity_strict": 1.0
+      },
+      "ordinal": {
+        "accuracy": 0.5833333333333334,
+        "brier_mean": 0.4297174396617977,
+        "calibration_n": 12,
+        "cost_basis": [
+          "local_gpu_no_provider_tariff"
+        ],
+        "coverage": 1.0,
+        "ece": {
+          "bins": [
+            {
+              "accuracy": null,
+              "hi": 0.1,
+              "lo": 0.0,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.2,
+              "lo": 0.1,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.3,
+              "lo": 0.2,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.4,
+              "lo": 0.3,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 0.3333333333333333,
+              "hi": 0.5,
+              "lo": 0.4,
+              "mean_confidence": 0.4347889920075734,
+              "n": 3
+            },
+            {
+              "accuracy": 0.0,
+              "hi": 0.6,
+              "lo": 0.5,
+              "mean_confidence": 0.5180816451708475,
+              "n": 3
+            },
+            {
+              "accuracy": null,
+              "hi": 0.7,
+              "lo": 0.6,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.8,
+              "lo": 0.7,
+              "mean_confidence": 0.7547558744748434,
+              "n": 3
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.9,
+              "lo": 0.8,
+              "mean_confidence": 0.8488078713417053,
+              "n": 3
+            },
+            {
+              "accuracy": null,
+              "hi": 1.0,
+              "lo": 0.9,
+              "mean_confidence": null,
+              "n": 0
+            }
+          ],
+          "ece": 0.25399338950713474,
+          "n": 12
+        },
+        "latency": {
+          "n": 12,
+          "p50_s": 0.06068861088715494,
+          "p95_s": 0.09393381557893007
+        },
+        "latency_failures": {
+          "n": 0,
+          "p50_s": null,
+          "p95_s": null
+        },
+        "n_attempted": 12,
+        "n_correct": 7,
+        "n_planned": 12,
+        "n_renormalized": 0,
+        "n_scorable": 12,
+        "n_valid": 12,
+        "operational_success": 1.0,
+        "ordinal_mae": 0.556312729138881,
+        "price_per_1000_decisions_usd": null,
+        "schema_validity": 1.0,
+        "schema_validity_strict": 1.0
+      },
+      "policy": {
+        "accuracy": 0.6666666666666666,
+        "brier_mean": 0.5632518159517107,
+        "calibration_n": 12,
+        "cost_basis": [
+          "local_gpu_no_provider_tariff"
+        ],
+        "coverage": 1.0,
+        "ece": {
+          "bins": [
+            {
+              "accuracy": null,
+              "hi": 0.1,
+              "lo": 0.0,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.2,
+              "lo": 0.1,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.3,
+              "lo": 0.2,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.4,
+              "lo": 0.3,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.5,
+              "lo": 0.4,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.6,
+              "lo": 0.5,
+              "mean_confidence": 0.5495584607124329,
+              "n": 1
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.7,
+              "lo": 0.6,
+              "mean_confidence": 0.616141676902771,
+              "n": 1
+            },
+            {
+              "accuracy": 0.0,
+              "hi": 0.8,
+              "lo": 0.7,
+              "mean_confidence": 0.7557246387004852,
+              "n": 2
+            },
+            {
+              "accuracy": null,
+              "hi": 0.9,
+              "lo": 0.8,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 0.75,
+              "hi": 1.0,
+              "lo": 0.9,
+              "mean_confidence": 0.9881341382861137,
+              "n": 8
+            }
+          ],
+          "ece": 0.3542351871728897,
+          "n": 12
+        },
+        "latency": {
+          "n": 12,
+          "p50_s": 0.09179099393077195,
+          "p95_s": 0.17634947728365657
+        },
+        "latency_failures": {
+          "n": 0,
+          "p50_s": null,
+          "p95_s": null
+        },
+        "n_attempted": 12,
+        "n_correct": 8,
+        "n_planned": 12,
+        "n_renormalized": 0,
+        "n_scorable": 12,
+        "n_valid": 12,
+        "operational_success": 1.0,
+        "ordinal_mae": null,
+        "price_per_1000_decisions_usd": null,
+        "schema_validity": 1.0,
+        "schema_validity_strict": 1.0
+      },
+      "routing": {
+        "accuracy": 0.8333333333333334,
+        "brier_mean": 0.24032512955826416,
+        "calibration_n": 12,
+        "cost_basis": [
+          "local_gpu_no_provider_tariff"
+        ],
+        "coverage": 1.0,
+        "ece": {
+          "bins": [
+            {
+              "accuracy": null,
+              "hi": 0.1,
+              "lo": 0.0,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.2,
+              "lo": 0.1,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.3,
+              "lo": 0.2,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.4,
+              "lo": 0.3,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 0.0,
+              "hi": 0.5,
+              "lo": 0.4,
+              "mean_confidence": 0.49998921155929565,
+              "n": 1
+            },
+            {
+              "accuracy": 0.5,
+              "hi": 0.6,
+              "lo": 0.5,
+              "mean_confidence": 0.5483307540416718,
+              "n": 2
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.7,
+              "lo": 0.6,
+              "mean_confidence": 0.637911856174469,
+              "n": 1
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.8,
+              "lo": 0.7,
+              "mean_confidence": 0.7353148460388184,
+              "n": 2
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 0.9,
+              "lo": 0.8,
+              "mean_confidence": 0.819983184337616,
+              "n": 2
+            },
+            {
+              "accuracy": 1.0,
+              "hi": 1.0,
+              "lo": 0.9,
+              "mean_confidence": 0.9291361421346664,
+              "n": 4
+            }
+          ],
+          "ece": 0.17763318618138627,
+          "n": 12
+        },
+        "latency": {
+          "n": 12,
+          "p50_s": 0.04560648952610791,
+          "p95_s": 0.08044822961091994
+        },
+        "latency_failures": {
+          "n": 0,
+          "p50_s": null,
+          "p95_s": null
+        },
+        "n_attempted": 12,
+        "n_correct": 10,
+        "n_planned": 12,
+        "n_renormalized": 0,
+        "n_scorable": 12,
+        "n_valid": 12,
+        "operational_success": 1.0,
+        "ordinal_mae": null,
+        "price_per_1000_decisions_usd": null,
+        "schema_validity": 1.0,
+        "schema_validity_strict": 1.0
+      }
+    },
+    "price_per_1000_decisions_usd": null,
+    "probability_sources": [
+      "native"
+    ],
+    "schema_validity": 1.0,
+    "schema_validity_strict": 1.0,
+    "splits": {
+      "public": {
+        "accuracy": 0.6944444444444444,
+        "brier_mean": 0.39750463081241993,
+        "calibration_n": 72,
+        "cost_basis": [
+          "local_gpu_no_provider_tariff"
+        ],
+        "coverage": 1.0,
+        "ece": {
+          "bins": [
+            {
+              "accuracy": null,
+              "hi": 0.1,
+              "lo": 0.0,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.2,
+              "lo": 0.1,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.3,
+              "lo": 0.2,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": null,
+              "hi": 0.4,
+              "lo": 0.3,
+              "mean_confidence": null,
+              "n": 0
+            },
+            {
+              "accuracy": 0.4,
+              "hi": 0.5,
+              "lo": 0.4,
+              "mean_confidence": 0.45405682921409607,
+              "n": 5
+            },
+            {
+              "accuracy": 0.25,
+              "hi": 0.6,
+              "lo": 0.5,
+              "mean_confidence": 0.538318544626236,
+              "n": 8
+            },
+            {
+              "accuracy": 0.4,
+              "hi": 0.7,
+              "lo": 0.6,
+              "mean_confidence": 0.6290562272071838,
+              "n": 5
+            },
+            {
+              "accuracy": 0.7272727272727273,
+              "hi": 0.8,
+              "lo": 0.7,
+              "mean_confidence": 0.7492249011993408,
+              "n": 11
+            },
+            {
+              "accuracy": 0.7,
+              "hi": 0.9,
+              "lo": 0.8,
+              "mean_confidence": 0.8450712203979492,
+              "n": 10
+            },
+            {
+              "accuracy": 0.8787878787878788,
+              "hi": 1.0,
+              "lo": 0.9,
+              "mean_confidence": 0.9637182950973511,
+              "n": 33
+            }
+          ],
+          "ece": 0.11412504853473769,
+          "n": 72
+        },
+        "latency": {
+          "n": 72,
+          "p50_s": 0.06066564843058586,
+          "p95_s": 0.12478509345091895
+        },
+        "latency_failures": {
+          "n": 0,
+          "p50_s": null,
+          "p95_s": null
+        },
+        "n_attempted": 72,
+        "n_correct": 50,
+        "n_planned": 72,
+        "n_renormalized": 0,
+        "n_scorable": 72,
+        "n_valid": 72,
+        "operational_success": 1.0,
+        "ordinal_mae": 0.556312729138881,
+        "price_per_1000_decisions_usd": null,
+        "schema_validity": 1.0,
+        "schema_validity_strict": 1.0
+      }
+    },
+    "version": "jevbench-v1"
+  }
+}

probe_results.json

@@ -0,0 +1,155 @@
+{
+  "T": 1.1242100350620867,
+  "null_offset": 0.0,
+  "calib_sets": [
+    "val"
+  ],
+  "val_nll": 0.37577103864577555,
+  "args": {
+    "name": "probe_ts1b",
+    "backbone": "Qwen/Qwen3-1.7B-Base",
+    "lora_layers": 8,
+    "lora_r": 16,
+    "steps": 12000,
+    "bs": 64,
+    "grad_accum": 1,
+    "lr": 0.0003,
+    "lora_lr": 0.0001,
+    "seed": 0,
+    "no_hybrid": false,
+    "no_cand_null": false,
+    "no_null": false,
+    "null": "factored",
+    "hard_only": false,
+    "mix": "full",
+    "eval_every": 4000,
+    "val_every": 1000,
+    "ckpt_every": 2000,
+    "eval_bs": 128,
+    "device": "auto",
+    "wandb": false,
+    "hf_repo": null,
+    "ckpt_upload": false,
+    "data": "data_probe",
+    "extra_data": null,
+    "eval_cap": null,
+    "calib_sets": null,
+    "family_weights": null,
+    "bucket_map": null,
+    "null_aug": null,
+    "smoke": false,
+    "eval_limit": 0,
+    "tap_layer": 20,
+    "zscore": false,
+    "eval_only": true,
+    "dump_logits": null,
+    "tower_d": 512,
+    "tower_layers": 2,
+    "tower_heads": 8,
+    "joint": false,
+    "listwise": false,
+    "extra_tap": 0,
+    "init_from": null,
+    "readout": "native",
+    "nc_head": "n3",
+    "nc_render": "letters_nonull",
+    "score_head": "choice",
+    "noul_head": "bern",
+    "qtype_filter": null,
+    "ordinal_smooth": 0.0,
+    "max_state": 1024,
+    "perm_lambda": 0.0,
+    "no_shuffle": false,
+    "zero_shot": false,
+    "shots": 0,
+    "cand_encoder": "backbone",
+    "head": "mlp",
+    "z_dim": 128,
+    "z_probes": 8,
+    "tiny_layers": 2,
+    "distill_alpha": 1.0,
+    "distill_beta": 0.0,
+    "distill_T": 2.0,
+    "delta_gamma": 0.0
+  },
+  "eval": {
+    "mmlu_choicesonly": {
+      "raw": {
+        "nll": 2.132304014679386,
+        "brier": 0.855733086948981,
+        "jsd": 0.4787010918172532,
+        "acc": 0.22666666666666666,
+        "acc_k": 0.22666666666666666,
+        "ece": 0.05177726411571105,
+        "sel_acc80": 0.25625,
+        "coverage": 0.9975,
+        "false_abstain": 0.0025,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 2.119891606203886,
+        "brier": 0.8533568513213663,
+        "jsd": 0.4808686117250174,
+        "acc": 0.22666666666666666,
+        "acc_k": 0.22666666666666666,
+        "ece": 0.03572918727373083,
+        "sel_acc80": 0.25625,
+        "coverage": 0.9975,
+        "false_abstain": 0.0025,
+        "null_recall": "n/a"
+      }
+    },
+    "mmlu_pro": {
+      "raw": {
+        "nll": 1.870684202959633,
+        "brier": 0.7778605752958676,
+        "jsd": 0.4253066819218272,
+        "acc": 0.3425,
+        "acc_k": 0.3433333333333333,
+        "ece": 0.04600312157223621,
+        "sel_acc80": 0.39166666666666666,
+        "coverage": 0.995,
+        "false_abstain": 0.005,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 1.8650622347991608,
+        "brier": 0.7769553003598431,
+        "jsd": 0.4300648104283429,
+        "acc": 0.3425,
+        "acc_k": 0.3433333333333333,
+        "ece": 0.04282940872013569,
+        "sel_acc80": 0.39166666666666666,
+        "coverage": 0.995,
+        "false_abstain": 0.005,
+        "null_recall": "n/a"
+      }
+    },
+    "mmlu_shuffledq": {
+      "raw": {
+        "nll": 2.2841442391906717,
+        "brier": 0.8968269319049337,
+        "jsd": 0.49137674516409957,
+        "acc": 0.21083333333333334,
+        "acc_k": 0.21666666666666667,
+        "ece": 0.09170151713614662,
+        "sel_acc80": 0.23125,
+        "coverage": 0.9775,
+        "false_abstain": 0.0225,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 2.2550489918898133,
+        "brier": 0.8906454120623264,
+        "jsd": 0.492517543247486,
+        "acc": 0.21083333333333334,
+        "acc_k": 0.21666666666666667,
+        "ece": 0.0703738091637691,
+        "sel_acc80": 0.23125,
+        "coverage": 0.9766666666666667,
+        "false_abstain": 0.023333333333333334,
+        "null_recall": "n/a"
+      }
+    }
+  }
+}

results.json

@@ -0,0 +1,2248 @@
+{
+  "T": 1.1242100350620867,
+  "null_offset": 0.0,
+  "calib_sets": [
+    "val"
+  ],
+  "val_nll": 0.37577103864577555,
+  "args": {
+    "name": "ts1b",
+    "backbone": "Qwen/Qwen3-1.7B-Base",
+    "lora_layers": 8,
+    "lora_r": 16,
+    "steps": 12000,
+    "bs": 64,
+    "grad_accum": 1,
+    "lr": 0.0003,
+    "lora_lr": 0.0001,
+    "seed": 0,
+    "no_hybrid": false,
+    "no_cand_null": false,
+    "no_null": false,
+    "null": "factored",
+    "hard_only": false,
+    "mix": "full",
+    "eval_every": 4000,
+    "val_every": 1000,
+    "ckpt_every": 2000,
+    "eval_bs": 128,
+    "device": "auto",
+    "wandb": false,
+    "hf_repo": null,
+    "ckpt_upload": false,
+    "data": "data_v5",
+    "extra_data": "data_kb,data_wf,data_wf_hf,data_wf_long,data_wh,data_u",
+    "eval_cap": 1500,
+    "calib_sets": null,
+    "family_weights": "E:0.40,K:0.15,W:0.35,U:0.10",
+    "bucket_map": "data_wf_long=W,data_wh=W,data_u=U",
+    "null_aug": "W:0.20",
+    "smoke": false,
+    "eval_limit": 0,
+    "tap_layer": 20,
+    "zscore": false,
+    "eval_only": true,
+    "dump_logits": null,
+    "tower_d": 512,
+    "tower_layers": 2,
+    "tower_heads": 8,
+    "joint": false,
+    "listwise": false,
+    "extra_tap": 0,
+    "init_from": null,
+    "readout": "native",
+    "nc_head": "n3",
+    "nc_render": "letters_nonull",
+    "score_head": "choice",
+    "noul_head": "bern",
+    "qtype_filter": null,
+    "ordinal_smooth": 0.0,
+    "max_state": 1024,
+    "perm_lambda": 0.0,
+    "no_shuffle": false,
+    "zero_shot": false,
+    "shots": 0,
+    "cand_encoder": "backbone",
+    "head": "mlp",
+    "z_dim": 128,
+    "z_probes": 8,
+    "tiny_layers": 2,
+    "distill_alpha": 1.0,
+    "distill_beta": 0.0,
+    "distill_T": 2.0,
+    "delta_gamma": 0.0
+  },
+  "eval": {
+    "anli_test": {
+      "raw": {
+        "nll": 1.1289180740916946,
+        "brier": 0.6692979299798677,
+        "jsd": 0.2759239949154016,
+        "acc": 0.496875,
+        "acc_k": 0.496875,
+        "ece": 0.19597449547611173,
+        "sel_acc80": 0.516015625,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 1.0832124084790902,
+        "brier": 0.6531642279580523,
+        "jsd": 0.2744905806400761,
+        "acc": 0.496875,
+        "acc_k": 0.496875,
+        "ece": 0.1716771886870265,
+        "sel_acc80": 0.51640625,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      }
+    },
+    "banking77_k": {
+      "raw": {
+        "nll": 0.8717148169361592,
+        "brier": 0.4279012928249291,
+        "jsd": 0.1819415852368955,
+        "acc": 0.7100130039011704,
+        "acc_k": 0.8606340819022457,
+        "ece": 0.10486168391101536,
+        "sel_acc80": 0.7724502234863877,
+        "conf_wrong": 0.09282970550576185,
+        "coverage": 0.6501950585175553,
+        "false_abstain": 0.08586525759577279,
+        "null_recall": 0.6056338028169014,
+        "auroc_null": 0.8788824577101132,
+        "auroc_conf": 0.8729193341869398
+      },
+      "scaled": {
+        "nll": 0.8565775965678362,
+        "brier": 0.42266858395142376,
+        "jsd": 0.18248111365256584,
+        "acc": 0.711963589076723,
+        "acc_k": 0.8606340819022457,
+        "ece": 0.09658078322252464,
+        "sel_acc80": 0.7757009345794392,
+        "conf_wrong": 0.08450704225352113,
+        "coverage": 0.6449934980494147,
+        "false_abstain": 0.08982826948480846,
+        "null_recall": 0.6120358514724712,
+        "auroc_null": 0.8788824577101132,
+        "auroc_conf": 0.8727353915736523
+      }
+    },
+    "banking77_test": {
+      "raw": {
+        "nll": 1.8774157424809907,
+        "brier": 0.5718742123352633,
+        "jsd": 0.2792573519790713,
+        "acc": 0.5929778933680104,
+        "acc_k": 0.6134590377113134,
+        "ece": 0.11646726103535868,
+        "sel_acc80": 0.6834620073141,
+        "coverage": 0.9307542262678804,
+        "false_abstain": 0.06924577373211964,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 1.7716108606732313,
+        "brier": 0.565376058808758,
+        "jsd": 0.28325008418116127,
+        "acc": 0.5923276983094928,
+        "acc_k": 0.6134590377113134,
+        "ece": 0.09275831222069124,
+        "sel_acc80": 0.6822429906542056,
+        "coverage": 0.9245773732119635,
+        "false_abstain": 0.0754226267880364,
+        "null_recall": "n/a"
+      }
+    },
+    "boolq_val": {
+      "raw": {
+        "nll": 0.43645680106840656,
+        "brier": 0.2618663419126855,
+        "jsd": 0.11080636584099614,
+        "acc": 0.8235474006116208,
+        "acc_k": 0.825993883792049,
+        "ece": 0.047257785731499316,
+        "sel_acc80": 0.8799694189602446,
+        "coverage": 0.9957186544342508,
+        "false_abstain": 0.004281345565749235,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 0.42464742925907756,
+        "brier": 0.2590639951058971,
+        "jsd": 0.11250903904001809,
+        "acc": 0.8235474006116208,
+        "acc_k": 0.825993883792049,
+        "ece": 0.03589964046937609,
+        "sel_acc80": 0.8799694189602446,
+        "coverage": 0.9957186544342508,
+        "false_abstain": 0.004281345565749235,
+        "null_recall": "n/a"
+      }
+    },
+    "chaos_mnli": {
+      "raw": {
+        "nll": 1.2939700895771997,
+        "brier": 0.24465358531101472,
+        "jsd": 0.0949094297745448
+      },
+      "scaled": {
+        "nll": 1.212854225335715,
+        "brier": 0.22637623031446555,
+        "jsd": 0.08678660793866103
+      }
+    },
+    "clinc_heldout": {
+      "raw": {
+        "nll": 0.23143039751182512,
+        "brier": 0.09878319172558447,
+        "jsd": 0.04650508883232555,
+        "acc": 0.9320712694877505,
+        "acc_k": 0.9632516703786191,
+        "ece": 0.024297224585887893,
+        "sel_acc80": 0.9958275382475661,
+        "coverage": 0.9576837416481069,
+        "false_abstain": 0.042316258351893093,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 0.2261652916434549,
+        "brier": 0.09926372397049718,
+        "jsd": 0.04731882559377883,
+        "acc": 0.9320712694877505,
+        "acc_k": 0.9632516703786191,
+        "ece": 0.024705316879701483,
+        "sel_acc80": 0.9958275382475661,
+        "coverage": 0.9565701559020044,
+        "false_abstain": 0.043429844097995544,
+        "null_recall": "n/a"
+      }
+    },
+    "clinc_k": {
+      "raw": {
+        "nll": 0.3231793638375014,
+        "brier": 0.1685297802997737,
+        "jsd": 0.07669964194250567,
+        "acc": 0.8873651079136691,
+        "acc_k": 0.9664729548502459,
+        "ece": 0.018513263035699633,
+        "sel_acc80": 0.9530767069401517,
+        "conf_wrong": 0.04115784712799638,
+        "coverage": 0.5613758992805755,
+        "false_abstain": 0.042914617791685296,
+        "null_recall": 0.8389868837630031,
+        "auroc_null": 0.9699786110290582,
+        "auroc_conf": 0.9684840721009897
+      },
+      "scaled": {
+        "nll": 0.3216351683687122,
+        "brier": 0.16711785999498344,
+        "jsd": 0.07704117516737093,
+        "acc": 0.8887140287769785,
+        "acc_k": 0.9664729548502459,
+        "ece": 0.01572394735856555,
+        "sel_acc80": 0.9530767069401517,
+        "conf_wrong": 0.03844414292175486,
+        "coverage": 0.5584532374100719,
+        "false_abstain": 0.044702726866338846,
+        "null_recall": 0.8430574400723655,
+        "auroc_null": 0.9699786110290585,
+        "auroc_conf": 0.9683471940092281
+      }
+    },
+    "clinc_oos": {
+      "raw": {
+        "nll": 0.5756657426096591,
+        "brier": 0.2989822245764066,
+        "jsd": 0.16344119860379913,
+        "acc": 0.8268268268268268,
+        "ece": 0.14330651938497604,
+        "sel_acc80": 0.8525,
+        "conf_wrong": 0.02002002002002002,
+        "coverage": 0.17317317317317316,
+        "false_abstain": "n/a",
+        "null_recall": 0.8268268268268268
+      },
+      "scaled": {
+        "nll": 0.5756657362777501,
+        "brier": 0.28923916904188607,
+        "jsd": 0.1634411964162575,
+        "acc": 0.8328328328328328,
+        "ece": 0.15310207864006717,
+        "sel_acc80": 0.8625,
+        "conf_wrong": 0.02002002002002002,
+        "coverage": 0.16716716716716717,
+        "false_abstain": "n/a",
+        "null_recall": 0.8328328328328328
+      }
+    },
+    "clinc_test": {
+      "raw": {
+        "nll": 0.7784299687760449,
+        "brier": 0.28365467007926637,
+        "jsd": 0.14319114862151858,
+        "acc": 0.8037320143884892,
+        "acc_k": 0.8340827338129496,
+        "ece": 0.028675001282110875,
+        "sel_acc80": 0.9033436358527677,
+        "coverage": 0.9332284172661871,
+        "false_abstain": 0.06677158273381295,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 0.7536637095175133,
+        "brier": 0.2846120396481659,
+        "jsd": 0.14652556864953595,
+        "acc": 0.8005845323741008,
+        "acc_k": 0.8340827338129496,
+        "ece": 0.02309598260820451,
+        "sel_acc80": 0.9013767912334926,
+        "coverage": 0.9280575539568345,
+        "false_abstain": 0.07194244604316546,
+        "null_recall": "n/a"
+      }
+    },
+    "cse_banking77": {
+      "raw": {
+        "nll": 0.9038375048530173,
+        "brier": 0.36887379787688285,
+        "jsd": 0.16302877525225848,
+        "acc": 0.7076666666666667,
+        "acc_k": 0.768,
+        "ece": 0.1064401980837186,
+        "sel_acc80": 0.78875,
+        "conf_wrong": 0.116,
+        "coverage": 0.8143333333333334,
+        "false_abstain": 0.0952,
+        "null_recall": 0.638,
+        "auroc_null": 0.8856160000000001,
+        "auroc_conf": 0.8466096000000001
+      },
+      "scaled": {
+        "nll": 0.8694398602835821,
+        "brier": 0.36427865239843843,
+        "jsd": 0.16326378234678152,
+        "acc": 0.7063333333333334,
+        "acc_k": 0.768,
+        "ece": 0.09903382284442586,
+        "sel_acc80": 0.7875,
+        "conf_wrong": 0.1,
+        "coverage": 0.8113333333333334,
+        "false_abstain": 0.0984,
+        "null_recall": 0.64,
+        "auroc_null": 0.8856160000000001,
+        "auroc_conf": 0.84588
+      },
+      "cse": {
+        "add_irr/p_irr": 0.0005295094670840995,
+        "add_irr/dlo_top2": 0.13645354047403815,
+        "remove_gold/dp_null": 0.4498149465108654,
+        "dup/mass_err": 0.0331556449693544,
+        "dup/slot_gap": 0.17666359726031608,
+        "dup/acc_dedup": 0.844,
+        "reorder/max_dp": 0.08150279471576005,
+        "near_dup/cluster_mass_err": 0.03125877748820778
+      }
+    },
+    "cse_clinc": {
+      "raw": {
+        "nll": 0.30950664569064634,
+        "brier": 0.12839356722975465,
+        "jsd": 0.05918977870640326,
+        "acc": 0.8746124698587668,
+        "acc_k": 0.9073535521395929,
+        "ece": 0.02284423466431781,
+        "sel_acc80": 0.9599655617735686,
+        "conf_wrong": 0.028225806451612902,
+        "coverage": 0.8284533241474337,
+        "false_abstain": 0.03489821354383049,
+        "null_recall": 0.8346774193548387,
+        "auroc_null": 0.9766046946406314,
+        "auroc_conf": 0.9623075170537546
+      },
+      "scaled": {
+        "nll": 0.3052672934138403,
+        "brier": 0.12660908099041643,
+        "jsd": 0.059311244384245086,
+        "acc": 0.875645883568722,
+        "acc_k": 0.9073535521395929,
+        "ece": 0.01759377376523214,
+        "sel_acc80": 0.9595350839431769,
+        "conf_wrong": 0.028225806451612902,
+        "coverage": 0.8274199104374784,
+        "false_abstain": 0.03489821354383049,
+        "null_recall": 0.8407258064516129,
+        "auroc_null": 0.9766046946406314,
+        "auroc_conf": 0.9617245399841859
+      },
+      "cse": {
+        "add_irr/p_irr": 0.001569051850666318,
+        "add_irr/dlo_top2": 0.16152436470361048,
+        "remove_gold/dp_null": 0.7188482229822042,
+        "dup/mass_err": 0.012801834568350948,
+        "dup/slot_gap": 0.16565532824922566,
+        "dup/acc_dedup": 0.9798387096774194,
+        "reorder/max_dp": 0.03195492150344274,
+        "near_dup/cluster_mass_err": 0.016617430864665605
+      }
+    },
+    "cse_hwu64": {
+      "raw": {
+        "nll": 0.5609047915995059,
+        "brier": 0.2507698667896729,
+        "jsd": 0.1167981555537652,
+        "acc": 0.7933333333333333,
+        "acc_k": 0.8428,
+        "ece": 0.04368377149105075,
+        "sel_acc80": 0.8616666666666667,
+        "conf_wrong": 0.068,
+        "coverage": 0.845,
+        "false_abstain": 0.0504,
+        "null_recall": 0.678,
+        "auroc_null": 0.9338304,
+        "auroc_conf": 0.89552
+      },
+      "scaled": {
+        "nll": 0.5507840743009456,
+        "brier": 0.24834142217870656,
+        "jsd": 0.11789211835745053,
+        "acc": 0.792,
+        "acc_k": 0.8428,
+        "ece": 0.0344938989629348,
+        "sel_acc80": 0.8608333333333333,
+        "conf_wrong": 0.062,
+        "coverage": 0.843,
+        "false_abstain": 0.0524,
+        "null_recall": 0.68,
+        "auroc_null": 0.9338304,
+        "auroc_conf": 0.8944143999999999
+      },
+      "cse": {
+        "add_irr/p_irr": 0.011361859426103703,
+        "add_irr/dlo_top2": 0.13446737565746777,
+        "remove_gold/dp_null": 0.4905430292236051,
+        "dup/mass_err": 0.027505983864113463,
+        "dup/slot_gap": 0.17891635448112775,
+        "dup/acc_dedup": 0.938,
+        "reorder/max_dp": 0.0636793900321718,
+        "near_dup/cluster_mass_err": 0.03313319766287608
+      }
+    },
+    "cse_snli": {
+      "raw": {
+        "nll": 0.40686523131470476,
+        "brier": 0.16987205012507944,
+        "jsd": 0.08462270844792318,
+        "acc": 0.8890134529147982,
+        "acc_k": 0.9016544117647058,
+        "ece": 0.07036731607087164,
+        "sel_acc80": 0.9271368519383466,
+        "conf_wrong": 0.032,
+        "coverage": 0.844170403587444,
+        "false_abstain": 0.0,
+        "null_recall": 0.834,
+        "auroc_null": 0.9999007352941176,
+        "auroc_conf": 0.9475193014705883
+      },
+      "scaled": {
+        "nll": 0.41134177711307934,
+        "brier": 0.17059116434583468,
+        "jsd": 0.08824297050300778,
+        "acc": 0.8893871449925261,
+        "acc_k": 0.9016544117647058,
+        "ece": 0.08119960839794711,
+        "sel_acc80": 0.9285380663241476,
+        "conf_wrong": 0.03,
+        "coverage": 0.843796711509716,
+        "false_abstain": 0.0,
+        "null_recall": 0.836,
+        "auroc_null": 0.9999007352941176,
+        "auroc_conf": 0.9443722426470589
+      },
+      "cse": {
+        "add_irr/p_irr": 0.01940807816808774,
+        "add_irr/dlo_top2": 0.09028009353347251,
+        "remove_gold/dp_null": 0.7515602824201983,
+        "dup/mass_err": 0.04235328816168476,
+        "dup/slot_gap": 0.1555379865783034,
+        "dup/acc_dedup": 0.904,
+        "reorder/max_dp": 0.01188999435165897,
+        "near_dup/cluster_mass_err": 0.04870520358202735
+      }
+    },
+    "hwu64_test": {
+      "raw": {
+        "nll": 0.9291563079845718,
+        "brier": 0.3372229757452212,
+        "jsd": 0.18254796335437554,
+        "acc": 0.7676579925650557,
+        "acc_k": 0.7843866171003717,
+        "ece": 0.03418452451940364,
+        "sel_acc80": 0.8641114982578397,
+        "coverage": 0.9563197026022305,
+        "false_abstain": 0.04368029739776952,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 0.9142610185219991,
+        "brier": 0.3406498241436756,
+        "jsd": 0.18930016280648101,
+        "acc": 0.7611524163568774,
+        "acc_k": 0.7843866171003717,
+        "ece": 0.04919961878690576,
+        "sel_acc80": 0.8652729384436701,
+        "coverage": 0.9451672862453532,
+        "false_abstain": 0.05483271375464684,
+        "null_recall": "n/a"
+      }
+    },
+    "ksweep_banking77": {
+      "raw": {
+        "nll": 0.9349302343615308,
+        "brier": 0.4471019757458735,
+        "jsd": 0.19401309938809777,
+        "acc": 0.68875,
+        "acc_k": 0.8521428571428571,
+        "ece": 0.11756176422217063,
+        "sel_acc80": 0.7609375,
+        "conf_wrong": 0.08214285714285714,
+        "coverage": 0.6689285714285714,
+        "false_abstain": 0.08392857142857142,
+        "null_recall": 0.5782142857142857,
+        "auroc_null": 0.8703889030612244,
+        "auroc_conf": 0.8664147959183673
+      },
+      "scaled": {
+        "nll": 0.9189812145302394,
+        "brier": 0.4404521904122212,
+        "jsd": 0.19510637309634174,
+        "acc": 0.6907142857142857,
+        "acc_k": 0.8521428571428571,
+        "ece": 0.10724775899201631,
+        "sel_acc80": 0.7647321428571429,
+        "conf_wrong": 0.07571428571428572,
+        "coverage": 0.6648214285714286,
+        "false_abstain": 0.08714285714285715,
+        "null_recall": 0.5832142857142857,
+        "auroc_null": 0.8703889030612244,
+        "auroc_conf": 0.8663109056122449
+      },
+      "ksweep": {
+        "K2/p_null_absent": 0.8690049877914134,
+        "K2/p_null_present": 0.1643589718966723,
+        "K2/acc": 0.975,
+        "K2/auroc": 0.9450375,
+        "K3/p_null_absent": 0.7012581462779781,
+        "K3/p_null_present": 0.08284207384213005,
+        "K3/acc": 0.96,
+        "K3/auroc": 0.9444312499999999,
+        "K5/p_null_absent": 0.6039111433527432,
+        "K5/p_null_present": 0.08178361789279734,
+        "K5/acc": 0.935,
+        "K5/auroc": 0.9240437499999999,
+        "K10/p_null_absent": 0.5488354222249473,
+        "K10/p_null_present": 0.10999041035836854,
+        "K10/acc": 0.89,
+        "K10/auroc": 0.8832062500000001,
+        "K20/p_null_absent": 0.46607297291746363,
+        "K20/p_null_present": 0.13052993028948548,
+        "K20/acc": 0.84,
+        "K20/auroc": 0.8385499999999998,
+        "K50/p_null_absent": 0.2712666229833849,
+        "K50/p_null_present": 0.10264942378737033,
+        "K50/acc": 0.715,
+        "K50/auroc": 0.7575625,
+        "K77/p_null_absent": 0.22605883536394686,
+        "K77/p_null_present": 0.10786349694710225,
+        "K77/acc": 0.65,
+        "K77/auroc": 0.7228375,
+        "p_null_absent_range": 0.6429461524274666
+      }
+    },
+    "ksweep_clinc": {
+      "raw": {
+        "nll": 0.4705098322558086,
+        "brier": 0.24142269373822897,
+        "jsd": 0.10927668251896748,
+        "acc": 0.8366228070175439,
+        "acc_k": 0.943295739348371,
+        "ece": 0.03536469359791146,
+        "sel_acc80": 0.9007439310884886,
+        "conf_wrong": 0.05169172932330827,
+        "coverage": 0.5897556390977443,
+        "false_abstain": 0.05263157894736842,
+        "null_recall": 0.7678571428571429,
+        "auroc_null": 0.9494151654198153,
+        "auroc_conf": 0.9456508596051532
+      },
+      "scaled": {
+        "nll": 0.4667588320710177,
+        "brier": 0.23902214623321588,
+        "jsd": 0.11002446956487777,
+        "acc": 0.837875939849624,
+        "acc_k": 0.943295739348371,
+        "ece": 0.03253249252649176,
+        "sel_acc80": 0.9025058731401723,
+        "conf_wrong": 0.04887218045112782,
+        "coverage": 0.5850563909774436,
+        "false_abstain": 0.05670426065162907,
+        "null_recall": 0.7731829573934837,
+        "auroc_null": 0.9494151654198151,
+        "auroc_conf": 0.9453110768776578
+      },
+      "ksweep": {
+        "K2/p_null_absent": 0.9109689841002581,
+        "K2/p_null_present": 0.05722885545324601,
+        "K2/acc": 0.9974937343358395,
+        "K2/auroc": 0.9872048542408653,
+        "K3/p_null_absent": 0.8574261630425477,
+        "K3/p_null_present": 0.03833966100025525,
+        "K3/acc": 0.9949874686716792,
+        "K3/auroc": 0.9842714555813091,
+        "K5/p_null_absent": 0.8189782498793055,
+        "K5/p_null_present": 0.04019082001239549,
+        "K5/acc": 0.9899749373433584,
+        "K5/auroc": 0.9828707106111142,
+        "K10/p_null_absent": 0.7780514757080648,
+        "K10/p_null_present": 0.06639160508924383,
+        "K10/acc": 0.9799498746867168,
+        "K10/auroc": 0.9722363552992757,
+        "K20/p_null_absent": 0.7468801522364882,
+        "K20/p_null_present": 0.10790303507195388,
+        "K20/acc": 0.9649122807017544,
+        "K20/auroc": 0.9462566189910867,
+        "K50/p_null_absent": 0.6084068263040617,
+        "K50/p_null_present": 0.11852430529603011,
+        "K50/acc": 0.924812030075188,
+        "K50/auroc": 0.9115206562772847,
+        "K100/p_null_absent": 0.4608676842318143,
+        "K100/p_null_present": 0.10504461155228671,
+        "K100/acc": 0.8972431077694235,
+        "K100/auroc": 0.8819479777137078,
+        "K150/p_null_absent": 0.3821950357776126,
+        "K150/p_null_present": 0.13089805873773785,
+        "K150/acc": 0.7969924812030075,
+        "K150/auroc": 0.8273377679788443,
+        "p_null_absent_range": 0.5287739483226455
+      }
+    },
+    "mmlu_cf": {
+      "raw": {
+        "nll": 1.9652890876376825,
+        "brier": 0.7101132646736971,
+        "jsd": 0.40397400241739434,
+        "acc": 0.34933333333333333,
+        "acc_k": 0.36,
+        "ece": 0.06165287426610789,
+        "sel_acc80": 0.4008333333333333,
+        "coverage": 0.9833333333333333,
+        "false_abstain": 0.016666666666666666,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 1.9540729414231996,
+        "brier": 0.7075911562875681,
+        "jsd": 0.40846717789843123,
+        "acc": 0.34933333333333333,
+        "acc_k": 0.36,
+        "ece": 0.05156281360983849,
+        "sel_acc80": 0.40166666666666667,
+        "coverage": 0.9833333333333333,
+        "false_abstain": 0.016666666666666666,
+        "null_recall": "n/a"
+      },
+      "cf": {
+        "orig/acc": 0.376,
+        "orig/among_k": 0.376,
+        "remove3/acc": 0.376,
+        "remove3/among_k": 0.432,
+        "add3_unrel/acc": 0.356,
+        "add3_unrel/among_k": 0.356,
+        "replace_hard/acc": 0.372,
+        "replace_hard/among_k": 0.376,
+        "add_neardup/acc": 0.272,
+        "add_neardup/among_k": 0.276,
+        "reorder/acc": 0.344,
+        "reorder/among_k": 0.344,
+        "add3_unrel/dlo_top2": 0.13401622239778724,
+        "reorder/max_dp": 0.12416648952616378,
+        "add_neardup/slot_gap": 0.12370567691186443,
+        "add_neardup/cluster_mass_err": 0.09759602142916993
+      }
+    },
+    "mmlu_cf_teacher": {
+      "raw": {
+        "nll": 1.9799224626151832,
+        "brier": 0.718764453695287,
+        "jsd": 0.411908770920552,
+        "acc": 0.33555555555555555,
+        "acc_k": 0.3458333333333333,
+        "ece": 0.04779739408546853,
+        "sel_acc80": 0.3767361111111111,
+        "coverage": 0.9838888888888889,
+        "false_abstain": 0.01611111111111111,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 1.9711311573138646,
+        "brier": 0.716484928918705,
+        "jsd": 0.4160687618841204,
+        "acc": 0.33555555555555555,
+        "acc_k": 0.3458333333333333,
+        "ece": 0.036748226951393806,
+        "sel_acc80": 0.3770833333333333,
+        "coverage": 0.9838888888888889,
+        "false_abstain": 0.01611111111111111,
+        "null_recall": "n/a"
+      },
+      "cf": {
+        "orig/acc": 0.3425,
+        "orig/among_k": 0.3433333333333333,
+        "orig/kl_teacher": 0.25423722638876495,
+        "remove3/acc": 0.37916666666666665,
+        "remove3/among_k": 0.4325,
+        "remove3/kl_teacher": 0.9529805054700989,
+        "add3_unrel/acc": 0.34,
+        "add3_unrel/among_k": 0.3408333333333333,
+        "add3_unrel/kl_teacher": 0.32834803689669667,
+        "replace_hard/acc": 0.3475,
+        "replace_hard/among_k": 0.35,
+        "replace_hard/kl_teacher": 0.2868563655580847,
+        "add_neardup/acc": 0.26,
+        "add_neardup/among_k": 0.26166666666666666,
+        "add_neardup/kl_teacher": 0.2687358245348599,
+        "reorder/acc": 0.3441666666666667,
+        "reorder/among_k": 0.3466666666666667,
+        "reorder/kl_teacher": 0.24609989547374833,
+        "remove3/delta_mae": 1.1517046652173903,
+        "remove3/delta_teacher_abs": 1.135840148062828,
+        "add3_unrel/delta_mae": 0.7575456402215782,
+        "add3_unrel/delta_teacher_abs": 0.7390631803199154,
+        "replace_hard/delta_mae": 0.7262603232459742,
+        "replace_hard/delta_teacher_abs": 0.6681882120617108,
+        "add_neardup/delta_mae": 0.7387218217655764,
+        "add_neardup/delta_teacher_abs": 0.7305883395327554,
+        "reorder/delta_mae": 0.9534334357618881,
+        "reorder/delta_teacher_abs": 0.7763704285195197,
+        "add3_unrel/dlo_top2": 0.1328711094665931,
+        "reorder/max_dp": 0.11963965538597525,
+        "add_neardup/slot_gap": 0.10855303687150201,
+        "add_neardup/cluster_mass_err": 0.0905932489552591
+      }
+    },
+    "mmlu_choicesonly": {
+      "raw": {
+        "nll": 2.132304014679386,
+        "brier": 0.855733086948981,
+        "jsd": 0.4787010918172532,
+        "acc": 0.22666666666666666,
+        "acc_k": 0.22666666666666666,
+        "ece": 0.05177726411571105,
+        "sel_acc80": 0.25625,
+        "coverage": 0.9975,
+        "false_abstain": 0.0025,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 2.119891606203886,
+        "brier": 0.8533568513213663,
+        "jsd": 0.4808686117250174,
+        "acc": 0.22666666666666666,
+        "acc_k": 0.22666666666666666,
+        "ece": 0.03572918727373083,
+        "sel_acc80": 0.25625,
+        "coverage": 0.9975,
+        "false_abstain": 0.0025,
+        "null_recall": "n/a"
+      }
+    },
+    "mmlu_pro": {
+      "raw": {
+        "nll": 1.870684202959633,
+        "brier": 0.7778605752958676,
+        "jsd": 0.4253066819218272,
+        "acc": 0.3425,
+        "acc_k": 0.3433333333333333,
+        "ece": 0.04600312157223621,
+        "sel_acc80": 0.39166666666666666,
+        "coverage": 0.995,
+        "false_abstain": 0.005,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 1.8650622347991608,
+        "brier": 0.7769553003598431,
+        "jsd": 0.4300648104283429,
+        "acc": 0.3425,
+        "acc_k": 0.3433333333333333,
+        "ece": 0.04282940872013569,
+        "sel_acc80": 0.39166666666666666,
+        "coverage": 0.995,
+        "false_abstain": 0.005,
+        "null_recall": "n/a"
+      }
+    },
+    "mmlu_shuffledq": {
+      "raw": {
+        "nll": 2.2841442391906717,
+        "brier": 0.8968269319049337,
+        "jsd": 0.49137674516409957,
+        "acc": 0.21083333333333334,
+        "acc_k": 0.21666666666666667,
+        "ece": 0.09170151713614662,
+        "sel_acc80": 0.23125,
+        "coverage": 0.9775,
+        "false_abstain": 0.0225,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 2.2550489918898133,
+        "brier": 0.8906454120623264,
+        "jsd": 0.492517543247486,
+        "acc": 0.21083333333333334,
+        "acc_k": 0.21666666666666667,
+        "ece": 0.0703738091637691,
+        "sel_acc80": 0.23125,
+        "coverage": 0.9766666666666667,
+        "false_abstain": 0.023333333333333334,
+        "null_recall": "n/a"
+      }
+    },
+    "mnli_val": {
+      "raw": {
+        "nll": 0.38298786152412717,
+        "brier": 0.2116872493208543,
+        "jsd": 0.09760297139339706,
+        "acc": 0.8591836734693877,
+        "acc_k": 0.8591836734693877,
+        "ece": 0.016026962665068034,
+        "sel_acc80": 0.9226190476190477,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 0.3825671840790251,
+        "brier": 0.21233959847225758,
+        "jsd": 0.1018273888841336,
+        "acc": 0.8591836734693877,
+        "acc_k": 0.8591836734693877,
+        "ece": 0.02029442191123957,
+        "sel_acc80": 0.9226190476190477,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      }
+    },
+    "ng20_test": {
+      "raw": {
+        "nll": 1.594920015134779,
+        "brier": 0.6067909948170968,
+        "jsd": 0.30844571120119146,
+        "acc": 0.5449077238550922,
+        "acc_k": 0.5807245386192754,
+        "ece": 0.07311073689328751,
+        "sel_acc80": 0.6211551606288448,
+        "coverage": 0.8474367737525632,
+        "false_abstain": 0.15256322624743676,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 1.56044065734942,
+        "brier": 0.6067970962992972,
+        "jsd": 0.31531418996922944,
+        "acc": 0.5403964456596035,
+        "acc_k": 0.5807245386192754,
+        "ece": 0.051888617539112475,
+        "sel_acc80": 0.6134654818865345,
+        "coverage": 0.833902939166097,
+        "false_abstain": 0.16609706083390294,
+        "null_recall": "n/a"
+      }
+    },
+    "null_irrq_intent": {
+      "raw": {
+        "nll": 0.28509880355066847,
+        "brier": 0.12926910897196567,
+        "jsd": 0.07821245099591534,
+        "acc": 0.9241658240647118,
+        "acc_k": 0.9683673469387755,
+        "ece": 0.07439453771791758,
+        "sel_acc80": 0.9747315224257739,
+        "conf_wrong": 0.001002004008016032,
+        "coverage": 0.519211324570273,
+        "false_abstain": 0.04285714285714286,
+        "null_recall": 0.9108216432865731,
+        "auroc_null": 0.9787196842664921,
+        "auroc_conf": 0.9848288413561817
+      },
+      "scaled": {
+        "nll": 0.28319577466198464,
+        "brier": 0.1267643414616399,
+        "jsd": 0.07851246372610352,
+        "acc": 0.9282103134479271,
+        "acc_k": 0.9683673469387755,
+        "ece": 0.08138514553451921,
+        "sel_acc80": 0.9766266582438408,
+        "conf_wrong": 0.0,
+        "coverage": 0.5146612740141557,
+        "false_abstain": 0.04387755102040816,
+        "null_recall": 0.9188376753507014,
+        "auroc_null": 0.9787196842664921,
+        "auroc_conf": 0.9855026379289191
+      }
+    },
+    "null_irrq_squad": {
+      "raw": {
+        "nll": 0.10103190551973668,
+        "brier": 0.05798018973415887,
+        "jsd": 0.029012514334021317,
+        "acc": 0.9585,
+        "acc_k": 0.995,
+        "ece": 0.011917174428701351,
+        "sel_acc80": 0.998125,
+        "conf_wrong": 0.007,
+        "coverage": 0.522,
+        "false_abstain": 0.017,
+        "null_recall": 0.939,
+        "auroc_null": 0.996387,
+        "auroc_conf": 0.9962260000000001
+      },
+      "scaled": {
+        "nll": 0.1011740032118126,
+        "brier": 0.05775404134531006,
+        "jsd": 0.02910070699993618,
+        "acc": 0.959,
+        "acc_k": 0.995,
+        "ece": 0.011337469652295146,
+        "sel_acc80": 0.99875,
+        "conf_wrong": 0.007,
+        "coverage": 0.5215000000000001,
+        "false_abstain": 0.017,
+        "null_recall": 0.94,
+        "auroc_null": 0.996387,
+        "auroc_conf": 0.99622
+      }
+    },
+    "null_nearmiss_banking77": {
+      "raw": {
+        "nll": 0.9477015185824751,
+        "brier": 0.48575886111373834,
+        "jsd": 0.20246781100242447,
+        "acc": 0.672,
+        "acc_k": 0.88,
+        "ece": 0.14288236629962925,
+        "sel_acc80": 0.71625,
+        "conf_wrong": 0.12,
+        "coverage": 0.7090000000000001,
+        "false_abstain": 0.078,
+        "null_recall": 0.504,
+        "auroc_null": 0.8642439999999999,
+        "auroc_conf": 0.848276
+      },
+      "scaled": {
+        "nll": 0.9372510745375565,
+        "brier": 0.4799510588310396,
+        "jsd": 0.20350421208345068,
+        "acc": 0.674,
+        "acc_k": 0.88,
+        "ece": 0.13095481739938258,
+        "sel_acc80": 0.71875,
+        "conf_wrong": 0.116,
+        "coverage": 0.706,
+        "false_abstain": 0.08,
+        "null_recall": 0.508,
+        "auroc_null": 0.8642439999999999,
+        "auroc_conf": 0.847356
+      }
+    },
+    "null_nearmiss_clinc": {
+      "raw": {
+        "nll": 0.7261187274036407,
+        "brier": 0.382317397417127,
+        "jsd": 0.1502412147763253,
+        "acc": 0.756838905775076,
+        "acc_k": 0.9494949494949495,
+        "ece": 0.1270041166528259,
+        "sel_acc80": 0.7949367088607595,
+        "conf_wrong": 0.17886178861788618,
+        "coverage": 0.6808510638297872,
+        "false_abstain": 0.04242424242424243,
+        "null_recall": 0.5975609756097561,
+        "auroc_null": 0.9221318879855465,
+        "auroc_conf": 0.89660014781966
+      },
+      "scaled": {
+        "nll": 0.7226177400781414,
+        "brier": 0.3803847641898642,
+        "jsd": 0.15083584162884187,
+        "acc": 0.7588652482269503,
+        "acc_k": 0.9494949494949495,
+        "ece": 0.1236053464260507,
+        "sel_acc80": 0.7949367088607595,
+        "conf_wrong": 0.17886178861788618,
+        "coverage": 0.6788247213779128,
+        "false_abstain": 0.04242424242424243,
+        "null_recall": 0.6016260162601627,
+        "auroc_null": 0.9221318879855465,
+        "auroc_conf": 0.8943048369877638
+      }
+    },
+    "null_nearmiss_hwu64": {
+      "raw": {
+        "nll": 0.8017006188180456,
+        "brier": 0.44291562677417917,
+        "jsd": 0.18860549976513902,
+        "acc": 0.689,
+        "acc_k": 0.906,
+        "ece": 0.11632219871878623,
+        "sel_acc80": 0.74125,
+        "conf_wrong": 0.12,
+        "coverage": 0.728,
+        "false_abstain": 0.048,
+        "null_recall": 0.496,
+        "auroc_null": 0.8993880000000001,
+        "auroc_conf": 0.8742839999999998
+      },
+      "scaled": {
+        "nll": 0.8007376150333095,
+        "brier": 0.43729978598013314,
+        "jsd": 0.1897504311153405,
+        "acc": 0.689,
+        "acc_k": 0.906,
+        "ece": 0.10554402610659597,
+        "sel_acc80": 0.745,
+        "conf_wrong": 0.108,
+        "coverage": 0.725,
+        "false_abstain": 0.052,
+        "null_recall": 0.498,
+        "auroc_null": 0.8993880000000001,
+        "auroc_conf": 0.8721200000000001
+      }
+    },
+    "snli_null": {
+      "raw": {
+        "nll": 0.36114479966903223,
+        "brier": 0.19934539594747502,
+        "jsd": 0.09212641990572218,
+        "acc": 0.8649027861294849,
+        "acc_k": 0.9433198380566802,
+        "ece": 0.014997484241226794,
+        "sel_acc80": 0.9321142284569138,
+        "conf_wrong": 0.03136165144898769,
+        "coverage": 0.581479254359591,
+        "false_abstain": 0.022672064777327937,
+        "null_recall": 0.8066693132195316,
+        "auroc_null": 0.9770666015207499,
+        "auroc_conf": 0.9634363453140746
+      },
+      "scaled": {
+        "nll": 0.36211979227144847,
+        "brier": 0.19853896379199165,
+        "jsd": 0.09328289433312073,
+        "acc": 0.8653036680697535,
+        "acc_k": 0.9433198380566802,
+        "ece": 0.014350581560996995,
+        "sel_acc80": 0.9333667334669339,
+        "conf_wrong": 0.02818578801111552,
+        "coverage": 0.5810783724193225,
+        "false_abstain": 0.022672064777327937,
+        "null_recall": 0.8074632790789996,
+        "auroc_null": 0.9770666015207501,
+        "auroc_conf": 0.9622099573604974
+      }
+    },
+    "snli_test": {
+      "raw": {
+        "nll": 0.284213361907045,
+        "brier": 0.15704357356528473,
+        "jsd": 0.07379276085972616,
+        "acc": 0.894167167769092,
+        "acc_k": 0.894167167769092,
+        "ece": 0.008250219062326106,
+        "sel_acc80": 0.9561623246492986,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 0.2871341337130327,
+        "brier": 0.1579343204560731,
+        "jsd": 0.07789926571026709,
+        "acc": 0.894167167769092,
+        "acc_k": 0.894167167769092,
+        "ece": 0.02058065520615534,
+        "sel_acc80": 0.9556613226452906,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      }
+    },
+    "snli_test_hyponly": {
+      "raw": {
+        "nll": 1.7306926384767332,
+        "brier": 0.8538196615095388,
+        "jsd": 0.3389282732869213,
+        "acc": 0.39787532571657647,
+        "acc_k": 0.39787532571657647,
+        "ece": 0.3512127329314441,
+        "sel_acc80": 0.4155811623246493,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 1.605434509078102,
+        "brier": 0.8275545011588878,
+        "jsd": 0.3336658913312569,
+        "acc": 0.39787532571657647,
+        "acc_k": 0.39787532571657647,
+        "ece": 0.3278745979004954,
+        "sel_acc80": 0.4168336673346693,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      }
+    },
+    "snli_test_paraphrase": {
+      "raw": {
+        "nll": 0.33001314107494534,
+        "brier": 0.17626339669192392,
+        "jsd": 0.093560031291203,
+        "acc": 0.8879534976949288,
+        "acc_k": 0.8879534976949288,
+        "ece": 0.056906451303702234,
+        "sel_acc80": 0.9493987975951904,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 0.345213491059501,
+        "brier": 0.18243332009021412,
+        "jsd": 0.10055492307245627,
+        "acc": 0.8879534976949288,
+        "acc_k": 0.8879534976949288,
+        "ece": 0.07956015918531549,
+        "sel_acc80": 0.9483967935871743,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      }
+    },
+    "snli_test_qpara": {
+      "raw": {
+        "nll": 0.28813321351334215,
+        "brier": 0.15915075021311392,
+        "jsd": 0.07611697804220868,
+        "acc": 0.8916750250752257,
+        "acc_k": 0.8916750250752257,
+        "ece": 0.011274306875174358,
+        "sel_acc80": 0.9530075187969925,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 0.2928096177722551,
+        "brier": 0.1605470109592482,
+        "jsd": 0.08055557756133541,
+        "acc": 0.8916750250752257,
+        "acc_k": 0.8916750250752257,
+        "ece": 0.03255550470670221,
+        "sel_acc80": 0.9523809523809523,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      }
+    },
+    "snli_test_soft": {
+      "raw": {
+        "nll": 0.5765725980284668,
+        "brier": 0.11069785863315745,
+        "jsd": 0.058387127716102265,
+        "acc": 0.894167167769092,
+        "acc_k": 0.894167167769092,
+        "ece": 0.008250219062326106,
+        "sel_acc80": 0.9561623246492986,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 0.5471916195654647,
+        "brier": 0.1071951342615467,
+        "jsd": 0.05873654566137795,
+        "acc": 0.894167167769092,
+        "acc_k": 0.894167167769092,
+        "ece": 0.02058065520615534,
+        "sel_acc80": 0.9556613226452906,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      }
+    },
+    "squad_null": {
+      "raw": {
+        "nll": 0.08034793192118085,
+        "brier": 0.04045550145126374,
+        "jsd": 0.01782847579404271,
+        "acc": 0.97425,
+        "acc_k": 0.997,
+        "ece": 0.006439283616840791,
+        "sel_acc80": 0.995,
+        "conf_wrong": 0.0165,
+        "coverage": 0.51125,
+        "false_abstain": 0.013,
+        "null_recall": 0.9645,
+        "auroc_null": 0.994786,
+        "auroc_conf": 0.9945622500000002
+      },
+      "scaled": {
+        "nll": 0.08038613771297327,
+        "brier": 0.0404679182690968,
+        "jsd": 0.017924676484503386,
+        "acc": 0.97425,
+        "acc_k": 0.997,
+        "ece": 0.006344715304672706,
+        "sel_acc80": 0.995,
+        "conf_wrong": 0.0165,
+        "coverage": 0.51125,
+        "false_abstain": 0.013,
+        "null_recall": 0.9645,
+        "auroc_null": 0.994786,
+        "auroc_conf": 0.9945115000000001
+      }
+    },
+    "trec_coarse": {
+      "raw": {
+        "nll": 1.4410608559203524,
+        "brier": 0.5647949710098153,
+        "jsd": 0.2594204842852714,
+        "acc": 0.598,
+        "acc_k": 0.64,
+        "ece": 0.13209271228313446,
+        "sel_acc80": 0.6525,
+        "coverage": 0.858,
+        "false_abstain": 0.142,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 1.3737689758814966,
+        "brier": 0.5630469420780047,
+        "jsd": 0.2625928078499603,
+        "acc": 0.598,
+        "acc_k": 0.64,
+        "ece": 0.1101637490391731,
+        "sel_acc80": 0.65,
+        "coverage": 0.856,
+        "false_abstain": 0.144,
+        "null_recall": "n/a"
+      }
+    },
+    "trec_fine": {
+      "raw": {
+        "nll": 1.8286524021989985,
+        "brier": 0.6856642809905157,
+        "jsd": 0.33030271212248546,
+        "acc": 0.516,
+        "acc_k": 0.556,
+        "ece": 0.11209276500344276,
+        "sel_acc80": 0.5675,
+        "coverage": 0.866,
+        "false_abstain": 0.134,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 1.7610366472313366,
+        "brier": 0.6826349232338287,
+        "jsd": 0.33616273860591916,
+        "acc": 0.508,
+        "acc_k": 0.556,
+        "ece": 0.099832693696022,
+        "sel_acc80": 0.56,
+        "coverage": 0.85,
+        "false_abstain": 0.15,
+        "null_recall": "n/a"
+      }
+    },
+    "unli_test": {
+      "raw": {
+        "nll": 0.5229310656880769,
+        "brier": 0.12426689562340587,
+        "jsd": 0.04685473064779179,
+        "auroc_null": 0.8959275410935928,
+        "auroc_conf": 0.8959275410935928
+      },
+      "scaled": {
+        "nll": 0.5229310634122645,
+        "brier": 0.124266895213907,
+        "jsd": 0.04685473041870062,
+        "auroc_null": 0.8959275410935928,
+        "auroc_conf": 0.8959275410935928
+      }
+    },
+    "data_kb_val": {
+      "raw": {
+        "nll": 0.9172184406482693,
+        "brier": 0.4730059211402316,
+        "jsd": 0.23839771802896959,
+        "acc": 0.6173333333333333,
+        "acc_k": 0.5984528832630098,
+        "ece": 0.03232765065630277,
+        "sel_acc80": 0.6975,
+        "conf_wrong": 0.01282051282051282,
+        "coverage": 0.9486666666666667,
+        "false_abstain": 0.0007032348804500703,
+        "null_recall": 0.9743589743589743,
+        "auroc_null": 0.9997655883731833,
+        "auroc_conf": 0.9838796927404522
+      },
+      "scaled": {
+        "nll": 0.9193650865536224,
+        "brier": 0.47460153079949197,
+        "jsd": 0.24302611810488342,
+        "acc": 0.6173333333333333,
+        "acc_k": 0.5984528832630098,
+        "ece": 0.03246050986647605,
+        "sel_acc80": 0.6975,
+        "conf_wrong": 0.01282051282051282,
+        "coverage": 0.9486666666666667,
+        "false_abstain": 0.0007032348804500703,
+        "null_recall": 0.9743589743589743,
+        "auroc_null": 0.9997655883731833,
+        "auroc_conf": 0.9838255977496484
+      }
+    },
+    "truthfulqa_mc1": {
+      "raw": {
+        "nll": 2.5604446606049165,
+        "brier": 0.9727379180282026,
+        "jsd": 0.44773304381342327,
+        "acc": 0.25703794369645044,
+        "acc_k": 0.32068543451652387,
+        "ece": 0.27811915158422956,
+        "sel_acc80": 0.2782874617737003,
+        "coverage": 0.8763769889840881,
+        "false_abstain": 0.12362301101591187,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 2.48766464211813,
+        "brier": 0.960296728999466,
+        "jsd": 0.4468052354590374,
+        "acc": 0.25703794369645044,
+        "acc_k": 0.32068543451652387,
+        "ece": 0.25495981387657224,
+        "sel_acc80": 0.28134556574923547,
+        "coverage": 0.8763769889840881,
+        "false_abstain": 0.12362301101591187,
+        "null_recall": "n/a"
+      }
+    },
+    "data_wf_val": {
+      "raw": {
+        "nll": 0.37583161847228896,
+        "brier": 0.20703477167290157,
+        "jsd": 0.105633798882721,
+        "acc": 0.8586666666666667,
+        "acc_k": 0.8568507157464212,
+        "ece": 0.05674472664793331,
+        "sel_acc80": 0.9383333333333334,
+        "conf_wrong": 0.0,
+        "coverage": 0.9786666666666667,
+        "false_abstain": 0.0006816632583503749,
+        "null_recall": 0.9393939393939394,
+        "auroc_null": 0.9999380306128771,
+        "auroc_conf": 0.9979343537625747
+      },
+      "scaled": {
+        "nll": 0.3865502429414146,
+        "brier": 0.21285080875890586,
+        "jsd": 0.11058233121499325,
+        "acc": 0.8586666666666667,
+        "acc_k": 0.8568507157464212,
+        "ece": 0.07054323360323908,
+        "sel_acc80": 0.9358333333333333,
+        "conf_wrong": 0.0,
+        "coverage": 0.9786666666666667,
+        "false_abstain": 0.0006816632583503749,
+        "null_recall": 0.9393939393939394,
+        "auroc_null": 0.9999380306128771,
+        "auroc_conf": 0.998223544235814
+      },
+      "ordinal": {
+        "ordinal_mae": 0.2672811059907834,
+        "exp_score_err": 0.4225480312823225,
+        "n": 434
+      },
+      "noul_reversed": {
+        "max_abs_diff": 0.0,
+        "mean_abs_diff": 0.0
+      }
+    },
+    "wf_heldout_choice": {
+      "raw": {
+        "nll": 0.02237159506845183,
+        "brier": 0.009990251291453516,
+        "jsd": 0.006968188747770273,
+        "acc": 0.9946666666666667,
+        "acc_k": 0.9957537154989384,
+        "ece": 0.010020962238311813,
+        "sel_acc80": 1.0,
+        "conf_wrong": 0.0,
+        "coverage": 0.9433333333333334,
+        "false_abstain": 0.0,
+        "null_recall": 0.9770114942528736,
+        "auroc_null": 1.0,
+        "auroc_conf": 0.9994143055860606
+      },
+      "scaled": {
+        "nll": 0.02658096210267664,
+        "brier": 0.0108593449895006,
+        "jsd": 0.00840652061124444,
+        "acc": 0.9946666666666667,
+        "acc_k": 0.9957537154989384,
+        "ece": 0.01406175897518795,
+        "sel_acc80": 1.0,
+        "conf_wrong": 0.0,
+        "coverage": 0.9433333333333334,
+        "false_abstain": 0.0,
+        "null_recall": 0.9770114942528736,
+        "auroc_null": 1.0,
+        "auroc_conf": 0.9993492284289561
+      }
+    },
+    "wf_heldout_noul": {
+      "raw": {
+        "nll": 0.6509058026223263,
+        "brier": 0.4112349650016078,
+        "jsd": 0.1613226242259403,
+        "acc": 0.71,
+        "acc_k": 0.71,
+        "ece": 0.12073413105805718,
+        "sel_acc80": 0.7566666666666667,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 0.6225472483414641,
+        "brier": 0.4031137822900303,
+        "jsd": 0.1609770686535907,
+        "acc": 0.71,
+        "acc_k": 0.71,
+        "ece": 0.10552653594811762,
+        "sel_acc80": 0.7566666666666667,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "noul_reversed": {
+        "max_abs_diff": 0.0,
+        "mean_abs_diff": 0.0
+      }
+    },
+    "wf_heldout_score": {
+      "raw": {
+        "nll": 1.0077400803811332,
+        "brier": 0.573195142899105,
+        "jsd": 0.2761805759176359,
+        "acc": 0.522,
+        "acc_k": 0.5226666666666666,
+        "ece": 0.040125139494736985,
+        "sel_acc80": 0.5766666666666667,
+        "coverage": 0.9993333333333333,
+        "false_abstain": 0.0006666666666666666,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 1.010836001086897,
+        "brier": 0.5717227830762833,
+        "jsd": 0.2801758694525442,
+        "acc": 0.522,
+        "acc_k": 0.5226666666666666,
+        "ece": 0.05136033900578818,
+        "sel_acc80": 0.575,
+        "coverage": 0.9993333333333333,
+        "false_abstain": 0.0006666666666666666,
+        "null_recall": "n/a"
+      },
+      "ordinal": {
+        "ordinal_mae": 0.5693333333333334,
+        "exp_score_err": 0.5842081432726915,
+        "n": 1500
+      }
+    },
+    "wf_heldout_style": {
+      "raw": {
+        "nll": 0.375583335464391,
+        "brier": 0.21027239215428048,
+        "jsd": 0.10794280937778464,
+        "acc": 0.8606666666666667,
+        "acc_k": 0.8584969532836831,
+        "ece": 0.07419633140166605,
+        "sel_acc80": 0.9383333333333334,
+        "conf_wrong": 0.0,
+        "coverage": 0.9846666666666667,
+        "false_abstain": 0.0,
+        "null_recall": 1.0,
+        "auroc_null": 1.0,
+        "auroc_conf": 1.0
+      },
+      "scaled": {
+        "nll": 0.38826847038172296,
+        "brier": 0.21664383115975253,
+        "jsd": 0.11323410183609288,
+        "acc": 0.8606666666666667,
+        "acc_k": 0.8584969532836831,
+        "ece": 0.08574621800581617,
+        "sel_acc80": 0.9358333333333333,
+        "conf_wrong": 0.0,
+        "coverage": 0.9846666666666667,
+        "false_abstain": 0.0,
+        "null_recall": 1.0,
+        "auroc_null": 1.0,
+        "auroc_conf": 1.0
+      },
+      "ordinal": {
+        "ordinal_mae": 0.2709677419354839,
+        "exp_score_err": 0.44317797320777375,
+        "n": 465
+      },
+      "noul_reversed": {
+        "max_abs_diff": 0.008650124073028564,
+        "mean_abs_diff": 4.512667779076415e-05
+      }
+    },
+    "wf_rubric_flip": {
+      "raw": {
+        "nll": 0.5544689662864979,
+        "brier": 0.3214072550849167,
+        "jsd": 0.14485513508589654,
+        "acc": 0.7566666666666667,
+        "acc_k": 0.7566666666666667,
+        "ece": 0.03826767514149347,
+        "sel_acc80": 0.8383333333333334,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 0.547252401613635,
+        "brier": 0.3193832961798829,
+        "jsd": 0.14676300963290417,
+        "acc": 0.7566666666666667,
+        "acc_k": 0.7566666666666667,
+        "ece": 0.04043171815077463,
+        "sel_acc80": 0.8391666666666666,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "flip": {
+        "n_pairs": 750,
+        "flip_rate": 0.7573333333333333,
+        "both_correct": 0.5866666666666667,
+        "acc": 0.7566666666666667
+      },
+      "ordinal": {
+        "ordinal_mae": 0.547808764940239,
+        "exp_score_err": 0.5861706615906308,
+        "n": 502
+      },
+      "noul_reversed": {
+        "max_abs_diff": 0.0,
+        "mean_abs_diff": 0.0
+      }
+    },
+    "wf_rubric_shuffled": {
+      "raw": {
+        "nll": 2.583553418746041,
+        "brier": 0.928155771024346,
+        "jsd": 0.3631805122798969,
+        "acc": 0.376,
+        "acc_k": 0.376,
+        "ece": 0.36781600866715114,
+        "sel_acc80": 0.3925,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 2.364992132114415,
+        "brier": 0.9104185657935674,
+        "jsd": 0.3598282600140078,
+        "acc": 0.376,
+        "acc_k": 0.376,
+        "ece": 0.3502550947666168,
+        "sel_acc80": 0.39416666666666667,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "ordinal": {
+        "ordinal_mae": 0.9470899470899471,
+        "exp_score_err": 0.8813899273937501,
+        "n": 567
+      },
+      "noul_reversed": {
+        "max_abs_diff": 0.0,
+        "mean_abs_diff": 0.0
+      }
+    },
+    "data_wf_hf_val": {
+      "raw": {
+        "nll": 0.019401891005933914,
+        "brier": 0.009413470640887905,
+        "jsd": 0.004283260887802352,
+        "acc": 0.9946666666666667,
+        "acc_k": 0.9946666666666667,
+        "ece": 0.0031396085421244255,
+        "sel_acc80": 1.0,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 0.01934158623401211,
+        "brier": 0.009498351599644392,
+        "jsd": 0.004642312540112629,
+        "acc": 0.9946666666666667,
+        "acc_k": 0.9946666666666667,
+        "ece": 0.0030115491151810297,
+        "sel_acc80": 1.0,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      }
+    },
+    "cua_s1_forms_demo": {
+      "raw": {
+        "nll": 0.0009300660857717562,
+        "brier": 1.7813199239036247e-05,
+        "jsd": 0.0003218505523506755,
+        "acc": 1.0,
+        "acc_k": 1.0,
+        "ece": 0.0009250634787033585,
+        "sel_acc80": 1.0,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 0.0018790738508953008,
+        "brier": 4.7467508047515636e-05,
+        "jsd": 0.0006498546450564701,
+        "acc": 1.0,
+        "acc_k": 1.0,
+        "ece": 0.0018648833644633322,
+        "sel_acc80": 1.0,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      }
+    },
+    "cua_s1_forms_test": {
+      "raw": {
+        "nll": 0.032046872477530054,
+        "brier": 0.017368318078177302,
+        "jsd": 0.006974812412770343,
+        "acc": 0.9873333333333333,
+        "acc_k": 0.9873333333333333,
+        "ece": 0.006146776457627566,
+        "sel_acc80": 1.0,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 0.03077882151868218,
+        "brier": 0.017053987064367247,
+        "jsd": 0.007157656705056871,
+        "acc": 0.9873333333333333,
+        "acc_k": 0.9873333333333333,
+        "ece": 0.006392546931902525,
+        "sel_acc80": 1.0,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      }
+    },
+    "jev4b_adversarial": {
+      "raw": {
+        "nll": 0.40543914653847096,
+        "brier": 0.21213837393283155,
+        "jsd": 0.10257905139738542,
+        "acc": 0.8566666666666667,
+        "acc_k": 0.8566666666666667,
+        "ece": 0.04504729241132734,
+        "sel_acc80": 0.9458333333333333,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 0.40307816177058725,
+        "brier": 0.21442104499654838,
+        "jsd": 0.10539861463346857,
+        "acc": 0.8566666666666667,
+        "acc_k": 0.8566666666666667,
+        "ece": 0.05155258302887279,
+        "sel_acc80": 0.9458333333333333,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "ordinal": {
+        "ordinal_mae": 0.25833333333333336,
+        "exp_score_err": 0.44842133761425884,
+        "n": 120
+      },
+      "noul_reversed": {
+        "max_abs_diff": 0.0,
+        "mean_abs_diff": 0.0
+      }
+    },
+    "jevlogs_triage": {
+      "raw": {
+        "nll": 1.371274447707115,
+        "brier": 0.8498478108011697,
+        "jsd": 0.29231064922915007,
+        "acc": 0.5,
+        "acc_k": 0.5,
+        "ece": 0.4159393855730693,
+        "sel_acc80": 0.5116666666666667,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 1.252508511219465,
+        "brier": 0.8168997236293741,
+        "jsd": 0.28321615671899836,
+        "acc": 0.5,
+        "acc_k": 0.5,
+        "ece": 0.39538653079668684,
+        "sel_acc80": 0.5116666666666667,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "noul_reversed": {
+        "max_abs_diff": 0.0,
+        "mean_abs_diff": 0.0
+      }
+    },
+    "mind2web_choice": {
+      "raw": {
+        "nll": 1.5252458937008002,
+        "brier": 0.7495051757484386,
+        "jsd": 0.3487572217093445,
+        "acc": 0.35133333333333333,
+        "acc_k": 0.354,
+        "ece": 0.20950158571203548,
+        "sel_acc80": 0.38,
+        "coverage": 0.9953333333333333,
+        "false_abstain": 0.004666666666666667,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 1.492309018165003,
+        "brier": 0.7332142893414118,
+        "jsd": 0.34893241092756905,
+        "acc": 0.3506666666666667,
+        "acc_k": 0.354,
+        "ece": 0.19169397105276587,
+        "sel_acc80": 0.38,
+        "coverage": 0.994,
+        "false_abstain": 0.006,
+        "null_recall": "n/a"
+      },
+      "noul_reversed": {
+        "max_abs_diff": 0.0,
+        "mean_abs_diff": 0.0
+      }
+    },
+    "pagerduty_trigger": {
+      "raw": {
+        "nll": 0.9310243398123302,
+        "brier": 0.6155340506142242,
+        "jsd": 0.23931832213418777,
+        "acc": 0.56,
+        "acc_k": 0.56,
+        "ece": 0.25797174461682637,
+        "sel_acc80": 0.5675,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 0.8901599970352302,
+        "brier": 0.5932938649817051,
+        "jsd": 0.236096828884083,
+        "acc": 0.56,
+        "acc_k": 0.56,
+        "ece": 0.27468403406937913,
+        "sel_acc80": 0.5675,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "noul_reversed": {
+        "max_abs_diff": 0.0,
+        "mean_abs_diff": 0.0
+      }
+    },
+    "systemone_lite_hard": {
+      "raw": {
+        "nll": 0.4891307900271223,
+        "brier": 0.30110738295596046,
+        "jsd": 0.14204759917310972,
+        "acc": 0.7786666666666666,
+        "acc_k": 0.7786666666666666,
+        "ece": 0.04153631351391474,
+        "sel_acc80": 0.8391666666666666,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 0.4946905537496238,
+        "brier": 0.3048013448634015,
+        "jsd": 0.1455159848651888,
+        "acc": 0.7786666666666666,
+        "acc_k": 0.7786666666666666,
+        "ece": 0.054403663555781016,
+        "sel_acc80": 0.8375,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "ordinal": {
+        "ordinal_mae": 0.332,
+        "exp_score_err": 0.4491760475578015,
+        "n": 500
+      },
+      "noul_reversed": {
+        "max_abs_diff": 0.0,
+        "mean_abs_diff": 0.0
+      }
+    },
+    "tree_choice_cap": {
+      "raw": {
+        "nll": 1.9912322046858726,
+        "brier": 0.776633232130246,
+        "jsd": 0.3516807077694661,
+        "acc": 0.4736111111111111,
+        "acc_k": 0.7055555555555556,
+        "ece": 0.2546793927076376,
+        "sel_acc80": 0.5295138888888888,
+        "coverage": 0.6069444444444445,
+        "false_abstain": 0.39305555555555555,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 2.037652770209596,
+        "brier": 0.7879233374142699,
+        "jsd": 0.36161560719221636,
+        "acc": 0.46944444444444444,
+        "acc_k": 0.7055555555555556,
+        "ece": 0.25013160817842517,
+        "sel_acc80": 0.5208333333333334,
+        "coverage": 0.5916666666666667,
+        "false_abstain": 0.4083333333333333,
+        "null_recall": "n/a"
+      }
+    },
+    "typed_decisions_test": {
+      "raw": {
+        "nll": 1.308004753461238,
+        "brier": 0.2680494162037552,
+        "jsd": 0.12193913841109022,
+        "acc": 0.4786666666666667,
+        "acc_k": 0.4786666666666667,
+        "ece": 0.11097379342714944,
+        "sel_acc80": 0.5066666666666667,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 1.2651857313405155,
+        "brier": 0.2577007533013979,
+        "jsd": 0.11840589262700886,
+        "acc": 0.4786666666666667,
+        "acc_k": 0.4786666666666667,
+        "ece": 0.09702249457438787,
+        "sel_acc80": 0.5058333333333334,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "ordinal": {
+        "ordinal_mae": 0.8266666666666667,
+        "exp_score_err": 0.8259128936029989,
+        "n": 600
+      },
+      "noul_reversed": {
+        "max_abs_diff": 0.0,
+        "mean_abs_diff": 0.0
+      }
+    },
+    "typed_decisions_train": {
+      "raw": {
+        "nll": 1.3072811380004457,
+        "brier": 0.21419036544099046,
+        "jsd": 0.09930688225243353,
+        "acc": 0.362,
+        "acc_k": 0.362,
+        "ece": 0.14027928841114043,
+        "sel_acc80": 0.38,
+        "coverage": 0.9993333333333333,
+        "false_abstain": 0.0006666666666666666,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 1.2813463910992928,
+        "brier": 0.20369022460075886,
+        "jsd": 0.09572205414893548,
+        "acc": 0.362,
+        "acc_k": 0.362,
+        "ece": 0.12121769414345425,
+        "sel_acc80": 0.38,
+        "coverage": 0.9993333333333333,
+        "false_abstain": 0.0006666666666666666,
+        "null_recall": "n/a"
+      },
+      "ordinal": {
+        "ordinal_mae": 1.03,
+        "exp_score_err": 0.8523423746745429,
+        "n": 600
+      },
+      "noul_reversed": {
+        "max_abs_diff": 0.0,
+        "mean_abs_diff": 0.0
+      }
+    },
+    "data_wh_val": {
+      "raw": {
+        "nll": 0.234013824826771,
+        "brier": 0.13775511773788127,
+        "jsd": 0.06296474492075899,
+        "acc": 0.8973333333333333,
+        "acc_k": 0.932381667918858,
+        "ece": 0.020509268840153996,
+        "sel_acc80": 0.9766666666666667,
+        "conf_wrong": 0.0,
+        "coverage": 0.8686666666666667,
+        "false_abstain": 0.03906836964688205,
+        "null_recall": 0.8579881656804734,
+        "auroc_null": 0.9832799114426577,
+        "auroc_conf": 0.9764780673871583
+      },
+      "scaled": {
+        "nll": 0.23435895126674677,
+        "brier": 0.1380138971177509,
+        "jsd": 0.06413277962415562,
+        "acc": 0.8973333333333333,
+        "acc_k": 0.932381667918858,
+        "ece": 0.01645974771181742,
+        "sel_acc80": 0.9758333333333333,
+        "conf_wrong": 0.0,
+        "coverage": 0.8686666666666667,
+        "false_abstain": 0.03906836964688205,
+        "null_recall": 0.8579881656804734,
+        "auroc_null": 0.9832799114426577,
+        "auroc_conf": 0.9760735132635959
+      }
+    },
+    "wh_heldout_family": {
+      "raw": {
+        "nll": 0.5463474661337482,
+        "brier": 0.25969181917455386,
+        "jsd": 0.1057542581702437,
+        "acc": 0.8199052132701422,
+        "acc_k": 0.8461538461538461,
+        "ece": 0.078609590917402,
+        "sel_acc80": 0.9023668639053254,
+        "conf_wrong": 0.0,
+        "coverage": 0.8767772511848342,
+        "false_abstain": 0.034482758620689655,
+        "null_recall": 0.8666666666666667,
+        "auroc_null": 0.9859711170055997,
+        "auroc_conf": 0.9732390215148836
+      },
+      "scaled": {
+        "nll": 0.515709105678624,
+        "brier": 0.25745610922373874,
+        "jsd": 0.10622799758517455,
+        "acc": 0.8175355450236966,
+        "acc_k": 0.8461538461538461,
+        "ece": 0.08652745971182513,
+        "sel_acc80": 0.9053254437869822,
+        "conf_wrong": 0.0,
+        "coverage": 0.8744075829383886,
+        "false_abstain": 0.03713527851458886,
+        "null_recall": 0.8666666666666667,
+        "auroc_null": 0.9859711170055997,
+        "auroc_conf": 0.9724727379899794
+      }
+    },
+    "wh_heldout_grammar": {
+      "raw": {
+        "nll": 0.24869155709362914,
+        "brier": 0.14870406423381147,
+        "jsd": 0.06852834314444578,
+        "acc": 0.883578431372549,
+        "acc_k": 0.9226519337016574,
+        "ece": 0.0202937848939031,
+        "sel_acc80": 0.9693721286370597,
+        "conf_wrong": 0.0,
+        "coverage": 0.866421568627451,
+        "false_abstain": 0.04419889502762431,
+        "null_recall": 0.8369565217391305,
+        "auroc_null": 0.9835755464809033,
+        "auroc_conf": 0.9737719192889743
+      },
+      "scaled": {
+        "nll": 0.250087791983527,
+        "brier": 0.1492171979474074,
+        "jsd": 0.06994133383184256,
+        "acc": 0.883578431372549,
+        "acc_k": 0.9226519337016574,
+        "ece": 0.022323701019380575,
+        "sel_acc80": 0.9663093415007658,
+        "conf_wrong": 0.0,
+        "coverage": 0.8639705882352942,
+        "false_abstain": 0.04558011049723757,
+        "null_recall": 0.8478260869565217,
+        "auroc_null": 0.9835755464809033,
+        "auroc_conf": 0.9725708623588758
+      }
+    },
+    "wh_heldout_style": {
+      "raw": {
+        "nll": 0.27159999686744934,
+        "brier": 0.15087037945392912,
+        "jsd": 0.0702443405562412,
+        "acc": 0.8883374689826302,
+        "acc_k": 0.8883374689826302,
+        "ece": 0.02315892800503868,
+        "sel_acc80": 0.9814241486068112,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 0.26962945683446005,
+        "brier": 0.1503456861985424,
+        "jsd": 0.07193390448881899,
+        "acc": 0.8883374689826302,
+        "acc_k": 0.8883374689826302,
+        "ece": 0.0237619669058778,
+        "sel_acc80": 0.9814241486068112,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      }
+    },
+    "wh_level7": {
+      "raw": {
+        "nll": 1.0941978558128005,
+        "brier": 0.6996876956981805,
+        "jsd": 0.2755464850522858,
+        "acc": 0.49318181818181817,
+        "acc_k": 0.49318181818181817,
+        "ece": 0.21915304772555827,
+        "sel_acc80": 0.5014204545454546,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 1.0441702379342557,
+        "brier": 0.6830152541002035,
+        "jsd": 0.27213881565412834,
+        "acc": 0.49318181818181817,
+        "acc_k": 0.49318181818181817,
+        "ece": 0.2006248111074621,
+        "sel_acc80": 0.5,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      }
+    },
+    "wh_rubric_flip": {
+      "raw": {
+        "nll": 0.4852592661059105,
+        "brier": 0.2752466288148543,
+        "jsd": 0.11332718659046732,
+        "acc": 0.8026666666666666,
+        "acc_k": 0.8235719450469993,
+        "ece": 0.06838929454485576,
+        "sel_acc80": 0.8791666666666667,
+        "conf_wrong": 0.0,
+        "coverage": 0.906,
+        "false_abstain": 0.02964569775849602,
+        "null_recall": 0.8547008547008547,
+        "auroc_null": 0.9890427721230324,
+        "auroc_conf": 0.9718931345829395
+      },
+      "scaled": {
+        "nll": 0.4660126084095091,
+        "brier": 0.2717966241329151,
+        "jsd": 0.11361927024392757,
+        "acc": 0.8013333333333333,
+        "acc_k": 0.8235719450469993,
+        "ece": 0.06078574814399086,
+        "sel_acc80": 0.8783333333333333,
+        "conf_wrong": 0.0,
+        "coverage": 0.9046666666666666,
+        "false_abstain": 0.03109182935647144,
+        "null_recall": 0.8547008547008547,
+        "auroc_null": 0.9890427721230324,
+        "auroc_conf": 0.970243061349352
+      }
+    },
+    "wh_rubric_shuffled": {
+      "raw": {
+        "nll": 2.9113006100472916,
+        "brier": 0.9620256743702243,
+        "jsd": 0.3528958761487168,
+        "acc": 0.398,
+        "acc_k": 0.4255469301340861,
+        "ece": 0.4284619954824448,
+        "sel_acc80": 0.3975,
+        "conf_wrong": 0.12048192771084337,
+        "coverage": 0.9359999999999999,
+        "false_abstain": 0.04163726182074806,
+        "null_recall": 0.4457831325301205,
+        "auroc_null": 0.9583244764520326,
+        "auroc_conf": 0.7902747192014352
+      },
+      "scaled": {
+        "nll": 2.6478036458048013,
+        "brier": 0.9498309075359751,
+        "jsd": 0.3500576999611389,
+        "acc": 0.39866666666666667,
+        "acc_k": 0.4255469301340861,
+        "ece": 0.41628739579518637,
+        "sel_acc80": 0.39916666666666667,
+        "conf_wrong": 0.12048192771084337,
+        "coverage": 0.9353333333333333,
+        "false_abstain": 0.04163726182074806,
+        "null_recall": 0.4578313253012048,
+        "auroc_null": 0.9583244764520324,
+        "auroc_conf": 0.7830730118781406
+      }
+    },
+    "data_u_val": {
+      "raw": {
+        "nll": 0.49041288665397725,
+        "brier": 0.04423868525139621,
+        "jsd": 0.01857737823312269,
+        "acc": 0.8953333333333333,
+        "acc_k": 0.8953333333333333,
+        "ece": 0.12396540609995522,
+        "sel_acc80": 0.9591666666666666,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 0.49306508783951614,
+        "brier": 0.04551667209993445,
+        "jsd": 0.019483829755564286,
+        "acc": 0.8953333333333333,
+        "acc_k": 0.8953333333333333,
+        "ece": 0.1426694559454918,
+        "sel_acc80": 0.9591666666666666,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      }
+    },
+    "u_chaosnli": {
+      "raw": {
+        "nll": 0.9992255345818895,
+        "brier": 0.15592243687493582,
+        "jsd": 0.06276678856554775,
+        "acc": 0.548,
+        "acc_k": 0.548,
+        "ece": 0.05644560275475185,
+        "sel_acc80": 0.5625,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 0.989961406583137,
+        "brier": 0.150204655222619,
+        "jsd": 0.062144876814656025,
+        "acc": 0.548,
+        "acc_k": 0.548,
+        "ece": 0.06379163038730622,
+        "sel_acc80": 0.5641666666666667,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      }
+    },
+    "u_real_heldout": {
+      "raw": {
+        "nll": 0.6606320054345184,
+        "brier": 0.13458670329345712,
+        "jsd": 0.05980471728660867,
+        "acc": 0.6494661921708185,
+        "acc_k": 0.6494661921708185,
+        "ece": 0.062161013006739735,
+        "sel_acc80": 0.6911111111111111,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 0.6586741027584431,
+        "brier": 0.13248622800527662,
+        "jsd": 0.060664774416809425,
+        "acc": 0.6494661921708185,
+        "acc_k": 0.6494661921708185,
+        "ece": 0.0787051947409572,
+        "sel_acc80": 0.6933333333333334,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      }
+    },
+    "u_synthetic_heldout": {
+      "raw": {
+        "nll": 0.6666258308359944,
+        "brier": 0.031941487551321104,
+        "jsd": 0.014305540425327239,
+        "acc": 0.9157894736842105,
+        "acc_k": 0.9157894736842105,
+        "ece": 0.16438576234014407,
+        "sel_acc80": 0.968421052631579,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      },
+      "scaled": {
+        "nll": 0.6595814980676692,
+        "brier": 0.029197536715013496,
+        "jsd": 0.013290023567297079,
+        "acc": 0.9157894736842105,
+        "acc_k": 0.9157894736842105,
+        "ece": 0.18903793711411326,
+        "sel_acc80": 0.968421052631579,
+        "coverage": 1.0,
+        "false_abstain": 0.0,
+        "null_recall": "n/a"
+      }
+    }
+  }
+}