Text Classification
Transformers
Safetensors
English
Korean
qwen3_5
image-text-to-text
ztc
answer-verification
hallucination-detection
zero-token
confidence-estimation
Instructions to use FINAL-Bench/ZTC-Judge-4B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use FINAL-Bench/ZTC-Judge-4B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="FINAL-Bench/ZTC-Judge-4B")# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("FINAL-Bench/ZTC-Judge-4B") model = AutoModelForMultimodalLM.from_pretrained("FINAL-Bench/ZTC-Judge-4B", device_map="auto") - Notebooks
- Google Colab
- Kaggle
정정: 폐기값(0.7364/0.7282) → 정본(0.7272/0.7289) + 정정 고지 추가
Browse files
README.md
CHANGED
|
@@ -1,305 +1,312 @@
|
|
| 1 |
-
---
|
| 2 |
-
license: apache-2.0
|
| 3 |
-
language:
|
| 4 |
-
- en
|
| 5 |
-
- ko
|
| 6 |
-
tags:
|
| 7 |
-
- ztc
|
| 8 |
-
- answer-verification
|
| 9 |
-
- hallucination-detection
|
| 10 |
-
- zero-token
|
| 11 |
-
- confidence-estimation
|
| 12 |
-
library_name: transformers
|
| 13 |
-
pipeline_tag: text-classification
|
| 14 |
-
---
|
| 15 |
-
|
| 16 |
-
# ZTC-Judge-4B
|
| 17 |
-
|
| 18 |
-
**Answer verification from a single forward pass, with zero generated tokens — at 4B.**
|
| 19 |
-
|
| 20 |
-
ZTC-Judge-4B takes a question and an answer written by *any* model and scores whether that
|
| 21 |
-
answer can be trusted. It is the bottom rung of a four-point size ladder measured under one
|
| 22 |
-
identical protocol, and it is published so the shape of that ladder can be checked rather than
|
| 23 |
-
asserted.
|
| 24 |
-
|
| 25 |
-
> **ZTC** — Zero-Token Confidence · **Judge** — it evaluates *someone else's* answer, not its own
|
| 26 |
-
|
| 27 |
-
---
|
| 28 |
-
|
| 29 |
-
## Read this before deploying
|
| 30 |
-
|
| 31 |
-
This model is **not** the strongest member of the family, and the card says so with numbers.
|
| 32 |
-
|
| 33 |
-
| Model | Leaderboard AUC |
|
| 34 |
-
|---|---|
|
| 35 |
-
| Darwin-397B-ZTC | 0.
|
| 36 |
-
| ZTC-Judge-27B | 0.
|
| 37 |
-
| **ZTC-Judge-9B** | **0.6506** |
|
| 38 |
-
| **ZTC-Judge-4B** | **0.6360** |
|
| 39 |
-
| *Answer length and formatting only* | *0.6223* |
|
| 40 |
-
|
| 41 |
-
**
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
|
| 117 |
-
|
| 118 |
-
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
|
| 122 |
-
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
|
| 126 |
-
|
| 127 |
-
|
| 128 |
-
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
|
| 133 |
-
{"model": "vidraft/ztc
|
| 134 |
-
"
|
| 135 |
-
|
| 136 |
-
|
| 137 |
-
|
| 138 |
-
|
| 139 |
-
|
| 140 |
-
|
| 141 |
-
|
| 142 |
-
"
|
| 143 |
-
|
| 144 |
-
|
| 145 |
-
|
| 146 |
-
|
| 147 |
-
|
| 148 |
-
|
| 149 |
-
|
| 150 |
-
|
| 151 |
-
|
| 152 |
-
|
| 153 |
-
|
| 154 |
-
|
| 155 |
-
|
| 156 |
-
|
| 157 |
-
|
| 158 |
-
|
| 159 |
-
|
| 160 |
-
|
| 161 |
-
|
| 162 |
-
|
| 163 |
-
|
| 164 |
-
|
| 165 |
-
|
| 166 |
-
|
| 167 |
-
|
| 168 |
-
|
| 169 |
-
|
| 170 |
-
|
| 171 |
-
|
| 172 |
-
|
| 173 |
-
|
| 174 |
-
|
| 175 |
-
|
| 176 |
-
|
| 177 |
-
|
| 178 |
-
|
| 179 |
-
|
| 180 |
-
|
| 181 |
-
|
| 182 |
-
|
| 183 |
-
|
| 184 |
-
|
| 185 |
-
|
| 186 |
-
|
| 187 |
-
|
| 188 |
-
|
| 189 |
-
|
|
| 190 |
-
|
|
| 191 |
-
|
|
| 192 |
-
|
|
| 193 |
-
|
| 194 |
-
|
| 195 |
-
|
| 196 |
-
|
| 197 |
-
|
| 198 |
-
|
| 199 |
-
|
| 200 |
-
|
| 201 |
-
|
| 202 |
-
|
| 203 |
-
|
| 204 |
-
|
| 205 |
-
|
| 206 |
-
|
| 207 |
-
|
| 208 |
-
|
|
| 209 |
-
|
|
| 210 |
-
|
| 211 |
-
|
| 212 |
-
|
| 213 |
-
|
| 214 |
-
|
| 215 |
-
|
| 216 |
-
*
|
| 217 |
-
|
| 218 |
-
|
| 219 |
-
|
| 220 |
-
|
| 221 |
-
|
| 222 |
-
|
| 223 |
-
|
| 224 |
-
|
| 225 |
-
|
| 226 |
-
|
| 227 |
-
|
| 228 |
-
|
| 229 |
-
|
| 230 |
-
|
| 231 |
-
|
| 232 |
-
|
| 233 |
-
|
| 234 |
-
|
| 235 |
-
|
| 236 |
-
|
| 237 |
-
|
| 238 |
-
|
| 239 |
-
|
| 240 |
-
|
| 241 |
-
|
| 242 |
-
|
| 243 |
-
|
| 244 |
-
|
| 245 |
-
|
| 246 |
-
|
| 247 |
-
|
| 248 |
-
|
| 249 |
-
|
| 250 |
-
|
| 251 |
-
|
| 252 |
-
|
| 253 |
-
|
| 254 |
-
|
| 255 |
-
|
| 256 |
-
|
| 257 |
-
|
| 258 |
-
|
| 259 |
-
|
| 260 |
-
|
| 261 |
-
|
| 262 |
-
|
| 263 |
-
|
| 264 |
-
|
| 265 |
-
|
| 266 |
-
|
| 267 |
-
|
| 268 |
-
- **
|
| 269 |
-
- **
|
| 270 |
-
|
| 271 |
-
|
| 272 |
-
|
| 273 |
-
|
| 274 |
-
|
| 275 |
-
|
| 276 |
-
|
| 277 |
-
|
| 278 |
-
|
| 279 |
-
|
| 280 |
-
|
| 281 |
-
|
| 282 |
-
|
| 283 |
-
|
| 284 |
-
|
| 285 |
-
|
| 286 |
-
|
| 287 |
-
|
| 288 |
-
|
| 289 |
-
|
| 290 |
-
|
|
| 291 |
-
|
|
| 292 |
-
|
|
| 293 |
-
|
|
| 294 |
-
|
| 295 |
-
##
|
| 296 |
-
|
| 297 |
-
|
| 298 |
-
|
| 299 |
-
|
| 300 |
-
|
| 301 |
-
|
| 302 |
-
##
|
| 303 |
-
|
| 304 |
-
|
| 305 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
language:
|
| 4 |
+
- en
|
| 5 |
+
- ko
|
| 6 |
+
tags:
|
| 7 |
+
- ztc
|
| 8 |
+
- answer-verification
|
| 9 |
+
- hallucination-detection
|
| 10 |
+
- zero-token
|
| 11 |
+
- confidence-estimation
|
| 12 |
+
library_name: transformers
|
| 13 |
+
pipeline_tag: text-classification
|
| 14 |
+
---
|
| 15 |
+
|
| 16 |
+
# ZTC-Judge-4B
|
| 17 |
+
|
| 18 |
+
**Answer verification from a single forward pass, with zero generated tokens — at 4B.**
|
| 19 |
+
|
| 20 |
+
ZTC-Judge-4B takes a question and an answer written by *any* model and scores whether that
|
| 21 |
+
answer can be trusted. It is the bottom rung of a four-point size ladder measured under one
|
| 22 |
+
identical protocol, and it is published so the shape of that ladder can be checked rather than
|
| 23 |
+
asserted.
|
| 24 |
+
|
| 25 |
+
> **ZTC** — Zero-Token Confidence · **Judge** — it evaluates *someone else's* answer, not its own
|
| 26 |
+
|
| 27 |
+
---
|
| 28 |
+
|
| 29 |
+
## Read this before deploying
|
| 30 |
+
|
| 31 |
+
This model is **not** the strongest member of the family, and the card says so with numbers.
|
| 32 |
+
|
| 33 |
+
| Model | Leaderboard AUC |
|
| 34 |
+
|---|---|
|
| 35 |
+
| Darwin-397B-ZTC | 0.7272 |
|
| 36 |
+
| ZTC-Judge-27B | 0.7289 |
|
| 37 |
+
| **ZTC-Judge-9B** | **0.6506** |
|
| 38 |
+
| **ZTC-Judge-4B** | **0.6360** |
|
| 39 |
+
| *Answer length and formatting only* | *0.6223* |
|
| 40 |
+
|
| 41 |
+
> **Revised 2026-09-21.** Earlier revisions of this card reported **0.7364** for Darwin-397B-ZTC
|
| 42 |
+
> and **0.7282** for ZTC-Judge-27B. Those figures were produced by a run whose standardisation
|
| 43 |
+
> statistics were computed over all five domains, including the held-out one, which leaks a small
|
| 44 |
+
> amount of the evaluation domain into every figure. Re-run with the statistics fitted inside the
|
| 45 |
+
> training domains only, the figures are **0.7272** and **0.7289**. Cite the current values.
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
**The ladder does not decline smoothly — it steps.** Between 9B and 27B the score moves 0.078,
|
| 49 |
+
while between 4B and 9B it moves 0.015. Whatever carries verification quality is largely absent
|
| 50 |
+
below 27B on this axis.
|
| 51 |
+
|
| 52 |
+
**Where this model is worth deploying is one specific place**, and it is a real one:
|
| 53 |
+
|
| 54 |
+
| Domain | Surface baseline | **4B** | Margin |
|
| 55 |
+
|---|---|---|---|
|
| 56 |
+
| **Professional exams (law · math · biology)** | 0.7138 | **0.7787** | ****+0.0649**** |
|
| 57 |
+
| Scientific reasoning | 0.7272 | 0.5576 | 🔴 -0.1696 |
|
| 58 |
+
| Biology & medicine | 0.5908 | 0.6223 | **+0.0315** |
|
| 59 |
+
| Disaster & safety procedures | 0.5949 | 0.5842 | 🔴 -0.0107 |
|
| 60 |
+
| General multi-step reasoning | 0.5420 | 0.5738 | **+0.0318** |
|
| 61 |
+
| **Size-weighted mean** | **0.6223** | **0.6360** | ****+0.0137**** |
|
| 62 |
+
|
| 63 |
+
🔴 **Do not use this model for disaster and safety content.** In that domain it does not clear the
|
| 64 |
+
surface baseline, which means it is reading answer shape rather than correctness there.
|
| 65 |
+
|
| 66 |
+
✅ **Professional-exam style content is where it earns its size.** It runs on a laptop, on CPU, and
|
| 67 |
+
inside networks that never reach the internet — places a hosted API cannot go.
|
| 68 |
+
|
| 69 |
+
## How it works
|
| 70 |
+
|
| 71 |
+
```
|
| 72 |
+
[question + answer] → one forward pass
|
| 73 |
+
→ final-layer hidden state at the last position (2560-d)
|
| 74 |
+
→ probe
|
| 75 |
+
→ score
|
| 76 |
+
```
|
| 77 |
+
|
| 78 |
+
**Generated tokens: 0.** No access to the answering model's weights or logits is required; the text
|
| 79 |
+
of the answer is the only input. Latency is one forward pass, and batching converts directly into
|
| 80 |
+
throughput.
|
| 81 |
+
|
| 82 |
+
## Usage
|
| 83 |
+
|
| 84 |
+
```python
|
| 85 |
+
import json
|
| 86 |
+
import numpy as np, torch
|
| 87 |
+
from huggingface_hub import hf_hub_download, snapshot_download
|
| 88 |
+
from transformers import AutoModel, AutoTokenizer
|
| 89 |
+
|
| 90 |
+
REPO = "FINAL-Bench/ZTC-Judge-4B"
|
| 91 |
+
cfg = json.load(open(hf_hub_download(REPO, "ztc_config.json"), encoding="utf-8"))
|
| 92 |
+
path = snapshot_download(REPO)
|
| 93 |
+
|
| 94 |
+
tok = AutoTokenizer.from_pretrained(path)
|
| 95 |
+
model = AutoModel.from_pretrained(path, dtype=torch.bfloat16, low_cpu_mem_usage=True).eval()
|
| 96 |
+
|
| 97 |
+
def hidden(question, answer):
|
| 98 |
+
b = tok([cfg["template"] % (question, answer)], return_tensors="pt",
|
| 99 |
+
truncation=True, max_length=cfg["max_length"])
|
| 100 |
+
dev = next(model.parameters()).device
|
| 101 |
+
with torch.no_grad():
|
| 102 |
+
h = model(input_ids=b["input_ids"].to(dev),
|
| 103 |
+
attention_mask=b["attention_mask"].to(dev)).last_hidden_state
|
| 104 |
+
return h[0, int(b["attention_mask"].sum()) - 1].float().numpy().astype(np.float64)
|
| 105 |
+
|
| 106 |
+
# linear probe — one dot product
|
| 107 |
+
p = np.load(hf_hub_download(REPO, "ztc_probe.npz"))
|
| 108 |
+
v = hidden("Which defensive chemical does an insect release?", "C. Allomone")
|
| 109 |
+
print(float(((v - p["mu"]) / p["sd"]) @ p["w"]))
|
| 110 |
+
```
|
| 111 |
+
|
| 112 |
+
The score is an unbounded real number; higher means more likely correct. It is a **ranking signal**,
|
| 113 |
+
not a calibrated probability — choose a threshold from your own review budget.
|
| 114 |
+
|
| 115 |
+
## Two probes ship with this model
|
| 116 |
+
|
| 117 |
+
| File | Produces |
|
| 118 |
+
|---|---|
|
| 119 |
+
| `ztc_probe.npz` | linear readout — one dot product |
|
| 120 |
+
| `ztc_curve_probe.npz` | **the reported figure 0.6360** — 256 anchors, RBF kernel |
|
| 121 |
+
|
| 122 |
+
Both read the same input. The curved probe is the one to use when the number matters.
|
| 123 |
+
|
| 124 |
+
## API — drop-in for an existing JEV integration
|
| 125 |
+
|
| 126 |
+
The endpoint takes the same request shape and returns the same response shape, so switching an
|
| 127 |
+
existing integration is a URL change.
|
| 128 |
+
|
| 129 |
+
```bash
|
| 130 |
+
POST /v1/evaluate
|
| 131 |
+
Authorization: Bearer <token>
|
| 132 |
+
|
| 133 |
+
{"model": "vidraft/ztc",
|
| 134 |
+
"state": {"question": "...", "answer": "..."},
|
| 135 |
+
"questions": {"correct": {"type": "boolean",
|
| 136 |
+
"instructions": "Is the ANSWER factually correct?"}}}
|
| 137 |
+
```
|
| 138 |
+
|
| 139 |
+
```json
|
| 140 |
+
{"model": "vidraft/ztc-judge-4b",
|
| 141 |
+
"answers": {"correct": {
|
| 142 |
+
"probability": 0.1043,
|
| 143 |
+
"verdict": "review",
|
| 144 |
+
"score": -0.72,
|
| 145 |
+
"position": 0.268,
|
| 146 |
+
|
| 147 |
+
"band": "low",
|
| 148 |
+
"action": "hold_or_escalate",
|
| 149 |
+
"measured": {
|
| 150 |
+
"band_accuracy": 0.485,
|
| 151 |
+
"base_accuracy": 0.748,
|
| 152 |
+
"if_lowest_20pct_dropped": 0.814,
|
| 153 |
+
"escalate_gain_at_20pct_budget": 0.0134,
|
| 154 |
+
"do_not": "resample_same_model",
|
| 155 |
+
"why_not": "measured: fixes 6.7% of wrong answers, breaks 13.1% of right ones"}}},
|
| 156 |
+
"usage": {"generated_tokens": 0}}
|
| 157 |
+
```
|
| 158 |
+
|
| 159 |
+
`type` accepts `boolean` and `noul`. Existing clients read `answers.<key>.probability` and ignore
|
| 160 |
+
the rest; the additional fields are there for clients that want to act on the score rather than
|
| 161 |
+
merely record it. **0.19 s per call, zero generated tokens.**
|
| 162 |
+
|
| 163 |
+
### What `probability` means
|
| 164 |
+
|
| 165 |
+
The raw score is unbounded. The shipped calibration maps it to P(answer is correct), fitted
|
| 166 |
+
**leave-one-domain-out** — the mapping never sees the domain it is applied to.
|
| 167 |
+
|
| 168 |
+
| | Expected calibration error |
|
| 169 |
+
|---|---|
|
| 170 |
+
| **ZTC-Judge-27B (after calibration)** | **0.0245** |
|
| 171 |
+
| JEV, as shipped | 0.0381 |
|
| 172 |
+
| JEV, after the same calibration | 0.0261 |
|
| 173 |
+
| Laya-Multilingual, as shipped | 0.4985 |
|
| 174 |
+
| Laya-Typed-Decisions, as shipped | 0.2641 |
|
| 175 |
+
|
| 176 |
+
Measured on the same 2,018 items. **ZTC and JEV are effectively tied on calibration; the difference
|
| 177 |
+
of 0.0016 is not meaningful.** Figures published elsewhere for these systems were measured on other
|
| 178 |
+
test sets and do not reproduce here.
|
| 179 |
+
|
| 180 |
+
🔴 Calibration is uneven across domains: 0.0225 on biology & medicine, but 0.2941 on scientific
|
| 181 |
+
reasoning and 0.2381 on general reasoning. Treat the probability as reliable in the first case and
|
| 182 |
+
as a ranking signal only in the other two.
|
| 183 |
+
|
| 184 |
+
## What to do when the score is low
|
| 185 |
+
|
| 186 |
+
The score alone is not actionable, so the response carries the measured consequence of each choice.
|
| 187 |
+
|
| 188 |
+
| Band | Share | **Actual accuracy of answers in this band** | Recommended |
|
| 189 |
+
|---|---|---|---|
|
| 190 |
+
| `low` | 20% | **48.5%** | hold, or escalate to a stronger model |
|
| 191 |
+
| `mid` | 40% | 73.2% | escalate if budget allows |
|
| 192 |
+
| `high` | 40% | **89.6%** | accept |
|
| 193 |
+
|
| 194 |
+
**Three things that work, measured on 2,018 items:**
|
| 195 |
+
|
| 196 |
+
| Action | Effect |
|
| 197 |
+
|---|---|
|
| 198 |
+
| **Drop the lowest-scoring 20%** | accuracy of what remains: **74.8% → 81.4%** |
|
| 199 |
+
| **Escalate the lowest 20% to a stronger model** | **+1.34 pp** end-to-end |
|
| 200 |
+
| **Send the lowest 20% to human review** | catches **47.2%** of all errors — 2.4× random |
|
| 201 |
+
|
| 202 |
+
### Generate several candidates and let the verifier pick
|
| 203 |
+
|
| 204 |
+
When the score is low, the most effective next step is not to escalate — it is to **produce more
|
| 205 |
+
candidate answers and select between them**. Measured on 346 questions with five candidates each
|
| 206 |
+
(1,730 candidates), all scored through this endpoint:
|
| 207 |
+
|
| 208 |
+
| Policy | Accuracy | Output tokens | vs. one attempt |
|
| 209 |
+
|---|---|---|---|
|
| 210 |
+
| One attempt | 49.13% | 72 | — |
|
| 211 |
+
| Majority vote over 3 | 46.82% | 212 | **−2.31 pp** |
|
| 212 |
+
| Majority vote over 5 | 46.53% | 353 | **−2.60 pp** |
|
| 213 |
+
| **Pick best of 5 with this model** | **51.45%** | 353 | **+2.31 pp** |
|
| 214 |
+
| Pick best of 5 with JEV | 53.47% | 353 | +4.34 pp |
|
| 215 |
+
| Pick one of 5 at random | 46.82% | 353 | −2.31 pp |
|
| 216 |
+
| *Oracle — any correct candidate counts* | *63.87%* | *353* | *+14.74 pp* |
|
| 217 |
+
|
| 218 |
+
**The same five candidates swing by 5 points depending on how one is chosen.** Majority voting is
|
| 219 |
+
worse than not resampling at all: when a model prefers a wrong answer, more samples make that wrong
|
| 220 |
+
consensus more certain. A verifier that ranks the candidates is what turns extra samples into
|
| 221 |
+
accuracy.
|
| 222 |
+
|
| 223 |
+
**Spend the budget only where it is needed.** Generating extra candidates only for low-scoring
|
| 224 |
+
first attempts captures most of the gain at a fraction of the cost:
|
| 225 |
+
|
| 226 |
+
| Triggered on | Accuracy | Output tokens | vs. one attempt |
|
| 227 |
+
|---|---|---|---|
|
| 228 |
+
| 10% of items | 49.71% | 80 | +0.58 pp |
|
| 229 |
+
| **30% of items** | **50.87%** | **132** | **+1.73 pp** |
|
| 230 |
+
| 100% of items | 51.45% | 353 | +2.31 pp |
|
| 231 |
+
|
| 232 |
+
**At a 30% trigger rate you get three quarters of the benefit for 1.8× the tokens**, where always
|
| 233 |
+
generating costs 4.9× for 1.3× the benefit.
|
| 234 |
+
|
| 235 |
+
*Scope: one generator (GPT-4o-mini), one item set, five candidates. The oracle row shows the
|
| 236 |
+
headroom that remains — a correct candidate is present far more often than any policy recovers it.*
|
| 237 |
+
|
| 238 |
+
**One thing that does not work:**
|
| 239 |
+
|
| 240 |
+
🔴 **Do not take a majority vote over resamples.** Measured: five resamples with majority voting score **46.53%** where a single attempt scores **49.13%**. More candidates make a wrong consensus more certain unless something picks between them — see the table above.
|
| 241 |
+
|
| 242 |
+
|
| 243 |
+
|
| 244 |
+
*Escalation pays for itself through precision, not recall. Re-answering repairs about 38% of wrong
|
| 245 |
+
answers and damages about 30% of right ones, so a gate is only worth its budget if it mostly calls
|
| 246 |
+
answers that are actually wrong.*
|
| 247 |
+
|
| 248 |
+
---
|
| 249 |
+
|
| 250 |
+
## Protocol
|
| 251 |
+
|
| 252 |
+
| | |
|
| 253 |
+
|---|---|
|
| 254 |
+
| Items | 2,018 · 508 incorrect · 5 domains · answers written by 4 different models |
|
| 255 |
+
| Metric | AUC — how well wrong answers sort to the bottom. Threshold-free. 0.5 = coin flip |
|
| 256 |
+
| Selection | **Leave-one-domain-out.** Every figure comes from a domain the probe never saw; hyper-parameters are chosen inside the training domains only |
|
| 257 |
+
| Aggregation | Per domain, then size-weighted. Pooling all items into one AUC inflates the result |
|
| 258 |
+
|
| 259 |
+
The same protocol, item set and grading code are applied to every rung of the ladder and to the
|
| 260 |
+
other systems on the independent leaderboard:
|
| 261 |
+
<https://huggingface.co/spaces/mayafree/typed-decision-leaderboard>
|
| 262 |
+
|
| 263 |
+
## Out of scope
|
| 264 |
+
|
| 265 |
+
- **Not a grounding checker.** It does not take a source document and decide whether the answer
|
| 266 |
+
follows from it.
|
| 267 |
+
- **Not a safety, toxicity or policy classifier.**
|
| 268 |
+
- **Not a calibrated probability.** Use it to rank and threshold.
|
| 269 |
+
- **Not a general-purpose verifier at this size.** See the domain table above.
|
| 270 |
+
|
| 271 |
+
## Limitations
|
| 272 |
+
|
| 273 |
+
- **Domain coverage.** Scores are meaningful only for the five domains listed. Outside them nothing
|
| 274 |
+
has been measured and no guarantee is published.
|
| 275 |
+
- **Below the surface baseline on disaster and safety.** Stated in the table rather than omitted.
|
| 276 |
+
- **Sensitive to which model wrote the answer.** The probe is fitted on answers from four models.
|
| 277 |
+
Adding 1,772 answers from a single additional model shifted the mixture and **lowered** the
|
| 278 |
+
size-weighted score from 0.7278 to 0.7177 — professional exams rose to 0.8575 while every other
|
| 279 |
+
domain fell. Treat "works on any model's output" as a design goal, not a measured guarantee: if
|
| 280 |
+
your generator differs sharply from the training mixture, measure before relying on the number.
|
| 281 |
+
- **Revision lock.** The probe is fitted to one specific revision of the base model. Running it on a
|
| 282 |
+
different revision produces **no error and silently wrong scores**; this repository ships the
|
| 283 |
+
matching weights so that failure mode cannot occur.
|
| 284 |
+
- **It reports the verifier's judgement**, which is not the answering model's own confidence — that
|
| 285 |
+
quantity measures 0.5000 on this set.
|
| 286 |
+
|
| 287 |
+
## Lineage
|
| 288 |
+
|
| 289 |
+
| | |
|
| 290 |
+
|---|---|
|
| 291 |
+
| Base model | `Qwen/Qwen3.5-4B`, Apache-2.0 |
|
| 292 |
+
| Modification to base weights | none — the probes are separate files |
|
| 293 |
+
| Added by FINAL-Bench | probes, inference code, evaluation protocol and tables |
|
| 294 |
+
|
| 295 |
+
## What this repository contains
|
| 296 |
+
|
| 297 |
+
| | |
|
| 298 |
+
|---|---|
|
| 299 |
+
| Included | Base weights · tokenizer · linear probe · curved probe · configuration |
|
| 300 |
+
| Not included | Training corpus · hidden-state matrices · fitting pipeline |
|
| 301 |
+
|
| 302 |
+
## The rest of the ladder
|
| 303 |
+
|
| 304 |
+
[ZTC-Judge-27B](https://huggingface.co/FINAL-Bench/ZTC-Judge-27B) ·
|
| 305 |
+
[Darwin-397B-ZTC](https://huggingface.co/FINAL-Bench/Darwin-397B-ZTC) ·
|
| 306 |
+
[ZTC-Judge-9B](https://huggingface.co/FINAL-Bench/ZTC-Judge-9B) ·
|
| 307 |
+
[ZTC-Judge-4B](https://huggingface.co/FINAL-Bench/ZTC-Judge-4B)
|
| 308 |
+
|
| 309 |
+
## License
|
| 310 |
+
|
| 311 |
+
The base model is Apache-2.0 and redistributable. The probes, the inference code and the evaluation
|
| 312 |
+
tables are assets of FINAL-Bench / VIDRAFT.
|