SeaWolf-AI commited on
Commit
b0bb160
·
verified ·
1 Parent(s): fd6d4de

정정: 폐기값(0.7364/0.7282) → 정본(0.7272/0.7289) + 정정 고지 추가

Browse files
Files changed (1) hide show
  1. README.md +312 -305
README.md CHANGED
@@ -1,305 +1,312 @@
1
- ---
2
- license: apache-2.0
3
- language:
4
- - en
5
- - ko
6
- tags:
7
- - ztc
8
- - answer-verification
9
- - hallucination-detection
10
- - zero-token
11
- - confidence-estimation
12
- library_name: transformers
13
- pipeline_tag: text-classification
14
- ---
15
-
16
- # ZTC-Judge-4B
17
-
18
- **Answer verification from a single forward pass, with zero generated tokens — at 4B.**
19
-
20
- ZTC-Judge-4B takes a question and an answer written by *any* model and scores whether that
21
- answer can be trusted. It is the bottom rung of a four-point size ladder measured under one
22
- identical protocol, and it is published so the shape of that ladder can be checked rather than
23
- asserted.
24
-
25
- > **ZTC** — Zero-Token Confidence · **Judge** — it evaluates *someone else's* answer, not its own
26
-
27
- ---
28
-
29
- ## Read this before deploying
30
-
31
- This model is **not** the strongest member of the family, and the card says so with numbers.
32
-
33
- | Model | Leaderboard AUC |
34
- |---|---|
35
- | Darwin-397B-ZTC | 0.7364 |
36
- | ZTC-Judge-27B | 0.7282 |
37
- | **ZTC-Judge-9B** | **0.6506** |
38
- | **ZTC-Judge-4B** | **0.6360** |
39
- | *Answer length and formatting only* | *0.6223* |
40
-
41
- **The ladder does not decline smoothly — it steps.** Between 9B and 27B the score moves 0.078,
42
- while between 4B and 9B it moves 0.015. Whatever carries verification quality is largely absent
43
- below 27B on this axis.
44
-
45
- **Where this model is worth deploying is one specific place**, and it is a real one:
46
-
47
- | Domain | Surface baseline | **4B** | Margin |
48
- |---|---|---|---|
49
- | **Professional exams (law · math · biology)** | 0.7138 | **0.7787** | ****+0.0649**** |
50
- | Scientific reasoning | 0.7272 | 0.5576 | 🔴 -0.1696 |
51
- | Biology & medicine | 0.5908 | 0.6223 | **+0.0315** |
52
- | Disaster & safety procedures | 0.5949 | 0.5842 | 🔴 -0.0107 |
53
- | General multi-step reasoning | 0.5420 | 0.5738 | **+0.0318** |
54
- | **Size-weighted mean** | **0.6223** | **0.6360** | ****+0.0137**** |
55
-
56
- 🔴 **Do not use this model for disaster and safety content.** In that domain it does not clear the
57
- surface baseline, which means it is reading answer shape rather than correctness there.
58
-
59
- ✅ **Professional-exam style content is where it earns its size.** It runs on a laptop, on CPU, and
60
- inside networks that never reach the internet — places a hosted API cannot go.
61
-
62
- ## How it works
63
-
64
- ```
65
- [question + answer] → one forward pass
66
- → final-layer hidden state at the last position (2560-d)
67
- → probe
68
- → score
69
- ```
70
-
71
- **Generated tokens: 0.** No access to the answering model's weights or logits is required; the text
72
- of the answer is the only input. Latency is one forward pass, and batching converts directly into
73
- throughput.
74
-
75
- ## Usage
76
-
77
- ```python
78
- import json
79
- import numpy as np, torch
80
- from huggingface_hub import hf_hub_download, snapshot_download
81
- from transformers import AutoModel, AutoTokenizer
82
-
83
- REPO = "FINAL-Bench/ZTC-Judge-4B"
84
- cfg = json.load(open(hf_hub_download(REPO, "ztc_config.json"), encoding="utf-8"))
85
- path = snapshot_download(REPO)
86
-
87
- tok = AutoTokenizer.from_pretrained(path)
88
- model = AutoModel.from_pretrained(path, dtype=torch.bfloat16, low_cpu_mem_usage=True).eval()
89
-
90
- def hidden(question, answer):
91
- b = tok([cfg["template"] % (question, answer)], return_tensors="pt",
92
- truncation=True, max_length=cfg["max_length"])
93
- dev = next(model.parameters()).device
94
- with torch.no_grad():
95
- h = model(input_ids=b["input_ids"].to(dev),
96
- attention_mask=b["attention_mask"].to(dev)).last_hidden_state
97
- return h[0, int(b["attention_mask"].sum()) - 1].float().numpy().astype(np.float64)
98
-
99
- # linear probe — one dot product
100
- p = np.load(hf_hub_download(REPO, "ztc_probe.npz"))
101
- v = hidden("Which defensive chemical does an insect release?", "C. Allomone")
102
- print(float(((v - p["mu"]) / p["sd"]) @ p["w"]))
103
- ```
104
-
105
- The score is an unbounded real number; higher means more likely correct. It is a **ranking signal**,
106
- not a calibrated probability — choose a threshold from your own review budget.
107
-
108
- ## Two probes ship with this model
109
-
110
- | File | Produces |
111
- |---|---|
112
- | `ztc_probe.npz` | linear readout — one dot product |
113
- | `ztc_curve_probe.npz` | **the reported figure 0.6360** — 256 anchors, RBF kernel |
114
-
115
- Both read the same input. The curved probe is the one to use when the number matters.
116
-
117
- ## API — drop-in for an existing JEV integration
118
-
119
- The endpoint takes the same request shape and returns the same response shape, so switching an
120
- existing integration is a URL change.
121
-
122
- ```bash
123
- POST /v1/evaluate
124
- Authorization: Bearer <token>
125
-
126
- {"model": "vidraft/ztc",
127
- "state": {"question": "...", "answer": "..."},
128
- "questions": {"correct": {"type": "boolean",
129
- "instructions": "Is the ANSWER factually correct?"}}}
130
- ```
131
-
132
- ```json
133
- {"model": "vidraft/ztc-judge-4b",
134
- "answers": {"correct": {
135
- "probability": 0.1043,
136
- "verdict": "review",
137
- "score": -0.72,
138
- "position": 0.268,
139
-
140
- "band": "low",
141
- "action": "hold_or_escalate",
142
- "measured": {
143
- "band_accuracy": 0.485,
144
- "base_accuracy": 0.748,
145
- "if_lowest_20pct_dropped": 0.814,
146
- "escalate_gain_at_20pct_budget": 0.0134,
147
- "do_not": "resample_same_model",
148
- "why_not": "measured: fixes 6.7% of wrong answers, breaks 13.1% of right ones"}}},
149
- "usage": {"generated_tokens": 0}}
150
- ```
151
-
152
- `type` accepts `boolean` and `noul`. Existing clients read `answers.<key>.probability` and ignore
153
- the rest; the additional fields are there for clients that want to act on the score rather than
154
- merely record it. **0.19 s per call, zero generated tokens.**
155
-
156
- ### What `probability` means
157
-
158
- The raw score is unbounded. The shipped calibration maps it to P(answer is correct), fitted
159
- **leave-one-domain-out** — the mapping never sees the domain it is applied to.
160
-
161
- | | Expected calibration error |
162
- |---|---|
163
- | **ZTC-Judge-27B (after calibration)** | **0.0245** |
164
- | JEV, as shipped | 0.0381 |
165
- | JEV, after the same calibration | 0.0261 |
166
- | Laya-Multilingual, as shipped | 0.4985 |
167
- | Laya-Typed-Decisions, as shipped | 0.2641 |
168
-
169
- Measured on the same 2,018 items. **ZTC and JEV are effectively tied on calibration; the difference
170
- of 0.0016 is not meaningful.** Figures published elsewhere for these systems were measured on other
171
- test sets and do not reproduce here.
172
-
173
- 🔴 Calibration is uneven across domains: 0.0225 on biology & medicine, but 0.2941 on scientific
174
- reasoning and 0.2381 on general reasoning. Treat the probability as reliable in the first case and
175
- as a ranking signal only in the other two.
176
-
177
- ## What to do when the score is low
178
-
179
- The score alone is not actionable, so the response carries the measured consequence of each choice.
180
-
181
- | Band | Share | **Actual accuracy of answers in this band** | Recommended |
182
- |---|---|---|---|
183
- | `low` | 20% | **48.5%** | hold, or escalate to a stronger model |
184
- | `mid` | 40% | 73.2% | escalate if budget allows |
185
- | `high` | 40% | **89.6%** | accept |
186
-
187
- **Three things that work, measured on 2,018 items:**
188
-
189
- | Action | Effect |
190
- |---|---|
191
- | **Drop the lowest-scoring 20%** | accuracy of what remains: **74.8% → 81.4%** |
192
- | **Escalate the lowest 20% to a stronger model** | **+1.34 pp** end-to-end |
193
- | **Send the lowest 20% to human review** | catches **47.2%** of all errors — 2.4× random |
194
-
195
- ### Generate several candidates and let the verifier pick
196
-
197
- When the score is low, the most effective next step is not to escalate — it is to **produce more
198
- candidate answers and select between them**. Measured on 346 questions with five candidates each
199
- (1,730 candidates), all scored through this endpoint:
200
-
201
- | Policy | Accuracy | Output tokens | vs. one attempt |
202
- |---|---|---|---|
203
- | One attempt | 49.13% | 72 | — |
204
- | Majority vote over 3 | 46.82% | 212 | **−2.31 pp** |
205
- | Majority vote over 5 | 46.53% | 353 | **−2.60 pp** |
206
- | **Pick best of 5 with this model** | **51.45%** | 353 | **+2.31 pp** |
207
- | Pick best of 5 with JEV | 53.47% | 353 | +4.34 pp |
208
- | Pick one of 5 at random | 46.82% | 353 | −2.31 pp |
209
- | *Oracle — any correct candidate counts* | *63.87%* | *353* | *+14.74 pp* |
210
-
211
- **The same five candidates swing by 5 points depending on how one is chosen.** Majority voting is
212
- worse than not resampling at all: when a model prefers a wrong answer, more samples make that wrong
213
- consensus more certain. A verifier that ranks the candidates is what turns extra samples into
214
- accuracy.
215
-
216
- **Spend the budget only where it is needed.** Generating extra candidates only for low-scoring
217
- first attempts captures most of the gain at a fraction of the cost:
218
-
219
- | Triggered on | Accuracy | Output tokens | vs. one attempt |
220
- |---|---|---|---|
221
- | 10% of items | 49.71% | 80 | +0.58 pp |
222
- | **30% of items** | **50.87%** | **132** | **+1.73 pp** |
223
- | 100% of items | 51.45% | 353 | +2.31 pp |
224
-
225
- **At a 30% trigger rate you get three quarters of the benefit for 1.8× the tokens**, where always
226
- generating costs 4.9× for 1.3× the benefit.
227
-
228
- *Scope: one generator (GPT-4o-mini), one item set, five candidates. The oracle row shows the
229
- headroom that remains — a correct candidate is present far more often than any policy recovers it.*
230
-
231
- **One thing that does not work:**
232
-
233
- 🔴 **Do not take a majority vote over resamples.** Measured: five resamples with majority voting score **46.53%** where a single attempt scores **49.13%**. More candidates make a wrong consensus more certain unless something picks between them — see the table above.
234
-
235
-
236
-
237
- *Escalation pays for itself through precision, not recall. Re-answering repairs about 38% of wrong
238
- answers and damages about 30% of right ones, so a gate is only worth its budget if it mostly calls
239
- answers that are actually wrong.*
240
-
241
- ---
242
-
243
- ## Protocol
244
-
245
- | | |
246
- |---|---|
247
- | Items | 2,018 · 508 incorrect · 5 domains · answers written by 4 different models |
248
- | Metric | AUC — how well wrong answers sort to the bottom. Threshold-free. 0.5 = coin flip |
249
- | Selection | **Leave-one-domain-out.** Every figure comes from a domain the probe never saw; hyper-parameters are chosen inside the training domains only |
250
- | Aggregation | Per domain, then size-weighted. Pooling all items into one AUC inflates the result |
251
-
252
- The same protocol, item set and grading code are applied to every rung of the ladder and to the
253
- other systems on the independent leaderboard:
254
- <https://huggingface.co/spaces/mayafree/typed-decision-leaderboard>
255
-
256
- ## Out of scope
257
-
258
- - **Not a grounding checker.** It does not take a source document and decide whether the answer
259
- follows from it.
260
- - **Not a safety, toxicity or policy classifier.**
261
- - **Not a calibrated probability.** Use it to rank and threshold.
262
- - **Not a general-purpose verifier at this size.** See the domain table above.
263
-
264
- ## Limitations
265
-
266
- - **Domain coverage.** Scores are meaningful only for the five domains listed. Outside them nothing
267
- has been measured and no guarantee is published.
268
- - **Below the surface baseline on disaster and safety.** Stated in the table rather than omitted.
269
- - **Sensitive to which model wrote the answer.** The probe is fitted on answers from four models.
270
- Adding 1,772 answers from a single additional model shifted the mixture and **lowered** the
271
- size-weighted score from 0.7278 to 0.7177 — professional exams rose to 0.8575 while every other
272
- domain fell. Treat "works on any model's output" as a design goal, not a measured guarantee: if
273
- your generator differs sharply from the training mixture, measure before relying on the number.
274
- - **Revision lock.** The probe is fitted to one specific revision of the base model. Running it on a
275
- different revision produces **no error and silently wrong scores**; this repository ships the
276
- matching weights so that failure mode cannot occur.
277
- - **It reports the verifier's judgement**, which is not the answering model's own confidence — that
278
- quantity measures 0.5000 on this set.
279
-
280
- ## Lineage
281
-
282
- | | |
283
- |---|---|
284
- | Base model | `Qwen/Qwen3.5-4B`, Apache-2.0 |
285
- | Modification to base weights | none — the probes are separate files |
286
- | Added by FINAL-Bench | probes, inference code, evaluation protocol and tables |
287
-
288
- ## What this repository contains
289
-
290
- | | |
291
- |---|---|
292
- | Included | Base weights · tokenizer · linear probe · curved probe · configuration |
293
- | Not included | Training corpus · hidden-state matrices · fitting pipeline |
294
-
295
- ## The rest of the ladder
296
-
297
- [ZTC-Judge-27B](https://huggingface.co/FINAL-Bench/ZTC-Judge-27B) ·
298
- [Darwin-397B-ZTC](https://huggingface.co/FINAL-Bench/Darwin-397B-ZTC) ·
299
- [ZTC-Judge-9B](https://huggingface.co/FINAL-Bench/ZTC-Judge-9B) ·
300
- [ZTC-Judge-4B](https://huggingface.co/FINAL-Bench/ZTC-Judge-4B)
301
-
302
- ## License
303
-
304
- The base model is Apache-2.0 and redistributable. The probes, the inference code and the evaluation
305
- tables are assets of FINAL-Bench / VIDRAFT.
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - en
5
+ - ko
6
+ tags:
7
+ - ztc
8
+ - answer-verification
9
+ - hallucination-detection
10
+ - zero-token
11
+ - confidence-estimation
12
+ library_name: transformers
13
+ pipeline_tag: text-classification
14
+ ---
15
+
16
+ # ZTC-Judge-4B
17
+
18
+ **Answer verification from a single forward pass, with zero generated tokens — at 4B.**
19
+
20
+ ZTC-Judge-4B takes a question and an answer written by *any* model and scores whether that
21
+ answer can be trusted. It is the bottom rung of a four-point size ladder measured under one
22
+ identical protocol, and it is published so the shape of that ladder can be checked rather than
23
+ asserted.
24
+
25
+ > **ZTC** — Zero-Token Confidence · **Judge** — it evaluates *someone else's* answer, not its own
26
+
27
+ ---
28
+
29
+ ## Read this before deploying
30
+
31
+ This model is **not** the strongest member of the family, and the card says so with numbers.
32
+
33
+ | Model | Leaderboard AUC |
34
+ |---|---|
35
+ | Darwin-397B-ZTC | 0.7272 |
36
+ | ZTC-Judge-27B | 0.7289 |
37
+ | **ZTC-Judge-9B** | **0.6506** |
38
+ | **ZTC-Judge-4B** | **0.6360** |
39
+ | *Answer length and formatting only* | *0.6223* |
40
+
41
+ > **Revised 2026-09-21.** Earlier revisions of this card reported **0.7364** for Darwin-397B-ZTC
42
+ > and **0.7282** for ZTC-Judge-27B. Those figures were produced by a run whose standardisation
43
+ > statistics were computed over all five domains, including the held-out one, which leaks a small
44
+ > amount of the evaluation domain into every figure. Re-run with the statistics fitted inside the
45
+ > training domains only, the figures are **0.7272** and **0.7289**. Cite the current values.
46
+
47
+
48
+ **The ladder does not decline smoothly — it steps.** Between 9B and 27B the score moves 0.078,
49
+ while between 4B and 9B it moves 0.015. Whatever carries verification quality is largely absent
50
+ below 27B on this axis.
51
+
52
+ **Where this model is worth deploying is one specific place**, and it is a real one:
53
+
54
+ | Domain | Surface baseline | **4B** | Margin |
55
+ |---|---|---|---|
56
+ | **Professional exams (law · math · biology)** | 0.7138 | **0.7787** | ****+0.0649**** |
57
+ | Scientific reasoning | 0.7272 | 0.5576 | 🔴 -0.1696 |
58
+ | Biology & medicine | 0.5908 | 0.6223 | **+0.0315** |
59
+ | Disaster & safety procedures | 0.5949 | 0.5842 | 🔴 -0.0107 |
60
+ | General multi-step reasoning | 0.5420 | 0.5738 | **+0.0318** |
61
+ | **Size-weighted mean** | **0.6223** | **0.6360** | ****+0.0137**** |
62
+
63
+ 🔴 **Do not use this model for disaster and safety content.** In that domain it does not clear the
64
+ surface baseline, which means it is reading answer shape rather than correctness there.
65
+
66
+ ✅ **Professional-exam style content is where it earns its size.** It runs on a laptop, on CPU, and
67
+ inside networks that never reach the internet — places a hosted API cannot go.
68
+
69
+ ## How it works
70
+
71
+ ```
72
+ [question + answer] → one forward pass
73
+ → final-layer hidden state at the last position (2560-d)
74
+ → probe
75
+ → score
76
+ ```
77
+
78
+ **Generated tokens: 0.** No access to the answering model's weights or logits is required; the text
79
+ of the answer is the only input. Latency is one forward pass, and batching converts directly into
80
+ throughput.
81
+
82
+ ## Usage
83
+
84
+ ```python
85
+ import json
86
+ import numpy as np, torch
87
+ from huggingface_hub import hf_hub_download, snapshot_download
88
+ from transformers import AutoModel, AutoTokenizer
89
+
90
+ REPO = "FINAL-Bench/ZTC-Judge-4B"
91
+ cfg = json.load(open(hf_hub_download(REPO, "ztc_config.json"), encoding="utf-8"))
92
+ path = snapshot_download(REPO)
93
+
94
+ tok = AutoTokenizer.from_pretrained(path)
95
+ model = AutoModel.from_pretrained(path, dtype=torch.bfloat16, low_cpu_mem_usage=True).eval()
96
+
97
+ def hidden(question, answer):
98
+ b = tok([cfg["template"] % (question, answer)], return_tensors="pt",
99
+ truncation=True, max_length=cfg["max_length"])
100
+ dev = next(model.parameters()).device
101
+ with torch.no_grad():
102
+ h = model(input_ids=b["input_ids"].to(dev),
103
+ attention_mask=b["attention_mask"].to(dev)).last_hidden_state
104
+ return h[0, int(b["attention_mask"].sum()) - 1].float().numpy().astype(np.float64)
105
+
106
+ # linear probe — one dot product
107
+ p = np.load(hf_hub_download(REPO, "ztc_probe.npz"))
108
+ v = hidden("Which defensive chemical does an insect release?", "C. Allomone")
109
+ print(float(((v - p["mu"]) / p["sd"]) @ p["w"]))
110
+ ```
111
+
112
+ The score is an unbounded real number; higher means more likely correct. It is a **ranking signal**,
113
+ not a calibrated probability — choose a threshold from your own review budget.
114
+
115
+ ## Two probes ship with this model
116
+
117
+ | File | Produces |
118
+ |---|---|
119
+ | `ztc_probe.npz` | linear readout — one dot product |
120
+ | `ztc_curve_probe.npz` | **the reported figure 0.6360** — 256 anchors, RBF kernel |
121
+
122
+ Both read the same input. The curved probe is the one to use when the number matters.
123
+
124
+ ## API — drop-in for an existing JEV integration
125
+
126
+ The endpoint takes the same request shape and returns the same response shape, so switching an
127
+ existing integration is a URL change.
128
+
129
+ ```bash
130
+ POST /v1/evaluate
131
+ Authorization: Bearer <token>
132
+
133
+ {"model": "vidraft/ztc",
134
+ "state": {"question": "...", "answer": "..."},
135
+ "questions": {"correct": {"type": "boolean",
136
+ "instructions": "Is the ANSWER factually correct?"}}}
137
+ ```
138
+
139
+ ```json
140
+ {"model": "vidraft/ztc-judge-4b",
141
+ "answers": {"correct": {
142
+ "probability": 0.1043,
143
+ "verdict": "review",
144
+ "score": -0.72,
145
+ "position": 0.268,
146
+
147
+ "band": "low",
148
+ "action": "hold_or_escalate",
149
+ "measured": {
150
+ "band_accuracy": 0.485,
151
+ "base_accuracy": 0.748,
152
+ "if_lowest_20pct_dropped": 0.814,
153
+ "escalate_gain_at_20pct_budget": 0.0134,
154
+ "do_not": "resample_same_model",
155
+ "why_not": "measured: fixes 6.7% of wrong answers, breaks 13.1% of right ones"}}},
156
+ "usage": {"generated_tokens": 0}}
157
+ ```
158
+
159
+ `type` accepts `boolean` and `noul`. Existing clients read `answers.<key>.probability` and ignore
160
+ the rest; the additional fields are there for clients that want to act on the score rather than
161
+ merely record it. **0.19 s per call, zero generated tokens.**
162
+
163
+ ### What `probability` means
164
+
165
+ The raw score is unbounded. The shipped calibration maps it to P(answer is correct), fitted
166
+ **leave-one-domain-out** — the mapping never sees the domain it is applied to.
167
+
168
+ | | Expected calibration error |
169
+ |---|---|
170
+ | **ZTC-Judge-27B (after calibration)** | **0.0245** |
171
+ | JEV, as shipped | 0.0381 |
172
+ | JEV, after the same calibration | 0.0261 |
173
+ | Laya-Multilingual, as shipped | 0.4985 |
174
+ | Laya-Typed-Decisions, as shipped | 0.2641 |
175
+
176
+ Measured on the same 2,018 items. **ZTC and JEV are effectively tied on calibration; the difference
177
+ of 0.0016 is not meaningful.** Figures published elsewhere for these systems were measured on other
178
+ test sets and do not reproduce here.
179
+
180
+ 🔴 Calibration is uneven across domains: 0.0225 on biology & medicine, but 0.2941 on scientific
181
+ reasoning and 0.2381 on general reasoning. Treat the probability as reliable in the first case and
182
+ as a ranking signal only in the other two.
183
+
184
+ ## What to do when the score is low
185
+
186
+ The score alone is not actionable, so the response carries the measured consequence of each choice.
187
+
188
+ | Band | Share | **Actual accuracy of answers in this band** | Recommended |
189
+ |---|---|---|---|
190
+ | `low` | 20% | **48.5%** | hold, or escalate to a stronger model |
191
+ | `mid` | 40% | 73.2% | escalate if budget allows |
192
+ | `high` | 40% | **89.6%** | accept |
193
+
194
+ **Three things that work, measured on 2,018 items:**
195
+
196
+ | Action | Effect |
197
+ |---|---|
198
+ | **Drop the lowest-scoring 20%** | accuracy of what remains: **74.8% → 81.4%** |
199
+ | **Escalate the lowest 20% to a stronger model** | **+1.34 pp** end-to-end |
200
+ | **Send the lowest 20% to human review** | catches **47.2%** of all errors — 2.4× random |
201
+
202
+ ### Generate several candidates and let the verifier pick
203
+
204
+ When the score is low, the most effective next step is not to escalate — it is to **produce more
205
+ candidate answers and select between them**. Measured on 346 questions with five candidates each
206
+ (1,730 candidates), all scored through this endpoint:
207
+
208
+ | Policy | Accuracy | Output tokens | vs. one attempt |
209
+ |---|---|---|---|
210
+ | One attempt | 49.13% | 72 | — |
211
+ | Majority vote over 3 | 46.82% | 212 | **−2.31 pp** |
212
+ | Majority vote over 5 | 46.53% | 353 | **−2.60 pp** |
213
+ | **Pick best of 5 with this model** | **51.45%** | 353 | **+2.31 pp** |
214
+ | Pick best of 5 with JEV | 53.47% | 353 | +4.34 pp |
215
+ | Pick one of 5 at random | 46.82% | 353 | −2.31 pp |
216
+ | *Oracle — any correct candidate counts* | *63.87%* | *353* | *+14.74 pp* |
217
+
218
+ **The same five candidates swing by 5 points depending on how one is chosen.** Majority voting is
219
+ worse than not resampling at all: when a model prefers a wrong answer, more samples make that wrong
220
+ consensus more certain. A verifier that ranks the candidates is what turns extra samples into
221
+ accuracy.
222
+
223
+ **Spend the budget only where it is needed.** Generating extra candidates only for low-scoring
224
+ first attempts captures most of the gain at a fraction of the cost:
225
+
226
+ | Triggered on | Accuracy | Output tokens | vs. one attempt |
227
+ |---|---|---|---|
228
+ | 10% of items | 49.71% | 80 | +0.58 pp |
229
+ | **30% of items** | **50.87%** | **132** | **+1.73 pp** |
230
+ | 100% of items | 51.45% | 353 | +2.31 pp |
231
+
232
+ **At a 30% trigger rate you get three quarters of the benefit for 1.8× the tokens**, where always
233
+ generating costs 4.9× for 1.3× the benefit.
234
+
235
+ *Scope: one generator (GPT-4o-mini), one item set, five candidates. The oracle row shows the
236
+ headroom that remains — a correct candidate is present far more often than any policy recovers it.*
237
+
238
+ **One thing that does not work:**
239
+
240
+ 🔴 **Do not take a majority vote over resamples.** Measured: five resamples with majority voting score **46.53%** where a single attempt scores **49.13%**. More candidates make a wrong consensus more certain unless something picks between them — see the table above.
241
+
242
+
243
+
244
+ *Escalation pays for itself through precision, not recall. Re-answering repairs about 38% of wrong
245
+ answers and damages about 30% of right ones, so a gate is only worth its budget if it mostly calls
246
+ answers that are actually wrong.*
247
+
248
+ ---
249
+
250
+ ## Protocol
251
+
252
+ | | |
253
+ |---|---|
254
+ | Items | 2,018 · 508 incorrect · 5 domains · answers written by 4 different models |
255
+ | Metric | AUC — how well wrong answers sort to the bottom. Threshold-free. 0.5 = coin flip |
256
+ | Selection | **Leave-one-domain-out.** Every figure comes from a domain the probe never saw; hyper-parameters are chosen inside the training domains only |
257
+ | Aggregation | Per domain, then size-weighted. Pooling all items into one AUC inflates the result |
258
+
259
+ The same protocol, item set and grading code are applied to every rung of the ladder and to the
260
+ other systems on the independent leaderboard:
261
+ <https://huggingface.co/spaces/mayafree/typed-decision-leaderboard>
262
+
263
+ ## Out of scope
264
+
265
+ - **Not a grounding checker.** It does not take a source document and decide whether the answer
266
+ follows from it.
267
+ - **Not a safety, toxicity or policy classifier.**
268
+ - **Not a calibrated probability.** Use it to rank and threshold.
269
+ - **Not a general-purpose verifier at this size.** See the domain table above.
270
+
271
+ ## Limitations
272
+
273
+ - **Domain coverage.** Scores are meaningful only for the five domains listed. Outside them nothing
274
+ has been measured and no guarantee is published.
275
+ - **Below the surface baseline on disaster and safety.** Stated in the table rather than omitted.
276
+ - **Sensitive to which model wrote the answer.** The probe is fitted on answers from four models.
277
+ Adding 1,772 answers from a single additional model shifted the mixture and **lowered** the
278
+ size-weighted score from 0.7278 to 0.7177 — professional exams rose to 0.8575 while every other
279
+ domain fell. Treat "works on any model's output" as a design goal, not a measured guarantee: if
280
+ your generator differs sharply from the training mixture, measure before relying on the number.
281
+ - **Revision lock.** The probe is fitted to one specific revision of the base model. Running it on a
282
+ different revision produces **no error and silently wrong scores**; this repository ships the
283
+ matching weights so that failure mode cannot occur.
284
+ - **It reports the verifier's judgement**, which is not the answering model's own confidence — that
285
+ quantity measures 0.5000 on this set.
286
+
287
+ ## Lineage
288
+
289
+ | | |
290
+ |---|---|
291
+ | Base model | `Qwen/Qwen3.5-4B`, Apache-2.0 |
292
+ | Modification to base weights | none — the probes are separate files |
293
+ | Added by FINAL-Bench | probes, inference code, evaluation protocol and tables |
294
+
295
+ ## What this repository contains
296
+
297
+ | | |
298
+ |---|---|
299
+ | Included | Base weights · tokenizer · linear probe · curved probe · configuration |
300
+ | Not included | Training corpus · hidden-state matrices · fitting pipeline |
301
+
302
+ ## The rest of the ladder
303
+
304
+ [ZTC-Judge-27B](https://huggingface.co/FINAL-Bench/ZTC-Judge-27B) ·
305
+ [Darwin-397B-ZTC](https://huggingface.co/FINAL-Bench/Darwin-397B-ZTC) ·
306
+ [ZTC-Judge-9B](https://huggingface.co/FINAL-Bench/ZTC-Judge-9B) ·
307
+ [ZTC-Judge-4B](https://huggingface.co/FINAL-Bench/ZTC-Judge-4B)
308
+
309
+ ## License
310
+
311
+ The base model is Apache-2.0 and redistributable. The probes, the inference code and the evaluation
312
+ tables are assets of FINAL-Bench / VIDRAFT.