VIDraft/ai-civilization-storage
3.46 GB
openfree
openfree
AI & ML interests
Contact: arxivgpt@gmail.com
Recent Activity
updated a bucket about 7 hours ago
VIDraft/ai-civilization-storage updated a Space about 15 hours ago
VIDraft/AI liked a Space 2 days ago
FINAL-Bench/Tetris-JEV-LAYA-ZTCOrganizations
updated a
bucket about 7 hours ago
updated a
Space about 15 hours ago
liked a
Space 2 days ago
reacted to SeaWolf-AI's post with 👍🤗 2 days ago
Post
3342
The cost of a judging gate is usually quoted as a number. This puts it on a Tetris board.
Three boards get the same piece order, and on every move the same proposal and the same noise — a paired comparison. The gate decides one thing: keep this move, or draw again. Each board gets the same 60 seconds of gate time.
The text-writing gates get through 15–22 moves. The generation-free gate gets through 40–50. The boards that stop simply run out of clock.
It does not win on accuracy: on the same 2,018-question LODO set, JEV scores AUC 0.7350 against ZTC-Judge-27B's 0.7289. The separation is elsewhere. Clock — 2.1 s vs 0.0615 s per call, and on a 200-candidate agent screen one judging call measured 3.206 s generative vs 0.033 s readout, same server. Calibration — a gate is a threshold, and at ECE 0.4985 (vs ZTC 0.0245) a threshold stops carrying information. Mechanism — a text judge can name option 42 when there is no option 42; a scoring readout cannot. Not a lower error rate. No path.
The curve in the ZTC panel is real online fitting, scored prequentially — predict first, learn after — with base weights untouched. Not recursive self-improvement.
Limits, also stated on the page: Laya's AUC and latency are not our measurements and are set equal to JEV's, so calibration is the only measured axis it differs on. The page is a simulation driven by measured constants.
KO / EN / ZH.
FINAL-Bench/Tetris-JEV-LAYA-ZTC
FINAL-Bench/ZTC-Judge-27B
Three boards get the same piece order, and on every move the same proposal and the same noise — a paired comparison. The gate decides one thing: keep this move, or draw again. Each board gets the same 60 seconds of gate time.
The text-writing gates get through 15–22 moves. The generation-free gate gets through 40–50. The boards that stop simply run out of clock.
It does not win on accuracy: on the same 2,018-question LODO set, JEV scores AUC 0.7350 against ZTC-Judge-27B's 0.7289. The separation is elsewhere. Clock — 2.1 s vs 0.0615 s per call, and on a 200-candidate agent screen one judging call measured 3.206 s generative vs 0.033 s readout, same server. Calibration — a gate is a threshold, and at ECE 0.4985 (vs ZTC 0.0245) a threshold stops carrying information. Mechanism — a text judge can name option 42 when there is no option 42; a scoring readout cannot. Not a lower error rate. No path.
The curve in the ZTC panel is real online fitting, scored prequentially — predict first, learn after — with base weights untouched. Not recursive self-improvement.
Limits, also stated on the page: Laya's AUC and latency are not our measurements and are set equal to JEV's, so calibration is the only measured axis it differs on. The page is a simulation driven by measured constants.
KO / EN / ZH.
FINAL-Bench/Tetris-JEV-LAYA-ZTC
FINAL-Bench/ZTC-Judge-27B
- 5 replies
liked 2
models 4 days ago
reacted to SeaWolf-AI's post with 🔥 4 days ago
Post
3142
Ask a language model how confident it is and you get an AUC of 0.5000. Exactly a coin flip. We measured it across 2,018 items.
FINAL-Bench/gate-tetris
https://huggingface.co/blog/FINAL-Bench/ztc
Collection: https://huggingface.co/collections/FINAL-Bench/ztc-models-jev-ecosystems
Zero-Token Confidence (ZTC) reads it. One forward pass over the model's hidden state returns a calibrated probability that the answer is correct. Zero generated tokens.
It sits at the top of the shared board. Same 2,018 items, same harness for every entry: ZTC on Darwin-397B 0.7394, JEV 0.7335, ZTC-Judge-27B 0.7255, a surface baseline that reads only answer length and formatting 0.7036, Lynx 8B 0.5157, the model's own self-reported confidence 0.5000, HHEM 0.4852. First and third place both emit nothing at all.
The number worth staring at is 0.7036. That is a baseline reading no content whatsoever, just how long the answer is and how it is formatted. Any verifier scoring below it is not reading content either.
On speed, one gate call costs 0.0615 seconds, measured on four B200s across 2,000 items. Generating a single candidate answer takes 1.631 seconds, so the gate is 26 times cheaper than the work it guards. A verifier that generates competes with your agent for the same budget. A verifier that only reads can be attached to every action instead of a sampled few.
We built it so you can watch it decide. Three lanes receive the same stream of proposed actions and the same time budget. One has no gate and must execute everything. One uses a text-reading verifier. One uses ZTC. Right action plus one, wrong action minus one, hold zero. Over 400 matches: no gate minus 3.9, text verifier plus 13.0, ZTC plus 29.1, with ZTC taking 98 percent of matches. Gating lifts executed accuracy from 49 percent to 65 percent.
FINAL-Bench/gate-tetris
https://huggingface.co/blog/FINAL-Bench/ztc
Collection: https://huggingface.co/collections/FINAL-Bench/ztc-models-jev-ecosystems
Zero-Token Confidence (ZTC) reads it. One forward pass over the model's hidden state returns a calibrated probability that the answer is correct. Zero generated tokens.
It sits at the top of the shared board. Same 2,018 items, same harness for every entry: ZTC on Darwin-397B 0.7394, JEV 0.7335, ZTC-Judge-27B 0.7255, a surface baseline that reads only answer length and formatting 0.7036, Lynx 8B 0.5157, the model's own self-reported confidence 0.5000, HHEM 0.4852. First and third place both emit nothing at all.
The number worth staring at is 0.7036. That is a baseline reading no content whatsoever, just how long the answer is and how it is formatted. Any verifier scoring below it is not reading content either.
On speed, one gate call costs 0.0615 seconds, measured on four B200s across 2,000 items. Generating a single candidate answer takes 1.631 seconds, so the gate is 26 times cheaper than the work it guards. A verifier that generates competes with your agent for the same budget. A verifier that only reads can be attached to every action instead of a sampled few.
We built it so you can watch it decide. Three lanes receive the same stream of proposed actions and the same time budget. One has no gate and must execute everything. One uses a text-reading verifier. One uses ZTC. Right action plus one, wrong action minus one, hold zero. Over 400 matches: no gate minus 3.9, text verifier plus 13.0, ZTC plus 29.1, with ZTC taking 98 percent of matches. Gating lifts executed accuracy from 49 percent to 65 percent.
liked a
Space 5 days ago
liked a
model 5 days ago
upvoted a collection 5 days ago
liked 2
Spaces 5 days ago
reacted to mayafree's post with 🚀🔥 5 days ago
Post
3455
JEV Ecosystems — every answer-verification vendor publishes a benchmark, and every one of them wins it. So we ran 13 of them on one test set: 2,018 items, identical labels, same grading code.
🎯 Leaderboard
mayafree/typed-decision-leaderboard
📄 Full write-up (method, mechanism, limits)
https://huggingface.co/blog/mayafree/jve-ecosystems
🧪 Try it — ZTC, JEV and Laya on the same input, side by side
mayafree/verifier-playground
Three results
1️⃣ Only three systems clear 0.70 — ZTC (397B) 0.7364 · JEV 0.7350 · ZTC (27B) 0.7282. First and second differ by 0.0014, so no rank is assigned.
2️⃣ A baseline that reads nothing but answer length and formatting scores 0.7036. Eight of the thirteen fall below it. A leaderboard without that line is flattering its entrants.
3️⃣ Bigger does not win. On scientific reasoning, 27B 0.7410 beats 397B 0.6287 — a model fourteen times larger scoring 0.11 lower.
And AUC is not the number you deploy on.
Same 20% retry budget, wired into an agent loop, against a 74.83% no-gate baseline:
ZTC +1.34 pp · JEV −0.07 pp · random −0.25 pp.
The mechanism is the interesting part. Re-answering is double-edged: 38% of wrong answers get fixed, and 30% of right answers get broken. So a gate is paid for by precision, not recall. Of the 403 items JEV routed for a retry, 216 were already correct.
0.0014 AUC apart; 1.4 points of end-to-end agent accuracy apart.
Scores, labels and grading code are published in full. Four public reproductions that would not run from their released artefacts are listed too, with the failure and a link, and no score.
Don't take the table's word for it — paste your own case into the playground and watch all three answer at once. Want a system added? Open a discussion on the Space.
🎯 Leaderboard
mayafree/typed-decision-leaderboard
📄 Full write-up (method, mechanism, limits)
https://huggingface.co/blog/mayafree/jve-ecosystems
🧪 Try it — ZTC, JEV and Laya on the same input, side by side
mayafree/verifier-playground
Three results
1️⃣ Only three systems clear 0.70 — ZTC (397B) 0.7364 · JEV 0.7350 · ZTC (27B) 0.7282. First and second differ by 0.0014, so no rank is assigned.
2️⃣ A baseline that reads nothing but answer length and formatting scores 0.7036. Eight of the thirteen fall below it. A leaderboard without that line is flattering its entrants.
3️⃣ Bigger does not win. On scientific reasoning, 27B 0.7410 beats 397B 0.6287 — a model fourteen times larger scoring 0.11 lower.
And AUC is not the number you deploy on.
Same 20% retry budget, wired into an agent loop, against a 74.83% no-gate baseline:
ZTC +1.34 pp · JEV −0.07 pp · random −0.25 pp.
The mechanism is the interesting part. Re-answering is double-edged: 38% of wrong answers get fixed, and 30% of right answers get broken. So a gate is paid for by precision, not recall. Of the 403 items JEV routed for a retry, 216 were already correct.
0.0014 AUC apart; 1.4 points of end-to-end agent accuracy apart.
Scores, labels and grading code are published in full. Four public reproductions that would not run from their released artefacts are listed too, with the failure and a link, and no score.
Don't take the table's word for it — paste your own case into the playground and watch all three answer at once. Want a system added? Open a discussion on the Space.
updated a
Space 6 days ago
liked a
model 6 days ago
updated a
bucket 6 days ago
updated a
dataset 7 days ago
published a
dataset 7 days ago