Ockev-3B: Ultra-Accurate Discriminative Decision Engine for AI Agent Deliverables

Ship the result, not the conversation. Ending the "Tomato-Egg Problem".

---

"I asked an AI agent to write a recipe for tomato scrambled eggs. It proudly delivered: 'Here is your tomato scrambled eggs (note: this dish contains no pork, no beef, no chicken, and no fish).' Why are AI agents so obsessed with telling us what they didn't do? Ship the result, not the conversation."
— Shangyin Tan, UC Berkeley

---

Model Summary

Ockev-3B is the flagship decision model in the Ockev family, built on Qwen/Qwen2.5-3B. It achieves 95.8% accuracy on TomatoEggBench-120, outperforming official TypeSafe Jev 1.13.0 (93.3%) while running completely locally in ~86ms.

Key Capabilities

• State-of-the-Art Accuracy (95.8%): Highest accuracy recorded on TomatoEggBench-120 across all tested systems.
• Zero Generated Tokens: Direct pointer-head decision readout.
• Local & Private: Zero data sent to third-party clouds.

---

Full Benchmark Breakdown (TomatoEggBench-120)

Domain	Total Items	Ockev-3B (Ours)	TypeSafe Jev 1.13.0	Ockev-1.5B (Ours)	SemIf 4B (Logits)
Code Repository	29	93.1% (27/29)	90.0% (26/29)	86.2% (25/29)	69.0% (20/29)
Excel / CSV Data	20	100.0% (20/20)	95.0% (19/20)	95.0% (19/20)	75.0% (15/20)
PDF Documents	20	100.0% (20/20)	100.0% (20/20)	100.0% (20/20)	80.0% (16/20)
Form RPA	30	100.0% (30/30)	100.0% (30/30)	100.0% (30/30)	76.7% (23/30)
Executive Slides	21	90.5% (19/21)	85.7% (18/21)	85.7% (18/21)	71.4% (15/21)
Overall Accuracy	120	95.8% (115/120)	93.3% (112/120)	92.5% (111/120)	74.7% (89/120)
Inference Latency	-	86 ms	250 ms	35 ms	850 ms
Execution Cost	-	$0.00 (Local)	$42.00 / 1B tok	$0.00 (Local)	$0.00 (Local)

Dataset: 🤗 fancyboi999/tomato-egg-bench

Benchmark Domain	Ockev-3B Accuracy	Official Jev 1.13.0
Code Repositories	93.1%	90.0%
Excel / CSV	100.0%	95.0%
PDF Documents	100.0%	100.0%
Form RPA	100.0%	100.0%
Executive Slide Decks	90.5%	85.7%
Overall (120 Tasks)	95.8%	93.3%

---

Quickstart

pip install ockev
ockev check path/to/deliverable.md

Python usage:

from ockev import OckevGate

gate = OckevGate(model_name_or_path="Qwen/Qwen2.5-3B")
res = gate.review_deliverable("Added JWT authentication.")
print(res)
# {'verdict': 'pass', 'confidence': 0.98, 'latency_ms': 86.4}

For full codebase and benchmarks, visit the GitHub Repository.