fancyboi999 commited on
Commit
5b1f702
·
verified ·
1 Parent(s): 2e32515

docs: enrich model card with systems comparison and benchmark breakdown

Browse files
Files changed (1) hide show
  1. README.md +68 -7
README.md CHANGED
@@ -9,17 +9,65 @@ tags:
9
  - jev
10
  - agent-guard
11
  - tomato-egg-problem
 
 
12
  ---
13
 
14
- # Ockev: 35ms Discriminative Decision Engine for AI Agent Deliverables
15
 
16
- Ship the result, not the conversation. Ending the "Tomato-Egg Problem" in AI agent deliverables.
17
 
18
- - **GitHub Repository**: [https://github.com/fancyboi999/ockev](https://github.com/fancyboi999/ockev)
19
- - **Base Model**: `Qwen/Qwen2.5-1.5B`
20
- - **Architecture**: Causal LM Backbone + Block-Causal Mask + Pointer Head
21
- - **TomatoEggBench-120 Accuracy**: 92.5% (Ockev-1.5B) / 95.8% (Ockev-3B)
22
- - **Inference Latency**: ~35ms (Apple Silicon MLX) / ~50ms (PyTorch CUDA)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
23
 
24
  ## Quickstart
25
 
@@ -27,3 +75,16 @@ Ship the result, not the conversation. Ending the "Tomato-Egg Problem" in AI age
27
  pip install ockev
28
  ockev check path/to/deliverable.md
29
  ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
9
  - jev
10
  - agent-guard
11
  - tomato-egg-problem
12
+ - mlx
13
+ - apple-silicon
14
  ---
15
 
16
+ <div align="center">
17
 
18
+ # Ockev-1.5B: 35ms Discriminative Decision Engine for AI Agent Deliverables
19
 
20
+ ### Ship the result, not the conversation. Ending the "Tomato-Egg Problem".
21
+
22
+ [![GitHub](https://img.shields.io/badge/GitHub-fancyboi999%2Fockev-blue.svg)](https://github.com/fancyboi999/ockev)
23
+ [![Benchmark](https://img.shields.io/badge/TomatoEggBench--120-92.5%25-emerald.svg)](https://github.com/fancyboi999/ockev)
24
+ [![Latency](https://img.shields.io/badge/Latency-35ms-orange.svg)]()
25
+
26
+ </div>
27
+
28
+ ---
29
+
30
+ > *"I asked an AI agent to write a recipe for tomato scrambled eggs. It proudly delivered: 'Here is your tomato scrambled eggs (note: this dish contains no pork, no beef, no chicken, and no fish).' Why are AI agents so obsessed with telling us what they didn't do? Ship the result, not the conversation."*
31
+ > — **Shangyin Tan**, UC Berkeley
32
+
33
+ ---
34
+
35
+ ## Model Summary
36
+
37
+ **Ockev-1.5B** is a lightweight, zero-generation discriminative decision model built on `Qwen/Qwen2.5-1.5B` and fine-tuned with a **Pointer Readout Head** and Block-Causal Masking.
38
+
39
+ It intercepts conversational residue, discarded alternatives, and negative echoes in AI agent deliverables before they reach users or production codebases.
40
+
41
+ ### Key Capabilities
42
+
43
+ - **Zero Generated Tokens**: Decisions are computed directly via scaled dot-product between decision query vectors and option key vectors. Zero hallucination, zero JSON repair.
44
+ - **35ms Latency**: 5x-7x faster than cloud decision services, 100% offline, free, and private.
45
+ - **Native Apple Silicon MLX**: Optimized for Metal GPU acceleration on MacBooks.
46
+
47
+ ---
48
+
49
+ ## Systems Comparison
50
+
51
+ | Execution Path | Generated Tokens | Time (ms) | JSON Parsing Failure Risk | Cost / 1M Checks |
52
+ | :--- | :---: | :---: | :---: | :---: |
53
+ | **Ockev-1.5B (Local Pointer Head)** | **0** | **35 ms** | **0% (Pure Softmax)** | **$0.00** |
54
+ | **TypeSafe Jev 1.13.0 (Cloud API)** | 0 | 250 ms | 0% (Typed) | ~$42.00 |
55
+ | **Generative LLM (e.g. GPT-4o-mini)** | 85–140 | 1,450 ms | 1.8% (Malformed JSON) | ~$8.50 |
56
+
57
+ ---
58
+
59
+ ## Benchmark Results (TomatoEggBench-120)
60
+
61
+ | Benchmark Domain | Ockev-1.5B Accuracy | Official Jev 1.13.0 |
62
+ | :--- | :---: | :---: |
63
+ | Code Repositories | 86.2% | 90.0% |
64
+ | Excel / CSV | 95.0% | 95.0% |
65
+ | PDF Documents | 100.0% | 100.0% |
66
+ | Form RPA | 100.0% | 100.0% |
67
+ | Executive Slide Decks | 85.7% | 85.7% |
68
+ | **Overall (120 Tasks)** | **92.5%** | **93.3%** |
69
+
70
+ ---
71
 
72
  ## Quickstart
73
 
 
75
  pip install ockev
76
  ockev check path/to/deliverable.md
77
  ```
78
+
79
+ Python usage:
80
+
81
+ ```python
82
+ from ockev import OckevGate
83
+
84
+ gate = OckevGate()
85
+ res = gate.review_deliverable("Added JWT authentication without session cookies.")
86
+ print(res)
87
+ # {'verdict': 'revise', 'confidence': 0.96, 'latency_ms': 34.2}
88
+ ```
89
+
90
+ For full codebase and benchmarks, visit the [GitHub Repository](https://github.com/fancyboi999/ockev).