AnkitAI commited on
Commit
90def02
·
verified ·
1 Parent(s): bf802cd

update readme

Browse files
Files changed (1) hide show
  1. README.md +49 -6
README.md CHANGED
@@ -50,19 +50,36 @@ else, because it never generates text; it scores the options you gave it and sto
50
  ## Watch it decide
51
 
52
  <div align="center">
53
- <img alt="TinyJev answering 12 typed decisions about one support ticket in one forward pass, next to GPT-6 Sol writing the same answers as JSON token by token" src="https://raw.githubusercontent.com/ankit-aglawe/tinyjev/main/assets/demo_batch.gif" width="860">
54
  </div>
55
 
56
- One support ticket, twelve typed decisions: which team, what priority, what the customer wants. TinyJev scores every option of every question in a single forward pass and returns all twelve together, 596 ms on a base M1, 10 of 12 right. GPT-6 Sol writes the same twelve as JSON, token by token: 2,198 ms, 12 of 12 right. Both lanes are real runs replayed at real speed; the ticket, the questions, the expected answers and both measurements are in the [repository](https://github.com/ankit-aglawe/tinyjev).
 
 
 
 
 
 
 
 
 
57
 
58
  <details>
59
- <summary><b>Six single questions against GPT-6 Sol</b> — click to expand</summary>
60
 
61
  <div align="center">
62
- <img alt="TinyJev and GPT-6 Sol answering the same six never-seen questions from a shared start" src="https://raw.githubusercontent.com/ankit-aglawe/tinyjev/main/assets/demo_race.gif" width="860">
63
  </div>
64
 
65
- Six decisions from six domains it never trained on, the same question to both at the same instant. TinyJev 6 of 6 at 86 ms a question; GPT-6 Sol 6 of 6 at 2,042 ms.
 
 
 
 
 
 
 
 
66
 
67
  </details>
68
 
@@ -73,7 +90,12 @@ Six decisions from six domains it never trained on, the same question to both at
73
  <img alt="TinyJev triaging support tickets" src="https://raw.githubusercontent.com/ankit-aglawe/tinyjev/main/assets/demo_triage.gif" width="860">
74
  </div>
75
 
76
- Eight real support tickets, three questions each in a single forward pass, about 110 ms per ticket on a base M1. Every number in that recording came from a live run.
 
 
 
 
 
77
 
78
  </details>
79
 
@@ -95,6 +117,27 @@ This is a demo, not a benchmark: on structured numeric state the answer barely m
95
 
96
  </details>
97
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
98
  ## Measured
99
 
100
  On OpenDecision's Original Choice 500, a suite of 25 domains that was not in the training data, with the same 500 inputs for every model:
 
50
  ## Watch it decide
51
 
52
  <div align="center">
53
+ <img alt="TinyJev and GPT-6 Sol answering the same six never-seen questions from a shared start; TinyJev finishes each in under 130 ms, GPT-6 Sol writes JSON for about two seconds" src="https://raw.githubusercontent.com/ankit-aglawe/tinyjev/main/assets/demo_race.gif" width="860">
54
  </div>
55
 
56
+ Six decisions from six domains it never trained on, the same question to both models at the same
57
+ instant. TinyJev answers in one forward pass: 6 of 6 right, 86 ms a question on a base M1. GPT-6 Sol
58
+ writes the answer as JSON, token by token: 6 of 6 right, 2,042 ms a question. Both lanes are real
59
+ runs replayed at real speed; the API lane is a recording with wall-clock timestamps. The questions,
60
+ the answers and both timings are in [`assets/recordings`](assets/recordings).
61
+
62
+ ```bash
63
+ pip install 'tinyjev[mlx,demo]'
64
+ python demos/race.py --recording assets/recordings/gpt-6-sol.jsonl --llm-name "GPT-6 Sol" --gif demo.gif
65
+ ```
66
 
67
  <details>
68
+ <summary><b>Twelve decisions about one ticket, in one pass</b> — click to expand</summary>
69
 
70
  <div align="center">
71
+ <img alt="TinyJev answering 12 typed decisions about one support ticket in one forward pass, next to GPT-6 Sol writing the same answers as JSON token by token" src="https://raw.githubusercontent.com/ankit-aglawe/tinyjev/main/assets/demo_batch.gif" width="860">
72
  </div>
73
 
74
+ One support ticket, twelve typed decisions: which team, what priority, what the customer wants.
75
+ TinyJev scores every option of every question in a single forward pass and returns all twelve
76
+ together, 596 ms, 10 of 12 right. GPT-6 Sol writes the same twelve as JSON: 2,198 ms, 12 of 12 right.
77
+ The ticket, the questions, the expected answers and both measurements are in
78
+ [`demos/cases`](demos/cases) and [`assets/recordings`](assets/recordings).
79
+
80
+ ```bash
81
+ python demos/batch_race.py --data assets/recordings/batch-support-ticket-gpt-6-sol.json --gif demo.gif
82
+ ```
83
 
84
  </details>
85
 
 
90
  <img alt="TinyJev triaging support tickets" src="https://raw.githubusercontent.com/ankit-aglawe/tinyjev/main/assets/demo_triage.gif" width="860">
91
  </div>
92
 
93
+ Eight real support tickets, one after another, on a base M1. Three questions per ticket in a
94
+ single forward pass, about 110 ms each. Every number in that recording came from a live run.
95
+
96
+ ```bash
97
+ python demos/triage_desk.py --gif demo.gif
98
+ ```
99
 
100
  </details>
101
 
 
117
 
118
  </details>
119
 
120
+ ## Models
121
+
122
+ Two models so far, same head, same training data, scored on the same 500 never-seen cases from 25
123
+ domains ([`benchmarks/opendecision`](https://github.com/ankit-aglawe/tinyjev/tree/main/benchmarks/opendecision), every case and probability logged).
124
+ Latency is a base M1 (16 GB) via MLX, one forward pass per case.
125
+
126
+ | Model | Params | OD-500 | ECE | Gate 0.85 | Cov. @ 2% err | transfer-v4 | ms / case | Size | Weights |
127
+ |---|---:|---:|---:|---|---:|---:|---:|---:|---|
128
+ | <img src="https://raw.githubusercontent.com/ankit-aglawe/tinyjev/main/assets/logos/tinyjev.png" width="18"> **TinyJev&nbsp;0.6B** | 596M | 440 (88.0%) | 0.071 | 59% @ 98.0% | 63% | 0.625 | 85 | 1.2 GB | 🤗 [AnkitAI/tinyjev-0.6b](https://huggingface.co/AnkitAI/tinyjev-0.6b) |
129
+ | <img src="https://raw.githubusercontent.com/ankit-aglawe/tinyjev/main/assets/logos/tinyjev.png" width="18"> **TinyJev&nbsp;4B** | 4.0B | 474 (94.8%) | 0.022 | 87% @ 99.1% | 92% | 0.762 | 628 | 8.0 GB | 🤗 release pending |
130
+
131
+ OD-500 is correct answers out of 500. Gate 0.85 is the share of decisions answered on its own at
132
+ confidence ≥ 0.85, and how often those were right. Load either with `tinyjev.load("tinyjev-0.6b")`
133
+ or `tinyjev.load("tinyjev-4b")` once released.
134
+
135
+ Both rows are fp16. Loading with `quantize=8` keeps the same weights in half the memory and changes
136
+ almost nothing: the 0.6B scores 440 at 90 ms, the 4B 473 at 845 ms, one answer in 500 different from
137
+ fp16. The gate is the number that matters in production; the rest of the queue goes to a person or a
138
+ bigger model. On the same 500: Kev-0.8B 463, Claude Opus 5.5 496, the same Qwen3-0.6B weights read
139
+ through letter logits with no head 354.
140
+
141
  ## Measured
142
 
143
  On OpenDecision's Original Choice 500, a suite of 25 domains that was not in the training data, with the same 500 inputs for every model: