JevBench public 231 accuracy and latency (Mac, 2026-09-24)
Browse filesSame subsection as coreai-model-zoo a708c2e (models/decider-0.8b/README.md).
README.md
CHANGED
|
@@ -116,6 +116,22 @@ a CPU-bound job from another lane ran on the same machine during the measurement
|
|
| 116 |
prefill is S=1 on this graph, a System One request costs about `rows × (state + question
|
| 117 |
tokens) / decode rate` — ten independent questions over a 300-token state are ~3,500 steps.
|
| 118 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 119 |
## Swift side
|
| 120 |
|
| 121 |
`coreai-kit` has no `systemOne` op yet. The design —
|
|
|
|
| 116 |
prefill is S=1 on this graph, a System One request costs about `rows × (state + question
|
| 117 |
tokens) / decode rate` — ten independent questions over a 300-token state are ~3,500 steps.
|
| 118 |
|
| 119 |
+
### JevBench public 231 (Mac, 2026-09-24)
|
| 120 |
+
|
| 121 |
+
| easy 48 | standard 72 | hard 111 | ECE hard | p50 | p95 | hard max |
|
| 122 |
+
|---:|---:|---:|---:|---:|---:|---:|
|
| 123 |
+
| 1.000 | 0.833 | 0.414 | 0.320 | 1.69 s | 29.10 s | 90.6 s |
|
| 124 |
+
|
| 125 |
+
The benchmark's own harness ([fstandhartinger/jevbench](https://github.com/fstandhartinger/jevbench)
|
| 126 |
+
`2fa63fa`, v1.4.0, `typesafe` adapter) ran the 231 public items against coreai-kit `adbc755`
|
| 127 |
+
`decide-cli serve` with the ship bundle, one question per request. Accuracy per tier and the hard
|
| 128 |
+
tier's ECE are JevBench's own scoring (argmax of the returned probabilities); p50 and p95 are
|
| 129 |
+
per-request latency over all 231 requests, hard max the maximum over the hard tier. Latency was
|
| 130 |
+
measured without an exclusive GPU window (contended), with this model's server running alone; the
|
| 131 |
+
graph's prefill is S=1, and at that kit commit the state was prefilled again for every question.
|
| 132 |
+
JevBench's published scores (Intelligence and the rest) are chance-corrected over 534 items, sealed
|
| 133 |
+
ones included, and are not comparable to these accuracies.
|
| 134 |
+
|
| 135 |
## Swift side
|
| 136 |
|
| 137 |
`coreai-kit` has no `systemOne` op yet. The design —
|