mlboydaisuke commited on
Commit
e08ef78
·
verified ·
1 Parent(s): c4a2952

JevBench public 231 accuracy and latency (Mac, 2026-09-24)

Browse files

Same subsection as coreai-model-zoo a708c2e (models/decider-0.8b/README.md).

Files changed (1) hide show
  1. README.md +16 -0
README.md CHANGED
@@ -116,6 +116,22 @@ a CPU-bound job from another lane ran on the same machine during the measurement
116
  prefill is S=1 on this graph, a System One request costs about `rows × (state + question
117
  tokens) / decode rate` — ten independent questions over a 300-token state are ~3,500 steps.
118
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
119
  ## Swift side
120
 
121
  `coreai-kit` has no `systemOne` op yet. The design —
 
116
  prefill is S=1 on this graph, a System One request costs about `rows × (state + question
117
  tokens) / decode rate` — ten independent questions over a 300-token state are ~3,500 steps.
118
 
119
+ ### JevBench public 231 (Mac, 2026-09-24)
120
+
121
+ | easy 48 | standard 72 | hard 111 | ECE hard | p50 | p95 | hard max |
122
+ |---:|---:|---:|---:|---:|---:|---:|
123
+ | 1.000 | 0.833 | 0.414 | 0.320 | 1.69 s | 29.10 s | 90.6 s |
124
+
125
+ The benchmark's own harness ([fstandhartinger/jevbench](https://github.com/fstandhartinger/jevbench)
126
+ `2fa63fa`, v1.4.0, `typesafe` adapter) ran the 231 public items against coreai-kit `adbc755`
127
+ `decide-cli serve` with the ship bundle, one question per request. Accuracy per tier and the hard
128
+ tier's ECE are JevBench's own scoring (argmax of the returned probabilities); p50 and p95 are
129
+ per-request latency over all 231 requests, hard max the maximum over the hard tier. Latency was
130
+ measured without an exclusive GPU window (contended), with this model's server running alone; the
131
+ graph's prefill is S=1, and at that kit commit the state was prefilled again for every question.
132
+ JevBench's published scores (Intelligence and the rest) are chance-corrected over 534 items, sealed
133
+ ones included, and are not comparable to these accuracies.
134
+
135
  ## Swift side
136
 
137
  `coreai-kit` has no `systemOne` op yet. The design —