JevBench public 231 accuracy and latency (Mac, 2026-09-24)
Browse filesSame subsection as coreai-model-zoo a708c2e (models/minicpm5-2b/README.md).
README.md
CHANGED
|
@@ -144,6 +144,14 @@ slow dequant path on the Mac GPU; block-32 lands on the fast quantized-matmul pa
|
|
| 144 |
per-channel decode and 1.6× fp16's, at +155 MB. On the phone the two decode the same
|
| 145 |
(bandwidth-bound), so block-32 wins on both.
|
| 146 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 147 |
## Quantization
|
| 148 |
|
| 149 |
Weight-only **symmetric int8, per-block-32** (a scale per 32-wide block along the input dim; no
|
|
|
|
| 144 |
per-channel decode and 1.6× fp16's, at +155 MB. On the phone the two decode the same
|
| 145 |
(bandwidth-bound), so block-32 wins on both.
|
| 146 |
|
| 147 |
+
### JevBench public 231 (Mac, 2026-09-24)
|
| 148 |
+
|
| 149 |
+
| easy 48 | standard 72 | hard 111 | ECE hard | p50 | p95 | hard max |
|
| 150 |
+
|---:|---:|---:|---:|---:|---:|---:|
|
| 151 |
+
| 0.979 | 0.708 | 0.459 | 0.241 | 0.14 s | 1.19 s | 1.8 s |
|
| 152 |
+
|
| 153 |
+
The benchmark's own harness ([fstandhartinger/jevbench](https://github.com/fstandhartinger/jevbench) `2fa63fa`, v1.4.0, `typesafe` adapter) ran the 231 public items against coreai-kit `adbc755` `decide-cli serve` with the `int8/` bundle, one question per request. This chat model was asked zero-shot under the kit's JSON decision prompt, with the catalog's calibration temperature 2.93 (fit on SemIf perturbations108). Accuracy per tier and the hard tier's ECE are JevBench's own scoring (argmax of the returned probabilities); p50 and p95 are per-request latency over all 231 requests, hard max the maximum over the hard tier. Latency was measured without an exclusive GPU window (contended), with this model's server running alone. JevBench's published scores (Intelligence and the rest) are chance-corrected over 534 items, sealed ones included, and are not comparable to these accuracies.
|
| 154 |
+
|
| 155 |
## Quantization
|
| 156 |
|
| 157 |
Weight-only **symmetric int8, per-block-32** (a scale per 32-wide block along the input dim; no
|