mlboydaisuke commited on
Commit
54b7b13
·
verified ·
1 Parent(s): ac8cbe3

JevBench public 231 accuracy and latency (Mac, 2026-09-24)

Browse files

Same subsection as coreai-model-zoo a708c2e (models/minicpm5-2b/README.md).

Files changed (1) hide show
  1. README.md +8 -0
README.md CHANGED
@@ -144,6 +144,14 @@ slow dequant path on the Mac GPU; block-32 lands on the fast quantized-matmul pa
144
  per-channel decode and 1.6× fp16's, at +155 MB. On the phone the two decode the same
145
  (bandwidth-bound), so block-32 wins on both.
146
 
 
 
 
 
 
 
 
 
147
  ## Quantization
148
 
149
  Weight-only **symmetric int8, per-block-32** (a scale per 32-wide block along the input dim; no
 
144
  per-channel decode and 1.6× fp16's, at +155 MB. On the phone the two decode the same
145
  (bandwidth-bound), so block-32 wins on both.
146
 
147
+ ### JevBench public 231 (Mac, 2026-09-24)
148
+
149
+ | easy 48 | standard 72 | hard 111 | ECE hard | p50 | p95 | hard max |
150
+ |---:|---:|---:|---:|---:|---:|---:|
151
+ | 0.979 | 0.708 | 0.459 | 0.241 | 0.14 s | 1.19 s | 1.8 s |
152
+
153
+ The benchmark's own harness ([fstandhartinger/jevbench](https://github.com/fstandhartinger/jevbench) `2fa63fa`, v1.4.0, `typesafe` adapter) ran the 231 public items against coreai-kit `adbc755` `decide-cli serve` with the `int8/` bundle, one question per request. This chat model was asked zero-shot under the kit's JSON decision prompt, with the catalog's calibration temperature 2.93 (fit on SemIf perturbations108). Accuracy per tier and the hard tier's ECE are JevBench's own scoring (argmax of the returned probabilities); p50 and p95 are per-request latency over all 231 requests, hard max the maximum over the hard tier. Latency was measured without an exclusive GPU window (contended), with this model's server running alone. JevBench's published scores (Intelligence and the rest) are chance-corrected over 534 items, sealed ones included, and are not comparable to these accuracies.
154
+
155
  ## Quantization
156
 
157
  Weight-only **symmetric int8, per-block-32** (a scale per 32-wide block along the input dim; no