mlboydaisuke commited on
Commit
a70d4da
·
verified ·
1 Parent(s): e8c3d5e

JevBench public 231 accuracy and latency (Mac, 2026-09-24)

Browse files

Same subsection as coreai-model-zoo a708c2e (models/laya-multilingual/README.md).

Files changed (1) hide show
  1. README.md +17 -0
README.md CHANGED
@@ -123,6 +123,23 @@ candidate questions it was not trained for), as `laya.load(...).predict` answers
123
  balanced accuracy **0.6114**, accuracy 0.5903 (the same at T = 1 and with the fitted calibration, and at
124
  both windows). That is the reference a port's decisions are compared with, not a claim about the port.
125
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
126
  ## Numerics gate
127
 
128
  The oracle is the publisher's package itself (`laya.load(<pinned snapshot>, subfolder="multilingual")`,
 
123
  balanced accuracy **0.6114**, accuracy 0.5903 (the same at T = 1 and with the fitted calibration, and at
124
  both windows). That is the reference a port's decisions are compared with, not a claim about the port.
125
 
126
+ ### JevBench public 231 (Mac, 2026-09-24)
127
+
128
+ | easy 48 | standard 72 | hard 111 | ECE hard | p50 | p95 | hard max |
129
+ |---:|---:|---:|---:|---:|---:|---:|
130
+ | 0.896 | 0.403 | 0.342 | 0.273 | 0.02 s | 0.20 s | 0.3 s |
131
+
132
+ The benchmark's own harness ([fstandhartinger/jevbench](https://github.com/fstandhartinger/jevbench)
133
+ `2fa63fa`, v1.4.0, `typesafe` adapter) ran the 231 public items against coreai-kit `adbc755`
134
+ `decide-cli serve` with the `macos/wfp16-s256` bundle, one question per request. The bundle's input
135
+ window is 256 tokens: 95 of the 111 hard items were truncated to it (hard states are up to 3,677 tokens
136
+ long), and no easy or standard item was (the longest is 107 tokens). Accuracy per tier and the hard
137
+ tier's ECE are JevBench's own scoring (argmax of the returned probabilities); p50 and p95 are per-request
138
+ latency over all 231 requests, hard max the maximum over the hard tier. Latency was measured without an
139
+ exclusive GPU window (contended), with this model's server running alone. JevBench's published scores
140
+ (Intelligence and the rest) are chance-corrected over 534 items, sealed ones included, and are not
141
+ comparable to these accuracies.
142
+
143
  ## Numerics gate
144
 
145
  The oracle is the publisher's package itself (`laya.load(<pinned snapshot>, subfolder="multilingual")`,