Card: JevBench public items and speed measured again (GitHub b44b4c9)
Browse files
README.md
CHANGED
|
@@ -218,37 +218,41 @@ Per-task accuracy / ECE on the held-out datasets of the rebuilt set, v8 against
|
|
| 218 |
| TypeSafe workflow decisions, 102 rows, accuracy / NLL | 78.4% / 0.594 | 80.4% / 0.585 | +2.0 (β2.0 to +5.9) |
|
| 219 |
| 847 in-task validation rows, accuracy / NLL | 83.6% / 0.443 | 83.2% / 0.444 | β0.4 (β1.3 to +0.6) |
|
| 220 |
| Bespoke's public suite, 13 subsets, macro accuracy | 0.706 | 0.704 | |
|
| 221 |
-
| JevBench public items, easy / standard / hard accuracy | 1.000 / 0.
|
| 222 |
| OpenJev, 5,252 rows, accuracy / NLL | 64.1% / 0.906 | 63.3% / 0.916 | β0.8 (β1.3 to β0.3) |
|
| 223 |
|
| 224 |
The browser gain is in the served distribution rather than in the argmax: sampled play improves by ten points, greedy play by
|
| 225 |
under one. Tic-tac-toe and minesweeper play did not change; a 2B model without search loses most of those games either
|
| 226 |
-
way. The one measured regression is OpenJev, under one point.
|
|
|
|
|
|
|
| 227 |
|
| 228 |
**Bespoke's public suite** (13 human-labelled subsets, 3,880 records in Jev's wire format, answered through `system_one` as
|
| 229 |
shipped). decider-2b v10 macro 0.704 / micro 0.711; v9 0.701 / 0.711; Nimble-9B 0.748 / 0.759; Jev 1.13.0 0.760 / 0.773
|
| 230 |
-
(the last two copied from Bespoke's report). Per-subset numbers, the JevBench public-item comparison (decider-2b v10 is at 1.000 / 0.
|
| 231 |
hard public items, against Jev 1.13.0 at 1.000 / 0.986 / 0.730) and recordings of both versions on the same browser pages and
|
| 232 |
game boards are in the GitHub README.
|
| 233 |
|
| 234 |
## Speed
|
| 235 |
|
| 236 |
-
One NVIDIA
|
| 237 |
-
|
| 238 |
-
|
| 239 |
|
| 240 |
| setting | p50 latency | throughput |
|
| 241 |
|---|---|---|
|
| 242 |
-
| single request, eager PyTorch |
|
| 243 |
-
| single request, CUDA graphs + torch.compile (helper default) |
|
| 244 |
-
| batch of 32, in-process, bf16 |
|
| 245 |
-
| batch of 32, in-process, FP8 linears |
|
| 246 |
-
| HTTP server (
|
| 247 |
-
| HTTP server (
|
| 248 |
-
|
| 249 |
-
|
| 250 |
-
|
| 251 |
-
|
|
|
|
|
|
|
| 252 |
|
| 253 |
## Limitations
|
| 254 |
|
|
|
|
| 218 |
| TypeSafe workflow decisions, 102 rows, accuracy / NLL | 78.4% / 0.594 | 80.4% / 0.585 | +2.0 (β2.0 to +5.9) |
|
| 219 |
| 847 in-task validation rows, accuracy / NLL | 83.6% / 0.443 | 83.2% / 0.444 | β0.4 (β1.3 to +0.6) |
|
| 220 |
| Bespoke's public suite, 13 subsets, macro accuracy | 0.706 | 0.704 | |
|
| 221 |
+
| JevBench public items, easy / standard / hard accuracy | 1.000 / 0.875 / 0.441 | 1.000 / 0.889 / 0.459 | +1 / +2 items |
|
| 222 |
| OpenJev, 5,252 rows, accuracy / NLL | 64.1% / 0.906 | 63.3% / 0.916 | β0.8 (β1.3 to β0.3) |
|
| 223 |
|
| 224 |
The browser gain is in the served distribution rather than in the argmax: sampled play improves by ten points, greedy play by
|
| 225 |
under one. Tic-tac-toe and minesweeper play did not change; a 2B model without search loses most of those games either
|
| 226 |
+
way. The one measured regression is OpenJev, under one point. The JevBench row was read again for both versions on 2026-09-23, in process, bf16, decider-ai
|
| 227 |
+
1.2.1. The values first published here (standard / hard: v8 0.861 / 0.459, v10 0.847 / 0.459) came from the FP8 server of
|
| 228 |
+
2026-09-19 and do not reproduce item for item.
|
| 229 |
|
| 230 |
**Bespoke's public suite** (13 human-labelled subsets, 3,880 records in Jev's wire format, answered through `system_one` as
|
| 231 |
shipped). decider-2b v10 macro 0.704 / micro 0.711; v9 0.701 / 0.711; Nimble-9B 0.748 / 0.759; Jev 1.13.0 0.760 / 0.773
|
| 232 |
+
(the last two copied from Bespoke's report). Per-subset numbers, the JevBench public-item comparison (decider-2b v10 is at 1.000 / 0.889 / 0.459 on the easy / standard /
|
| 233 |
hard public items, against Jev 1.13.0 at 1.000 / 0.986 / 0.730) and recordings of both versions on the same browser pages and
|
| 234 |
game boards are in the GitHub README.
|
| 235 |
|
| 236 |
## Speed
|
| 237 |
|
| 238 |
+
One NVIDIA B300, decider-ai 1.2.1, measured 2026-09-23. Support-ticket states of about 230 tokens with 3 typed questions each
|
| 239 |
+
(the first 64 `support_tickets` examples). `decider.infer.Decider` uses shape-bucketed CUDA graphs; the batching server is
|
| 240 |
+
`decider/serve.py`, whose default since 1.1 is bf16.
|
| 241 |
|
| 242 |
| setting | p50 latency | throughput |
|
| 243 |
|---|---|---|
|
| 244 |
+
| single request, eager PyTorch | 18.9 ms | |
|
| 245 |
+
| single request, CUDA graphs + torch.compile (helper default) | 3.2 ms | |
|
| 246 |
+
| batch of 32, in-process, bf16 | 35.5 ms | about 2,700 decisions/s |
|
| 247 |
+
| batch of 32, in-process, FP8 linears | 32.3 ms | about 2,980 decisions/s |
|
| 248 |
+
| HTTP server `/decide` (bf16, default), 1 client | 6.1 ms | 158 req/s |
|
| 249 |
+
| HTTP server `/decide` (bf16, default), 64 clients | 134 ms | 436 req/s, 2,181 decisions/s |
|
| 250 |
+
| HTTP server `/decide` with `DECIDER_FP8=1`, 64 clients | 212 ms | 286 req/s, 1,429 decisions/s |
|
| 251 |
+
|
| 252 |
+
On the B300, FP8 is faster in process but slower through the server, so the server default is bf16. The schema-cache figures
|
| 253 |
+
were measured earlier on one GH200 and not repeated: with `Decider.schema`, 10 described questions on short chat messages ran at
|
| 254 |
+
11,180 decisions/s in a batch, and one question with 151 options at 19x the full-forward rate. FP8 (e4m3 weights, per-token
|
| 255 |
+
activation scales) changes accuracy and calibration by less than the evaluation noise.
|
| 256 |
|
| 257 |
## Limitations
|
| 258 |
|