Post
68
I benchmarked TypeSafe's Jev against open, CPU-only stacks. 10,000 decisions, every raw output published.
Jev returns a typed choice with calibrated probabilities, and the options arrive per request — so it can't be a fixed classifier head. It has to score (state, option) pairs. I rebuilt that from open parts and measured instead of guessing.
Zero-shot, neither side given labels:
• typesafe/jev (hosted): 79.3%, 381 ms
• ModernBERT-base-zeroshot cross-encoder, 149M, CPU: 78.7%
• bi-encoder + 8-float head: 69.9%, 15.7 ms
• Qwen2.5-0.5B generative: 49.6%, with 4.0% unparseable outputs
A 149M open model on a laptop CPU lands 0.6 points behind a hosted commercial model.
With ~2,000 labels per task, an 8M static embedding plus a decision layer of literally eight floats reaches 78.2% at 0.1 ms in 82 MB, and trains in 0.1 s. On BANKING77 (77-way routing) it gets 88.8% vs Jev's 77.8%.
I also published the eight things that did NOT work: template ensembling 0.0pp, scaling 149M to 395M net zero, distillation -5.4pp, listwise reranking -1.0pp. The only thing that worked was architectural — scoring (text, option) jointly instead of comparing two embeddings. +12 to +24pp.
And one I didn't expect: there is no neutral set of option descriptions. The same rewrite moves Jev +0.5pp, the bi-encoder +6.3pp, the cross-encoder -4.3pp.
Model (the eight floats): dylantom2012/fly-head-potion-8m
Per-item predictions from 5 stacks incl. Jev: dylantom2012/open-system-one-bench
Interactive results: dylantom2012/open-system-one
Code: https://github.com/zhlei07/open-system-one
I don't program. The hunch was mine; every experiment was designed and run by Claude Code in one session. Everything is committed so the numbers can be checked rather than trusted.
Jev returns a typed choice with calibrated probabilities, and the options arrive per request — so it can't be a fixed classifier head. It has to score (state, option) pairs. I rebuilt that from open parts and measured instead of guessing.
Zero-shot, neither side given labels:
• typesafe/jev (hosted): 79.3%, 381 ms
• ModernBERT-base-zeroshot cross-encoder, 149M, CPU: 78.7%
• bi-encoder + 8-float head: 69.9%, 15.7 ms
• Qwen2.5-0.5B generative: 49.6%, with 4.0% unparseable outputs
A 149M open model on a laptop CPU lands 0.6 points behind a hosted commercial model.
With ~2,000 labels per task, an 8M static embedding plus a decision layer of literally eight floats reaches 78.2% at 0.1 ms in 82 MB, and trains in 0.1 s. On BANKING77 (77-way routing) it gets 88.8% vs Jev's 77.8%.
I also published the eight things that did NOT work: template ensembling 0.0pp, scaling 149M to 395M net zero, distillation -5.4pp, listwise reranking -1.0pp. The only thing that worked was architectural — scoring (text, option) jointly instead of comparing two embeddings. +12 to +24pp.
And one I didn't expect: there is no neutral set of option descriptions. The same rewrite moves Jev +0.5pp, the bi-encoder +6.3pp, the cross-encoder -4.3pp.
Model (the eight floats): dylantom2012/fly-head-potion-8m
Per-item predictions from 5 stacks incl. Jev: dylantom2012/open-system-one-bench
Interactive results: dylantom2012/open-system-one
Code: https://github.com/zhlei07/open-system-one
I don't program. The hunch was mine; every experiment was designed and run by Claude Code in one session. Everything is committed so the numbers can be checked rather than trusted.
