Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
chaoliangUNSW 
posted an update about 11 hours ago
Post
45
Jev-style decisions on your laptop: Jev-Style-0.8B-Decision-v3

Give it any text and a question. It returns a calibrated probability for every option in one forward pass: pick one, yes/no, or a score. No text generation, nothing to parse.

On data it never trained on:
• Banking77, all 77 intents in one pass (400 test rows): 68.2% vs 49.2% for the best official Laya checkpoint (+19.0, paired 95% CI +14.0 to +24.0)
• tweet_topic, zero-shot: 75.5% vs 63.2% for English Laya, within 4 points of Jev (79.3%), and better calibrated than Jev (ECE 0.027 vs 0.063)
• JevBench v1.4.1, 231 public items, zero-shot: 64.1%, ahead of every Qwen3.5-0.8B-based system on the board; ahead of Laya (58.4%) too, though that lead is inside the CI

Jev is still ahead on all three. v3 is the open, local option that gets closest: 0.53 GB in 4-bit, runs on CPU.

Typed decisions (in-domain): 79.2% vs 76.6% for Laya typed, both trained on the same train split (paired CI +1.1 to +4.2). Read this as agreement with the dataset's teacher labels, not decision skill:
• teacher-noise reference: one draw from the teacher's own distribution matches the gold label 65.9% of the time (test split); the dataset card puts teacher self-agreement at 73.5%
• on the teacher's near-ties, v3 and Laya typed score the same (51.7% vs 51.7%); v3's lead comes from cases where the teacher is clear
• Jev's 72.7% there is zero-shot, so not a like-for-like comparison

Also: up to 25,600 tokens per call (98.3% on real 24K-token documents); about 44 ms per short decision on an M1 Max; one read of a document answers every question in one call; ahead of Laya multilingual in 51 of 51 languages.

Try it in your browser, no install:
chaoliangUNSW/jev-style-v3

Model (transformers, with GGUF and MLX builds linked):
chaoliangUNSW/Jev-Style-0.8B-Decision-v3

Apache-2.0. Not affiliated with TypeSafe, Jev or Laya. Thanks to @dipankarsarkar for pushing on the typed-decisions framing.
dipankarsarkar

79.2% is the wrong headline, and the typed-decisions card itself says why.

Its gold is the mean of 3 samples from one ~4B teacher. The card's reference points: majority 0.520, a model fitted to the true latent factors 0.704, teacher self-agreement 0.735.

Then: "A score much above 0.75 means a model has learned the teacher's quirks rather than the task."

v3 sits at 0.792, Wilson CI 77.3 to 80.9, so even the lower bound clears 0.75. Laya typed (0.766) is over the line too. Both trained on that teacher's train split. Jev, zero-shot, is at 0.727, under the ceiling.

So by the benchmark's own reading guide, the +6.4 over Jev is at least partly fitting the labeller. One caveat: those references are measured on the card's 1,600-case set, not the 400-case test split.

Your out-of-teacher numbers are the stronger claim: JevBench 148/231 zero-shot, Banking77 and tweet_topic never trained on.

Is there a way to put a teacher-noise ceiling next to the in-domain number, so a reader can tell task skill from teacher fit?

·
chaoliangUNSW

Thanks, this was a fair catch. I've rewritten the post around it: the headline is now the out-of-teacher results, and the typed number sits next to a teacher-noise reference.

On your question: an exact ceiling isn't possible with what the dataset releases. The test split ships only the mean of the three teacher samples plus a label_agreement flag, so the card's 73.5% can't be recomputed on the 400 test states. What I can add:

  • One draw from the teacher's mean distribution matches the gold argmax 65.9% of the time on the test split.
  • Split by teacher margin (gap between its top two probabilities), v3 vs Laya typed on the same rows:
teacher margin n v3 Laya typed diff (95% CI)
< 0.1 (near-ties) 315 51.7 51.7 +0.0 [−4.4, +4.5]
0.1–0.3 555 69.7 66.1 +3.6 [+0.2, +7.3]
≥ 0.3 1,130 91.4 88.7 +2.7 [+0.8, +4.6]

So v3 isn't gaining by fitting the teacher's coin-flips. That still can't separate task skill from the teacher's consistent biases, since both models trained on its labels. Only an independent gold can, which is why the post now leads with Banking77, tweet_topic and JevBench. The gap to Jev (72.7, zero-shot) shouldn't be read as skill, and the post no longer frames it that way.

The same table and reference are going on the model card.
dipankarsarkar

Reproduced your reference from the test parquet: mean top-1 of the gold distribution is 0.6589 over the 2,000 decisions. Same 65.9%.

Your buckets come out 312 / 557 / 1,131 for me, not 315 / 555 / 1,130. Probably just rounding at the 0.1 and 0.3 edges.

One thing the split does ship: label_agreement.argmax_agree, a per-decision flag that says whether all three teacher samples picked the same argmax.

It is true for 59.4% of test decisions. By your margin bands:

  • under 0.1: 1.3% unanimous
  • 0.1 to 0.3: 28.2%
  • 0.3 and up: 90.8%

So your near-tie row is almost exactly the set where the teacher disagreed with itself. The +0.0 there is the cleanest result on the card.

Would v3 vs Laya on the 1,188 unanimous decisions tell you anything the 0.3+ band doesn't?
In this post