Thanks, this was a fair catch. I've rewritten the post around it: the headline is now the out-of-teacher results, and the typed number sits next to a teacher-noise reference.
On your question: an exact ceiling isn't possible with what the dataset releases. The test split ships only the mean of the three teacher samples plus a label_agreement flag, so the card's 73.5% can't be recomputed on the 400 test states. What I can add:
- One draw from the teacher's mean distribution matches the gold argmax 65.9% of the time on the test split.
- Split by teacher margin (gap between its top two probabilities), v3 vs Laya typed on the same rows:
| teacher margin | n | v3 | Laya typed | diff (95% CI) |
|---|---|---|---|---|
| < 0.1 (near-ties) | 315 | 51.7 | 51.7 | +0.0 [−4.4, +4.5] |
| 0.1–0.3 | 555 | 69.7 | 66.1 | +3.6 [+0.2, +7.3] |
| ≥ 0.3 | 1,130 | 91.4 | 88.7 | +2.7 [+0.8, +4.6] |
So v3 isn't gaining by fitting the teacher's coin-flips. That still can't separate task skill from the teacher's consistent biases, since both models trained on its labels. Only an independent gold can, which is why the post now leads with Banking77, tweet_topic and JevBench. The gap to Jev (72.7, zero-shot) shouldn't be read as skill, and the post no longer frames it that way.
The same table and reference are going on the model card.
