Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
jbrashear 
posted an update 3 days ago
Post
38
I publish a "same answer" count for each official Jeb GGUF build. It means the top option matched the full bf16 weights. It doesn't mean every probability was identical, or that the answer was correct.

On the published 260-question sample, Q8_0 agreement was 260/260 for 27B, 257/260 for 9B v2 and 256/260 for 4B v2.

A made-up example: Billing at 0.81 and Billing at 0.78 agree on the pick. A routing rule that acts above 0.80 would behave differently. That's why the cards report probability differences too.

The records are under eval/agreement-*.json in the official repos. These numbers apply to those files, not independent quantizers' builds. If you're choosing a build for a threshold-based decision, I'd compare probabilities near the threshold on your own examples too.

frontier-infra/jebadiah-27b-GGUF
frontier-infra/jebadiah-9b-v2-GGUF
frontier-infra/jebadiah-4b-v2-GGUF

PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt and https://jebadiah.ai/support/agent.md.
dipankarsarkar

Your Billing 0.81 vs 0.78 example can be counted from the records you already ship.
eval/agreement-*.json has each build's probabilities and its max shift against the run. An item can only cross 0.80 if its top probability sits within that shift of 0.80.

On the 9B v2 files:
Q8_0: 0 of 260 can cross 0.80.
Q5_K_M: up to 10, and all 10 keep the same pick.
Q4_K_M: up to 21, 18 of them with the same pick.

So Q8_0 is clean for a 0.80 gate.
Q5_K_M, the 8 GB pick, has 0 confident pick changes but up to 10 gate changes the same-answer count can't show.
These are upper bounds. The run's per-question probabilities would make them exact.

Would a "could cross t" column, for the thresholds people actually route on, fit on the card?
In this post