Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
jbrashear 
posted an update about 1 month ago
Post
69
I publish Jeb's weights, but loading them in a chat window isn't enough to get a decision. The scripts read probabilities over the answer labels. A chat reply doesn't expose those.

For the official 9B Q8_0 build with Ollama, the local path is:

pip install jebadiah-decide
jeb doctor
jeb serve

Doctor pulls and checks the model. Serve gives your code POST http://localhost:8100/v1/systemone. It renders the prompt, reads the label probabilities and applies temperatures.json.

The published Ollama 0.34.4 check matched the full bf16 weights on 257 of 260 top answers. That's agreement on that sample, not an accuracy score. There are at most 20 options per question on this path.

If you're already running the weights locally, which decision would you want to put behind that endpoint?

The commands, records and caveats are here:
frontier-infra/jebadiah-9b-v2-GGUF

PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt and https://jebadiah.ai/support/agent.md.
dipankarsarkar

The Q4_K_M changes are not tie-breaks, and that is the part I'd want on the card.

From eval/agreement-Q4_K_M.json: 9 of its 20 changed answers have a top-2 gap above 0.1 in the GGUF's own probabilities.

pubmedqa-217 and pubmedqa-895 go to true at 0.69 and 0.70, where bf16 picked false.

Q8_0 looks different. Its 3 changes sit at gaps of 0.038, 0.0056 and 0.0013. Near-ties, and the same count as your bf16-vs-run row.

So an abstain-below-0.05 gate catches all 3 Q8_0 changes and only 9 of Q4_K_M's 20.

The rows carry probabilities but no gold label. If you have gold for the 260, do Q4's confident changes move toward the right answer or away from it?

·
jbrashear

Thanks @dipankarsarkar , good catch, and you're right.

I had gold for all 260, so I went back and scored every build against it. Your figures check out against the agreement files: 9 of the Q4_K_M changes are above a 0.1 gap, and the three Q8_0 gaps are what you said.

And the confident Q4_K_M changes mostly go the wrong way.

Build Changed Gap > 0.1 Toward gold Away from gold Right on the 260
bf16 174
Q8_0 3 0 0 1 173
Q4_K_M 20 9 5 (2 confident) 10 (6 confident) 169

The rest have both answers wrong. Of the 9 confident Q4_K_M changes, 2 move toward gold, 6 move away and 1 is wrong either way.

pubmedqa-217 and pubmedqa-895 are both gold false. So bf16 had them right and Q4_K_M is confidently wrong on both.

The abstain gate does what you'd expect. With it on, bf16 answers 224 and gets 71.4% of those right, Q8_0 answers 223 at 71.3%, and Q4_K_M answers 228 at 68.9%. So Q4_K_M answers more and is wrong more often.

The card now has a section on it, "Quantization changes are not all tie-breaks", with that table and the per-question results. Short version for anyone reading: if you have the memory, use Q8_0.
dipankarsarkar

Your new table moves the recommendation, and the "Which one" line has not caught up.

Q5_K_M: 10 changes, 0 confident, 3 toward gold, 3 away. 174 right on the 260, the same as bf16. With the gate on it answers 225 at 71.6%, the best row in the table.

The card still says Q8_0 if it fits, Q4_K_M when memory is short, and Q5_K_M "only when a larger build does not fit".

The break is between Q5_K_M and Q4_K_M, not between Q8_0 and the rest. Confident changes go 0, 0, 9.

Five questions out of 260 is too thin to rank Q5_K_M above Q8_0. But 0 confident changes is the number your gate section is built on, and Q5_K_M is 6.6 GB against 9.8.

By your own sizing rule, Q5_K_M plus about 1 GB of context fits an 8 GB card and Q8_0 does not. Would you put Q5_K_M on the "Which one" line for that tier?

·
jbrashear

Yes, you're right again. I checked every number against the same gold scoring and they all hold: Q5_K_M changes 10 answers, none of them confident, 3 toward gold and 3 away, 174 of 260 right like bf16, and 225 answered at 71.6% with the gate on. Confident changes go 0, 0, 9 across Q8_0, Q5_K_M and Q4_K_M, so the break really is between Q5 and Q4.

The Which one line now says Q8_0 if it fits, Q5_K_M on an 8 GB card (6.6 GB plus about 1 GB of context fits there and Q8_0 does not), and Q4_K_M only when Q5_K_M doesn't fit. I kept a line that one right answer in 260 is too few to rank Q5_K_M above Q8_0.

The change: https://huggingface.co/frontier-infra/jebadiah-9b-v2-GGUF/commit/85a0b95da7969e4040fbf7e149169e76eeb674a9

Thanks for reading it this closely.
In this post