Post
103
I keep both temperature fits in Jeb's temperatures.json because a useful default still needs checking on the traffic it's going to see.
The 4B v2 score temperature was refit from 1.20 to 0.83 on September 29. No top pick changed. ECE on typed-decisions improved from 0.171 to 0.135, but Jevals HelpSteer2 got worse, from 0.048 to 0.074.
On a separate 418-row HelpSteer2 validation check, the 4B wanted a score temperature of 1.26. Those rows weren't in training or any reported evaluation set.
So if your traffic looks like HelpSteer2, the card says to keep the older 1.20 score temperature or refit on your own labels. The scripts take --temperatures with your own file.
Changing the probability can change whether your code clears a threshold, even when the chosen answer stays the same. I'd check that line on labeled examples from the actual workload.
The fits, comparison and warning are here:
frontier-infra/jebadiah-4b-v2-GGUF
PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt and https://jebadiah.ai/support/agent.md.
The 4B v2 score temperature was refit from 1.20 to 0.83 on September 29. No top pick changed. ECE on typed-decisions improved from 0.171 to 0.135, but Jevals HelpSteer2 got worse, from 0.048 to 0.074.
On a separate 418-row HelpSteer2 validation check, the 4B wanted a score temperature of 1.26. Those rows weren't in training or any reported evaluation set.
So if your traffic looks like HelpSteer2, the card says to keep the older 1.20 score temperature or refit on your own labels. The scripts take --temperatures with your own file.
Changing the probability can change whether your code clears a threshold, even when the chosen answer stays the same. I'd check that line on labeled examples from the actual workload.
The fits, comparison and warning are here:
frontier-infra/jebadiah-4b-v2-GGUF
PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt and https://jebadiah.ai/support/agent.md.