AI & ML interests
None defined yet.
Recent Activity
View all activity
Articles
Post
37
I publish a "same answer" count for each official Jeb GGUF build. It means the top option matched the full bf16 weights. It doesn't mean every probability was identical, or that the answer was correct.
On the published 260-question sample, Q8_0 agreement was 260/260 for 27B, 257/260 for 9B v2 and 256/260 for 4B v2.
A made-up example: Billing at 0.81 and Billing at 0.78 agree on the pick. A routing rule that acts above 0.80 would behave differently. That's why the cards report probability differences too.
The records are under eval/agreement-*.json in the official repos. These numbers apply to those files, not independent quantizers' builds. If you're choosing a build for a threshold-based decision, I'd compare probabilities near the threshold on your own examples too.
frontier-infra/jebadiah-27b-GGUF
frontier-infra/jebadiah-9b-v2-GGUF
frontier-infra/jebadiah-4b-v2-GGUF
PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt and https://jebadiah.ai/support/agent.md.
On the published 260-question sample, Q8_0 agreement was 260/260 for 27B, 257/260 for 9B v2 and 256/260 for 4B v2.
A made-up example: Billing at 0.81 and Billing at 0.78 agree on the pick. A routing rule that acts above 0.80 would behave differently. That's why the cards report probability differences too.
The records are under eval/agreement-*.json in the official repos. These numbers apply to those files, not independent quantizers' builds. If you're choosing a build for a threshold-based decision, I'd compare probabilities near the threshold on your own examples too.
frontier-infra/jebadiah-27b-GGUF
frontier-infra/jebadiah-9b-v2-GGUF
frontier-infra/jebadiah-4b-v2-GGUF
PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt and https://jebadiah.ai/support/agent.md.
- 1 reply
Post
1,163
Jebadiah 27B v2.1 with the T1e inference recipe scores Full 64.4 on Jev Decision Index 0.4.1: rank 25 overall and tie rank 6, above the Jev 1.14 reference at 60.24. Board data: October 22, 2026.
The smaller 9B v2.1 with T1e scores Full 50.9, rank 80 overall and tie rank 18, with a board median latency of 125.1 ms versus 303.2 ms for the 27B. These are board reproduction timings on one RTX PRO 6000, not a promise for another setup.
The 27B is a LoRA fine-tune of Qwen3.8-27B. These are benchmark scores on the board's scale, not accuracy percentages or a guarantee for every runtime or quantized build.
The public index is 60.46. Private new-domain tasks map to 72.52 on the public scale. The full result combines private same-skill tests and private new-domain tasks at 60/40. The 0.9-point tie band defines groups; the overall position is 25.
Announcement and recipe background (earlier October 20 snapshot): https://titaniumcomputing.com/insights/jebadiah-decision-index-0-4/
Board: multimodalart/jev-decision-index
The board describes these as the same published v2.1 weights with a new inference recipe. Its lab check reproduced the submitted decisions with 100.00% agreement for the 9B and 99.90% for the 27B, each on 831 rows. That is reproduction agreement, not accuracy.
PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt and https://jebadiah.ai/support/agent.md.
The smaller 9B v2.1 with T1e scores Full 50.9, rank 80 overall and tie rank 18, with a board median latency of 125.1 ms versus 303.2 ms for the 27B. These are board reproduction timings on one RTX PRO 6000, not a promise for another setup.
The 27B is a LoRA fine-tune of Qwen3.8-27B. These are benchmark scores on the board's scale, not accuracy percentages or a guarantee for every runtime or quantized build.
The public index is 60.46. Private new-domain tasks map to 72.52 on the public scale. The full result combines private same-skill tests and private new-domain tasks at 60/40. The 0.9-point tie band defines groups; the overall position is 25.
Announcement and recipe background (earlier October 20 snapshot): https://titaniumcomputing.com/insights/jebadiah-decision-index-0-4/
Board: multimodalart/jev-decision-index
The board describes these as the same published v2.1 weights with a new inference recipe. Its lab check reproduced the submitted decisions with 100.00% agreement for the 9B and 99.90% for the 27B, each on 831 rows. That is reproduction agreement, not accuracy.
PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt and https://jebadiah.ai/support/agent.md.
jbrashear
updated 8
models 9 days ago
frontier-infra/jebadiah-9b-v2-1-MLX-8bit
Text Generation • 9B • Updated • 105
frontier-infra/jebadiah-9b-v2-1-MLX-4bit
Text Generation • 9B • Updated • 86
frontier-infra/jebadiah-9b-v2-1-GGUF
9B • Updated • 613
frontier-infra/jebadiah-9b-v2-1
Image-Text-to-Text • 10B • Updated • 270
frontier-infra/jebadiah-27b-v2-1-MLX-4bit
Text Generation • 27B • Updated • 78
frontier-infra/jebadiah-27b-v2-1-MLX-8bit
Text Generation • 27B • Updated • 87
frontier-infra/jebadiah-27b-v2-1-GGUF
27B • Updated • 459
frontier-infra/jebadiah-27b-v2-1
Image-Text-to-Text • 28B • Updated • 306 • 1
Post
103
I keep both temperature fits in Jeb's temperatures.json because a useful default still needs checking on the traffic it's going to see.
The 4B v2 score temperature was refit from 1.20 to 0.83 on September 29. No top pick changed. ECE on typed-decisions improved from 0.171 to 0.135, but Jevals HelpSteer2 got worse, from 0.048 to 0.074.
On a separate 418-row HelpSteer2 validation check, the 4B wanted a score temperature of 1.26. Those rows weren't in training or any reported evaluation set.
So if your traffic looks like HelpSteer2, the card says to keep the older 1.20 score temperature or refit on your own labels. The scripts take --temperatures with your own file.
Changing the probability can change whether your code clears a threshold, even when the chosen answer stays the same. I'd check that line on labeled examples from the actual workload.
The fits, comparison and warning are here:
frontier-infra/jebadiah-4b-v2-GGUF
PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt and https://jebadiah.ai/support/agent.md.
The 4B v2 score temperature was refit from 1.20 to 0.83 on September 29. No top pick changed. ECE on typed-decisions improved from 0.171 to 0.135, but Jevals HelpSteer2 got worse, from 0.048 to 0.074.
On a separate 418-row HelpSteer2 validation check, the 4B wanted a score temperature of 1.26. Those rows weren't in training or any reported evaluation set.
So if your traffic looks like HelpSteer2, the card says to keep the older 1.20 score temperature or refit on your own labels. The scripts take --temperatures with your own file.
Changing the probability can change whether your code clears a threshold, even when the chosen answer stays the same. I'd check that line on labeled examples from the actual workload.
The fits, comparison and warning are here:
frontier-infra/jebadiah-4b-v2-GGUF
PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt and https://jebadiah.ai/support/agent.md.
jbrashear
updated 8
models 10 days ago
frontier-infra/jebadiah-4b-v2-MLX-8bit
Text Generation • 4B • Updated • 91
frontier-infra/jebadiah-4b-v2-MLX-4bit
Text Generation • 4B • Updated • 81
frontier-infra/jebadiah-9b-v2-MLX-8bit
Text Generation • 9B • Updated • 95
frontier-infra/jebadiah-27b-MLX-8bit
Text Generation • 27B • Updated • 75
frontier-infra/jebadiah-27b-MLX-4bit
Text Generation • 27B • Updated • 181
frontier-infra/jebadiah-4b-v2-MLX
Text Generation • Updated
frontier-infra/jebadiah-9b-v2-MLX
Text Generation • Updated
frontier-infra/jebadiah-27b-MLX
Text Generation • Updated