Hugging Face's logo
Activity Feed

AI & ML interests

AI Machine Learning and Nvidia
jbrashear 
posted an update about 11 hours ago
view post
Post
20
I need more than the model name to make a decision bug reproducible.

For Jeb, the useful starting point is jeb --version, the backend and its version, the exact model and quantization, and the output of jeb doctor with the same backend setting you use for the server.

Then keep one small request that shows the problem. Include the probabilities you received and the answer you expected. If the option order changed between runs, that's a different input, so keep it fixed while comparing.

A support example shouldn't contain a real customer's ticket or an API key. Replace private material with a made-up case that still fails. If it no longer fails, say that too. It's a useful limit on what the example demonstrates.

For a model-file question, use that model repository's Community tab. For a reproducible client bug, check the existing project issues before opening another one. Keep new evidence in the same thread.

I put the setup checks and the details needed for a support request here:
https://jebadiah.ai/support/llms.txt
jbrashear 
posted an update 2 days ago
view post
Post
2,749
I publish Jeb as a decision model. It returns numbers your application can read. Your application still needs a rule for what happens next.

Take a support route. A choice question can return a probability for billing, technical support and account access. Those labels are an example, not a fixed set you have to use.

A top answer tells you which label won. It doesn't tell you whether the result clears your own bar for an automatic route. Keep that second decision in the application, where you can inspect and change it.

I'd start by saving the question, the option names, the probabilities and the action rule together. Review the cases near the boundary. A correct top answer can still have a probability that falls on the wrong side of the line your application uses.

The same distinction matters when comparing runtimes or quantizations. Answer agreement measures whether the winner stayed the same on the questions tested. It doesn't establish that the probabilities stayed identical. If those probabilities drive an action, compare them too.

The standalone server exposes POST /v1/systemone. The decide scripts are another way to get decision outputs. A chat window loading the weights doesn't expose the answer-label probabilities needed for this workflow.

Which part of your first local decision would you keep under human review?

https://jebadiah.ai/

PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt and https://jebadiah.ai/support/agent.md.
  • 9 replies
·
jbrashear 
posted an update 6 days ago
view post
Post
84
I publish three decision shapes in Jeb: choice, noul and score. The distinction matters before you wire an answer into an action.

Choice gives a probability for each named option. Noul gives the probability that a statement is true. Score gives a probability for each level of an ordered scale, plus the expected level.

The support-ticket example on jebadiah.ai shows a frustration score of 2.29 on a 0 to 3 scale. That's an expected level between labels, not a fourth label or a claim that the answer is 229 percent certain. That example was recorded on September 26, 2026.

The standalone server exposes POST /v1/systemone. The decision scripts read answer-label probabilities; a chat reply doesn't expose those. Your application still has to decide what to do with the result.

Which shape fits the first decision you want to make locally?

https://jebadiah.ai/

PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt and https://jebadiah.ai/support/agent.md.
jbrashear 
posted an update 16 days ago
view post
Post
44
I publish a "same answer" count for each official Jeb GGUF build. It means the top option matched the full bf16 weights. It doesn't mean every probability was identical, or that the answer was correct.

On the published 260-question sample, Q8_0 agreement was 260/260 for 27B, 257/260 for 9B v2 and 256/260 for 4B v2.

A made-up example: Billing at 0.81 and Billing at 0.78 agree on the pick. A routing rule that acts above 0.80 would behave differently. That's why the cards report probability differences too.

The records are under eval/agreement-*.json in the official repos. These numbers apply to those files, not independent quantizers' builds. If you're choosing a build for a threshold-based decision, I'd compare probabilities near the threshold on your own examples too.

frontier-infra/jebadiah-27b-GGUF
frontier-infra/jebadiah-9b-v2-GGUF
frontier-infra/jebadiah-4b-v2-GGUF

PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt and https://jebadiah.ai/support/agent.md.
  • 1 reply
·
jbrashear 
posted an update 22 days ago
view post
Post
1,165
Jebadiah 27B v2.1 with the T1e inference recipe scores Full 64.4 on Jev Decision Index 0.4.1: rank 25 overall and tie rank 6, above the Jev 1.14 reference at 60.24. Board data: October 22, 2026.

The smaller 9B v2.1 with T1e scores Full 50.9, rank 80 overall and tie rank 18, with a board median latency of 125.1 ms versus 303.2 ms for the 27B. These are board reproduction timings on one RTX PRO 6000, not a promise for another setup.

The 27B is a LoRA fine-tune of Qwen3.8-27B. These are benchmark scores on the board's scale, not accuracy percentages or a guarantee for every runtime or quantized build.

The public index is 60.46. Private new-domain tasks map to 72.52 on the public scale. The full result combines private same-skill tests and private new-domain tasks at 60/40. The 0.9-point tie band defines groups; the overall position is 25.

Announcement and recipe background (earlier October 20 snapshot): https://titaniumcomputing.com/insights/jebadiah-decision-index-0-4/
Board: multimodalart/jev-decision-index

The board describes these as the same published v2.1 weights with a new inference recipe. Its lab check reproduced the submitted decisions with 100.00% agreement for the 9B and 99.90% for the 27B, each on 831 rows. That is reproduction agreement, not accuracy.

PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt and https://jebadiah.ai/support/agent.md.
jbrashear 
posted an update 23 days ago
view post
Post
104
I keep both temperature fits in Jeb's temperatures.json because a useful default still needs checking on the traffic it's going to see.

The 4B v2 score temperature was refit from 1.20 to 0.83 on September 29. No top pick changed. ECE on typed-decisions improved from 0.171 to 0.135, but Jevals HelpSteer2 got worse, from 0.048 to 0.074.

On a separate 418-row HelpSteer2 validation check, the 4B wanted a score temperature of 1.26. Those rows weren't in training or any reported evaluation set.

So if your traffic looks like HelpSteer2, the card says to keep the older 1.20 score temperature or refit on your own labels. The scripts take --temperatures with your own file.

Changing the probability can change whether your code clears a threshold, even when the chosen answer stays the same. I'd check that line on labeled examples from the actual workload.

The fits, comparison and warning are here:
frontier-infra/jebadiah-4b-v2-GGUF

PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt and https://jebadiah.ai/support/agent.md.
jbrashear 
posted an update 28 days ago
view post
Post
121
I wanted to share where the Jeb community downloads stand this Friday. These are the recorded Hugging Face counts from October 16 at 08:30 Central, so they may have moved by the time you read this.

bartowski's 9B v2 GGUF repo is at 6,440 all-time downloads, up 124 from the previous morning. Across that repo and mradermacher's 4B v2, 9B v2 and 27B repos, the total is 11,133, up 157 over the same interval.

Those are downloads, not a count of people or evidence that the model is right on a particular task. The counts also belong to those specific repos. They don't measure adoption of the newer v2.1 weights.

If you're using one of these quants for a routing or classification step, I'd like to hear what the question looks like and where the decision fails. A small example with the options and the expected answer is more useful than a general score.

The community repos:
bartowski/frontier-infra_jebadiah-9b-v2-GGUF
mradermacher/jebadiah-4b-v2-GGUF
mradermacher/jebadiah-9b-v2-GGUF
mradermacher/jebadiah-27b-GGUF

PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt and https://jebadiah.ai/support/agent.md.
jbrashear 
posted an update about 1 month ago
view post
Post
69
I publish Jeb's weights, but loading them in a chat window isn't enough to get a decision. The scripts read probabilities over the answer labels. A chat reply doesn't expose those.

For the official 9B Q8_0 build with Ollama, the local path is:

pip install jebadiah-decide
jeb doctor
jeb serve

Doctor pulls and checks the model. Serve gives your code POST http://localhost:8100/v1/systemone. It renders the prompt, reads the label probabilities and applies temperatures.json.

The published Ollama 0.34.4 check matched the full bf16 weights on 257 of 260 top answers. That's agreement on that sample, not an accuracy score. There are at most 20 options per question on this path.

If you're already running the weights locally, which decision would you want to put behind that endpoint?

The commands, records and caveats are here:
frontier-infra/jebadiah-9b-v2-GGUF

PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt and https://jebadiah.ai/support/agent.md.
  • 4 replies
·
jbrashear 
posted an update about 1 month ago
view post
Post
165
Jebadiah v2.1 is out, 27B and 9B.

Decision Index 0.3 public suite, my own runs, submitted to the board:
- 27B: 57.03, up from 55.11
- 9B: 47.20, up from 44.09

Knowledge, Language and Retrieval went up on both. Tools went down: 2.10 on the 27B (When2Call -6.38, home appliance -3.41) and 0.84 on the 9B. If you route tool calls with it, read the card before you switch.

GGUF (Q8_0, Q5_K_M, Q4_K_M) and MLX (8-bit, 4-bit) are up for both, with agreement counts against the full weights on each card.

27B: frontier-infra/jebadiah-27b-v2-1
9B: frontier-infra/jebadiah-9b-v2-1
Results: frontier-infra/jebadiah-v2-1-decision-index-results

Thanks to bartowski and mradermacher for the community quants of v2.

PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt and https://jebadiah.ai/support/agent.md
jbrashear 
posted an update about 1 month ago
view post
Post
58
Quick Jeb update. This week was mostly about making the models easier to find and run, and the community builds kept moving faster than ours.

On the October 5 snapshot, the four community GGUF repos had 7,211 downloads between them. They were at 5,781 on September 30. bartowski's 9B alone went from 3,542 to 4,431. mradermacher's 4B, 9B and 27B builds were at 751, 1,089 and 940.

Our 13 Hugging Face repos went from 1,892 downloads on September 30 to 3,697 on October 5. jebadiah-decide 0.2.0 had 192 PyPI downloads in its first reporting week.

Thank you again to bartowski and mradermacher. The GGUF builds are where most people are meeting Jeb.

The project also landed in a few useful directories this week: awesome-mlx, awesome-jev, the LLM Engineer Toolkit, and Awesome-LLMOps. I appreciate the maintainers reviewing and merging those additions.

One practical reminder for anyone trying a quant: Jeb is not a chat model. It makes a typed choice, yes-or-no call, or score from label-token probabilities at one position, with thinking off. The easiest path is still:

pip install jebadiah-decide
jeb serve


That applies the prompt format and per-question-type temperatures for you, whether the model is behind Ollama, LM Studio, llama-server, vLLM or MLX.

Run-locally guide: https://github.com/getainode/jebadiah/blob/main/docs/run-locally.md

If you are running Jeb outside a demo, what did you put it behind: routing, triage, scoring, or something else? I would especially like to hear where the smaller 4B model is enough.

PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt and https://jebadiah.ai/support/agent.md.
jbrashear 
posted an update about 1 month ago
view post
Post
112
Jeb update for the week. Most of what changed came out of a review.

Dipankar went back to the per-question eval records we publish, re-tempered them himself, and found the 4B and 9B were still shipping the score temperature fit to the training target. The 27B had already moved off it.

So on 2026-09-29 we refit the 4B and 9B without retraining. Score questions now use the hard fit, T 0.83 on both. Choice and noul stay where they were, and accuracy doesn't move. Across the 20 evaluation sets, question-weighted ECE goes from 0.0863 to 0.0800 on the 4B and from 0.0796 to 0.0709 on the 9B.

It isn't free. Jevals HelpSteer2 gets worse, 0.048 to 0.074 on the 4B and 0.039 to 0.082 on the 9B.

Then we ran the check he pushed us toward. HelpSteer2 has 418 validation rows that none of our reported eval sets use and that never went near the training pool. Those rows want a score temperature of 1.26 on the 4B and 1.29 on the 9B, not 0.83.

So the problem is how our calibration split drew HelpSteer2. If your traffic looks like HelpSteer2, keep the older train fit (1.20 on the 4B, 1.22 on the 9B, both in temperatures.json) or refit on your own labels. Both model cards say that now. For v3, the HelpSteer2 part of calibration comes from held-out validation data.

The community quants have also gone further than I expected. bartowski's 9B GGUF is at 3,542 downloads, and mradermacher's 4B, 9B and 27B GGUFs are at 600, 860 and 779. That's more than our own repos put together. Thank you both. And thank you Dipankar, that was exactly the kind of review I hoped for when we put the records up.

A question for anyone running Jeb on real traffic: are you using the shipped temperatures as they are, or refitting on your own labels? If you refit, I'd like to know what you ended up with.

The review thread: https://huggingface.co/posts/jbrashear/856060098865438
jbrashear 
posted an update about 2 months ago
view post
Post
125
I've been training Jebadiah, small models that don't chat. You give one a typed question about some JSON state and it gives you back a probability for every option, from one forward pass.

There are three sizes, 4B, 9B and 27B, all Apache 2.0, with GGUF and MLX builds. Each repo ships a temperatures.json fitted on held-out data, so the probabilities are calibrated per question type.

New this week: pip install jebadiah-decide, then jeb serve. It runs them through Ollama, LM Studio, llama-server, vLLM or MLX and puts /v1/systemone and /v1/decide on localhost. The setup for each runtime is here: https://github.com/getainode/jebadiah/blob/main/docs/run-locally.md

If you try it on a routing or triage step of your own, I'd like to hear how it does.

https://huggingface.co/collections/frontier-infra/jebadiah-open-system-one-decision-models-6ab80765ddd3fa0b3eba5213
  • 5 replies
·