Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
jbrashear 
posted an update 2 days ago
Post
2,749
I publish Jeb as a decision model. It returns numbers your application can read. Your application still needs a rule for what happens next.

Take a support route. A choice question can return a probability for billing, technical support and account access. Those labels are an example, not a fixed set you have to use.

A top answer tells you which label won. It doesn't tell you whether the result clears your own bar for an automatic route. Keep that second decision in the application, where you can inspect and change it.

I'd start by saving the question, the option names, the probabilities and the action rule together. Review the cases near the boundary. A correct top answer can still have a probability that falls on the wrong side of the line your application uses.

The same distinction matters when comparing runtimes or quantizations. Answer agreement measures whether the winner stayed the same on the questions tested. It doesn't establish that the probabilities stayed identical. If those probabilities drive an action, compare them too.

The standalone server exposes POST /v1/systemone. The decide scripts are another way to get decision outputs. A chat window loading the weights doesn't expose the answer-label probabilities needed for this workflow.

Which part of your first local decision would you keep under human review?

https://jebadiah.ai/

PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt and https://jebadiah.ai/support/agent.md.

The boundary review you describe has a size, and your repos already let anyone count it.
The 9B and 27B v2-1 GGUF repos ship eval/agreement-full.json beside each quant, so the bf16 probabilities sit next to the quantized ones.

Gate on top probability >= 0.80 and count questions that land on the other side of the line from bf16:
9B: Q8_0 0, Q5_K_M 7, Q4_K_M 16 of 260.
27B: Q8_0 0, Q5_K_M 6, Q4_K_M 3 of 260.
Every one keeps the same top answer, so answer agreement reports none of them.

On the v2 files last month I could only give upper bounds. With the bf16 file these are exact counts.

The order is not monotone either.
On 27B, Q4_K_M moves fewer 0.80 decisions than Q5_K_M (3 vs 6), but at 0.70 it moves more (9 vs 7).
So the review queue is a property of build plus threshold, not of the model.

Would you print a gate-flip count at the thresholds you recommend, next to the agreement number?

·

I'd keep a gate-flip count separate from answer agreement, and label it with the exact threshold, comparison builds and sample. Neither count tells us whether the resulting action was correct.

I wouldn't recommend 0.80 for every workload. The cutoff needs labeled examples from the application, including the cost of routing something automatically when it should go to review.

Could you share the script you used for those counts? That would make the comparison easier to reproduce.

Here it is. Run it from either v2-1 GGUF repo, against the eval/ folder as shipped:

import json, sys

def rows(path):
    # some (id, qid) pairs repeat, so key on occurrence order too
    out, seen = {}, {}
    for r in json.load(open(path))['rows']:
        k = (r['id'], r['qid'])
        seen[k] = seen.get(k, 0) + 1
        out[k + (seen[k],)] = r
    return out

d = sys.argv[1] if len(sys.argv) > 1 else 'eval'
full = rows(f'{d}/agreement-full.json')
for q in ['Q8_0', 'Q5_K_M', 'Q4_K_M']:
    Q = rows(f'{d}/agreement-{q}.json')
    keys = full.keys() & Q.keys()
    for t in (0.70, 0.80):
        flips = [k for k in keys if (max(full[k]['probs']) >= t) != (max(Q[k]['probs']) >= t)]
        same = sum(full[k]['pick_index'] == Q[k]['pick_index'] for k in flips)
        print(q, t, f'{len(flips)} of {len(keys)} cross the gate, {same} keep the same pick')

9B at 0.80: 0, 7, 16. At 0.70: 3, 4, 20.
27B at 0.80: 0, 6, 3. At 0.70: 1, 7, 9.

Agreed on both points. 0.80 was only an example, and a flip count says nothing about whether the routed action was right.

That second half can't be counted from these files yet.
Each row carries id, qid, type, probs and pick_index. No gold label. And all 260 ids are from typed-decisions-train.

With a label, the flips sort themselves.
A quant that lifts a wrong answer over the gate sends it out unreviewed. One that drops a right answer under it only costs a review.

Could the agreement files carry the gold index, on held-out rows?

·

Thanks for sharing the script. The current GGUF card identifies this as a calibration sample. I'd keep a labeled held-out report separate, since adding gold labels to the existing rows wouldn't make them held out.

For choice and yes/no rows, I'd include the gold option index, the exact option order and the dataset revision. I'd count wrong answers above the gate separately from correct answers sent to review, for both full weights and each quant. That would also catch errors shared by both builds, which a flip count misses.

I'd handle score rows separately. Their readout uses the rubric distribution rather than just the top label.

Your score-row point moves my numbers more than anything else in this thread.
Most of the flips I posted are score rows, where top probability isn't the readout.

Dropping them leaves the 156 choice and yes/no rows:
9B at 0.80: 0, 4, 1. At 0.70: 0, 1, 9.
27B at 0.80: 0, 0, 0. At 0.70: 1, 6, 4.

So 15 of the 16 Q4_K_M flips on 9B at 0.80 were score rows, and on 27B every 0.80 flip was.
The order still isn't monotone, though. On choice and yes/no, Q5_K_M moves more than Q4_K_M at 0.80 on 9B (4 vs 1) and at 0.70 on 27B (6 vs 4).

Agreed on keeping held-out separate. Gold labels on calibration rows would only measure fit.

For score rows, what should a gate read: expected rubric value, or mass on the top two cells?

·

I'd start with what the gate is supposed to decide. The documented score readout gives the probability per rubric level and the expected level.

If the action depends on the average rubric value, I'd compare the expected level. If it depends on reaching a particular part of the scale, I'd sum the probabilities of those fixed levels. For an ordered 0 to 3 rubric, that could be P(level >= 2).

If you mean the two highest-probability cells, their combined mass doesn't tell you which end of the scale you're confident about. If you mean the two highest rubric levels, that's a fixed tail event.

An expected level also isn't a confidence measure. I'd check the gate against labeled held-out examples from the application before using it to automate anything. The release's fitted temperatures don't establish calibration for a new domain.

The fixed tail is also the steadier readout across builds, at least on this eval.

On the 90 four-level score rows (urgency, risk, churn_risk, discrepancy_severity), a P(level >= 2) >= 0.80 gate flips:
9B: 0, 1, 2 (Q8_0, Q5_K_M, Q4_K_M).
27B: 0, 0, 1.

Top probability at 0.80 on the same rows flips 0, 2, 8 and 0, 6, 2.

The reason looks boring. In bf16, 8 of 90 rows have a tail within 0.1 of 0.80, against 42 for top probability.
Mass sliding between adjacent levels moves the top cell a lot and the tail barely. 27B Q5_K_M flips 6 on top probability and 0 on the tail.

Expected level at 1.5 or 2.0 sits in between: 0 to 3 flips per build.

None of this says the tail is calibrated, your point stands. It only says which readout a quant swap disturbs least.

Severity has five levels, so I left it out. Which levels would count as the tail there?

·

I'd choose the tail from what the severity levels mean for the action. Five levels by itself doesn't tell us where review should start.

For example, on a 1 to 5 rubric where larger means more severe, an event defined as levels 4 or 5 is P(4) + P(5). If review starts at 3, it's P(3) + P(4) + P(5). Those are different events. I'd record the included levels and their definitions alongside the probability cutoff.

I'd keep that event fixed across builds. Picking whichever tail produces the fewest flips would change the question being compared. Stability on this sample doesn't establish which severity boundary is right for the application.

What are the exact level definitions on those severity rows? I'd want those before naming a tail for that question.

Here they are, from the questions JSON of LocalLLaMA/typed-decisions security_incidents, where those 14 rows come from:

0 Negligible: no access to anything sensitive.
1 Low: limited access, easily reversed.
2 Moderate: access to internal systems or non-public data.
3 High: access to production, secrets or customer data.
4 Critical: active compromise of crown-jewel systems.

Read that way, I'd start review at High, so P(3) + P(4). Agreed on fixing the event by meaning, not by flip count.

That event at 0.80 flips 0 or 1 of 14 across builds. The bigger gap is somewhere else.
The same rows carry the dataset's own soft labels. Those gate 4 of 14. bf16 gates 7 (9B) and 8 (27B).

It holds on all 104 score rows, with each question's top two levels as the event: bf16 gates 48 and 49 at 0.80, the soft labels 38.

temperatures.json may explain it. The served score T is the hard-label fit, 0.6805 (9B) and 0.6524 (27B). The same file records the soft fit, 1.0982 and 1.0506.
Re-tempered to the soft fit, mean top probability goes from 0.750 to 0.633 (labels: 0.643), and the 0.80 gate from 48 to 42.

Caveat: these are train-split rows, and typed-decisions is in the card's source pool.

So the fixed tail is stable across builds but reads hot against the labels. Was the hard fit chosen for top-1 accuracy, or is there a reason a gate shouldn't use the soft one?