README.md

metadata

base_model: Qwen/Qwen2.5-0.5B-Instruct
license: apache-2.0
language:
  - en
library_name: transformers
pipeline_tag: text-generation
tags:
  - verifier
  - answer-verification
  - llm-cascade
  - model-cascade
  - escalation
  - llm-router
  - orchestration
  - multi-agent
model-index:
  - name: Jedi v1.0
    results:
      - task:
          type: text-classification
          name: answer verification (accept / escalate)
        dataset:
          type: custom
          name: held-out test answers (2
          532 answers to 633 unseen tasks): null
        metrics:
          - type: roc_auc
            value: 0.9195
            name: AUROC (correct vs wrong answers)
          - type: ece
            value: 0.017
            name: expected calibration error
      - task:
          type: text-generation
          name: LLM escalation ladder
        dataset:
          type: custom
          name: Shaktimaan held-out text and code tasks (491 tasks)
        metrics:
          - type: accuracy
            value: 0.9185
            name: end-to-end task accuracy (threshold 0.7)
          - type: relative_cost
            value: 0.4538
            name: cost vs always-strongest agent

Jedi v1.0 (0.5B)

Jedi is a 0.5B-parameter answer verifier that runs an LLM escalation ladder. A cheap model answers first. Jedi reads the task and that answer and returns a calibrated probability that the answer is correct. If the probability is high enough, the answer is accepted. Otherwise the task goes to the next, more expensive model, whose answer Jedi checks in turn. Jedi never answers tasks itself; it emits a single token.

Fine-tuned from Qwen2.5-0.5B-Instruct on 38,558 graded answers from three LLMs. It is the successor to Shaktimaan, which picks a model before any answer exists (see How Jedi differs from Shaktimaan).

At a glance

• 91.9% end-to-end accuracy on 491 held-out text and code tasks at 0.45x the cost of always using the strongest model. Always using the strongest model gets 84.7%; Shaktimaan v1.0 gets 84.9% at 0.32x on the same tasks.
• Beats every single model in the pool. At threshold 0.3 it also beats both standard cascade checks (a format check and self-consistency) on accuracy, cost and silent failures at once.
• Within 1.4 points of a perfect verifier (93.3%) on the same ladder.
• AUROC 0.92 at telling correct answers from wrong ones on unseen tasks; calibrated confidence (ECE 0.017).
• One forward pass and one output token per check: ~56 ms at batch size 1, ~450 input tokens.
• It judges how an answer looks, not whether its final value is right. Swapping the final answer of a real response barely moves its score. Much of its signal is surface cues (length, truncation, which model answered), though it still beats a gate built only on those cues by 1-2 points on the ladder. See What Jedi actually picks up on.
• Weak outside its training domains: AUROC 0.72 on BIG-Bench Hard (where answer length alone scores 0.80) and 0.59 on CommonsenseQA. See Limitations.

How it works

             task
               |
               v
   +-----------------------+      answer      +------------------+
   | rung 1: glm-5.3-flash | ---------------> |       Jedi       |  P(correct) >= threshold ?
   |      (cost 0.2)       |                  |  0.5B, 1 token   | --- yes --> return answer
   +-----------------------+                  +------------------+
               | no (escalate)
               v
   +-----------------------+      answer      +------------------+
   | rung 2: qwen3.8-27b   | ---------------> |       Jedi       | --- yes --> return answer
   |      (cost 0.5)       |                  +------------------+
   +-----------------------+
               | no (escalate)
               v
   +-----------------------+
   | rung 3: nemotron-ultra| -----------------------------------------------> return answer (always accepted)
   |      (cost 1.0)       |
   +-----------------------+

1. What Jedi reads. common.judge_view() builds a short header naming the agent, the task type, the answer length and whether the answer was cut off at the token limit. It then appends the task (first 2,500 + last 1,000 characters) and the candidate's visible answer (first 1,500 + last 2,500 characters, with <think> blocks removed). The system prompt is common.JEDI_SYSTEM.
2. What Jedi outputs. One word, accept or escalate. Jedi is never sampled. The runtime reads the logits of those two tokens at the first output position and takes the gap z = logit(accept) - logit(escalate).
3. Calibration. P(correct) = sigmoid(a·z + b), with a, b fitted by Platt scaling on a held-out validation split (calibration.json). This drops calibration error from 0.061 to 0.017 on the test split, so a threshold of 0.7 means roughly "accept when at least 70% likely to be right".
4. The single dial. The threshold trades cost for accuracy. Raise it to escalate more and catch more wrong answers; lower it to save money. The top rung is always accepted, so every task gets an answer.

What Jedi actually picks up on

The system prompt asks Jedi to look for broken reasoning, bad arithmetic, failing code, hedging and truncation. Training only rewards predicting the grade, though, so we tested what it learned (diagnostics.py, diagnostics_ladder.py).

check	in-domain test	unseen (BIG-Bench Hard)
Jedi AUROC	0.919	0.721
surface-feature baseline (logistic regression on answer length, truncated, empty final answer, agent, source)	0.871	0.826
answer length alone (shorter = more likely right)	0.810	0.801
Jedi, truncated answers excluded	0.879	0.682
Jedi ranks the right answer above a wrong one to the same task	93.1% of 690 pairs	84.6% of 1,168 pairs

Final-answer swap test. We took 1,583 real test answers (multiple choice and numeric, not truncated) and changed only the value after Final answer:. Right answers got a wrong value; wrong answers got the gold value.

	mean P(correct) before	after swap
right answers -> wrong final value (1,443)	0.912	0.902
wrong answers -> gold final value (140)	0.717	0.806

Jedi barely notices when the final answer contradicts the reasoning above it, or when a wrong answer is given the correct final value. It scores the body of the response: its length, its shape, whether it ends cleanly, and the style of reasoning that tends to accompany wrong answers from each model. Hand-written short answers with a planted arithmetic slip are usually accepted.

On the ladder, Jedi still beats the surface-feature gate (same 491 tasks, best settings):

gate	accuracy	relative cost
surface-feature baseline, threshold 0.05	89.8%	0.224
surface-feature baseline, threshold 0.1	89.8%	0.395
Jedi, threshold 0.05	91.0%	0.244
Jedi, threshold 0.7	91.9%	0.454

So in its training domains, Jedi learned something real beyond length and truncation. That gain is 1-2 points of end-to-end accuracy. On unseen task families, the surface features transfer better than Jedi does.

Quick start

Score one answer

import importlib.util, json, math, torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "podstack/Jedi"
spec = importlib.util.spec_from_file_location("common", hf_hub_download(repo, "common.py"))
common = importlib.util.module_from_spec(spec); spec.loader.exec_module(common)
cal = json.load(open(hf_hub_download(repo, "calibration.json")))
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16).eval()

def p_correct(task, answer, agent="glm-5.3-flash", kind="num", finish="stop"):
    msgs = [{"role": "system", "content": common.JEDI_SYSTEM},
            {"role": "user", "content": common.judge_view(task, answer, agent, kind, finish)}]
    enc = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True)
    with torch.no_grad():
        logits = model(**enc).logits[0, -1].float()
    gap = (logits[cal["accept_token"]] - logits[cal["escalate_token"]]).item()
    return 1 / (1 + math.exp(-(cal["a"] * gap + cal["b"])))

task = "A shop sells pencils at 3 for $1.20. How much do 15 pencils cost, in dollars?"
print(p_correct(task, "Each pencil costs 1.20 / 3 = 0.40. 15 x 0.40 = 6.00.\nFinal answer: 6"))     # 0.99 accept
print(p_correct(task, "Let me think step by step. First, 3 pencils cost 1.20, so", finish="length"))  # 0.07 escalate
print(p_correct(task, "15 / 3 = 5 groups. 5 x 1.20 = 5.00.\nFinal answer: 5"))                     # 0.98 - missed

The third call shows the main limitation: a fluent answer with an arithmetic slip is accepted. Jedi is a learned filter over real model outputs, not a checker of the math.

Always build the input with common.judge_view(...) and use common.JEDI_SYSTEM as the system prompt; Jedi was trained on exactly that format. kind is the task type it was trained with: num, math, mc (multiple choice), code, exact. finish is the agent's finish_reason ("length" means the answer was truncated).

Run the full ladder

jedi.py runs the ladder end to end against OpenAI-compatible endpoints:

export JEDI_ENDPOINTS='{"glm-5.3-flash": "http://host-a:8000/v1", "qwen3.8-27b": "http://host-b:8000/v1",
                       "nemotron-ultra": "http://host-c:8000/v1"}'
python jedi.py "If a train travels 120 km in 1.5 hours, what is its average speed in km/h?"
python jedi.py --threshold 0.9 --code "Write a function that checks whether a string is a palindrome."

It prints the final answer to stdout and a per-rung trace to stderr, e.g. {"agent": "glm-5.3-flash", "verdict": "accept", "confidence": 0.97, "escalate_to": null}. Rungs missing from JEDI_ENDPOINTS are skipped, so a two-rung ladder works too.

Agent pool (the ladder)

rung	agent	model	relative cost / token
1	glm-5.3-flash	zai-org/GLM-5.3-Flash	0.2
2	qwen3.8-27b	Qwen/Qwen3.8-27B	0.5
3	nemotron-ultra	nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4	1.0

kimi-vl (Kimi-VL-A3B-Thinking) is not on the ladder, but its answers are in the training data. It is the weakest of the three answering models, so it supplied most of the wrong answers Jedi learned from. "Relative cost" is a per-token weight for roughly relative serving cost, not a vendor price.

How Jedi differs from Shaktimaan

Both models are 0.5B fine-tunes of Qwen2.5-0.5B-Instruct built for the same agent pool. They solve the same problem, getting good answers without paying for the biggest model every time, at different moments.

	Shaktimaan v1.0 (router)	Jedi v1.0 (verifier)
When it decides	before any model answers	after each answer
What it reads	the task only	the task and a specific candidate answer
What it outputs	route JSON: domain, difficulty, strategy, agent list (~30 tokens)	one token -> calibrated P(correct)
Question it learns	"which agent is likely to solve this kind of task?"	"is this answer right?"
Model calls per task	exactly the routed agent(s), all up front	1 call, plus more only on escalation (1.15 on average at threshold 0.7)
Cost/quality control	fixed by training labels	one threshold, tunable at run time
Inputs it handles	text, code, images (routes to the vision agent), up to ~260k tokens (routes to long-context agents)	text and code; long inputs are truncated for judging
Extra latency	one router call before the agent	56 ms per check, plus the next rung's latency when it escalates (14 s on average at 0.7)

Why the switch. Shaktimaan's main finding was that, on ordinary text and code tasks, which mid-size model solves a given task is largely sampling noise that cannot be predicted from the prompt. A router has to guess blind. Jedi waits until a cheap answer exists and looks at that particular sample. Signs that a sample went wrong (rambling length, truncation, the reasoning style that comes with mistakes) only exist after it is written, so a router can never see them. Jedi does not re-derive the answer, so fluent answers that are wrong still get through (see What Jedi actually picks up on).

Head to head, on the same 491 held-out tasks with the same fresh agent samples:

policy	accuracy	relative cost (always-Ultra = 1.00)
Shaktimaan v1.0	84.9%	0.323
Jedi ladder, threshold 0.1	91.5%	0.263
Jedi ladder, threshold 0.7	91.9%	0.454

At threshold 0.1, Jedi is both more accurate (+6.5 points) and cheaper than Shaktimaan. (Shaktimaan's ensemble routes are scored as the aggregating agent's own answer here, which is slightly conservative for Shaktimaan.)

They are complementary. Jedi's ladder is text-only with a 32k-context first rung; it has no notion of images or 256k-token inputs. A sensible deployment routes by capability first (images -> the vision agent, very long inputs -> long-context agents, as Shaktimaan or simple rules do) and sends everything else through Jedi's ladder.

Training data

Every task was answered by glm-5.3-flash (2 samples), qwen3.8-27b (1) and kimi-vl (1) at temperature 0.6 with a "be concise, end with Final answer:" prompt. Each answer was graded automatically: exact, numeric or LaTeX-normalised matching, and execution of unit tests in a network-less Docker sandbox for code. That gave 28,296 graded answers on 7,074 tasks. A second borderline pass collected 8,580 more GLM and Qwen answers on the 2,145 tasks where the models disagreed. This is where the hard-to-judge answers are; those extra samples are used for training only.

source	category	tasks	grading	license
GSM8K	math word problems	1,000	numeric match	MIT
MATH	competition math	1,000	normalised LaTeX / numeric	MIT
MMLU-Pro	knowledge MCQ (10 options)	1,200	option letter	MIT
ARC-Challenge	science MCQ	800	option letter	CC-BY-SA-4.0
MBPP	Python coding	700	unit tests in sandbox	CC-BY-4.0
HumanEval	Python coding	164	unit tests in sandbox	MIT
AQuA-RAT	algebra word problems (MCQ)	800	option letter	Apache-2.0
CommonsenseQA	commonsense MCQ	600	option letter	MIT
BIG-Bench Hard	27 reasoning tasks	810	exact match	MIT

The first six sources use the same task ids as Shaktimaan-routing, so the two models can be compared on identical tasks. Charts and long documents were left out.

Solve rate per model (share of answers graded correct):

source	glm-5.3-flash	qwen3.8-27b	kimi-vl
GSM8K	0.96	0.95	0.81
MATH	0.75	0.75	0.42
MMLU-Pro	0.81	0.80	0.49
ARC-Challenge	0.97	0.97	0.92
MBPP	0.93	0.82	0.65
HumanEval	0.94	0.91	0.81
AQuA-RAT	0.89	0.92	0.64
CommonsenseQA	0.80	0.81	0.75
BIG-Bench Hard	0.90	0.91	0.54

Splits are by task, never by answer, so no task appears in two splits:

split	tasks	answers	wrong answers	used for
train	5,261	38,558	40.0% (oversampled from ~18%)	fine-tuning
val	370	1,480	17.6%	Platt calibration only
test	633	2,532	17.6%	Shaktimaan's held-out test tasks + 10% of AQuA / CommonsenseQA
ood	810	3,240	18.8%	all of BIG-Bench Hard, never trained on

Each example's label is accept if the answer was graded correct and escalate otherwise. Held-out splits use only the base samples, without the borderline extras, so every task is sampled the same way.

Training

setting	value
base model	Qwen2.5-0.5B-Instruct (494M parameters)
method	full fine-tune, supervised (TRL SFTTrainer); loss on the one-word verdict only
epochs	2 (2,380 steps)
learning rate	2e-5, cosine schedule, 30 warm-up steps
effective batch size	32
max sequence length	2,048 tokens
precision	bf16, gradient checkpointing
hardware / time	one MIG 1g.67gb slice of an NVIDIA B300, 1 h 54 min
eval loss (val)	0.095 after epoch 1, 0.142 after epoch 2 (the published weights are from epoch 2)

Results

As a verifier

Calibrated P(correct) on answers to tasks Jedi never saw. False accept = share of wrong answers scored >= 0.5; false escalate = share of correct answers scored < 0.5.

split	answers	wrong	AUROC	false accept	false escalate	ECE	Brier
test (in-domain)	2,532	17.6%	0.920	35.9%	1.9%	0.017	0.063
val	1,480	17.6%	0.884	42.3%	1.3%	0.017	0.072
ood (BIG-Bench Hard)	3,240	18.8%	0.720	70.2%	6.7%	0.060	0.140

Test split by answering model and by source:

slice	answers	wrong	AUROC	false accept
glm-5.3-flash	1,266	11.8%	0.910	41.6%
qwen3.8-27b	633	12.3%	0.915	50.0%
kimi-vl	633	34.6%	0.896	26.9%
HumanEval	72	9.7%	0.991	14.3%
MATH	424	31.8%	0.963	14.1%
MBPP	268	9.7%	0.955	38.5%
MMLU-Pro	480	29.4%	0.928	34.0%
AQuA-RAT	316	16.8%	0.896	30.2%
GSM8K	420	6.2%	0.854	53.8%
ARC-Challenge	300	4.0%	0.842	91.7%
CommonsenseQA	252	18.3%	0.593	89.1%

Jedi is strongest where wrong answers look different from right ones: multi-step math and code, where failures tend to be long, meandering or cut off. It is weakest where a wrong answer reads exactly like a right one, such as single-hop commonsense and easy science multiple choice.

As the gate of the ladder

491 held-out tasks: Shaktimaan's test tasks from GSM8K, MATH, MMLU-Pro, ARC, MBPP and HumanEval. GLM and Qwen answers are fresh samples; Ultra's outcome and token count come from its graded run in Shaktimaan-routing (same task ids). Cost = sum over called agents of (relative cost x output tokens), normalised so always-Ultra = 1.00. Silent failure = a wrong answer accepted by a lower rung, i.e. a mistake the ladder had a chance to escalate and didn't.

policy	accuracy	relative cost	silent failures	calls / task
Always glm-5.3-flash	88.8%	0.115	-	1.00
Always qwen3.8-27b	87.4%	0.421	-	1.00
Always nemotron-ultra	84.7%	1.000	-	1.00
Shaktimaan v1.0 routing	84.9%	0.323	-	-
Ladder + format check (escalate truncated / unparseable answers)	89.8%	0.406	7.3%	1.07
Ladder + self-consistency (escalate if two samples disagree)	90.4%	0.554	6.5%	1.14
Ladder + Jedi, threshold 0.7	91.9%	0.454	2.9%	1.15
Ladder + perfect verifier (oracle, upper bound)	93.3%	0.473	0.0%	1.19

Threshold sweep (same 491 tasks):

threshold	accuracy	relative cost	silent failures	calls / task	extra latency / task
0.05	91.0%	0.244	7.5%	1.09	8.9 s
0.1	91.5%	0.263	6.5%	1.10	9.5 s
0.3	91.5%	0.323	5.5%	1.11	10.7 s
0.5	91.7%	0.387	4.5%	1.12	12.1 s
0.7	91.9%	0.454	2.9%	1.15	14.3 s
0.8	91.0%	0.506	2.9%	1.20	16.4 s
0.9	90.2%	0.609	1.6%	1.32	20.8 s
0.95	87.0%	0.891	1.4%	1.65	34.5 s

Above 0.7, accuracy falls even as silent failures keep dropping. Escalating a correct answer is not free: the next rung can get it wrong, and in this pool Ultra's recorded solve rate (84.7%) is below GLM's. Use 0.7 for the best accuracy, or 0.1-0.3 for most of the gain at a quarter of always-Ultra's cost.

Outside the training domains (two-rung ladder: GLM -> Qwen)

Cost is relative to always using Qwen.

tasks	policy	accuracy	cost vs always-Qwen	silent failures
AQuA + CommonsenseQA test (142)	always GLM / always Qwen	85.2% / 88.7%	0.30 / 1.00	-
	self-consistency	88.7%	1.05	7.0%
	Jedi @ 0.5	88.7%	0.68	8.5%
	Jedi @ 0.7	87.3%	0.72	8.5%
	oracle	92.3%	0.71	0.0%
BIG-Bench Hard, unseen (810)	always GLM / always Qwen	90.1% / 90.6%	0.14 / 1.00	-
	self-consistency	91.4%	0.53	6.1%
	Jedi @ 0.7	91.1%	0.27	7.8%
	Jedi @ 0.9	92.0%	0.69	2.2%
	oracle	94.6%	0.37	0.0%

On unseen task families the ladder still improves on both single models, because the cheap model is usually right and Jedi rarely escalates correct answers. It catches far fewer wrong ones than it does in-domain, though. On the CommonsenseQA-heavy new-domain set, Jedi at 0.7 does not beat always using Qwen.

Overhead

~56 ms per check (batch size 1, one MIG 1g.67gb slice of a B300), ~451 input tokens, 1 output token. Checks are a single forward pass and batch well.

Limitations

• Not a checker. Jedi does not verify the final answer: changing it barely moves the score, and planted arithmetic slips in otherwise fluent answers usually pass. It relies heavily on surface cues, and on unseen task families a logistic regression on answer length, truncation and agent beats it (AUROC 0.83 vs 0.72).
• Domain-bound. Verification skill transfers poorly to unseen task families (BIG-Bench Hard AUROC 0.72) and is near chance on CommonsenseQA (0.59). Measure it on your own traffic before relying on a threshold.
• A third of wrong in-domain answers get through at 0.5. False-accept rates are highest for the strongest models, whose mistakes are rarer and more fluent. Jedi is a cost-saving filter, not a correctness guarantee.
• Pool-specific. Trained on answers from GLM-5.3-Flash, Qwen3.8-27B and Kimi-VL; the agent name is part of its input. Other models' answers are out of distribution, and a new pool needs new data and retraining.
• Text and code only, English only, single-turn tasks with a checkable final answer. Not trained on open-ended writing, agentic / tool use, multi-turn or image inputs. Long tasks and answers are truncated (head + tail) for judging.
• Labels inherit grading noise from automatic answer extraction and matching.
• Ladder numbers use recorded Ultra outcomes, a single sample per task from the Shaktimaan-routing runs, not fresh calls.
• Validation loss rose in epoch 2 (0.095 -> 0.142) while AUROC and calibration stayed good; a one-epoch model may be marginally better.

Files

file	purpose
model.safetensors, config.json, tokenizer files	the verifier
calibration.json	Platt scaling a, b and the accept / escalate token ids
common.py	agent pool, ladder, prompts, judge_view, answer extraction and graders
jedi.py	end-to-end runtime (answer -> verify -> escalate)
gen.py	collects and grades agent answers (stage 1)
build.py	builds the accept / escalate dataset and task-level splits (stage 2)
train.py	fine-tuning script (stage 3)
evaluate.py	verifier metrics, ladder, baselines and threshold sweep (stage 4)
eval_results.json	raw evaluation output behind every number on this card
diagnostics.py, diagnostics_ladder.py	surface-feature baselines, answer-swap test, ladder with a surface-feature gate

The graded answer data is not published; gen.py and build.py regenerate it given endpoints for the three models.

License

Model weights: Apache-2.0 (as the base model). Task sources keep their own licenses (table above).