README.md

---
base_model: Qwen/Qwen2.5-0.5B-Instruct
license: apache-2.0
language: [en]
library_name: transformers
pipeline_tag: text-generation
tags: [verifier, answer-verification, llm-cascade, model-cascade, escalation, llm-router, orchestration, multi-agent]
model-index:
- name: Jedi v1.0
  results:
  - task: {type: text-classification, name: answer verification (accept / escalate)}
    dataset: {type: custom, name: held-out test answers (2,532 answers to 633 unseen tasks)}
    metrics:
    - {type: roc_auc, value: 0.9195, name: AUROC (correct vs wrong answers)}
    - {type: ece, value: 0.0170, name: expected calibration error}
  - task: {type: text-generation, name: LLM escalation ladder}
    dataset: {type: custom, name: Shaktimaan held-out text and code tasks (491 tasks)}
    metrics:
    - {type: accuracy, value: 0.9185, name: end-to-end task accuracy (threshold 0.7)}
    - {type: relative_cost, value: 0.4538, name: cost vs always-strongest agent}
---
# Jedi v1.0 (0.5B)

**Jedi** is a 0.5B-parameter **answer verifier** that runs an LLM escalation ladder. A cheap model answers first.
Jedi reads the task *and that answer* and returns a calibrated probability that the answer is correct. If the
probability is high enough, the answer is accepted. Otherwise the task goes to the next, more expensive model,
whose answer Jedi checks in turn. Jedi never answers tasks itself; it emits a single token.

Fine-tuned from [Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) on 38,558 graded
answers from three LLMs. It is the successor to [Shaktimaan](https://huggingface.co/podstack/Shaktimaan), which
picks a model *before* any answer exists (see [How Jedi differs from Shaktimaan](#how-jedi-differs-from-shaktimaan)).

## At a glance
- **91.9%** end-to-end accuracy on 491 held-out text and code tasks at **0.45x** the cost of always using the
  strongest model. Always using the strongest model gets 84.7%; Shaktimaan v1.0 gets 84.9% at 0.32x on the same tasks.
- Beats every single model in the pool. At threshold 0.3 it also beats both standard cascade checks (a format
  check and self-consistency) on accuracy, cost and silent failures at once.
- Within 1.4 points of a *perfect* verifier (93.3%) on the same ladder.
- **AUROC 0.92** at telling correct answers from wrong ones on unseen tasks; calibrated confidence (ECE 0.017).
- One forward pass and **one output token** per check: ~56 ms at batch size 1, ~450 input tokens.
- **It judges how an answer looks, not whether its final value is right.** Swapping the final answer of a real
  response barely moves its score. Much of its signal is surface cues (length, truncation, which model answered),
  though it still beats a gate built only on those cues by 1-2 points on the ladder. See
  [What Jedi actually picks up on](#what-jedi-actually-picks-up-on).
- **Weak outside its training domains**: AUROC 0.72 on BIG-Bench Hard (where answer length alone scores 0.80) and
  0.59 on CommonsenseQA. See *Limitations*.

## How it works

```
             task
               |
               v
   +-----------------------+      answer      +------------------+
   | rung 1: glm-5.3-flash | ---------------> |       Jedi       |  P(correct) >= threshold ?
   |      (cost 0.2)       |                  |  0.5B, 1 token   | --- yes --> return answer
   +-----------------------+                  +------------------+
               | no (escalate)
               v
   +-----------------------+      answer      +------------------+
   | rung 2: qwen3.8-27b   | ---------------> |       Jedi       | --- yes --> return answer
   |      (cost 0.5)       |                  +------------------+
   +-----------------------+
               | no (escalate)
               v
   +-----------------------+
   | rung 3: nemotron-ultra| -----------------------------------------------> return answer (always accepted)
   |      (cost 1.0)       |
   +-----------------------+
```

1. **What Jedi reads.** `common.judge_view()` builds a short header naming the agent, the task type, the answer
   length and whether the answer was cut off at the token limit. It then appends the task (first 2,500 + last 1,000
   characters) and the candidate's visible answer (first 1,500 + last 2,500 characters, with `<think>` blocks
   removed). The system prompt is `common.JEDI_SYSTEM`.
2. **What Jedi outputs.** One word, `accept` or `escalate`. Jedi is never sampled. The runtime reads the logits of
   those two tokens at the first output position and takes the gap `z = logit(accept) - logit(escalate)`.
3. **Calibration.** `P(correct) = sigmoid(a·z + b)`, with `a`, `b` fitted by Platt scaling on a held-out validation
   split (`calibration.json`). This drops calibration error from 0.061 to 0.017 on the test split, so a threshold
   of 0.7 means roughly "accept when at least 70% likely to be right".
4. **The single dial.** The threshold trades cost for accuracy. Raise it to escalate more and catch more wrong
   answers; lower it to save money. The top rung is always accepted, so every task gets an answer.

### What Jedi actually picks up on
The system prompt asks Jedi to look for broken reasoning, bad arithmetic, failing code, hedging and truncation.
Training only rewards predicting the grade, though, so we tested what it learned (`diagnostics.py`,
`diagnostics_ladder.py`).

| check | in-domain test | unseen (BIG-Bench Hard) |
|---|---|---|
| Jedi AUROC | **0.919** | 0.721 |
| surface-feature baseline (logistic regression on answer length, truncated, empty final answer, agent, source) | 0.871 | **0.826** |
| answer length alone (shorter = more likely right) | 0.810 | 0.801 |
| Jedi, truncated answers excluded | 0.879 | 0.682 |
| Jedi ranks the right answer above a wrong one *to the same task* | 93.1% of 690 pairs | 84.6% of 1,168 pairs |

**Final-answer swap test.** We took 1,583 real test answers (multiple choice and numeric, not truncated) and changed
only the value after `Final answer:`. Right answers got a wrong value; wrong answers got the gold value.

| | mean P(correct) before | after swap |
|---|---|---|
| right answers -> wrong final value (1,443) | 0.912 | 0.902 |
| wrong answers -> gold final value (140) | 0.717 | 0.806 |

Jedi barely notices when the final answer contradicts the reasoning above it, or when a wrong answer is given the
correct final value. It scores the *body* of the response: its length, its shape, whether it ends cleanly, and the
style of reasoning that tends to accompany wrong answers from each model. Hand-written short answers with a planted
arithmetic slip are usually accepted.

**On the ladder, Jedi still beats the surface-feature gate** (same 491 tasks, best settings):

| gate | accuracy | relative cost |
|---|---|---|
| surface-feature baseline, threshold 0.05 | 89.8% | 0.224 |
| surface-feature baseline, threshold 0.1 | 89.8% | 0.395 |
| Jedi, threshold 0.05 | 91.0% | 0.244 |
| Jedi, threshold 0.7 | 91.9% | 0.454 |

So in its training domains, Jedi learned something real beyond length and truncation. That gain is 1-2 points of
end-to-end accuracy. On unseen task families, the surface features transfer better than Jedi does.

## Quick start

### Score one answer
```python
import importlib.util, json, math, torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "podstack/Jedi"
spec = importlib.util.spec_from_file_location("common", hf_hub_download(repo, "common.py"))
common = importlib.util.module_from_spec(spec); spec.loader.exec_module(common)
cal = json.load(open(hf_hub_download(repo, "calibration.json")))
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16).eval()

def p_correct(task, answer, agent="glm-5.3-flash", kind="num", finish="stop"):
    msgs = [{"role": "system", "content": common.JEDI_SYSTEM},
            {"role": "user", "content": common.judge_view(task, answer, agent, kind, finish)}]
    enc = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True)
    with torch.no_grad():
        logits = model(**enc).logits[0, -1].float()
    gap = (logits[cal["accept_token"]] - logits[cal["escalate_token"]]).item()
    return 1 / (1 + math.exp(-(cal["a"] * gap + cal["b"])))

task = "A shop sells pencils at 3 for $1.20. How much do 15 pencils cost, in dollars?"
print(p_correct(task, "Each pencil costs 1.20 / 3 = 0.40. 15 x 0.40 = 6.00.\nFinal answer: 6"))     # 0.99 accept
print(p_correct(task, "Let me think step by step. First, 3 pencils cost 1.20, so", finish="length"))  # 0.07 escalate
print(p_correct(task, "15 / 3 = 5 groups. 5 x 1.20 = 5.00.\nFinal answer: 5"))                     # 0.98 - missed
```
The third call shows the main limitation: a fluent answer with an arithmetic slip is accepted. Jedi is a learned
filter over real model outputs, not a checker of the math.

Always build the input with `common.judge_view(...)` and use `common.JEDI_SYSTEM` as the system prompt; Jedi was
trained on exactly that format. `kind` is the task type it was trained with: `num`, `math`, `mc`
(multiple choice), `code`, `exact`. `finish` is the agent's `finish_reason` (`"length"` means the answer was truncated).

### Run the full ladder
`jedi.py` runs the ladder end to end against OpenAI-compatible endpoints:

```bash
export JEDI_ENDPOINTS='{"glm-5.3-flash": "http://host-a:8000/v1", "qwen3.8-27b": "http://host-b:8000/v1",
                       "nemotron-ultra": "http://host-c:8000/v1"}'
python jedi.py "If a train travels 120 km in 1.5 hours, what is its average speed in km/h?"
python jedi.py --threshold 0.9 --code "Write a function that checks whether a string is a palindrome."
```

It prints the final answer to stdout and a per-rung trace to stderr, e.g.
`{"agent": "glm-5.3-flash", "verdict": "accept", "confidence": 0.97, "escalate_to": null}`. Rungs missing from
`JEDI_ENDPOINTS` are skipped, so a two-rung ladder works too.

## Agent pool (the ladder)
| rung | agent | model | relative cost / token |
|---|---|---|---|
| 1 | `glm-5.3-flash` | `zai-org/GLM-5.3-Flash` | 0.2 |
| 2 | `qwen3.8-27b` | `Qwen/Qwen3.8-27B` | 0.5 |
| 3 | `nemotron-ultra` | `nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4` | 1.0 |

`kimi-vl` (`Kimi-VL-A3B-Thinking`) is not on the ladder, but its answers are in the training data. It is the weakest
of the three answering models, so it supplied most of the wrong answers Jedi learned from. "Relative cost" is a
per-token weight for roughly relative serving cost, not a vendor price.

## How Jedi differs from Shaktimaan
Both models are 0.5B fine-tunes of Qwen2.5-0.5B-Instruct built for the same agent pool. They solve the same
problem, getting good answers without paying for the biggest model every time, at different moments.

| | **Shaktimaan v1.0** (router) | **Jedi v1.0** (verifier) |
|---|---|---|
| When it decides | **before** any model answers | **after** each answer |
| What it reads | the task only | the task **and** a specific candidate answer |
| What it outputs | route JSON: domain, difficulty, strategy, agent list (~30 tokens) | one token -> calibrated P(correct) |
| Question it learns | "which agent is likely to solve this kind of task?" | "is *this* answer right?" |
| Model calls per task | exactly the routed agent(s), all up front | 1 call, plus more only on escalation (1.15 on average at threshold 0.7) |
| Cost/quality control | fixed by training labels | one threshold, tunable at run time |
| Inputs it handles | text, code, images (routes to the vision agent), up to ~260k tokens (routes to long-context agents) | text and code; long inputs are truncated for judging |
| Extra latency | one router call before the agent | ~56 ms per check, plus the next rung's latency when it escalates (~14 s on average at 0.7) |

**Why the switch.** Shaktimaan's main finding was that, on ordinary text and code tasks, *which* mid-size model
solves a given task is largely sampling noise that cannot be predicted from the prompt. A router has to guess
blind. Jedi waits until a cheap answer exists and looks at that particular sample. Signs that a sample went wrong
(rambling length, truncation, the reasoning style that comes with mistakes) only exist after it is written, so a
router can never see them. Jedi does not re-derive the answer, so fluent answers that are wrong still get through
(see [What Jedi actually picks up on](#what-jedi-actually-picks-up-on)).

**Head to head**, on the same 491 held-out tasks with the same fresh agent samples:

| policy | accuracy | relative cost (always-Ultra = 1.00) |
|---|---|---|
| Shaktimaan v1.0 | 84.9% | 0.323 |
| Jedi ladder, threshold 0.1 | 91.5% | 0.263 |
| Jedi ladder, threshold 0.7 | **91.9%** | 0.454 |

At threshold 0.1, Jedi is both more accurate (+6.5 points) and cheaper than Shaktimaan. (Shaktimaan's ensemble
routes are scored as the aggregating agent's own answer here, which is slightly conservative for Shaktimaan.)

**They are complementary.** Jedi's ladder is text-only with a 32k-context first rung; it has no notion of images or
256k-token inputs. A sensible deployment routes by capability first (images -> the vision agent, very long inputs
-> long-context agents, as Shaktimaan or simple rules do) and sends everything else through Jedi's ladder.

## Training data
Every task was answered by `glm-5.3-flash` (2 samples), `qwen3.8-27b` (1) and `kimi-vl` (1) at temperature 0.6
with a "be concise, end with `Final answer:`" prompt. Each answer was graded automatically: exact, numeric or
LaTeX-normalised matching, and execution of unit tests in a network-less Docker sandbox for code. That gave 28,296
graded answers on 7,074 tasks. A second **borderline pass** collected 8,580 more GLM and Qwen answers on the 2,145
tasks where the models disagreed. This is where the hard-to-judge answers are; those extra samples are used for
training only.

| source | category | tasks | grading | license |
|---|---|---|---|---|
| [GSM8K](https://huggingface.co/datasets/openai/gsm8k) | math word problems | 1,000 | numeric match | MIT |
| [MATH](https://huggingface.co/datasets/EleutherAI/hendrycks_math) | competition math | 1,000 | normalised LaTeX / numeric | MIT |
| [MMLU-Pro](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) | knowledge MCQ (10 options) | 1,200 | option letter | MIT |
| [ARC-Challenge](https://huggingface.co/datasets/allenai/ai2_arc) | science MCQ | 800 | option letter | CC-BY-SA-4.0 |
| [MBPP](https://huggingface.co/datasets/google-research-datasets/mbpp) | Python coding | 700 | unit tests in sandbox | CC-BY-4.0 |
| [HumanEval](https://huggingface.co/datasets/openai/openai_humaneval) | Python coding | 164 | unit tests in sandbox | MIT |
| [AQuA-RAT](https://huggingface.co/datasets/deepmind/aqua_rat) | algebra word problems (MCQ) | 800 | option letter | Apache-2.0 |
| [CommonsenseQA](https://huggingface.co/datasets/tau/commonsense_qa) | commonsense MCQ | 600 | option letter | MIT |
| [BIG-Bench Hard](https://huggingface.co/datasets/lukaemon/bbh) | 27 reasoning tasks | 810 | exact match | MIT |

The first six sources use the same task ids as
[Shaktimaan-routing](https://huggingface.co/datasets/podstack/Shaktimaan-routing), so the two models can be
compared on identical tasks. Charts and long documents were left out.

**Solve rate per model** (share of answers graded correct):

| source | glm-5.3-flash | qwen3.8-27b | kimi-vl |
|---|---|---|---|
| GSM8K | 0.96 | 0.95 | 0.81 |
| MATH | 0.75 | 0.75 | 0.42 |
| MMLU-Pro | 0.81 | 0.80 | 0.49 |
| ARC-Challenge | 0.97 | 0.97 | 0.92 |
| MBPP | 0.93 | 0.82 | 0.65 |
| HumanEval | 0.94 | 0.91 | 0.81 |
| AQuA-RAT | 0.89 | 0.92 | 0.64 |
| CommonsenseQA | 0.80 | 0.81 | 0.75 |
| BIG-Bench Hard | 0.90 | 0.91 | 0.54 |

**Splits are by task, never by answer**, so no task appears in two splits:

| split | tasks | answers | wrong answers | used for |
|---|---|---|---|---|
| train | 5,261 | 38,558 | 40.0% (oversampled from ~18%) | fine-tuning |
| val | 370 | 1,480 | 17.6% | Platt calibration only |
| test | 633 | 2,532 | 17.6% | Shaktimaan's held-out test tasks + 10% of AQuA / CommonsenseQA |
| ood | 810 | 3,240 | 18.8% | all of BIG-Bench Hard, never trained on |

Each example's label is `accept` if the answer was graded correct and `escalate` otherwise. Held-out splits use
only the base samples, without the borderline extras, so every task is sampled the same way.

## Training
| setting | value |
|---|---|
| base model | Qwen2.5-0.5B-Instruct (494M parameters) |
| method | full fine-tune, supervised (TRL `SFTTrainer`); loss on the one-word verdict only |
| epochs | 2 (2,380 steps) |
| learning rate | 2e-5, cosine schedule, 30 warm-up steps |
| effective batch size | 32 |
| max sequence length | 2,048 tokens |
| precision | bf16, gradient checkpointing |
| hardware / time | one MIG 1g.67gb slice of an NVIDIA B300, 1 h 54 min |
| eval loss (val) | 0.095 after epoch 1, 0.142 after epoch 2 (the published weights are from epoch 2) |

## Results

### As a verifier
Calibrated P(correct) on answers to tasks Jedi never saw. *False accept* = share of wrong answers scored >= 0.5;
*false escalate* = share of correct answers scored < 0.5.

| split | answers | wrong | AUROC | false accept | false escalate | ECE | Brier |
|---|---|---|---|---|---|---|---|
| test (in-domain) | 2,532 | 17.6% | **0.920** | 35.9% | 1.9% | 0.017 | 0.063 |
| val | 1,480 | 17.6% | 0.884 | 42.3% | 1.3% | 0.017 | 0.072 |
| ood (BIG-Bench Hard) | 3,240 | 18.8% | 0.720 | 70.2% | 6.7% | 0.060 | 0.140 |

**Test split by answering model and by source:**

| slice | answers | wrong | AUROC | false accept |
|---|---|---|---|---|
| glm-5.3-flash | 1,266 | 11.8% | 0.910 | 41.6% |
| qwen3.8-27b | 633 | 12.3% | 0.915 | 50.0% |
| kimi-vl | 633 | 34.6% | 0.896 | 26.9% |
| HumanEval | 72 | 9.7% | 0.991 | 14.3% |
| MATH | 424 | 31.8% | 0.963 | 14.1% |
| MBPP | 268 | 9.7% | 0.955 | 38.5% |
| MMLU-Pro | 480 | 29.4% | 0.928 | 34.0% |
| AQuA-RAT | 316 | 16.8% | 0.896 | 30.2% |
| GSM8K | 420 | 6.2% | 0.854 | 53.8% |
| ARC-Challenge | 300 | 4.0% | 0.842 | 91.7% |
| CommonsenseQA | 252 | 18.3% | 0.593 | 89.1% |

Jedi is strongest where wrong answers *look* different from right ones: multi-step math and code, where
failures tend to be long, meandering or cut off. It is weakest where a wrong answer reads exactly like a right one,
such as single-hop commonsense and easy science multiple choice.

### As the gate of the ladder
491 held-out tasks: Shaktimaan's test tasks from GSM8K, MATH, MMLU-Pro, ARC, MBPP and HumanEval. GLM and Qwen
answers are fresh samples; Ultra's outcome and token count come from its graded run in Shaktimaan-routing (same task
ids). Cost = sum over called agents of (relative cost x output tokens), normalised so always-Ultra = 1.00. *Silent
failure* = a wrong answer accepted by a lower rung, i.e. a mistake the ladder had a chance to escalate and didn't.

| policy | accuracy | relative cost | silent failures | calls / task |
|---|---|---|---|---|
| Always `glm-5.3-flash` | 88.8% | 0.115 | - | 1.00 |
| Always `qwen3.8-27b` | 87.4% | 0.421 | - | 1.00 |
| Always `nemotron-ultra` | 84.7% | 1.000 | - | 1.00 |
| Shaktimaan v1.0 routing | 84.9% | 0.323 | - | - |
| Ladder + format check (escalate truncated / unparseable answers) | 89.8% | 0.406 | 7.3% | 1.07 |
| Ladder + self-consistency (escalate if two samples disagree) | 90.4% | 0.554 | 6.5% | 1.14 |
| **Ladder + Jedi, threshold 0.7** | **91.9%** | **0.454** | **2.9%** | 1.15 |
| Ladder + perfect verifier (oracle, upper bound) | 93.3% | 0.473 | 0.0% | 1.19 |

**Threshold sweep** (same 491 tasks):

| threshold | accuracy | relative cost | silent failures | calls / task | extra latency / task |
|---|---|---|---|---|---|
| 0.05 | 91.0% | 0.244 | 7.5% | 1.09 | 8.9 s |
| 0.1 | 91.5% | 0.263 | 6.5% | 1.10 | 9.5 s |
| 0.3 | 91.5% | 0.323 | 5.5% | 1.11 | 10.7 s |
| 0.5 | 91.7% | 0.387 | 4.5% | 1.12 | 12.1 s |
| **0.7** | **91.9%** | **0.454** | **2.9%** | 1.15 | 14.3 s |
| 0.8 | 91.0% | 0.506 | 2.9% | 1.20 | 16.4 s |
| 0.9 | 90.2% | 0.609 | 1.6% | 1.32 | 20.8 s |
| 0.95 | 87.0% | 0.891 | 1.4% | 1.65 | 34.5 s |

Above 0.7, accuracy *falls* even as silent failures keep dropping. Escalating a correct answer is not free: the
next rung can get it wrong, and in this pool Ultra's recorded solve rate (84.7%) is below GLM's. Use **0.7** for
the best accuracy, or **0.1-0.3** for most of the gain at a quarter of always-Ultra's cost.

### Outside the training domains (two-rung ladder: GLM -> Qwen)
Cost is relative to always using Qwen.

| tasks | policy | accuracy | cost vs always-Qwen | silent failures |
|---|---|---|---|---|
| AQuA + CommonsenseQA test (142) | always GLM / always Qwen | 85.2% / 88.7% | 0.30 / 1.00 | - |
| | self-consistency | 88.7% | 1.05 | 7.0% |
| | Jedi @ 0.5 | 88.7% | 0.68 | 8.5% |
| | Jedi @ 0.7 | 87.3% | 0.72 | 8.5% |
| | oracle | 92.3% | 0.71 | 0.0% |
| BIG-Bench Hard, unseen (810) | always GLM / always Qwen | 90.1% / 90.6% | 0.14 / 1.00 | - |
| | self-consistency | 91.4% | 0.53 | 6.1% |
| | Jedi @ 0.7 | 91.1% | 0.27 | 7.8% |
| | Jedi @ 0.9 | 92.0% | 0.69 | 2.2% |
| | oracle | 94.6% | 0.37 | 0.0% |

On unseen task families the ladder still improves on both single models, because the cheap model is usually right
and Jedi rarely escalates correct answers. It catches far fewer wrong ones than it does in-domain, though. On the
CommonsenseQA-heavy new-domain set, Jedi at 0.7 does not beat always using Qwen.

### Overhead
~56 ms per check (batch size 1, one MIG 1g.67gb slice of a B300), ~451 input tokens, 1 output token. Checks are a
single forward pass and batch well.

## Limitations
- **Not a checker.** Jedi does not verify the final answer: changing it barely moves the score, and planted
  arithmetic slips in otherwise fluent answers usually pass. It relies heavily on surface cues, and on unseen task
  families a logistic regression on answer length, truncation and agent beats it (AUROC 0.83 vs 0.72).
- **Domain-bound.** Verification skill transfers poorly to unseen task families (BIG-Bench Hard AUROC 0.72) and is
  near chance on CommonsenseQA (0.59). Measure it on your own traffic before relying on a threshold.
- **A third of wrong in-domain answers get through at 0.5.** False-accept rates are highest for the strongest
  models, whose mistakes are rarer and more fluent. Jedi is a cost-saving filter, not a correctness guarantee.
- **Pool-specific.** Trained on answers from GLM-5.3-Flash, Qwen3.8-27B and Kimi-VL; the agent name is part of its
  input. Other models' answers are out of distribution, and a new pool needs new data and retraining.
- **Text and code only**, English only, single-turn tasks with a checkable final answer. Not trained on open-ended
  writing, agentic / tool use, multi-turn or image inputs. Long tasks and answers are truncated (head + tail) for judging.
- **Labels inherit grading noise** from automatic answer extraction and matching.
- **Ladder numbers use recorded Ultra outcomes**, a single sample per task from the Shaktimaan-routing runs, not
  fresh calls.
- **Validation loss rose in epoch 2** (0.095 -> 0.142) while AUROC and calibration stayed good; a one-epoch model
  may be marginally better.

## Files
| file | purpose |
|---|---|
| `model.safetensors`, `config.json`, tokenizer files | the verifier |
| `calibration.json` | Platt scaling `a`, `b` and the `accept` / `escalate` token ids |
| `common.py` | agent pool, ladder, prompts, `judge_view`, answer extraction and graders |
| `jedi.py` | end-to-end runtime (answer -> verify -> escalate) |
| `gen.py` | collects and grades agent answers (stage 1) |
| `build.py` | builds the accept / escalate dataset and task-level splits (stage 2) |
| `train.py` | fine-tuning script (stage 3) |
| `evaluate.py` | verifier metrics, ladder, baselines and threshold sweep (stage 4) |
| `eval_results.json` | raw evaluation output behind every number on this card |
| `diagnostics.py`, `diagnostics_ladder.py` | surface-feature baselines, answer-swap test, ladder with a surface-feature gate |

The graded answer data is not published; `gen.py` and `build.py` regenerate it given endpoints for the three models.

## License
Model weights: Apache-2.0 (as the base model). Task sources keep their own licenses (table above).