Add files using upload-large-folder tool

.gitattributes

@@ -33,3 +33,9 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tekken.json filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
+docs/assets/05-gain-over-base.png filter=lfs diff=lfs merge=lfs -text
+docs/assets/00-benchmark-card.png filter=lfs diff=lfs merge=lfs -text
+docs/assets/06-vs-jev.png filter=lfs diff=lfs merge=lfs -text
+docs/assets/02-latency-vs-qwen.png filter=lfs diff=lfs merge=lfs -text

model-00001-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:41c1d711d80f96f9cd5ca8eb8b0b4375addd3174b2021bd8ab72dc1a9c9c7500
+size 4999724576

model-00002-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:57d50fd02ebafbb491d35d445c8279dda89ad8ad5034edf7637971564857b48d
+size 4999820896

model-00003-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:03616016aafb162714f7265f9e1a192f72bdc109e4d80d079c593fd0756c1cba
+size 4915917688

model-00004-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a77ea6878393937e46cead80bdfda98e090020cd93b746b89c38a5a81fbf823a
+size 2920659992

server/benchmarks/DECISION_BENCHMARK_SELECTION.md

@@ -0,0 +1,77 @@
+# JEV 형태의 판단 성능을 위한 추가 벤치 선택
+
+확인일: 2026-09-21. **현재 비교의 중심은 KEV의 고정 평가 세트와 JevBench**다.
+추가 공개 데이터는 아래 두 개를 추천한다. 이 문서는 선정 근거이며,
+CLINC150과 ContractNLI의 변환·실행·추가 지표는 아직 통합하지 않았다.
+모두 텍스트 입력에서 정해진 선택지의 확률만 읽어 평가할 수 있다.
+
+| 우선순위 | 데이터 | 측정하려는 판단 | 공식 테스트 규모 | 라이선스 |
+|---|---|---|---|---|
+| 1 | CLINC150 / oos-eval | 요청 라우팅과 지원 범위 밖 요청 거절 | 지원 범위 4,500 + 범위 밖 1,000 = 5,500문항 | CC BY 3.0 |
+| 2 | ContractNLI | 문서가 조건을 지지하는지, 반박하는지, 근거가 없는지 | 123개 계약 × 17개 명제 = 2,091판단 | CC BY 4.0 |
+
+## CLINC150: 라우팅과 범위 밖 요청
+
+150개 intent와 `oos` 한 개를 합쳐 **151개 선택지**를 사용한다.
+공식 `data/data_full.json`은 intent마다 train 100개·validation 20개·test 30개이며,
+별도 OOS 분할은 train 100개·validation 100개·test 1,000개다.
+따라서 검증 세트는 총 3,100개, 테스트는 5,500개다.
+영어 단일 intent 요청을 사람이 작성·패러프레이즈한 데이터다.
+[공식 설명](https://github.com/clinc/oos-eval/tree/828f8093932c8fe6ca7936c3d2e52903b1c523de)
+[라이선스](https://github.com/clinc/oos-eval/blob/828f8093932c8fe6ca7936c3d2e52903b1c523de/LICENSE)
+
+권장 매핑은 `state=사용자 발화`, `choice=150개 intent + oos`다.
+공식 intent 이름의 밑줄을 공백으로 바꾸는 정도의 결정적 설명을 쓰고,
+테스트 발화나 정답을 이용해 선택지 설명을 만들지 않는다.
+모델마다 동일한 후보 목록·순서를 사용하며, 후보 일부를 정답에 맞춰 제거하지 않는다.
+151개 선택지는 기존 어댑터의 255개 한도 내에 있다.
+
+통합할 때 추가할 지표:
+
+- 지원 범위 150개에 대한 accuracy·macro F1, 전체 151개 macro F1.
+- `p(oos)`를 점수, OOS를 양성으로 정한 AUROC·AUPRC.
+- FPR@95% OOS TPR: OOS의 95%를 탐지할 때 정상 요청을 잘못 거절하는 비율.
+- 고정 임계값에서 OOS 탐지율과 정상 요청 오거절률, 수락한 요청의 정확도·coverage.
+- Brier·NLL·ECE와 요청별 p50/p95 지연. 후보 151개의 비용을 함께 명시.
+
+운영 임계값을 선택한다면 validation만 사용하고 test에는 고정해서 적용한다.
+가중치 학습 없이 비교하되, 임계값 조정 여부는 별도로 기록한다.
+OOS는 **지원하지 않는 intent**라는 뜻이다. 입력 근거가 부족해서 답을 알 수 없는
+KEV `unknowable`과는 다른 능력을 측정하므로 둘을 합산하지 않는다.
+
+직접 검증한 원본: 커밋 `828f8093932c8fe6ca7936c3d2e52903b1c523de`,
+[`data/data_full.json`](https://github.com/clinc/oos-eval/blob/828f8093932c8fe6ca7936c3d2e52903b1c523de/data/data_full.json),
+SHA256 `36923c3705a59e08fe9c3883d8bc2dd966ef93e22cb78ac41171782a698d56e0`.
+다른 `small`·`plus` 변형과 혼합하지 않는다.
+
+## ContractNLI: 명시된 문서 근거에 따른 판단
+
+607개의 NDA에 동일한 17개 명제를 사람이 주석했다.
+분할은 train 423개·development 61개·test 123개 계약이다.
+정답은 `Entailment`, `Contradiction`, `NotMentioned`의 세 종류다.
+예외에 의한 부정, 문서에 없는 내용, 긴 문서의 근거를 찾는 판단을 점검하기 좋다.
+[공식 데이터·스키마·라이선스](https://stanfordnlp.github.io/contract-nli/)
+[원 논문의 분할 표](https://aclanthology.org/2021.findings-emnlp.164.pdf)
+
+`state=전체 계약 텍스트`, 각 명제를 `choice` 질문으로 바꾼다.
+출력은 세 선택지의 확률이며, 근거 문장 생성은 요구하지 않는다.
+`NotMentioned`를 `false`로 합치면 반박과 근거 부재를 구분할 수 없으므로
+기본 비교에는 `noul` 대신 3-way `choice`를 권장한다.
+원본의 정답 evidence span은 입력에 넣지 않는다.
+
+권장 지표는 3개 레이블 macro F1·accuracy, 명제별 성능, Brier·NLL·ECE,
+`NotMentioned` precision/recall이다. 계약 단위로 신뢰구간을 계산해 한 문서의
+17개 질문을 독립 표본으로 취급하지 않는다.
+명제당 지연과 계약 전체 17개 질문의 지연을 함께 보고한다.
+
+긴 계약을 임의로 잘라 평가하지 않는다. 각 모델의 tokenizer로 입력 길이를 먼저
+측정하고, 공통 context 한도 밖의 문서는 실패·제외 수를 명시한다.
+본문 전체 판단만 수행한 결과를 원 논문의 evidence identification까지 포함한
+전체 과제 결과와 동일하다고 표현하지 않는다.
+
+## 해석 범위
+
+두 데이터 모두 공개된 지 오래되어 최신 Qwen·Mistral의 사전학습 오염을 배제할 수 없다.
+학습하지 않은 모델을 평가한다는 것은 **이번 실험에서 추가 튜닝하지 않는다**는 뜻이다.
+공개 테스���를 처음 접했다는 보장은 아니다. 영어 중심 결과를 한국어 운영 성능으로
+일반화하지 않으며, 모델 비교에서는 동일 입력·동일 분할·동일 판단 형식을 유지한다.

server/benchmarks/METRIC_VALIDATION.md

@@ -0,0 +1,71 @@
+# KEV metric parity validation
+
+On 2026-09-21, the adapter's scalar quality metrics were compared numerically
+with the original KEV implementations at commit
+`4f8110a3f8620cc3a182ae9a708e4398492c4b1a`.
+All common scalar metrics matched within `1e-12` absolute error. The largest
+observed difference was `8.881784197001252e-16` (score MAE). Complete contrastive
+pair metrics matched exactly.
+
+This validates the metric calculations. It does not validate H200 execution,
+inference probabilities, model accuracy, or serving latency.
+
+## Method
+
+The optional [verify_metrics.py](verify_metrics.py) script reads these original
+files from a local KEV checkout and verifies their full SHA256 before execution:
+
+| File | Extracted functions | SHA256 |
+|---|---|---|
+| `kev/evaluate.py` | `ece` | `1b20e3f9edf417aa8dae924b1526e52f74b710cadf7213c5ec68334f6e7f8fe1` |
+| `kev/benchmark.py` | `coverage_at_error`, `metrics` | `4192ec3b26b065452f84bde38a091e6854a28fe185d2e0f39a0c91b2efe69df7` |
+| `kev/contrastive.py` | `paired_flip` | `cbb979aa5d40265ad0e64695f94b281d91751fa405811ddfa8212fede111edcf` |
+
+Python AST extraction keeps only those function definitions. KEV's package,
+PyTorch, Transformers and model-loading code are not imported. The audit uses
+NumPy in its own optional environment; NumPy is not an adapter dependency.
+The original successful audit used NumPy `2.3.5`.
+
+Input generation uses `numpy.random.default_rng(483)` for 100 batches of 50
+rows, totaling 5,000 rows. Rows mix choice, boolean and ordinal score questions,
+with 2–10 options. Probabilities come from Dirichlet distributions. The first
+10 rows in each batch additionally cover binary probability endpoints and
+decimal boundaries, including zero and one. Both implementations receive the
+same rows in the same order, including confidence ties.
+
+Compared metrics include accuracy, NLL, multiclass Brier score, ten-bin ECE,
+mean confidence, confidence bias, confident errors, coverage/accuracy at 0.9,
+coverage at 1%/5% empirical error, score MAE and ranked probability score.
+Ten complete two-sibling pairs additionally check relevant and invariant pair
+metrics. Five pairs have changed gold labels and five have unchanged labels.
+
+## Reproduce
+
+Use a Python environment that already has NumPy installed:
+
+```bash
+git clone https://github.com/jaredpalmer/kev.git /tmp/kev-reference
+git -C /tmp/kev-reference checkout 4f8110a3f8620cc3a182ae9a708e4398492c4b1a
+python benchmarks/verify_metrics.py --kev-root /tmp/kev-reference
+```
+
+The JSON output includes the reference hashes, current adapter metric-code hash,
+NumPy version, sample counts, per-metric maximum differences and pair results.
+A source mismatch or numerical discrepancy exits unsuccessfully. Reference
+files are checked even if the local checkout has uncommitted changes.
+
+## Scope differences
+
+- Accuracy headlines use clean question rows, not all submitted records.
+  Source `unknowable` is excluded from accuracy and scored for confidence.
+- Incomplete pairs are counted explicitly for smoke subsets; KEV's original
+  pair function rejects incomplete pairs.
+- The adapter omits meaningless unknowable accuracy from grouped reports;
+  KEV's original code includes it in some diagnostic subreports.
+- This audit compares common scalar metrics and complete pairs. It does not
+  certify every report field, input conversion, HTTP behavior, or SemIf's
+  separate family-balanced aggregation.
+
+Original definitions: [KEV benchmark.py](https://github.com/jaredpalmer/kev/blob/4f8110a3f8620cc3a182ae9a708e4398492c4b1a/kev/benchmark.py),
+[evaluate.py](https://github.com/jaredpalmer/kev/blob/4f8110a3f8620cc3a182ae9a708e4398492c4b1a/kev/evaluate.py),
+[contrastive.py](https://github.com/jaredpalmer/kev/blob/4f8110a3f8620cc3a182ae9a708e4398492c4b1a/kev/contrastive.py).

server/benchmarks/PUBLIC_DATASETS.md

@@ -0,0 +1,99 @@
+# Public decision benchmarks beyond the KEV suites
+
+Verified on 2026-09-21. JevBench's 231 public decisions are integrated into the preparation tool and default H200 matrix. The other entries below are research candidates, not automatically downloaded.
+
+The primary evaluation combines KEV and **JevBench's 231 public decisions** for explicit rule following and probability quality. **Typed Decisions' 400 test cases / 2,000 questions** is a secondary shared-state diagnostic because its labels measure teacher agreement. Image support is not a selection requirement. Additional objective intent/OOS and evidence-grounded candidates are assessed in [DECISION_BENCHMARK_SELECTION.md](DECISION_BENCHMARK_SELECTION.md). Keep each suite's results separate.
+
+## Official TypeSafe data versus community benchmarks
+
+No downloadable, labeled benchmark released by TypeSafe itself was located in its [official documentation](https://docs.typesafe.ai/introduction), [announcement](https://typesafe.ai/blog/introducing-system-one-models-and-jev), or [public GitHub organization](https://github.com/typesafe-ai). This is a search finding, not proof that no such data exists. TypeSafe publishes an [LLM comparison adapter](https://github.com/typesafe-ai/system-one-adapter-python), but an adapter is not an evaluation dataset.
+
+All Jev-specific datasets below are independent community work. A repository name containing `Jev` or `TypeSafe` does not make it official. The community [TypeSafeAI playground](https://github.com/TypeSafeAI/typesafe-playground) explicitly says it is independent and its examples are not validated accuracy benchmarks.
+
+KEV's current [4B model card](https://github.com/jaredpalmer/kev/blob/4f8110a3f8620cc3a182ae9a708e4398492c4b1a/docs/model-cards/kev-4b.md) already reports:
+
+- `decision-v7`, `transfer-v4`, and newer `transfer-v9` evaluations;
+- SemIf's 144 authored examples;
+- scienthoon's 900 support tickets;
+- ekzhang's 1,000-question MMLU-Pro sample.
+
+SemIf, scienthoon, and that MMLU-Pro sample are therefore **external to KEV's original training suite, but not additional benchmarks that KEV has never run**. The referenced KEV suites are text based; they do not measure the vision encoder. “Never trained” in KEV's terminology concerns its own fine-tuning sources and is not evidence that a foundation model or Jev never saw the data.
+
+## 1. JevBench public subset — first addition
+
+[JevBench](https://github.com/fstandhartinger/jevbench) covers routing, policy checks, intent, enum extraction, ordinal severity, answer judging, and harder reasoning over supplied state. It explicitly disclaims TypeSafe affiliation. The full published evaluation has 534 decisions, while the downloadable public subset has **231**. Use a separate public-subset result rather than claiming reproduction of the full leaderboard.
+
+Pinned repository: [`fd51755eb0c0b546ca206d764faf3302feca913e`](https://github.com/fstandhartinger/jevbench/tree/fd51755eb0c0b546ca206d764faf3302feca913e). Counts below were checked by parsing the pinned JSONL bytes.
+
+| Download | Decisions | Choice | Noul | Score | License |
+|---|---:|---:|---:|---:|---|
+| [original.jsonl](https://raw.githubusercontent.com/fstandhartinger/jevbench/fd51755eb0c0b546ca206d764faf3302feca913e/datasets/public/original.jsonl) | 72 | 36 | 24 | 12 | MIT |
+| [easy.jsonl](https://raw.githubusercontent.com/fstandhartinger/jevbench/fd51755eb0c0b546ca206d764faf3302feca913e/datasets/public/easy.jsonl) | 48 | 36 | 12 | 0 | MIT |
+| [hard.jsonl](https://raw.githubusercontent.com/fstandhartinger/jevbench/fd51755eb0c0b546ca206d764faf3302feca913e/datasets/public/hard.jsonl) | 111 | 67 | 38 | 6 | MIT |
+| Total | 231 | 139 | 74 | 18 | |
+
+SHA-256 checksums, in the same order:
+
+```text
+5c2414edb3006b8bfcb70fda433f0f9ca015759433849f8d3104328a1f7c4180
+231df3c2c8e88a1a8c137ebe85de96ba70fabd330849098ac7b3c52c70b7172b
+89e9e6becb33ed88c1de7d42dcc87531b2fb64cfaef4e1986faf7c37b3f80ebb
+```
+
+Each row has `state`, one `question`, ordered `labels`, `expected`, `id`, `group`, and `provenance`. Send only `state` and `question`. Noul gold is `"no"`/`"yes"`; Score gold is an integer rubric index. The hard tier's `provenance` can contain the answer rationale and `gold_probs`: neither belongs in model input. Some hard cases have exact reference probabilities, unlike ordinary hard-label classification. The [hard-tier protocol](https://github.com/fstandhartinger/jevbench/blob/fd51755eb0c0b546ca206d764faf3302feca913e/datasets/HARD-TIER.md) describes synthetic authoring, cross-model review, and the freeze.
+
+The 303 unavailable decisions comprise 157 deliberately private items and 146 imported items whose task text the maintainers do not redistribute. Hashes and aggregate results do not reconstruct those inputs or grant redistribution rights. See [third-party terms](https://github.com/fstandhartinger/jevbench/blob/fd51755eb0c0b546ca206d764faf3302feca913e/THIRD-PARTY.md) and the [license](https://github.com/fstandhartinger/jevbench/blob/fd51755eb0c0b546ca206d764faf3302feca913e/LICENSE).
+
+Implementation notes: retain paraphrase `group` identifiers for grouped uncertainty estimates. Label our adapter's output as **token-logprob-derived**, since JevBench's published ordinary-LLM adapter uses verbalized probabilities. Its leaderboard also applies assumed latency adjustments to some endpoints; our H200 results should report actual local measurements without adopting those adjustments.
+
+## 2. LocalLLaMA/typed-decisions — shared-state and soft labels
+
+The [dataset card](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/blob/ea9306458d6e9563628369a3d1e72e362fb381d2/README.md) specifies four workflows: agent traces, customer service, invoices, and security incidents. Each has 300 train and 100 test cases, with five Choice/Noul/Score questions per case. Evaluate **400 test cases / 2,000 questions**. The `all` configuration repeats the four workflow configurations; do not concatenate both.
+
+- Dataset revision: `ea9306458d6e9563628369a3d1e72e362fb381d2`.
+- [Combined test Parquet](https://huggingface.co/datasets/LocalLLaMA/typed-decisions/resolve/ea9306458d6e9563628369a3d1e72e362fb381d2/all/test-00000-of-00001.parquet), SHA-256 `4f294f218ea1da27f3efef936359389c62ea4d3973a41457732990f1d31b647c`.
+- License: Apache-2.0, as declared in the pinned card.
+- Parse JSON-string columns `state`, `questions`, and `gold`. Only the first two are model input; `factors` reveals the generating variables.
+
+Gold averages three teacher-model probability samples. Consequently, this measures **agreement with a synthetic teacher**, not independently established correctness. Report hard-label agreement separately from soft-target Brier/KL and Score MAE. The teacher can be wrong, so a stronger model can receive a lower score. Its published Jev result is measured on this test set; its specialist baselines were trained on the corresponding training workflows.
+
+## 3. NPC addressee benchmark — application behavior
+
+[wondertwins/jev-benchmark](https://github.com/wondertwins/jev-benchmark/tree/1c2509ac7d5508b8a7ce00ae4df6d7652d05de8b) provides **79 hand-labeled utterances**, each with clean, punctuation-free STT, and misheard-name variants. That is 237 case variants, not 237 independent utterances. Its strict addressee metrics exclude four explicitly ambiguous originals. Questions combine per-NPC Noul, primary-addressee Choice, intent, and urgency Score.
+
+- Pin: `1c2509ac7d5508b8a7ce00ae4df6d7652d05de8b`.
+- Fixtures: [npcaddress/dataset.py](https://github.com/wondertwins/jev-benchmark/blob/1c2509ac7d5508b8a7ce00ae4df6d7652d05de8b/npcaddress/dataset.py).
+- Request construction: [npcaddress/questions.py](https://github.com/wondertwins/jev-benchmark/blob/1c2509ac7d5508b8a7ce00ae4df6d7652d05de8b/npcaddress/questions.py).
+- [MIT license](https://github.com/wondertwins/jev-benchmark/blob/1c2509ac7d5508b8a7ce00ae4df6d7652d05de8b/LICENSE).
+
+The same repository contains 30 chess positions and 25 mate-in-one puzzles, with cached Stockfish references. Treat chess as a separate reasoning stress test. Keep raw-board, code-enriched, and filtered hybrid variants distinct. These inputs are text/structured state, not screenshots.
+
+## 4. Prompt-injection decisions — domain-specific addition
+
+[jev-sec-bench](https://github.com/Gaurav-Gosain/jev-sec-bench/tree/fdb16b94d37535db9bad77f8ef0faa971bd7d69a) evaluated all **662** public `deepset/prompt-injections` messages. The original data has 546 train and 116 test rows. Reproducing the 662-row community experiment requires explicitly naming the combined split; it must not be reported as 662 held-out test examples. Its policy context matters: the source labels concern a news assistant, not an unrestricted chatbot.
+
+[Dataset and Apache-2.0 declaration](https://huggingface.co/datasets/deepset/prompt-injections/tree/4f61ecb038e9c3fb77e21034b22511b523772cdd), pinned revision `4f61ecb038e9c3fb77e21034b22511b523772cdd`. Schema: `text`, binary `label`; map to Noul using the documented deployment context. Report F1, false positives/negatives, ROC-AUC, Brier, and latency. This is an application-specific classification benchmark, not proof of general guardrail security.
+
+## Lower-priority or overlapping sources
+
+[AbdelStark/jev-benchmarks](https://github.com/AbdelStark/jev-benchmarks/tree/0d610cc53e79bcbec691312b0c4adb4a0e371642) has a reproducible 300-example BTZSC pilot: AG News, Banking77, and Emotion. Those source families already occur in KEV training/evaluation, so the new harness is useful, but the data source is not independent of KEV's source selection. It pins BTZSC to `fef2a2ac62b69c58670047dddf045c53d7c3cb5e`; its Banking configuration has 72 available labels and excludes rows with no positive candidate. The harness is Apache-2.0; underlying datasets retain their own terms.
+
+Small scenario collections such as [souvikr/jev-test](https://github.com/souvikr/jev-test) are useful smoke tests, but 17 cases / 23 checks are too small and easy to serve as the main accuracy benchmark. Documentation-derived instruction corpora and unlabeled live Jev outputs are not independent gold evaluation data.
+
+## Future vision additions — general benchmarks, not Jev releases
+
+### POPE: object-presence Noul
+
+The [original POPE project](https://github.com/AoiDragon/POPE/tree/08d957b917e5a378a2f99d35b6293c536a66298b) supplies image filenames, questions, and yes/no labels. At the pinned revision, each of [random](https://raw.githubusercontent.com/AoiDragon/POPE/08d957b917e5a378a2f99d35b6293c536a66298b/output/coco/coco_pope_random.json), [popular](https://raw.githubusercontent.com/AoiDragon/POPE/08d957b917e5a378a2f99d35b6293c536a66298b/output/coco/coco_pope_popular.json), and [adversarial](https://raw.githubusercontent.com/AoiDragon/POPE/08d957b917e5a378a2f99d35b6293c536a66298b/output/coco/coco_pope_adversarial.json) contains **3,000 questions over 500 images**; counts were checked directly. These are related sampling variants, not three independent image datasets. Schema: `question_id`, `image`, `text`, `label`.
+
+Use Noul for “is this object present?” and report each variant separately. The [repository license](https://github.com/AoiDragon/POPE/blob/08d957b917e5a378a2f99d35b6293c536a66298b/LICENSE) is MIT; COCO image rights remain separate. Obtain the named COCO 2014 images according to the [COCO terms](https://cocodataset.org/#termsofuse). Start with a deterministic image-grouped subset before a full run.
+
+### VSR: spatial-relation Noul
+
+The original [VSR benchmark](https://github.com/cambridgeltl/visual-spatial-reasoning/tree/b27a0af0ee1462d2b6b92c8c83e869d9254a241a) asks whether a caption describes the spatial relationship in an image. Recommend the [random test file](https://huggingface.co/datasets/cambridgeltl/vsr_random/resolve/b2053328fafdd018ff56cf1dfa9643caaa4e69b8/test.jsonl): **2,195** image/caption pairs, SHA-256 `8ade82a0b93ac9dc1e53f6cf1f11e9d5536776a3715102b6b4d27d4f81d551cc`. Its [official dataset card](https://huggingface.co/datasets/cambridgeltl/vsr_random/tree/b2053328fafdd018ff56cf1dfa9643caaa4e69b8) declares CC-BY-4.0; the implementation repository is Apache-2.0, and COCO image terms remain separate.
+
+Schema includes `image`, `image_link`, `caption`, binary `label`, and `relation`. Fetch images using the original project's [image instructions](https://github.com/cambridgeltl/visual-spatial-reasoning/blob/b27a0af0ee1462d2b6b92c8c83e869d9254a241a/data/README.md); only image and caption belong in input. Score by relation and overall.
+
+Version pitfall: the repository README's zero-shot table says 616 test examples, but the pinned GitHub and [HF zero-shot](https://huggingface.co/datasets/cambridgeltl/vsr_zeroshot/tree/148b3777ceef4a1bfe46377614980836db7d12f2) `test.jsonl` both contain **1,222** rows, SHA-256 `914d9156723b912d8f794b179248af865a68c5f81c9d8e3a21d007778cacaad4`. Do not copy the README count into a report for those files.
+
+For vision timing, record image dimensions, processed image-token counts, resize policy, and cache condition. Measure the same fixed examples on every model. A no-image ablation tests image dependence but does not make the text-only task equally answerable. Keep prompt tuning, temperature calibration, and final test evaluation separate; public availability does not establish absence from any model's pretraining.

server/benchmarks/README.md

@@ -0,0 +1,144 @@
+# H200 한 장에서 학습 전 판단 성능 비교
+
+추가 학습·LoRA·확률 보정 없이 공식 모델을 실행해 KEV의 공개 평가 문항을 평가합니다. 모델과 SGLang은 서버에서 실행하고, 이 저장소의 어댑터와 벤치 클라이언트는 별도 Python 환경을 사용합니다. 기본 엔진을 수정하지 않습니다.
+
+현재 상태: **H200 실측 완료.** 네 모델 각각 4,006개 요청·4,852개 질문을 오류 없이 처리했습니다. [결과 보고서](../../outputs/h200-baseline-benchmark-2026-09-21/RESULTS.md)에서 정확도와 HTTP 지연을 확인할 수 있습니다. 제공된 모델은 추가 학습 전 공식 지시학습 배포본이며, 사전학습 전용 `*-Base` 모델을 뜻하지 않습니다.
+
+2026-09-21 로컬 검증: pytest 97개와 하위 사례 74개, H200 실행 도구 CPU 테스트 12개 통과. KEV 5종과 JevBench 공개 3종의 총 4,006개 요청·4,852개 질문을 준비했습니다. 합산 숫자는 실행 작업량이며 세트 간 중복을 제거한 독립 평가 문항 수가 아닙니다.
+
+## 모델과 비교 조건
+
+| 프로필 | 공식 모델 | 기본 정밀도 |
+|---|---|---|
+| `ministral3-3b-bf16` | `mistralai/Ministral-3-3B-Instruct-2512-BF16` | BF16 |
+| `qwen35-4b-bf16` | `Qwen/Qwen3.5-4B` | BF16 |
+| `qwen36-27b-bf16` | `Qwen/Qwen3.6-27B` | BF16 |
+| `qwen36-35b-a3b-bf16` | `Qwen/Qwen3.6-35B-A3B` | BF16 |
+
+원래 다운로드했던 Ministral의 공식 FP8 배포본은 `ministral3-3b-fp8` 선택형 프로필입니다. 기본 비교는 BF16으로 맞추며, FP8 결과는 별도 행으로 비교합니다. 모델 리비전, 엔진 리비전과 실행 옵션은 [h200/models.json](h200/models.json)에 고정했습니다. H200 1장, TP=1, 컨텍스트 8,192, 입력 절단 없음, speculative decoding/MTP 없음, 비전 인코더 유지 조건입니다. 최대 동시 요청은 32이며 기본 클라이언트 동시성은 1입니다.
+
+공식 지시학습 모델의 chat template에 `enable_thinking=False`를 전달하고, 선택지 라벨의 다음 토큰 확률을 읽습니다. 문장을 생성하지 않으며 `output_tokens=0`을 검증합니다. 질문마다 별도 프리필을 실행하므로 여러 질문을 하나의 특수 pointer head로 처리하는 KEV와 실행 구조는 다릅니다. 문항과 지표를 맞춘 **학습 전 어댑터 기준선**이며 KEV의 학습 결과나 기존 기본 모델 probe의 프롬프트를 그대로 복제한 실험은 아닙니다.
+
+Ministral은 SGLang 0.5.20의 native chat 문자열 재인코딩 오류를 피하기 위해
+어댑터에서 공식 토크나이저의 `tokenize=True` 결과를 전달합니다. 전체 4,852문항의
+토큰 경계와 최대 입력 길이(4,037토큰)를 CPU에서 확인했습니다. Qwen은 엔진의
+HTTP 토큰화 경로를 사용합니다. 따라서 모델 간 HTTP 지연에는 토큰화 위치와
+호출 횟수 차이가 포함되며, 순수 GPU 프리필 시간 비교가 아닙니다.
+초기 Ministral HTTP 점검은 `invalidated.json`으로 무효 표시하며 비교에서 제외합니다.
+
+## 준비된 KEV 데이터
+
+원본 커밋은 [`4f8110a3f8620cc3a182ae9a708e4398492c4b1a`](https://github.com/jaredpalmer/kev/tree/4f8110a3f8620cc3a182ae9a708e4398492c4b1a)입니다. 원본 manifest와 JSONL의 SHA256을 검증하고, 정답·출처·변형 정보를 모델 요청에서 제거합니다. 모든 질문을 유지하며 Banking77의 78개 선택지 변형도 포함합니다.
+
+| 개발 세트 | 요청 수 | 전체 질문 | 기본 정확도에 포함되는 질문 |
+|---|---:|---:|---:|
+| `decision-v7` | 1,204 | 1,468 | 1,264 |
+| `transfer-v4` | 764 | 764 | 656 |
+| `transfer-v9` | 1,264 | 1,264 | 1,046 |
+| `semif-v1` — 선택형 | 252 | 252 | 144 |
+| `scienthoon-v1` — 선택형 | 291 | 873 | 873, 작업별 구분 필요 |
+
+`decision-v7`은 KEV 학습에 사용된 출처의 평가 문항, `transfer-v4`는 KEV 미세조정에서 사용하지 않은 출처·정책 구조입니다. 이는 원래 Qwen/Mistral 사전학습에서 보지 않았다는 보장은 아닙니다. `transfer-v9`는 v4에 MMLU-Pro·긴 문맥·근거 제거 문항을 더한 것이므로 두 결과를 독립 데이터처럼 합산하지 않습니다.
+
+모델카드의 정확도와 비교할 값은 `report.json`의 `suites.<suite>/development.clean.acc`입니다. 순서 변형과 none-of-the-above 문항은 `variants`, 조건 반전은 `paired_flip`, 선택지 순서 영향은 `permutation`에 기록합니다. v9의 근거 제거 110문항은 정확도에서 제외하고 `unknowable`의 과신 비율로 평가합니다.
+
+SemIf는 원래 가족별 balanced accuracy를 사용하므로 여기의 clean micro accuracy를 원래 지표와 혼동하지 않습니다. Scienthoon의 원본 900개 질문 행은 KEV 변환 과정에서 291개 고유 상태·873개 질문이 됐습니다. priority 라벨은 입력에 없는 조직 규칙에 의존하므로 queue·angry와 분리해 `tasks`를 확인합니다. 두 외부 세트도 KEV가 이미 평가한 데이터���니다.
+
+## JevBench 공개 데이터 — 기본 실행에 추가
+
+[JevBench](https://github.com/fstandhartinger/jevbench/tree/fd51755eb0c0b546ca206d764faf3302feca913e)는 JEV 계열 판단 모델용 커뮤니티 벤치입니다. TypeSafe 공식 벤치는 아닙니다. 전체 결과표의 534문항 중 실제 공개된 231문항을 커밋과 SHA256으로 고정했습니다.
+
+| 공개 세트 | 문항 | 주요 평가 |
+|---|---:|---|
+| `jevbench-original` | 72 | 정책, 라우팅, 의도, 등급, 추출, 답변 적합성 |
+| `jevbench-easy` | 48 | 명확한 사실·의도·도구 선택 |
+| `jevbench-hard` | 111 | 복합 규칙, 시간·수량, 확률, 모호성, 함정, 답변 판단 |
+
+정답과 해설·생성 근거는 모델 입력에 넣지 않습니다. 데이터는 `public.jsonl`로 저장하며 테스트나 비공개 문항으로 부르지 않습니다. 확률을 정확히 계산할 수 있는 hard 10문항은 `reference_distribution`에서 확률 분포 오차를 별도로 측정합니다. 최고 확률 동률은 JevBench 원본처럼 라벨 사전순으로 처리합니다. 기존 KEV 지표와 공개 문항별 지표를 기록하며, JevBench의 전체 534문항 종합 점수나 네트워크 지연 보정 점수는 재현했다고 주장하지 않습니다. 확률 출력 방식도 `token-logprob-derived`로 구분합니다.
+
+평가 선정 기준은 **정책·분류·근거 부족·다중 질문 판단과 확률 품질**이며 비전 여부는 조건이 아닙니다. 기본 준비 검사는 텍스트만 사용하고, `--probe-vision`을 추가했을 때만 이미지도 점검합니다. Typed Decisions는 작은 교사 모델과의 일치도를 측정하므로 보조 후보입니다. 추가 후보와 선정 근거는 [PUBLIC_DATASETS.md](PUBLIC_DATASETS.md), [DECISION_BENCHMARK_SELECTION.md](DECISION_BENCHMARK_SELECTION.md)에 있습니다.
+
+## H200 서버에서 실행
+
+이 저장소를 서버로 옮긴 뒤 저장소 루트에서 실행합니다. 모델 가중치는 실행 시 Hugging Face에서 받으며, 네 모델을 동시에 GPU에 올리지 않습니다. 전체 BF16 체크포인트 캐시를 위해 충분한 로컬 디스크 공간을 확보합니다. 엔진 설치 조건과 CUDA 13 환경은 [h200/README.md](h200/README.md)를 따릅니다.
+
+어댑터 환경:
+
+```bash
+python3.12 -m venv .venv
+.venv/bin/python -m pip install -e '.[dev,native-tokenizer]'
+```
+
+평가 데이터만 다운로드합니다. 함께 전달된 `benchmarks/data`가 있다면 이 단계는 다시 실행해도 동일 파일인지 확인합니다.
+
+```bash
+.venv/bin/python -m jev_adapter.benchmarks.prepare \
+  --output benchmarks/data \
+  --suite decision-v7 --suite transfer-v4 --suite transfer-v9 \
+  --suite jevbench-original --suite jevbench-easy --suite jevbench-hard \
+  --suite semif-v1 --suite scienthoon-v1
+```
+
+오프라인에서는 같은 커밋의 KEV 체크아웃을 `--source-root /path/to/kev`, JevBench 체크아웃을 `--jevbench-source-root /path/to/jevbench`로 지정할 수 있습니다. 학습 데이터는 다운로드하지 않습니다.
+
+먼저 네 모델의 실행 명령만 확인합니다. `--execute`가 없으면 모델을 받거나 실행하지 않습니다.
+
+```bash
+.venv/bin/python benchmarks/run_matrix.py \
+  --engine-python .venv-sglang/bin/python \
+  --output benchmarks/results/preview --include-external
+```
+
+서버 통합 점검은 소량 문항으로 진행합니다. 이 결과는 정확도 비교용으로 쓰지 않습니다.
+
+```bash
+.venv/bin/python benchmarks/run_matrix.py \
+  --engine-python .venv-sglang/bin/python \
+  --output benchmarks/results/smoke \
+  --limit 8 --warmup 2 --execute
+```
+
+사전 점검이 통과하면 네 모델을 순서대로 평가합니다. 각 모델의 텍스트 입력, 토큰 경계, 0토큰 출력, 모델 리비전과 캐시 설정을 먼저 검증하고, 모델마다 어댑터를 다시 시작합니다. 기존 프로세스가 포트를 사용하면 중단합니다. JevBench만 실행할 때는 `--suite jevbench-original --suite jevbench-easy --suite jevbench-hard`를 지정하세요.
+
+```bash
+.venv/bin/python benchmarks/run_matrix.py \
+  --engine-python .venv-sglang/bin/python \
+  --output benchmarks/results/bf16-c1 \
+  --include-external --execute
+```
+
+동시성 8에서의 처리량도 보려면 별도 출력 폴더로 실행합니다. 동시성 1의 지연시간과 구분해서 해석합니다.
+
+```bash
+.venv/bin/python benchmarks/run_matrix.py \
+  --engine-python .venv-sglang/bin/python \
+  --output benchmarks/results/bf16-c8 \
+  --concurrency 8 --include-external --execute
+```
+
+`--profile qwen36-35b-a3b-bf16`처럼 모델 하나만 선택할 수 있습니다. `--repeats 3`은 지연 측정을 반복하고 정확도는 첫 반복만 사용합니다. 모든 결과 디렉터리는 새 경로여야 하며 기존 결과를 덮어쓰지 않습니다.
+
+## 결과와 측정 범위
+
+각 모델 아래 `engine/launch.json`, `engine/preflight.json`, `engine/engine.log`, `c1/manifest.json`, `c1/predictions.jsonl`, `c1/report.json`이 생성됩니다. 전체 실행이 성공하면 `comparison.csv`에 모델·세트별 결과를 모읍니다.
+
+- 정확도: clean 질문별 micro accuracy, 출처·작업별 정확도, 작업별 macro accuracy.
+- 확률 품질: multiclass Brier, NLL, 10-bin ECE, 최대 확률 0.9 이상 오답 비율, 경험적 오류 예산별 coverage. 어댑터의 entropy confidence 대신 **최대 선택지 확률**로 ECE를 계산합니다.
+- 순서·정책 반전: 라벨 정렬 후 flip rate, 최소 대조쌍 양쪽 정답률. 부분 집합의 누락된 짝은 별도 집계합니다.
+- 시간: 요청별 HTTP 왕복 p50/p95/p99, 어댑터 내부 벽시계 시간, 전체 요청·질문 처리량, 입력 토큰 수. 모델 로딩과 20회 준비 요청은 제외합니다.
+
+지연시간에는 HTTP, 토큰화, 엔진 스케줄링과 프리필이 포함됩니다. **순수 GPU 커널 시간이나 프리필 FLOPS 측정값이 아닙니다.** 질문 수와 입력 토큰 수도 함께 보세요. 어댑터는 질문마다 엔진 HTTP 호출 4회를 사용합니다. 정확도 평가를 위해 지정한 라벨 logprob를 반환받는 비용도 포함됩니다.
+
+기본 실행은 radix cache와 이미지 전처리·비전 특징 재사용을 끕니다. 반복 입력의 캐시 효과가 섞이지 않는 조건입니다. 반대로 실제 서비스의 캐시 효과를 측정하려면 별도 엔진 설정과 `--cache-mode server-default`의 개별 runner 실행으로 기록해야 합니다.
+
+오류·시간 초과·확률 누락·출력 토큰 발생 시 재시도나 문항 제외로 성공률을 높이지 않습니다. 실패한 실행은 `status=failed`로 저장하고 기본 정확도를 출력하지 않습니다. 데이터 SHA256·부분 집합 여부·코드 SHA256·실행 옵션·엔진/모델 정보를 기록합니다. 실패한 실행은 비교 CSV에 포함하지 않습니다.
+
+개발 세트로 조건을 정한 뒤 최종 확인에만 test를 사용합니다. test 준비와 개별 실행은 모두 `--allow-test`를 명시해야 합니다. 행렬 실행기는 KEV 개발 세트와 JevBench 공개 세트를 사용합니다. 학습·프롬프트 선택·온도 보정 없이 현재 고정 조건을 그대로 비교하는 것이 이 기준선의 목적입니다.
+
+```bash
+.venv/bin/python -m pytest
+.venv/bin/python -m unittest discover -s benchmarks/h200 -p 'test_*.py'
+.venv/bin/ruff check .
+```
+
+지표 수식의 원본 대조 결과와 재현 방법은 [METRIC_VALIDATION.md](METRIC_VALIDATION.md)를 참고하세요. 공개 데이터마다 원래 라이선스가 다르며 변환 manifest가 해당 출처를 보존합니다. JEV 유료 API 호출이나 학습은 실행 과정에 포함되지 않습니다.

server/benchmarks/data/jevbench-easy/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Florian Standhartinger and contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

server/benchmarks/data/jevbench-easy/THIRD-PARTY.md

@@ -0,0 +1,51 @@
+# Third-party projects, data and services
+
+MIT (see `LICENSE`) covers this harness and the 72 original public decisions in
+`datasets/public/original.jsonl`. Nothing else in this list is ours to license.
+
+## Systems evaluated
+
+Each was reached through the interface its author published. We used public
+endpoints as ordinary clients, at one request at a time, and never sent a
+provider's API key to anyone else's endpoint.
+
+| System | Author | Source |
+|---|---|---|
+| Jev 1.13.0 | TypeSafe AI | <https://docs.typesafe.ai> |
+| openjev-sglang | ekzhang | <https://github.com/ekzhang/openjev-sglang> |
+| system-one-open | mithalouni | <https://github.com/mithalouni/system-one-open> |
+| open-alternative-jev | IkerMoel | <https://github.com/ikermoel/open-alternative-jev>, <https://huggingface.co/spaces/IkerMoel/open-alternative-jev> |
+| open-jev-deberta-v3-large | Kotoba Labs | <https://huggingface.co/com-kotobalabs/open-jev-deberta-v3-large>, <https://github.com/kotoba-lang/typed-decisions> |
+| GPT-5.6 Luna | OpenAI | <https://platform.openai.com> |
+| Gemini 3.1 Flash-Lite | Google | <https://ai.google.dev> |
+| DeepSeek V4.1 Flash | DeepSeek | <https://api-docs.deepseek.com> |
+| Qwen3.8 27B | Qwen, served by Chutes | <https://chutes.ai> |
+
+Model weights, base models and each project's own code keep their own licences.
+A permissive licence on a repository is not a licence for the base model it
+fine-tunes, and we do not restate either.
+
+## Wire format
+
+The typed-decision request shape (`state`, `questions`, the `noul` / `choice` /
+`score` primitives) is TypeSafe's public HTTP API, documented at
+<https://docs.typesafe.ai/api>. Several of the open rebuilds implement it
+deliberately, which is why one adapter reaches more than one of them. JevBench is
+not affiliated with or endorsed by TypeSafe AI.
+
+## Imported decisions
+
+146 of the 242 decisions come from our own earlier auto-router experiment
+(<https://github.com/fstandhartinger/auto-model-router>): 78 routing requests and
+68 answer-adequacy judgements whose ground truth is a deterministic grader's
+verdict on a saved answer. The upstream task text is **not** redistributed here,
+because the datasets it was drawn from keep their own terms. `datasets/manifest.json`
+pins the hashes; `scripts/import_router.py` shows exactly what was taken and what
+was excluded.
+
+## Held-out decisions
+
+24 scenarios are written by us and deliberately unpublished, so the suite cannot
+be trained on in full. Only their whole-split hash and aggregate results appear
+here. They are sent to the services being evaluated, which is not the same thing
+as being public - see the limits section of the README.

server/benchmarks/data/jevbench-easy/public.jsonl

@@ -0,0 +1,48 @@
+{"id": "easy-intent-00", "suite": "jevbench-easy", "split": "public", "source": "intent", "variant": "clean", "record": {"state": "Where is my package? I ordered it last week and it still hasn't arrived.", "questions": {"decision": {"type": "choice", "instructions": "Which intent does the user's message express?", "criteria": {"billing_question": "Asks about a charge, invoice or payment", "cancel_order": "Wants to cancel an order", "change_address": "Wants to change the delivery address", "report_damage": "Received an item that is broken or damaged", "track_order": "Wants to know where an order is or when it arrives"}}}}, "expected": {"decision": {"labels": ["track_order", "cancel_order", "change_address", "report_damage", "billing_question"], "target": [1.0, 0.0, 0.0, 0.0, 0.0], "label": 0, "type": "choice", "task": "intent"}}, "metadata": {"id": "easy-intent-00", "source": "intent", "variant": "clean", "group_id": "easy-intent-00", "upstream_group": null, "canonical_labels": ["track_order", "cancel_order", "change_address", "report_damage", "billing_question"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-intent-01", "suite": "jevbench-easy", "split": "public", "source": "intent", "variant": "clean", "record": {"state": "Please cancel my order #4471, I don't need it anymore.", "questions": {"decision": {"type": "choice", "instructions": "Which intent does the user's message express?", "criteria": {"billing_question": "Asks about a charge, invoice or payment", "cancel_order": "Wants to cancel an order", "change_address": "Wants to change the delivery address", "report_damage": "Received an item that is broken or damaged", "track_order": "Wants to know where an order is or when it arrives"}}}}, "expected": {"decision": {"labels": ["track_order", "cancel_order", "change_address", "report_damage", "billing_question"], "target": [0.0, 1.0, 0.0, 0.0, 0.0], "label": 1, "type": "choice", "task": "intent"}}, "metadata": {"id": "easy-intent-01", "source": "intent", "variant": "clean", "group_id": "easy-intent-01", "upstream_group": null, "canonical_labels": ["track_order", "cancel_order", "change_address", "report_damage", "billing_question"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-intent-02", "suite": "jevbench-easy", "split": "public", "source": "intent", "variant": "clean", "record": {"state": "I moved. Can you ship my order to 12 Elm Street instead of the old address?", "questions": {"decision": {"type": "choice", "instructions": "Which intent does the user's message express?", "criteria": {"billing_question": "Asks about a charge, invoice or payment", "cancel_order": "Wants to cancel an order", "change_address": "Wants to change the delivery address", "report_damage": "Received an item that is broken or damaged", "track_order": "Wants to know where an order is or when it arrives"}}}}, "expected": {"decision": {"labels": ["track_order", "cancel_order", "change_address", "report_damage", "billing_question"], "target": [0.0, 0.0, 1.0, 0.0, 0.0], "label": 2, "type": "choice", "task": "intent"}}, "metadata": {"id": "easy-intent-02", "source": "intent", "variant": "clean", "group_id": "easy-intent-02", "upstream_group": null, "canonical_labels": ["track_order", "cancel_order", "change_address", "report_damage", "billing_question"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-intent-03", "suite": "jevbench-easy", "split": "public", "source": "intent", "variant": "clean", "record": {"state": "The mug I received arrived smashed into pieces.", "questions": {"decision": {"type": "choice", "instructions": "Which intent does the user's message express?", "criteria": {"billing_question": "Asks about a charge, invoice or payment", "cancel_order": "Wants to cancel an order", "change_address": "Wants to change the delivery address", "report_damage": "Received an item that is broken or damaged", "track_order": "Wants to know where an order is or when it arrives"}}}}, "expected": {"decision": {"labels": ["track_order", "cancel_order", "change_address", "report_damage", "billing_question"], "target": [0.0, 0.0, 0.0, 1.0, 0.0], "label": 3, "type": "choice", "task": "intent"}}, "metadata": {"id": "easy-intent-03", "source": "intent", "variant": "clean", "group_id": "easy-intent-03", "upstream_group": null, "canonical_labels": ["track_order", "cancel_order", "change_address", "report_damage", "billing_question"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-intent-04", "suite": "jevbench-easy", "split": "public", "source": "intent", "variant": "clean", "record": {"state": "Why was I charged twice on my credit card statement for one order?", "questions": {"decision": {"type": "choice", "instructions": "Which intent does the user's message express?", "criteria": {"billing_question": "Asks about a charge, invoice or payment", "cancel_order": "Wants to cancel an order", "change_address": "Wants to change the delivery address", "report_damage": "Received an item that is broken or damaged", "track_order": "Wants to know where an order is or when it arrives"}}}}, "expected": {"decision": {"labels": ["track_order", "cancel_order", "change_address", "report_damage", "billing_question"], "target": [0.0, 0.0, 0.0, 0.0, 1.0], "label": 4, "type": "choice", "task": "intent"}}, "metadata": {"id": "easy-intent-04", "source": "intent", "variant": "clean", "group_id": "easy-intent-04", "upstream_group": null, "canonical_labels": ["track_order", "cancel_order", "change_address", "report_damage", "billing_question"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-intent-05", "suite": "jevbench-easy", "split": "public", "source": "intent", "variant": "clean", "record": {"state": "When will my order be delivered? The tracking page shows nothing.", "questions": {"decision": {"type": "choice", "instructions": "Which intent does the user's message express?", "criteria": {"billing_question": "Asks about a charge, invoice or payment", "cancel_order": "Wants to cancel an order", "change_address": "Wants to change the delivery address", "report_damage": "Received an item that is broken or damaged", "track_order": "Wants to know where an order is or when it arrives"}}}}, "expected": {"decision": {"labels": ["track_order", "cancel_order", "change_address", "report_damage", "billing_question"], "target": [1.0, 0.0, 0.0, 0.0, 0.0], "label": 0, "type": "choice", "task": "intent"}}, "metadata": {"id": "easy-intent-05", "source": "intent", "variant": "clean", "group_id": "easy-intent-05", "upstream_group": null, "canonical_labels": ["track_order", "cancel_order", "change_address", "report_damage", "billing_question"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-intent-06", "suite": "jevbench-easy", "split": "public", "source": "intent", "variant": "clean", "record": {"state": "Wake me up at 6:30 tomorrow morning.", "questions": {"decision": {"type": "choice", "instructions": "Which intent does the user's message express?", "criteria": {"play_music": "Play a song, album, artist or playlist", "send_message": "Send a text message to someone", "set_alarm": "Set an alarm or wake-up time", "turn_off_lights": "Switch lights off", "weather": "Ask about the weather or forecast"}}}}, "expected": {"decision": {"labels": ["set_alarm", "play_music", "weather", "send_message", "turn_off_lights"], "target": [1.0, 0.0, 0.0, 0.0, 0.0], "label": 0, "type": "choice", "task": "intent"}}, "metadata": {"id": "easy-intent-06", "source": "intent", "variant": "clean", "group_id": "easy-intent-06", "upstream_group": null, "canonical_labels": ["set_alarm", "play_music", "weather", "send_message", "turn_off_lights"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-intent-07", "suite": "jevbench-easy", "split": "public", "source": "intent", "variant": "clean", "record": {"state": "Play some Taylor Swift.", "questions": {"decision": {"type": "choice", "instructions": "Which intent does the user's message express?", "criteria": {"play_music": "Play a song, album, artist or playlist", "send_message": "Send a text message to someone", "set_alarm": "Set an alarm or wake-up time", "turn_off_lights": "Switch lights off", "weather": "Ask about the weather or forecast"}}}}, "expected": {"decision": {"labels": ["set_alarm", "play_music", "weather", "send_message", "turn_off_lights"], "target": [0.0, 1.0, 0.0, 0.0, 0.0], "label": 1, "type": "choice", "task": "intent"}}, "metadata": {"id": "easy-intent-07", "source": "intent", "variant": "clean", "group_id": "easy-intent-07", "upstream_group": null, "canonical_labels": ["set_alarm", "play_music", "weather", "send_message", "turn_off_lights"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-intent-08", "suite": "jevbench-easy", "split": "public", "source": "intent", "variant": "clean", "record": {"state": "Will it rain in Berlin tomorrow?", "questions": {"decision": {"type": "choice", "instructions": "Which intent does the user's message express?", "criteria": {"play_music": "Play a song, album, artist or playlist", "send_message": "Send a text message to someone", "set_alarm": "Set an alarm or wake-up time", "turn_off_lights": "Switch lights off", "weather": "Ask about the weather or forecast"}}}}, "expected": {"decision": {"labels": ["set_alarm", "play_music", "weather", "send_message", "turn_off_lights"], "target": [0.0, 0.0, 1.0, 0.0, 0.0], "label": 2, "type": "choice", "task": "intent"}}, "metadata": {"id": "easy-intent-08", "source": "intent", "variant": "clean", "group_id": "easy-intent-08", "upstream_group": null, "canonical_labels": ["set_alarm", "play_music", "weather", "send_message", "turn_off_lights"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-intent-09", "suite": "jevbench-easy", "split": "public", "source": "intent", "variant": "clean", "record": {"state": "Text Anna that I'll be ten minutes late.", "questions": {"decision": {"type": "choice", "instructions": "Which intent does the user's message express?", "criteria": {"play_music": "Play a song, album, artist or playlist", "send_message": "Send a text message to someone", "set_alarm": "Set an alarm or wake-up time", "turn_off_lights": "Switch lights off", "weather": "Ask about the weather or forecast"}}}}, "expected": {"decision": {"labels": ["set_alarm", "play_music", "weather", "send_message", "turn_off_lights"], "target": [0.0, 0.0, 0.0, 1.0, 0.0], "label": 3, "type": "choice", "task": "intent"}}, "metadata": {"id": "easy-intent-09", "source": "intent", "variant": "clean", "group_id": "easy-intent-09", "upstream_group": null, "canonical_labels": ["set_alarm", "play_music", "weather", "send_message", "turn_off_lights"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-intent-10", "suite": "jevbench-easy", "split": "public", "source": "intent", "variant": "clean", "record": {"state": "Turn off the lights in the bedroom.", "questions": {"decision": {"type": "choice", "instructions": "Which intent does the user's message express?", "criteria": {"play_music": "Play a song, album, artist or playlist", "send_message": "Send a text message to someone", "set_alarm": "Set an alarm or wake-up time", "turn_off_lights": "Switch lights off", "weather": "Ask about the weather or forecast"}}}}, "expected": {"decision": {"labels": ["set_alarm", "play_music", "weather", "send_message", "turn_off_lights"], "target": [0.0, 0.0, 0.0, 0.0, 1.0], "label": 4, "type": "choice", "task": "intent"}}, "metadata": {"id": "easy-intent-10", "source": "intent", "variant": "clean", "group_id": "easy-intent-10", "upstream_group": null, "canonical_labels": ["set_alarm", "play_music", "weather", "send_message", "turn_off_lights"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-intent-11", "suite": "jevbench-easy", "split": "public", "source": "intent", "variant": "clean", "record": {"state": "Set an alarm for 7 am.", "questions": {"decision": {"type": "choice", "instructions": "Which intent does the user's message express?", "criteria": {"play_music": "Play a song, album, artist or playlist", "send_message": "Send a text message to someone", "set_alarm": "Set an alarm or wake-up time", "turn_off_lights": "Switch lights off", "weather": "Ask about the weather or forecast"}}}}, "expected": {"decision": {"labels": ["set_alarm", "play_music", "weather", "send_message", "turn_off_lights"], "target": [1.0, 0.0, 0.0, 0.0, 0.0], "label": 0, "type": "choice", "task": "intent"}}, "metadata": {"id": "easy-intent-11", "source": "intent", "variant": "clean", "group_id": "easy-intent-11", "upstream_group": null, "canonical_labels": ["set_alarm", "play_music", "weather", "send_message", "turn_off_lights"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-fact-00", "suite": "jevbench-easy", "split": "public", "source": "fact", "variant": "clean", "record": {"state": "Order #1182. Status: shipped on 3 September. Carrier: DHL.", "questions": {"decision": {"type": "noul", "instructions": "Has the order been shipped? Answer strictly from the facts stated in the text.", "criteria": {"false": "The text states that this is not so", "true": "The text states that this is so"}}}}, "expected": {"decision": {"labels": ["false", "true"], "target": [0.0, 1.0], "label": 1, "type": "noul", "task": "fact"}}, "metadata": {"id": "easy-fact-00", "source": "fact", "variant": "clean", "group_id": "easy-fact-00", "upstream_group": null, "canonical_labels": ["no", "yes"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-fact-01", "suite": "jevbench-easy", "split": "public", "source": "fact", "variant": "clean", "record": {"state": "Order #1183. Status: not yet shipped, waiting for stock.", "questions": {"decision": {"type": "noul", "instructions": "Has the order been shipped? Answer strictly from the facts stated in the text.", "criteria": {"false": "The text states that this is not so", "true": "The text states that this is so"}}}}, "expected": {"decision": {"labels": ["false", "true"], "target": [1.0, 0.0], "label": 0, "type": "noul", "task": "fact"}}, "metadata": {"id": "easy-fact-01", "source": "fact", "variant": "clean", "group_id": "easy-fact-01", "upstream_group": null, "canonical_labels": ["no", "yes"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-fact-02", "suite": "jevbench-easy", "split": "public", "source": "fact", "variant": "clean", "record": {"state": "Invoice 2026-044. Amount: 120 EUR. Payment status: paid in full.", "questions": {"decision": {"type": "noul", "instructions": "Is the invoice paid? Answer strictly from the facts stated in the text.", "criteria": {"false": "The text states that this is not so", "true": "The text states that this is so"}}}}, "expected": {"decision": {"labels": ["false", "true"], "target": [0.0, 1.0], "label": 1, "type": "noul", "task": "fact"}}, "metadata": {"id": "easy-fact-02", "source": "fact", "variant": "clean", "group_id": "easy-fact-02", "upstream_group": null, "canonical_labels": ["no", "yes"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-fact-03", "suite": "jevbench-easy", "split": "public", "source": "fact", "variant": "clean", "record": {"state": "Invoice 2026-045. Amount: 80 EUR. Payment status: unpaid, overdue since 1 August.", "questions": {"decision": {"type": "noul", "instructions": "Is the invoice paid? Answer strictly from the facts stated in the text.", "criteria": {"false": "The text states that this is not so", "true": "The text states that this is so"}}}}, "expected": {"decision": {"labels": ["false", "true"], "target": [1.0, 0.0], "label": 0, "type": "noul", "task": "fact"}}, "metadata": {"id": "easy-fact-03", "source": "fact", "variant": "clean", "group_id": "easy-fact-03", "upstream_group": null, "canonical_labels": ["no", "yes"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-fact-04", "suite": "jevbench-easy", "split": "public", "source": "fact", "variant": "clean", "record": {"state": "Guest profile: Maria Lopez. Allergies: peanuts. Diet: vegetarian.", "questions": {"decision": {"type": "noul", "instructions": "Is the guest allergic to peanuts? Answer strictly from the facts stated in the text.", "criteria": {"false": "The text states that this is not so", "true": "The text states that this is so"}}}}, "expected": {"decision": {"labels": ["false", "true"], "target": [0.0, 1.0], "label": 1, "type": "noul", "task": "fact"}}, "metadata": {"id": "easy-fact-04", "source": "fact", "variant": "clean", "group_id": "easy-fact-04", "upstream_group": null, "canonical_labels": ["no", "yes"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-fact-05", "suite": "jevbench-easy", "split": "public", "source": "fact", "variant": "clean", "record": {"state": "Guest profile: Jonas Weber. Allergies: none. Diet: no restrictions.", "questions": {"decision": {"type": "noul", "instructions": "Is the guest allergic to peanuts? Answer strictly from the facts stated in the text.", "criteria": {"false": "The text states that this is not so", "true": "The text states that this is so"}}}}, "expected": {"decision": {"labels": ["false", "true"], "target": [1.0, 0.0], "label": 0, "type": "noul", "task": "fact"}}, "metadata": {"id": "easy-fact-05", "source": "fact", "variant": "clean", "group_id": "easy-fact-05", "upstream_group": null, "canonical_labels": ["no", "yes"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-fact-06", "suite": "jevbench-easy", "split": "public", "source": "fact", "variant": "clean", "record": {"state": "Meeting room B: booked from 14:00 to 15:00 by the sales team.", "questions": {"decision": {"type": "noul", "instructions": "Is meeting room B booked from 14:00 to 15:00? Answer strictly from the facts stated in the text.", "criteria": {"false": "The text states that this is not so", "true": "The text states that this is so"}}}}, "expected": {"decision": {"labels": ["false", "true"], "target": [0.0, 1.0], "label": 1, "type": "noul", "task": "fact"}}, "metadata": {"id": "easy-fact-06", "source": "fact", "variant": "clean", "group_id": "easy-fact-06", "upstream_group": null, "canonical_labels": ["no", "yes"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-fact-07", "suite": "jevbench-easy", "split": "public", "source": "fact", "variant": "clean", "record": {"state": "Meeting room C: free all afternoon, no bookings.", "questions": {"decision": {"type": "noul", "instructions": "Is meeting room C booked this afternoon? Answer strictly from the facts stated in the text.", "criteria": {"false": "The text states that this is not so", "true": "The text states that this is so"}}}}, "expected": {"decision": {"labels": ["false", "true"], "target": [1.0, 0.0], "label": 0, "type": "noul", "task": "fact"}}, "metadata": {"id": "easy-fact-07", "source": "fact", "variant": "clean", "group_id": "easy-fact-07", "upstream_group": null, "canonical_labels": ["no", "yes"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-fact-08", "suite": "jevbench-easy", "split": "public", "source": "fact", "variant": "clean", "record": {"state": "Account settings: two-factor authentication is enabled.", "questions": {"decision": {"type": "noul", "instructions": "Is two-factor authentication enabled? Answer strictly from the facts stated in the text.", "criteria": {"false": "The text states that this is not so", "true": "The text states that this is so"}}}}, "expected": {"decision": {"labels": ["false", "true"], "target": [0.0, 1.0], "label": 1, "type": "noul", "task": "fact"}}, "metadata": {"id": "easy-fact-08", "source": "fact", "variant": "clean", "group_id": "easy-fact-08", "upstream_group": null, "canonical_labels": ["no", "yes"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-fact-09", "suite": "jevbench-easy", "split": "public", "source": "fact", "variant": "clean", "record": {"state": "Account settings: two-factor authentication is disabled.", "questions": {"decision": {"type": "noul", "instructions": "Is two-factor authentication enabled? Answer strictly from the facts stated in the text.", "criteria": {"false": "The text states that this is not so", "true": "The text states that this is so"}}}}, "expected": {"decision": {"labels": ["false", "true"], "target": [1.0, 0.0], "label": 0, "type": "noul", "task": "fact"}}, "metadata": {"id": "easy-fact-09", "source": "fact", "variant": "clean", "group_id": "easy-fact-09", "upstream_group": null, "canonical_labels": ["no", "yes"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-fact-10", "suite": "jevbench-easy", "split": "public", "source": "fact", "variant": "clean", "record": {"state": "The user wrote: 'Yes, please subscribe me to the newsletter.'", "questions": {"decision": {"type": "noul", "instructions": "Did the user agree to subscribe to the newsletter? Answer strictly from the facts stated in the text.", "criteria": {"false": "The text states that this is not so", "true": "The text states that this is so"}}}}, "expected": {"decision": {"labels": ["false", "true"], "target": [0.0, 1.0], "label": 1, "type": "noul", "task": "fact"}}, "metadata": {"id": "easy-fact-10", "source": "fact", "variant": "clean", "group_id": "easy-fact-10", "upstream_group": null, "canonical_labels": ["no", "yes"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-fact-11", "suite": "jevbench-easy", "split": "public", "source": "fact", "variant": "clean", "record": {"state": "The user wrote: 'No thanks, I do not want the newsletter.'", "questions": {"decision": {"type": "noul", "instructions": "Did the user agree to subscribe to the newsletter? Answer strictly from the facts stated in the text.", "criteria": {"false": "The text states that this is not so", "true": "The text states that this is so"}}}}, "expected": {"decision": {"labels": ["false", "true"], "target": [1.0, 0.0], "label": 0, "type": "noul", "task": "fact"}}, "metadata": {"id": "easy-fact-11", "source": "fact", "variant": "clean", "group_id": "easy-fact-11", "upstream_group": null, "canonical_labels": ["no", "yes"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-extraction-00", "suite": "jevbench-easy", "split": "public", "source": "extraction", "variant": "clean", "record": {"state": "I paid with PayPal yesterday evening.", "questions": {"decision": {"type": "choice", "instructions": "Which payment method does the customer name?", "criteria": {"bank_transfer": "Bank transfer / wire", "cash": "Cash", "credit_card": "Paid or wants to pay by credit card", "paypal": "PayPal"}}}}, "expected": {"decision": {"labels": ["credit_card", "paypal", "bank_transfer", "cash"], "target": [0.0, 1.0, 0.0, 0.0], "label": 1, "type": "choice", "task": "extraction"}}, "metadata": {"id": "easy-extraction-00", "source": "extraction", "variant": "clean", "group_id": "easy-extraction-00", "upstream_group": null, "canonical_labels": ["credit_card", "paypal", "bank_transfer", "cash"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-extraction-01", "suite": "jevbench-easy", "split": "public", "source": "extraction", "variant": "clean", "record": {"state": "I'll pay cash when the courier arrives.", "questions": {"decision": {"type": "choice", "instructions": "Which payment method does the customer name?", "criteria": {"bank_transfer": "Bank transfer / wire", "cash": "Cash", "credit_card": "Paid or wants to pay by credit card", "paypal": "PayPal"}}}}, "expected": {"decision": {"labels": ["credit_card", "paypal", "bank_transfer", "cash"], "target": [0.0, 0.0, 0.0, 1.0], "label": 3, "type": "choice", "task": "extraction"}}, "metadata": {"id": "easy-extraction-01", "source": "extraction", "variant": "clean", "group_id": "easy-extraction-01", "upstream_group": null, "canonical_labels": ["credit_card", "paypal", "bank_transfer", "cash"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-extraction-02", "suite": "jevbench-easy", "split": "public", "source": "extraction", "variant": "clean", "record": {"state": "I sent the money by bank transfer on Monday.", "questions": {"decision": {"type": "choice", "instructions": "Which payment method does the customer name?", "criteria": {"bank_transfer": "Bank transfer / wire", "cash": "Cash", "credit_card": "Paid or wants to pay by credit card", "paypal": "PayPal"}}}}, "expected": {"decision": {"labels": ["credit_card", "paypal", "bank_transfer", "cash"], "target": [0.0, 0.0, 1.0, 0.0], "label": 2, "type": "choice", "task": "extraction"}}, "metadata": {"id": "easy-extraction-02", "source": "extraction", "variant": "clean", "group_id": "easy-extraction-02", "upstream_group": null, "canonical_labels": ["credit_card", "paypal", "bank_transfer", "cash"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-extraction-03", "suite": "jevbench-easy", "split": "public", "source": "extraction", "variant": "clean", "record": {"state": "Ticket #88 - Priority: HIGH - Printer on floor 2 is jammed.", "questions": {"decision": {"type": "choice", "instructions": "Which priority does the ticket state?", "criteria": {"high": "High priority", "low": "Low priority", "medium": "Medium priority", "urgent": "Urgent / critical"}}}}, "expected": {"decision": {"labels": ["low", "medium", "high", "urgent"], "target": [0.0, 0.0, 1.0, 0.0], "label": 2, "type": "choice", "task": "extraction"}}, "metadata": {"id": "easy-extraction-03", "source": "extraction", "variant": "clean", "group_id": "easy-extraction-03", "upstream_group": null, "canonical_labels": ["low", "medium", "high", "urgent"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-extraction-04", "suite": "jevbench-easy", "split": "public", "source": "extraction", "variant": "clean", "record": {"state": "Ticket #89 - Priority: low - Please update my desk phone label.", "questions": {"decision": {"type": "choice", "instructions": "Which priority does the ticket state?", "criteria": {"high": "High priority", "low": "Low priority", "medium": "Medium priority", "urgent": "Urgent / critical"}}}}, "expected": {"decision": {"labels": ["low", "medium", "high", "urgent"], "target": [1.0, 0.0, 0.0, 0.0], "label": 0, "type": "choice", "task": "extraction"}}, "metadata": {"id": "easy-extraction-04", "source": "extraction", "variant": "clean", "group_id": "easy-extraction-04", "upstream_group": null, "canonical_labels": ["low", "medium", "high", "urgent"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-extraction-05", "suite": "jevbench-easy", "split": "public", "source": "extraction", "variant": "clean", "record": {"state": "Ticket #90 - Priority: URGENT - Production database is down.", "questions": {"decision": {"type": "choice", "instructions": "Which priority does the ticket state?", "criteria": {"high": "High priority", "low": "Low priority", "medium": "Medium priority", "urgent": "Urgent / critical"}}}}, "expected": {"decision": {"labels": ["low", "medium", "high", "urgent"], "target": [0.0, 0.0, 0.0, 1.0], "label": 3, "type": "choice", "task": "extraction"}}, "metadata": {"id": "easy-extraction-05", "source": "extraction", "variant": "clean", "group_id": "easy-extraction-05", "upstream_group": null, "canonical_labels": ["low", "medium", "high", "urgent"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-extraction-06", "suite": "jevbench-easy", "split": "public", "source": "extraction", "variant": "clean", "record": {"state": "Could I get the blue shirt in size M, please?", "questions": {"decision": {"type": "choice", "instructions": "Which shirt size does the customer ask for?", "criteria": {"L": "Large", "M": "Medium", "S": "Small", "XL": "Extra large"}}}}, "expected": {"decision": {"labels": ["S", "M", "L", "XL"], "target": [0.0, 1.0, 0.0, 0.0], "label": 1, "type": "choice", "task": "extraction"}}, "metadata": {"id": "easy-extraction-06", "source": "extraction", "variant": "clean", "group_id": "easy-extraction-06", "upstream_group": null, "canonical_labels": ["S", "M", "L", "XL"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-extraction-07", "suite": "jevbench-easy", "split": "public", "source": "extraction", "variant": "clean", "record": {"state": "I need the jacket in extra large (XL).", "questions": {"decision": {"type": "choice", "instructions": "Which shirt size does the customer ask for?", "criteria": {"L": "Large", "M": "Medium", "S": "Small", "XL": "Extra large"}}}}, "expected": {"decision": {"labels": ["S", "M", "L", "XL"], "target": [0.0, 0.0, 0.0, 1.0], "label": 3, "type": "choice", "task": "extraction"}}, "metadata": {"id": "easy-extraction-07", "source": "extraction", "variant": "clean", "group_id": "easy-extraction-07", "upstream_group": null, "canonical_labels": ["S", "M", "L", "XL"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-extraction-08", "suite": "jevbench-easy", "split": "public", "source": "extraction", "variant": "clean", "record": {"state": "Let's meet on Thursday at 10.", "questions": {"decision": {"type": "choice", "instructions": "Which day does the person propose for the meeting?", "criteria": {"friday": "Friday", "monday": "Monday", "thursday": "Thursday", "tuesday": "Tuesday", "wednesday": "Wednesday"}}}}, "expected": {"decision": {"labels": ["monday", "tuesday", "wednesday", "thursday", "friday"], "target": [0.0, 0.0, 0.0, 1.0, 0.0], "label": 3, "type": "choice", "task": "extraction"}}, "metadata": {"id": "easy-extraction-08", "source": "extraction", "variant": "clean", "group_id": "easy-extraction-08", "upstream_group": null, "canonical_labels": ["monday", "tuesday", "wednesday", "thursday", "friday"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-extraction-09", "suite": "jevbench-easy", "split": "public", "source": "extraction", "variant": "clean", "record": {"state": "Does Monday work for you? I'm free all day.", "questions": {"decision": {"type": "choice", "instructions": "Which day does the person propose for the meeting?", "criteria": {"friday": "Friday", "monday": "Monday", "thursday": "Thursday", "tuesday": "Tuesday", "wednesday": "Wednesday"}}}}, "expected": {"decision": {"labels": ["monday", "tuesday", "wednesday", "thursday", "friday"], "target": [1.0, 0.0, 0.0, 0.0, 0.0], "label": 0, "type": "choice", "task": "extraction"}}, "metadata": {"id": "easy-extraction-09", "source": "extraction", "variant": "clean", "group_id": "easy-extraction-09", "upstream_group": null, "canonical_labels": ["monday", "tuesday", "wednesday", "thursday", "friday"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-extraction-10", "suite": "jevbench-easy", "split": "public", "source": "extraction", "variant": "clean", "record": {"state": "How about Friday afternoon for the review?", "questions": {"decision": {"type": "choice", "instructions": "Which day does the person propose for the meeting?", "criteria": {"friday": "Friday", "monday": "Monday", "thursday": "Thursday", "tuesday": "Tuesday", "wednesday": "Wednesday"}}}}, "expected": {"decision": {"labels": ["monday", "tuesday", "wednesday", "thursday", "friday"], "target": [0.0, 0.0, 0.0, 0.0, 1.0], "label": 4, "type": "choice", "task": "extraction"}}, "metadata": {"id": "easy-extraction-10", "source": "extraction", "variant": "clean", "group_id": "easy-extraction-10", "upstream_group": null, "canonical_labels": ["monday", "tuesday", "wednesday", "thursday", "friday"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-extraction-11", "suite": "jevbench-easy", "split": "public", "source": "extraction", "variant": "clean", "record": {"state": "Please charge my credit card ending in 4242.", "questions": {"decision": {"type": "choice", "instructions": "Which payment method does the customer name?", "criteria": {"bank_transfer": "Bank transfer / wire", "cash": "Cash", "credit_card": "Paid or wants to pay by credit card", "paypal": "PayPal"}}}}, "expected": {"decision": {"labels": ["credit_card", "paypal", "bank_transfer", "cash"], "target": [1.0, 0.0, 0.0, 0.0], "label": 0, "type": "choice", "task": "extraction"}}, "metadata": {"id": "easy-extraction-11", "source": "extraction", "variant": "clean", "group_id": "easy-extraction-11", "upstream_group": null, "canonical_labels": ["credit_card", "paypal", "bank_transfer", "cash"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-tool_selection-00", "suite": "jevbench-easy", "split": "public", "source": "tool_selection", "variant": "clean", "record": {"state": "What's the weather like in Paris right now?", "questions": {"decision": {"type": "choice", "instructions": "Which single tool should be called to handle this request?", "criteria": {"convert_currency": "Convert an amount from one currency to another", "create_calendar_event": "Put an appointment or meeting into the calendar", "get_weather": "Current weather or forecast for a place", "send_email": "Send an email to a recipient", "translate_text": "Translate text into another language"}}}}, "expected": {"decision": {"labels": ["get_weather", "send_email", "create_calendar_event", "convert_currency", "translate_text"], "target": [1.0, 0.0, 0.0, 0.0, 0.0], "label": 0, "type": "choice", "task": "tool_selection"}}, "metadata": {"id": "easy-tool_selection-00", "source": "tool_selection", "variant": "clean", "group_id": "easy-tool_selection-00", "upstream_group": null, "canonical_labels": ["get_weather", "send_email", "create_calendar_event", "convert_currency", "translate_text"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-tool_selection-01", "suite": "jevbench-easy", "split": "public", "source": "tool_selection", "variant": "clean", "record": {"state": "Email Sarah the quarterly report and say it's attached.", "questions": {"decision": {"type": "choice", "instructions": "Which single tool should be called to handle this request?", "criteria": {"convert_currency": "Convert an amount from one currency to another", "create_calendar_event": "Put an appointment or meeting into the calendar", "get_weather": "Current weather or forecast for a place", "send_email": "Send an email to a recipient", "translate_text": "Translate text into another language"}}}}, "expected": {"decision": {"labels": ["get_weather", "send_email", "create_calendar_event", "convert_currency", "translate_text"], "target": [0.0, 1.0, 0.0, 0.0, 0.0], "label": 1, "type": "choice", "task": "tool_selection"}}, "metadata": {"id": "easy-tool_selection-01", "source": "tool_selection", "variant": "clean", "group_id": "easy-tool_selection-01", "upstream_group": null, "canonical_labels": ["get_weather", "send_email", "create_calendar_event", "convert_currency", "translate_text"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-tool_selection-02", "suite": "jevbench-easy", "split": "public", "source": "tool_selection", "variant": "clean", "record": {"state": "Add a dentist appointment on Friday at 3 pm to my calendar.", "questions": {"decision": {"type": "choice", "instructions": "Which single tool should be called to handle this request?", "criteria": {"convert_currency": "Convert an amount from one currency to another", "create_calendar_event": "Put an appointment or meeting into the calendar", "get_weather": "Current weather or forecast for a place", "send_email": "Send an email to a recipient", "translate_text": "Translate text into another language"}}}}, "expected": {"decision": {"labels": ["get_weather", "send_email", "create_calendar_event", "convert_currency", "translate_text"], "target": [0.0, 0.0, 1.0, 0.0, 0.0], "label": 2, "type": "choice", "task": "tool_selection"}}, "metadata": {"id": "easy-tool_selection-02", "source": "tool_selection", "variant": "clean", "group_id": "easy-tool_selection-02", "upstream_group": null, "canonical_labels": ["get_weather", "send_email", "create_calendar_event", "convert_currency", "translate_text"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-tool_selection-03", "suite": "jevbench-easy", "split": "public", "source": "tool_selection", "variant": "clean", "record": {"state": "How much is 250 US dollars in euros?", "questions": {"decision": {"type": "choice", "instructions": "Which single tool should be called to handle this request?", "criteria": {"convert_currency": "Convert an amount from one currency to another", "create_calendar_event": "Put an appointment or meeting into the calendar", "get_weather": "Current weather or forecast for a place", "send_email": "Send an email to a recipient", "translate_text": "Translate text into another language"}}}}, "expected": {"decision": {"labels": ["get_weather", "send_email", "create_calendar_event", "convert_currency", "translate_text"], "target": [0.0, 0.0, 0.0, 1.0, 0.0], "label": 3, "type": "choice", "task": "tool_selection"}}, "metadata": {"id": "easy-tool_selection-03", "source": "tool_selection", "variant": "clean", "group_id": "easy-tool_selection-03", "upstream_group": null, "canonical_labels": ["get_weather", "send_email", "create_calendar_event", "convert_currency", "translate_text"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-tool_selection-04", "suite": "jevbench-easy", "split": "public", "source": "tool_selection", "variant": "clean", "record": {"state": "Translate 'good morning' into Japanese.", "questions": {"decision": {"type": "choice", "instructions": "Which single tool should be called to handle this request?", "criteria": {"convert_currency": "Convert an amount from one currency to another", "create_calendar_event": "Put an appointment or meeting into the calendar", "get_weather": "Current weather or forecast for a place", "send_email": "Send an email to a recipient", "translate_text": "Translate text into another language"}}}}, "expected": {"decision": {"labels": ["get_weather", "send_email", "create_calendar_event", "convert_currency", "translate_text"], "target": [0.0, 0.0, 0.0, 0.0, 1.0], "label": 4, "type": "choice", "task": "tool_selection"}}, "metadata": {"id": "easy-tool_selection-04", "source": "tool_selection", "variant": "clean", "group_id": "easy-tool_selection-04", "upstream_group": null, "canonical_labels": ["get_weather", "send_email", "create_calendar_event", "convert_currency", "translate_text"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-tool_selection-05", "suite": "jevbench-easy", "split": "public", "source": "tool_selection", "variant": "clean", "record": {"state": "Schedule a team meeting next Tuesday at 10 in my calendar.", "questions": {"decision": {"type": "choice", "instructions": "Which single tool should be called to handle this request?", "criteria": {"convert_currency": "Convert an amount from one currency to another", "create_calendar_event": "Put an appointment or meeting into the calendar", "get_weather": "Current weather or forecast for a place", "send_email": "Send an email to a recipient", "translate_text": "Translate text into another language"}}}}, "expected": {"decision": {"labels": ["get_weather", "send_email", "create_calendar_event", "convert_currency", "translate_text"], "target": [0.0, 0.0, 1.0, 0.0, 0.0], "label": 2, "type": "choice", "task": "tool_selection"}}, "metadata": {"id": "easy-tool_selection-05", "source": "tool_selection", "variant": "clean", "group_id": "easy-tool_selection-05", "upstream_group": null, "canonical_labels": ["get_weather", "send_email", "create_calendar_event", "convert_currency", "translate_text"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-tool_selection-06", "suite": "jevbench-easy", "split": "public", "source": "tool_selection", "variant": "clean", "record": {"state": "Find me flights from Munich to Lisbon on 12 October.", "questions": {"decision": {"type": "choice", "instructions": "Which single tool should be called to handle this request?", "criteria": {"book_restaurant": "Reserve a table at a restaurant", "call_taxi": "Order a taxi to a location", "get_stock_price": "Look up the current price of a stock", "search_flights": "Find flights between two cities", "set_reminder": "Remind the user of something at a given time"}}}}, "expected": {"decision": {"labels": ["search_flights", "book_restaurant", "set_reminder", "get_stock_price", "call_taxi"], "target": [1.0, 0.0, 0.0, 0.0, 0.0], "label": 0, "type": "choice", "task": "tool_selection"}}, "metadata": {"id": "easy-tool_selection-06", "source": "tool_selection", "variant": "clean", "group_id": "easy-tool_selection-06", "upstream_group": null, "canonical_labels": ["search_flights", "book_restaurant", "set_reminder", "get_stock_price", "call_taxi"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-tool_selection-07", "suite": "jevbench-easy", "split": "public", "source": "tool_selection", "variant": "clean", "record": {"state": "Book a table for four at Luigi's tonight at 8.", "questions": {"decision": {"type": "choice", "instructions": "Which single tool should be called to handle this request?", "criteria": {"book_restaurant": "Reserve a table at a restaurant", "call_taxi": "Order a taxi to a location", "get_stock_price": "Look up the current price of a stock", "search_flights": "Find flights between two cities", "set_reminder": "Remind the user of something at a given time"}}}}, "expected": {"decision": {"labels": ["search_flights", "book_restaurant", "set_reminder", "get_stock_price", "call_taxi"], "target": [0.0, 1.0, 0.0, 0.0, 0.0], "label": 1, "type": "choice", "task": "tool_selection"}}, "metadata": {"id": "easy-tool_selection-07", "source": "tool_selection", "variant": "clean", "group_id": "easy-tool_selection-07", "upstream_group": null, "canonical_labels": ["search_flights", "book_restaurant", "set_reminder", "get_stock_price", "call_taxi"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-tool_selection-08", "suite": "jevbench-easy", "split": "public", "source": "tool_selection", "variant": "clean", "record": {"state": "Remind me to call my mother at 6 pm.", "questions": {"decision": {"type": "choice", "instructions": "Which single tool should be called to handle this request?", "criteria": {"book_restaurant": "Reserve a table at a restaurant", "call_taxi": "Order a taxi to a location", "get_stock_price": "Look up the current price of a stock", "search_flights": "Find flights between two cities", "set_reminder": "Remind the user of something at a given time"}}}}, "expected": {"decision": {"labels": ["search_flights", "book_restaurant", "set_reminder", "get_stock_price", "call_taxi"], "target": [0.0, 0.0, 1.0, 0.0, 0.0], "label": 2, "type": "choice", "task": "tool_selection"}}, "metadata": {"id": "easy-tool_selection-08", "source": "tool_selection", "variant": "clean", "group_id": "easy-tool_selection-08", "upstream_group": null, "canonical_labels": ["search_flights", "book_restaurant", "set_reminder", "get_stock_price", "call_taxi"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-tool_selection-09", "suite": "jevbench-easy", "split": "public", "source": "tool_selection", "variant": "clean", "record": {"state": "What is Apple's stock price right now?", "questions": {"decision": {"type": "choice", "instructions": "Which single tool should be called to handle this request?", "criteria": {"book_restaurant": "Reserve a table at a restaurant", "call_taxi": "Order a taxi to a location", "get_stock_price": "Look up the current price of a stock", "search_flights": "Find flights between two cities", "set_reminder": "Remind the user of something at a given time"}}}}, "expected": {"decision": {"labels": ["search_flights", "book_restaurant", "set_reminder", "get_stock_price", "call_taxi"], "target": [0.0, 0.0, 0.0, 1.0, 0.0], "label": 3, "type": "choice", "task": "tool_selection"}}, "metadata": {"id": "easy-tool_selection-09", "source": "tool_selection", "variant": "clean", "group_id": "easy-tool_selection-09", "upstream_group": null, "canonical_labels": ["search_flights", "book_restaurant", "set_reminder", "get_stock_price", "call_taxi"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-tool_selection-10", "suite": "jevbench-easy", "split": "public", "source": "tool_selection", "variant": "clean", "record": {"state": "Get me a taxi to the main station.", "questions": {"decision": {"type": "choice", "instructions": "Which single tool should be called to handle this request?", "criteria": {"book_restaurant": "Reserve a table at a restaurant", "call_taxi": "Order a taxi to a location", "get_stock_price": "Look up the current price of a stock", "search_flights": "Find flights between two cities", "set_reminder": "Remind the user of something at a given time"}}}}, "expected": {"decision": {"labels": ["search_flights", "book_restaurant", "set_reminder", "get_stock_price", "call_taxi"], "target": [0.0, 0.0, 0.0, 0.0, 1.0], "label": 4, "type": "choice", "task": "tool_selection"}}, "metadata": {"id": "easy-tool_selection-10", "source": "tool_selection", "variant": "clean", "group_id": "easy-tool_selection-10", "upstream_group": null, "canonical_labels": ["search_flights", "book_restaurant", "set_reminder", "get_stock_price", "call_taxi"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}
+{"id": "easy-tool_selection-11", "suite": "jevbench-easy", "split": "public", "source": "tool_selection", "variant": "clean", "record": {"state": "Remind me tomorrow at 9 to water the plants.", "questions": {"decision": {"type": "choice", "instructions": "Which single tool should be called to handle this request?", "criteria": {"book_restaurant": "Reserve a table at a restaurant", "call_taxi": "Order a taxi to a location", "get_stock_price": "Look up the current price of a stock", "search_flights": "Find flights between two cities", "set_reminder": "Remind the user of something at a given time"}}}}, "expected": {"decision": {"labels": ["search_flights", "book_restaurant", "set_reminder", "get_stock_price", "call_taxi"], "target": [0.0, 0.0, 1.0, 0.0, 0.0], "label": 2, "type": "choice", "task": "tool_selection"}}, "metadata": {"id": "easy-tool_selection-11", "source": "tool_selection", "variant": "clean", "group_id": "easy-tool_selection-11", "upstream_group": null, "canonical_labels": ["search_flights", "book_restaurant", "set_reminder", "get_stock_price", "call_taxi"], "provenance": {"exclude_reason": null, "label_basis": "Answer named or stated explicitly in the text; authored and reviewed before inference", "license": "MIT", "source": "JevBench v1.1 easy tier, original authored item"}, "gold_policy": {"hard_label": "authored_reviewed_rubric", "reference_probs": null, "argmax_tie_break": "lexicographic_label"}}}

server/benchmarks/data/jevbench-easy/public.manifest.json

@@ -0,0 +1,95 @@
+{
+  "schema_version": 1,
+  "suite": "jevbench-easy",
+  "split": "public",
+  "description": "Public easy tier: 48 explicit facts, intents and tool selections.",
+  "data_file": "public.jsonl",
+  "data_sha256": "e186c6dd90b3a7a177ef15f2fd3673d7bbfce95340bfdc6062dc15bfa0e01a6a",
+  "upstream": {
+    "repository": "https://github.com/fstandhartinger/jevbench",
+    "commit": "fd51755eb0c0b546ca206d764faf3302feca913e",
+    "path": "datasets/public/easy.jsonl",
+    "url": "https://raw.githubusercontent.com/fstandhartinger/jevbench/fd51755eb0c0b546ca206d764faf3302feca913e/datasets/public/easy.jsonl",
+    "sha256": "231df3c2c8e88a1a8c137ebe85de96ba70fabd330849098ac7b3c52c70b7172b",
+    "notice_sha256": {
+      "LICENSE": "3e5beed774bb0bcbfb2fcf24ba9554212c6ae112937d308040989465ab0c5784",
+      "THIRD-PARTY.md": "396422e29055bba4073f4a9e5a7163cc724f13e970a34ef58e8ea1fb439b9b38"
+    }
+  },
+  "full_partition": {
+    "records": 48,
+    "questions": 48,
+    "clean_records": 48,
+    "clean_questions": 48,
+    "headline_questions": 48,
+    "variants": {
+      "clean": 48
+    },
+    "clean_sources": {
+      "intent": 12,
+      "fact": 12,
+      "extraction": 12,
+      "tool_selection": 12
+    },
+    "question_types": {
+      "choice": 36,
+      "noul": 12
+    },
+    "maximum_options": 5
+  },
+  "selected": {
+    "records": 48,
+    "questions": 48,
+    "clean_records": 48,
+    "clean_questions": 48,
+    "headline_questions": 48,
+    "variants": {
+      "clean": 48
+    },
+    "clean_sources": {
+      "intent": 12,
+      "fact": 12,
+      "extraction": 12,
+      "tool_selection": 12
+    },
+    "question_types": {
+      "choice": 36,
+      "noul": 12
+    },
+    "maximum_options": 5
+  },
+  "selection": {
+    "method": "full",
+    "requested_limit": null,
+    "is_full_partition": true,
+    "note": "Full frozen public tier; not the complete published leaderboard."
+  },
+  "protocol": {
+    "calibration_applied": false,
+    "training_data_downloaded": false,
+    "gold_labels_sent_to_model": false,
+    "headline_variant": "clean",
+    "exclude_from_headline_sources": [],
+    "locked_test": false,
+    "notes": [
+      "Community benchmark; not an official TypeSafe dataset release.",
+      "Public original/easy/hard have 72/48/111 questions; report separately.",
+      "The 534-item leaderboard contains private/untracked data not downloaded here.",
+      "Choice request criteria retain their native order; scoring labels retain canonical order.",
+      "Native accuracy uses argmax with lexicographic label tie-break, including Score.",
+      "Score also supports expected-value MAE; original has 36 paraphrase groups.",
+      "One-hot labels and ten explicit mathematical probability references are separate targets.",
+      "Provenance, rationale and gold probabilities are never sent in requests."
+    ]
+  },
+  "provenance": {
+    "dataset_license": "MIT",
+    "license_url": "https://github.com/fstandhartinger/jevbench/blob/fd51755eb0c0b546ca206d764faf3302feca913e/LICENSE",
+    "notices": [
+      "LICENSE",
+      "THIRD-PARTY.md"
+    ],
+    "reference_probability_questions": 0,
+    "gold_policy": "Authored rubric labels reviewed before inference; hard items retain author and review metadata. Explicit mathematical distributions are not teacher model confidence or population frequency estimates."
+  }
+}

server/benchmarks/data/jevbench-hard/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Florian Standhartinger and contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

server/benchmarks/data/jevbench-hard/THIRD-PARTY.md

@@ -0,0 +1,51 @@
+# Third-party projects, data and services
+
+MIT (see `LICENSE`) covers this harness and the 72 original public decisions in
+`datasets/public/original.jsonl`. Nothing else in this list is ours to license.
+
+## Systems evaluated
+
+Each was reached through the interface its author published. We used public
+endpoints as ordinary clients, at one request at a time, and never sent a
+provider's API key to anyone else's endpoint.
+
+| System | Author | Source |
+|---|---|---|
+| Jev 1.13.0 | TypeSafe AI | <https://docs.typesafe.ai> |
+| openjev-sglang | ekzhang | <https://github.com/ekzhang/openjev-sglang> |
+| system-one-open | mithalouni | <https://github.com/mithalouni/system-one-open> |
+| open-alternative-jev | IkerMoel | <https://github.com/ikermoel/open-alternative-jev>, <https://huggingface.co/spaces/IkerMoel/open-alternative-jev> |
+| open-jev-deberta-v3-large | Kotoba Labs | <https://huggingface.co/com-kotobalabs/open-jev-deberta-v3-large>, <https://github.com/kotoba-lang/typed-decisions> |
+| GPT-5.6 Luna | OpenAI | <https://platform.openai.com> |
+| Gemini 3.1 Flash-Lite | Google | <https://ai.google.dev> |
+| DeepSeek V4.1 Flash | DeepSeek | <https://api-docs.deepseek.com> |
+| Qwen3.8 27B | Qwen, served by Chutes | <https://chutes.ai> |
+
+Model weights, base models and each project's own code keep their own licences.
+A permissive licence on a repository is not a licence for the base model it
+fine-tunes, and we do not restate either.
+
+## Wire format
+
+The typed-decision request shape (`state`, `questions`, the `noul` / `choice` /
+`score` primitives) is TypeSafe's public HTTP API, documented at
+<https://docs.typesafe.ai/api>. Several of the open rebuilds implement it
+deliberately, which is why one adapter reaches more than one of them. JevBench is
+not affiliated with or endorsed by TypeSafe AI.
+
+## Imported decisions
+
+146 of the 242 decisions come from our own earlier auto-router experiment
+(<https://github.com/fstandhartinger/auto-model-router>): 78 routing requests and
+68 answer-adequacy judgements whose ground truth is a deterministic grader's
+verdict on a saved answer. The upstream task text is **not** redistributed here,
+because the datasets it was drawn from keep their own terms. `datasets/manifest.json`
+pins the hashes; `scripts/import_router.py` shows exactly what was taken and what
+was excluded.
+
+## Held-out decisions
+
+24 scenarios are written by us and deliberately unpublished, so the suite cannot
+be trained on in full. Only their whole-split hash and aggregate results appear
+here. They are sent to the services being evaluated, which is not the same thing
+as being public - see the limits section of the README.

server/benchmarks/data/jevbench-hard/public.manifest.json

@@ -0,0 +1,109 @@
+{
+  "schema_version": 1,
+  "suite": "jevbench-hard",
+  "split": "public",
+  "description": "Public hard tier: 111 decisions; ten supply probability references.",
+  "data_file": "public.jsonl",
+  "data_sha256": "6c93c82edf10d08b70ee0c3addcf7c1ea8fd19957e94fb8021969ff4c0027f95",
+  "upstream": {
+    "repository": "https://github.com/fstandhartinger/jevbench",
+    "commit": "fd51755eb0c0b546ca206d764faf3302feca913e",
+    "path": "datasets/public/hard.jsonl",
+    "url": "https://raw.githubusercontent.com/fstandhartinger/jevbench/fd51755eb0c0b546ca206d764faf3302feca913e/datasets/public/hard.jsonl",
+    "sha256": "89e9e6becb33ed88c1de7d42dcc87531b2fb64cfaef4e1986faf7c37b3f80ebb",
+    "notice_sha256": {
+      "LICENSE": "3e5beed774bb0bcbfb2fcf24ba9554212c6ae112937d308040989465ab0c5784",
+      "THIRD-PARTY.md": "396422e29055bba4073f4a9e5a7163cc724f13e970a34ef58e8ea1fb439b9b38"
+    }
+  },
+  "full_partition": {
+    "records": 111,
+    "questions": 111,
+    "clean_records": 111,
+    "clean_questions": 111,
+    "headline_questions": 111,
+    "variants": {
+      "clean": 111
+    },
+    "clean_sources": {
+      "long_policy": 19,
+      "probability": 10,
+      "temporal_numeric": 15,
+      "ambiguous": 7,
+      "multi_hop": 18,
+      "tradeoff": 6,
+      "adversarial": 6,
+      "trap": 8,
+      "judge_hard": 17,
+      "routing_hard": 5
+    },
+    "question_types": {
+      "choice": 67,
+      "noul": 38,
+      "score": 6
+    },
+    "maximum_options": 6
+  },
+  "selected": {
+    "records": 111,
+    "questions": 111,
+    "clean_records": 111,
+    "clean_questions": 111,
+    "headline_questions": 111,
+    "variants": {
+      "clean": 111
+    },
+    "clean_sources": {
+      "long_policy": 19,
+      "probability": 10,
+      "temporal_numeric": 15,
+      "ambiguous": 7,
+      "multi_hop": 18,
+      "tradeoff": 6,
+      "adversarial": 6,
+      "trap": 8,
+      "judge_hard": 17,
+      "routing_hard": 5
+    },
+    "question_types": {
+      "choice": 67,
+      "noul": 38,
+      "score": 6
+    },
+    "maximum_options": 6
+  },
+  "selection": {
+    "method": "full",
+    "requested_limit": null,
+    "is_full_partition": true,
+    "note": "Full frozen public tier; not the complete published leaderboard."
+  },
+  "protocol": {
+    "calibration_applied": false,
+    "training_data_downloaded": false,
+    "gold_labels_sent_to_model": false,
+    "headline_variant": "clean",
+    "exclude_from_headline_sources": [],
+    "locked_test": false,
+    "notes": [
+      "Community benchmark; not an official TypeSafe dataset release.",
+      "Public original/easy/hard have 72/48/111 questions; report separately.",
+      "The 534-item leaderboard contains private/untracked data not downloaded here.",
+      "Choice request criteria retain their native order; scoring labels retain canonical order.",
+      "Native accuracy uses argmax with lexicographic label tie-break, including Score.",
+      "Score also supports expected-value MAE; original has 36 paraphrase groups.",
+      "One-hot labels and ten explicit mathematical probability references are separate targets.",
+      "Provenance, rationale and gold probabilities are never sent in requests."
+    ]
+  },
+  "provenance": {
+    "dataset_license": "MIT",
+    "license_url": "https://github.com/fstandhartinger/jevbench/blob/fd51755eb0c0b546ca206d764faf3302feca913e/LICENSE",
+    "notices": [
+      "LICENSE",
+      "THIRD-PARTY.md"
+    ],
+    "reference_probability_questions": 10,
+    "gold_policy": "Authored rubric labels reviewed before inference; hard items retain author and review metadata. Explicit mathematical distributions are not teacher model confidence or population frequency estimates."
+  }
+}

server/benchmarks/data/jevbench-original/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Florian Standhartinger and contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

server/benchmarks/data/jevbench-original/THIRD-PARTY.md

@@ -0,0 +1,51 @@
+# Third-party projects, data and services
+
+MIT (see `LICENSE`) covers this harness and the 72 original public decisions in
+`datasets/public/original.jsonl`. Nothing else in this list is ours to license.
+
+## Systems evaluated
+
+Each was reached through the interface its author published. We used public
+endpoints as ordinary clients, at one request at a time, and never sent a
+provider's API key to anyone else's endpoint.
+
+| System | Author | Source |
+|---|---|---|
+| Jev 1.13.0 | TypeSafe AI | <https://docs.typesafe.ai> |
+| openjev-sglang | ekzhang | <https://github.com/ekzhang/openjev-sglang> |
+| system-one-open | mithalouni | <https://github.com/mithalouni/system-one-open> |
+| open-alternative-jev | IkerMoel | <https://github.com/ikermoel/open-alternative-jev>, <https://huggingface.co/spaces/IkerMoel/open-alternative-jev> |
+| open-jev-deberta-v3-large | Kotoba Labs | <https://huggingface.co/com-kotobalabs/open-jev-deberta-v3-large>, <https://github.com/kotoba-lang/typed-decisions> |
+| GPT-5.6 Luna | OpenAI | <https://platform.openai.com> |
+| Gemini 3.1 Flash-Lite | Google | <https://ai.google.dev> |
+| DeepSeek V4.1 Flash | DeepSeek | <https://api-docs.deepseek.com> |
+| Qwen3.8 27B | Qwen, served by Chutes | <https://chutes.ai> |
+
+Model weights, base models and each project's own code keep their own licences.
+A permissive licence on a repository is not a licence for the base model it
+fine-tunes, and we do not restate either.
+
+## Wire format
+
+The typed-decision request shape (`state`, `questions`, the `noul` / `choice` /
+`score` primitives) is TypeSafe's public HTTP API, documented at
+<https://docs.typesafe.ai/api>. Several of the open rebuilds implement it
+deliberately, which is why one adapter reaches more than one of them. JevBench is
+not affiliated with or endorsed by TypeSafe AI.
+
+## Imported decisions
+
+146 of the 242 decisions come from our own earlier auto-router experiment
+(<https://github.com/fstandhartinger/auto-model-router>): 78 routing requests and
+68 answer-adequacy judgements whose ground truth is a deterministic grader's
+verdict on a saved answer. The upstream task text is **not** redistributed here,
+because the datasets it was drawn from keep their own terms. `datasets/manifest.json`
+pins the hashes; `scripts/import_router.py` shows exactly what was taken and what
+was excluded.
+
+## Held-out decisions
+
+24 scenarios are written by us and deliberately unpublished, so the suite cannot
+be trained on in full. Only their whole-split hash and aggregate results appear
+here. They are sent to the services being evaluated, which is not the same thing
+as being public - see the limits section of the README.

server/benchmarks/data/jevbench-original/public.manifest.json

@@ -0,0 +1,101 @@
+{
+  "schema_version": 1,
+  "suite": "jevbench-original",
+  "split": "public",
+  "description": "Public original tier: 72 decisions in 36 paraphrase pairs.",
+  "data_file": "public.jsonl",
+  "data_sha256": "ad453e78624b1c150d5eb3f0f47c5c14caa79542329b4f7e1bcc6199e2eb7007",
+  "upstream": {
+    "repository": "https://github.com/fstandhartinger/jevbench",
+    "commit": "fd51755eb0c0b546ca206d764faf3302feca913e",
+    "path": "datasets/public/original.jsonl",
+    "url": "https://raw.githubusercontent.com/fstandhartinger/jevbench/fd51755eb0c0b546ca206d764faf3302feca913e/datasets/public/original.jsonl",
+    "sha256": "5c2414edb3006b8bfcb70fda433f0f9ca015759433849f8d3104328a1f7c4180",
+    "notice_sha256": {
+      "LICENSE": "3e5beed774bb0bcbfb2fcf24ba9554212c6ae112937d308040989465ab0c5784",
+      "THIRD-PARTY.md": "396422e29055bba4073f4a9e5a7163cc724f13e970a34ef58e8ea1fb439b9b38"
+    }
+  },
+  "full_partition": {
+    "records": 72,
+    "questions": 72,
+    "clean_records": 72,
+    "clean_questions": 72,
+    "headline_questions": 72,
+    "variants": {
+      "clean": 72
+    },
+    "clean_sources": {
+      "policy": 12,
+      "intent": 12,
+      "ordinal": 12,
+      "extraction": 12,
+      "adequacy": 12,
+      "routing": 12
+    },
+    "question_types": {
+      "noul": 24,
+      "choice": 36,
+      "score": 12
+    },
+    "maximum_options": 6
+  },
+  "selected": {
+    "records": 72,
+    "questions": 72,
+    "clean_records": 72,
+    "clean_questions": 72,
+    "headline_questions": 72,
+    "variants": {
+      "clean": 72
+    },
+    "clean_sources": {
+      "policy": 12,
+      "intent": 12,
+      "ordinal": 12,
+      "extraction": 12,
+      "adequacy": 12,
+      "routing": 12
+    },
+    "question_types": {
+      "noul": 24,
+      "choice": 36,
+      "score": 12
+    },
+    "maximum_options": 6
+  },
+  "selection": {
+    "method": "full",
+    "requested_limit": null,
+    "is_full_partition": true,
+    "note": "Full frozen public tier; not the complete published leaderboard."
+  },
+  "protocol": {
+    "calibration_applied": false,
+    "training_data_downloaded": false,
+    "gold_labels_sent_to_model": false,
+    "headline_variant": "clean",
+    "exclude_from_headline_sources": [],
+    "locked_test": false,
+    "notes": [
+      "Community benchmark; not an official TypeSafe dataset release.",
+      "Public original/easy/hard have 72/48/111 questions; report separately.",
+      "The 534-item leaderboard contains private/untracked data not downloaded here.",
+      "Choice request criteria retain their native order; scoring labels retain canonical order.",
+      "Native accuracy uses argmax with lexicographic label tie-break, including Score.",
+      "Score also supports expected-value MAE; original has 36 paraphrase groups.",
+      "One-hot labels and ten explicit mathematical probability references are separate targets.",
+      "Provenance, rationale and gold probabilities are never sent in requests."
+    ]
+  },
+  "provenance": {
+    "dataset_license": "MIT",
+    "license_url": "https://github.com/fstandhartinger/jevbench/blob/fd51755eb0c0b546ca206d764faf3302feca913e/LICENSE",
+    "notices": [
+      "LICENSE",
+      "THIRD-PARTY.md"
+    ],
+    "reference_probability_questions": 0,
+    "gold_policy": "Authored rubric labels reviewed before inference; hard items retain author and review metadata. Explicit mathematical distributions are not teacher model confidence or population frequency estimates."
+  }
+}

server/benchmarks/run_matrix.py

@@ -0,0 +1,277 @@
+#!/usr/bin/env python3
+"""Run one pinned model at a time on one H200. Default is a command preview."""
+
+import argparse
+import json
+import os
+import shlex
+import signal
+import socket
+import subprocess
+import sys
+import time
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+RUNTIME = ROOT / "benchmarks/h200/runtime.py"
+
+
+def commands(args, manifest, profile):
+    root = args.output.resolve() / profile
+    engine = [
+        sys.executable,
+        str(RUNTIME),
+        "launch",
+        profile,
+        "--engine-python",
+        # Resolving a venv Python symlink loses pyvenv.cfg and its installed engine.
+        str(args.engine_python.absolute()),
+        "--gpu",
+        args.gpu,
+        "--output",
+        str(root / "engine"),
+        "--timeout",
+        str(args.startup_timeout),
+        "--execute",
+    ]
+    if args.probe_vision:
+        engine.append("--probe-vision")
+    adapter = [
+        sys.executable,
+        "-m",
+        "jev_adapter",
+        "--engine-url",
+        "http://127.0.0.1:30000",
+        "--model",
+        "decision-model",
+        "--port",
+        "30120",
+        "--max-concurrency",
+        "32",
+    ]
+    model = manifest["models"][profile]
+    if model.get("tokenization") == "native_official_text":
+        adapter += [
+            "--tokenizer-model",
+            model["repo_id"],
+            "--tokenizer-revision",
+            model["revision"],
+        ]
+    suites = [
+        "decision-v7",
+        "transfer-v4",
+        "transfer-v9",
+        "jevbench-original",
+        "jevbench-easy",
+        "jevbench-hard",
+    ]
+    if args.suite:
+        suites = args.suite
+    if args.include_external:
+        suites += [
+            suite for suite in ["semif-v1", "scienthoon-v1"] if suite not in suites
+        ]
+    evaluations = []
+    for concurrency in args.concurrency:
+        run = [
+            sys.executable,
+            "-m",
+            "jev_adapter.benchmarks.run",
+            "--output",
+            str(root / f"c{concurrency}"),
+            "--engine-manifest",
+            str(root / "engine/launch.json"),
+            "--concurrency",
+            str(concurrency),
+            "--warmup",
+            str(args.warmup),
+            "--repeats",
+            str(args.repeats),
+        ]
+        for suite in suites:
+            split = "public" if suite.startswith("jevbench-") else "development"
+            run += [
+                "--data",
+                str(args.data_root.resolve() / suite / f"{split}.jsonl"),
+            ]
+        if args.limit is not None:
+            run += ["--limit", str(args.limit)]
+        evaluations.append(run)
+    return {
+        "profile": profile,
+        "model": manifest["models"][profile],
+        "engine": engine,
+        "adapter": adapter,
+        "evaluations": evaluations,
+    }
+
+
+def stop_owned(process):
+    if process is not None and process.poll() is None:
+        # SIGINT can be inherited as ignored under nohup; the runtime handles TERM.
+        os.killpg(process.pid, signal.SIGTERM)
+        try:
+            process.wait(timeout=40)
+        except subprocess.TimeoutExpired as exc:
+            raise RuntimeError(
+                f"Owned process {process.pid} did not stop; check its log."
+            ) from exc
+
+
+def wait_engine(process, ready_path, timeout):
+    deadline = time.monotonic() + timeout
+    while not ready_path.exists():
+        if process.poll() is not None:
+            raise RuntimeError(
+                "Engine launcher exited; see launcher.log and engine/engine.log"
+            )
+        if time.monotonic() > deadline:
+            raise TimeoutError("Engine startup deadline exceeded")
+        time.sleep(1)
+
+
+def wait_adapter(process, timeout=60):
+    import httpx
+
+    deadline = time.monotonic() + timeout
+    headers = {}
+    if key := os.environ.get("JEV_API_KEY"):
+        headers["Authorization"] = f"Bearer {key}"
+    with httpx.Client(timeout=2) as client:
+        while True:
+            if process.poll() is not None:
+                raise RuntimeError("Adapter startup failed; see adapter.log")
+            try:
+                response = client.get(
+                    "http://127.0.0.1:30120/v1/models", headers=headers
+                )
+                if response.is_success:
+                    return
+            except httpx.HTTPError:
+                pass
+            if time.monotonic() > deadline:
+                raise TimeoutError("Adapter startup deadline exceeded")
+            time.sleep(1)
+
+
+def main():
+    manifest = json.loads((RUNTIME.parent / "models.json").read_text())
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument(
+        "--engine-python", type=Path, default=ROOT / ".venv-sglang/bin/python"
+    )
+    parser.add_argument("--data-root", type=Path, default=ROOT / "benchmarks/data")
+    parser.add_argument("--profile", action="append", choices=list(manifest["models"]))
+    parser.add_argument("--concurrency", type=int, action="append")
+    parser.add_argument("--warmup", type=int, default=20)
+    parser.add_argument("--repeats", type=int, default=1)
+    parser.add_argument("--limit", type=int)
+    parser.add_argument("--startup-timeout", type=int, default=3600)
+    parser.add_argument("--gpu", default="0")
+    parser.add_argument("--include-external", action="store_true")
+    parser.add_argument(
+        "--suite",
+        action="append",
+        choices=[
+            "decision-v7",
+            "transfer-v4",
+            "transfer-v9",
+            "jevbench-original",
+            "jevbench-easy",
+            "jevbench-hard",
+            "semif-v1",
+            "scienthoon-v1",
+        ],
+    )
+    parser.add_argument(
+        "--probe-vision", action="store_true", help="optional image readiness probe"
+    )
+    parser.add_argument("--execute", action="store_true")
+    args = parser.parse_args()
+    args.concurrency = args.concurrency or [1]
+    if (
+        any(c < 1 or c > 32 for c in args.concurrency)
+        or len(set(args.concurrency)) != len(args.concurrency)
+        or args.warmup < 0
+        or args.repeats < 1
+        or args.startup_timeout < 1
+        or (args.limit is not None and args.limit < 1)
+    ):
+        parser.error("invalid counts; concurrency must be unique values from 1 to 32")
+    profiles = args.profile or manifest["default_matrix"]
+    if len(set(profiles)) != len(profiles):
+        parser.error("profiles must be unique")
+    plans = [commands(args, manifest, profile) for profile in profiles]
+    for plan in plans:
+        print(f"\n{plan['profile']}")
+        for command in [plan["engine"], plan["adapter"], *plan["evaluations"]]:
+            print(shlex.join(command))
+    if not args.execute:
+        print("\nPreview only. Add --execute on the prepared H200 server.")
+        return
+    if not args.engine_python.is_file():
+        parser.error("engine Python is missing; follow benchmarks/h200/README.md")
+    for plan in plans:
+        for command in plan["evaluations"]:
+            for i, value in enumerate(command):
+                if value == "--data" and not Path(command[i + 1]).is_file():
+                    parser.error(f"missing prepared dataset: {command[i + 1]}")
+    args.output.mkdir(parents=True, exist_ok=False)
+    (args.output / "matrix.json").write_text(json.dumps(plans, indent=2) + "\n")
+    for plan in plans:
+        # Never attach to or stop unrelated services using the benchmark ports.
+        for port in (30000, 30120):
+            with socket.socket() as listener:
+                # Match server bind semantics: ignore TIME_WAIT, reject live listeners.
+                listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+                listener.bind(("127.0.0.1", port))
+        root = args.output.resolve() / plan["profile"]
+        root.mkdir()
+        engine, adapter = None, None
+        with (
+            (root / "launcher.log").open("x") as engine_log,
+            (root / "adapter.log").open("x") as adapter_log,
+        ):
+            try:
+                engine = subprocess.Popen(
+                    plan["engine"],
+                    cwd=ROOT,
+                    stdout=engine_log,
+                    stderr=subprocess.STDOUT,
+                    start_new_session=True,
+                )
+                wait_engine(
+                    engine, root / "engine/ready.json", args.startup_timeout + 90
+                )
+                adapter = subprocess.Popen(
+                    plan["adapter"],
+                    cwd=ROOT,
+                    stdout=adapter_log,
+                    stderr=subprocess.STDOUT,
+                    start_new_session=True,
+                )
+                wait_adapter(adapter)
+                for command in plan["evaluations"]:
+                    subprocess.run(command, cwd=ROOT, check=True)
+            finally:
+                try:
+                    stop_owned(adapter)
+                finally:
+                    stop_owned(engine)
+    subprocess.run(
+        [
+            sys.executable,
+            "-m",
+            "jev_adapter.benchmarks.compare",
+            str(args.output.resolve()),
+            "--output",
+            str(args.output.resolve() / "comparison.csv"),
+        ],
+        cwd=ROOT,
+        check=True,
+    )
+
+
+if __name__ == "__main__":
+    main()

server/benchmarks/verify_metrics.py

@@ -0,0 +1,140 @@
+"""Audit metric parity against verified KEV source, without importing KEV or torch.
+
+Requires NumPy only in the optional audit environment, not in jev-adapter.
+"""
+
+from __future__ import annotations
+
+import argparse
+import ast
+import hashlib
+import json
+import math
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
+from jev_adapter.benchmarks import metrics as adapter_metrics
+from jev_adapter.benchmarks.data import KEV_COMMIT
+
+REFERENCE_FILES = {
+    "kev/evaluate.py": (
+        "1b20e3f9edf417aa8dae924b1526e52f74b710cadf7213c5ec68334f6e7f8fe1",
+        {"ece"},
+    ),
+    "kev/benchmark.py": (
+        "4192ec3b26b065452f84bde38a091e6854a28fe185d2e0f39a0c91b2efe69df7",
+        {"coverage_at_error", "metrics"},
+    ),
+    "kev/contrastive.py": (
+        "cbb979aa5d40265ad0e64695f94b281d91751fa405811ddfa8212fede111edcf",
+        {"paired_flip"},
+    ),
+}
+
+
+def extract_reference(root, numpy):
+    scope = {"np": numpy, "math": math, "EPSILON": 1e-9}
+    for relative, (expected_hash, names) in REFERENCE_FILES.items():
+        payload = (root / relative).read_bytes()
+        if hashlib.sha256(payload).hexdigest() != expected_hash:
+            raise ValueError(
+                f"reference file differs from KEV {KEV_COMMIT}: {relative}"
+            )
+        tree = ast.parse(payload, filename=relative)
+        tree.body = [
+            node
+            for node in tree.body
+            if isinstance(node, ast.FunctionDef) and node.name in names
+        ]
+        if {node.name for node in tree.body} != names:
+            raise ValueError(f"missing reference functions in {relative}")
+        # Execute only the hash-verified function definitions. Top-level imports,
+        # model loading, decorators elsewhere and benchmark entry points are absent.
+        exec(compile(tree, relative, "exec"), scope)  # noqa: S102
+    return scope
+
+
+def audit(root, numpy):
+    reference = extract_reference(root, numpy)
+    rng = numpy.random.default_rng(483)
+    maximum_delta = {}
+    tolerance = 1e-12
+    for batch in range(100):
+        rows = []
+        for index in range(50):
+            kind = ["choice", "noul", "score"][index % 3]
+            options = 2 if kind == "noul" else int(rng.integers(2, 11))
+            probabilities = rng.dirichlet(numpy.ones(options)).tolist()
+            label = int(rng.integers(options))
+            if index < 10:
+                probabilities = [index / 10, 1 - index / 10]
+                label %= 2
+                kind = "noul"
+            rows.append({"p": probabilities, "label": label, "type": kind})
+        expected = reference["metrics"](rows)
+        actual = adapter_metrics.metrics(rows)
+        for key, value in actual.items():
+            if value is None or not isinstance(value, (int, float)):
+                continue
+            delta = abs(value - expected[key])
+            maximum_delta[key] = max(maximum_delta.get(key, 0), delta)
+            if delta > tolerance:
+                raise ValueError(
+                    f"batch {batch}, {key}: adapter={value}, KEV={expected[key]}"
+                )
+    rows = []
+    for index in range(10):
+        for sibling, label in (("a", 0), ("b", index % 2)):
+            rows.append(
+                {
+                    "pair_id": str(index),
+                    "question": "q",
+                    "keys": ["x", "y"],
+                    "label": label,
+                    "p": [0.7, 0.3] if index % 3 else [0.4, 0.6],
+                    "sibling": sibling,
+                }
+            )
+    expected = reference["paired_flip"](rows)
+    actual = adapter_metrics.paired_flip(rows)
+    for key, value in expected.items():
+        if actual[key] != value:
+            raise ValueError(f"pair metric differs: {key}: {actual[key]} != {value}")
+    return {
+        "status": "passed",
+        "kev_commit": KEV_COMMIT,
+        "reference_sha256": {path: entry[0] for path, entry in REFERENCE_FILES.items()},
+        "adapter_metrics_sha256": hashlib.sha256(
+            Path(adapter_metrics.__file__).read_bytes()
+        ).hexdigest(),
+        "numpy_version": numpy.__version__,
+        "seed": 483,
+        "batches": 100,
+        "rows_per_batch": 50,
+        "total_metric_rows": 5000,
+        "absolute_tolerance": tolerance,
+        "maximum_absolute_delta": maximum_delta,
+        "complete_pair_metrics_exact": expected,
+        "limitations": [
+            "Numerical metric parity, not GPU/model inference validation.",
+            "Only common returned scalar metrics and complete pair metrics compared.",
+            "Partial-pair handling and unknowable reporting intentionally differ.",
+        ],
+    }
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--kev-root", required=True, type=Path)
+    args = parser.parse_args()
+    try:
+        import numpy
+    except ImportError:
+        parser.error("NumPy is required in this optional audit environment")
+    print(json.dumps(audit(args.kev_root, numpy), indent=2, allow_nan=False))
+
+
+if __name__ == "__main__":
+    main()

server/examples/request.json

@@ -0,0 +1,13 @@
+{
+  "model": "jev-latest",
+  "state": "결제가 두 번 되었습니다. 중복 결제 건을 환불해 주세요.",
+  "questions": {
+    "route": {
+      "type": "choice",
+      "instructions": "어느 부서로 보내야 하나?",
+      "criteria": {"billing": "결제 및 환불", "technical": "기술 지원", "sales": "상품 문의"}
+    },
+    "refund": {"type": "noul", "instructions": "환불을 요청했나?"},
+    "urgency": {"type": "score", "instructions": "대응 긴급도", "criteria": ["낮음", "보통", "높음"]}
+  }
+}

server/examples/smoke.py

@@ -0,0 +1,49 @@
+"""Smoke-test a running adapter, optionally including actual image bytes."""
+
+import argparse
+import base64
+import json
+import mimetypes
+import os
+import time
+from pathlib import Path
+from urllib.request import Request, urlopen
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--base-url", default="http://127.0.0.1:30120")
+    parser.add_argument(
+        "--request", type=Path, default=Path(__file__).with_name("request.json")
+    )
+    parser.add_argument("--image", type=Path, action="append", default=[])
+    args = parser.parse_args()
+    body = json.loads(args.request.read_text())
+    for path in args.image:
+        mime = mimetypes.guess_type(path.name)[0]
+        if not mime or not mime.startswith("image/"):
+            parser.error(f"Cannot identify image type: {path.name}")
+        encoded = base64.b64encode(path.read_bytes()).decode("ascii")
+        body.setdefault("images", []).append(f"data:{mime};base64,{encoded}")
+    headers = {"Content-Type": "application/json"}
+    if key := os.environ.get("JEV_API_KEY"):
+        headers["Authorization"] = f"Bearer {key}"
+    request = Request(
+        args.base_url.rstrip("/") + "/v1/systemone",
+        data=json.dumps(body).encode(),
+        headers=headers,
+    )
+    started = time.perf_counter()
+    with urlopen(request, timeout=120) as response:
+        result = json.load(response)
+    elapsed = (time.perf_counter() - started) * 1000
+    if result.get("usage", {}).get("output_tokens") != 0:
+        raise RuntimeError("Expected zero generated tokens")
+    if set(result.get("answers", {})) != set(body["questions"]):
+        raise RuntimeError("Missing decision answers")
+    print(json.dumps(result, indent=2, ensure_ascii=False))
+    print(f"End-to-end HTTP latency: {elapsed:.1f} ms (not a GPU benchmark)")
+
+
+if __name__ == "__main__":
+    main()

server/jev_adapter/benchmarks/__init__.py

@@ -0,0 +1 @@
+"""Reproducible, training-free decision benchmarks over the adapter HTTP API."""

server/jev_adapter/benchmarks/compare.py

@@ -0,0 +1,72 @@
+"""Collect complete benchmark reports into a comparison table, one row per suite."""
+
+import argparse
+import csv
+import json
+from pathlib import Path
+
+
+def collect(root):
+    rows = []
+    for path in sorted(root.rglob("report.json")):
+        if path.with_name("invalidated.json").exists():
+            continue
+        report = json.loads(path.read_text())
+        if report.get("status") != "complete" or not report.get("suites"):
+            continue
+        manifest = json.loads(path.with_name("manifest.json").read_text())
+        launch = manifest.get("launch_manifest", {})
+        engine_model = (
+            manifest.get("engine", {}).get("model_info", {}).get("model_path")
+        )
+        engine_revision = (
+            manifest.get("engine", {}).get("server_info", {}).get("revision")
+        )
+        for suite, result in report["suites"].items():
+            clean = result.get("clean")
+            if clean is None:
+                continue
+            latency = result["latency_ms"]
+            rows.append(
+                {
+                    "profile": launch.get("profile", manifest["requested_model"]),
+                    "model_repo": launch.get("model", {}).get("repo_id", engine_model),
+                    "model_revision": launch.get("model", {}).get(
+                        "revision", engine_revision
+                    ),
+                    "suite": suite,
+                    "concurrency": manifest["concurrency"],
+                    "partial_dataset": report["partial_dataset"],
+                    "clean_questions": clean["n"],
+                    "accuracy": clean["acc"],
+                    "brier": clean["brier"],
+                    "ece": clean["ece"],
+                    "confident_error_rate": clean["confident_error_rate"],
+                    "p50_ms": latency["p50"],
+                    "p95_ms": latency["p95"],
+                    "p99_ms": latency["p99"],
+                    "cache_mode": manifest["cache_mode"],
+                    "report": str(path.resolve()),
+                }
+            )
+    return rows
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("root", type=Path)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    rows = collect(args.root)
+    if not rows:
+        parser.error("no complete benchmark reports found; failed runs are not ranked")
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    with args.output.open("x", newline="") as handle:
+        writer = csv.DictWriter(handle, fieldnames=list(rows[0]))
+        writer.writeheader()
+        writer.writerows(rows)
+    print(f"Wrote {len(rows)} suite rows to {args.output}; latency is HTTP wall time.")
+
+
+if __name__ == "__main__":
+    main()

server/jev_adapter/benchmarks/data.py

@@ -0,0 +1,217 @@
+"""Normalize pinned, public KEV evaluation records without changing their tasks.
+
+Only the request is sent to a model. Labels and source metadata remain outside
+that request, in a separate expected/metadata envelope used by the evaluator.
+This module is an independent format conversion, not imported KEV model code.
+"""
+
+from __future__ import annotations
+
+import copy
+import hashlib
+import json
+from collections import Counter
+from dataclasses import dataclass
+from typing import Any
+
+KEV_COMMIT = "4f8110a3f8620cc3a182ae9a708e4398492c4b1a"
+KEV_REPOSITORY = "https://github.com/jaredpalmer/kev"
+KEV_RAW = f"https://raw.githubusercontent.com/jaredpalmer/kev/{KEV_COMMIT}"
+DEFAULT_SUITES = ("decision-v7", "transfer-v4", "transfer-v9")
+
+
+@dataclass(frozen=True)
+class SuiteSpec:
+    path: str
+    manifest_sha256: str
+    description: str
+    notes: tuple[str, ...] = ()
+
+
+SUITES = {
+    "decision-v7": SuiteSpec(
+        "evals/v7/decision-v7",
+        "a8f50e481b7d90b97da049e0ff6a01cee2f1ed204aed61a8265af0edbb5514d2",
+        "Ten public sources and generated policies; KEV trained-source evaluation.",
+        ("Includes up to 78 choices with none-of-the-above variants.",),
+    ),
+    "transfer-v4": SuiteSpec(
+        "evals/v4/transfer-v4",
+        "31677c2256b406222e7d94ffdc0a02a70ce05746b9efe307876024c4e77291d1",
+        "Six sources unseen in KEV fine-tuning and held-out policy structures.",
+        ("Unseen means unseen in KEV fine-tuning, not in base-model pretraining.",),
+    ),
+    "transfer-v9": SuiteSpec(
+        "evals/v9/transfer-v9",
+        "3c4f0be94509a3612678bfd3a30fd99a8d0ca3c47ddfe7318075d95b2fa365e4",
+        "Transfer-v4 plus MMLU-Pro, buried evidence and unknowable/control pairs.",
+        (
+            "Contains transfer-v4 records; do not pool both suites as independent data.",
+            "Source 'unknowable' is evaluated for confidence, not accuracy.",
+        ),
+    ),
+    "semif-v1": SuiteSpec(
+        "evals/external/semif-v1",
+        "0de05eac16b0ddeeb2719c50a94a9148d6ae195f66303aec74aa103a3845ad11",
+        "SemIf's 144 authored choices plus 108 perturbations, frozen by KEV.",
+        (
+            "Already evaluated by KEV; this is not an additional independent suite.",
+            "SemIf's own headline is mean family balanced accuracy.",
+        ),
+    ),
+    "scienthoon-v1": SuiteSpec(
+        "evals/external/scienthoon-v1",
+        "ef31183425bf9d3c2d8ac5d245a14d30fa44d1e531c945e4405edcb1f75ae0d5",
+        "KEV's frozen conversion of scienthoon's synthetic support tickets.",
+        (
+            "Already evaluated by KEV; this is not an additional independent suite.",
+            "Original 900 question rows become 291 unique states / 873 questions in this conversion.",
+            "Priority labels depend on an organizational rule absent from the state; report separately.",
+        ),
+    ),
+}
+
+
+def sha256(data: bytes) -> str:
+    return hashlib.sha256(data).hexdigest()
+
+
+def require_sha256(data: bytes, expected: str, name: str) -> None:
+    actual = sha256(data)
+    if actual != expected:
+        raise ValueError(
+            f"SHA256 mismatch for {name}: expected {expected}, got {actual}"
+        )
+
+
+def normalize_record(raw: dict[str, Any], suite: str, split: str) -> dict[str, Any]:
+    """Retain option order, all sibling questions and KEV's original provenance."""
+    meta = copy.deepcopy(raw["_meta"])
+    if not isinstance(meta.get("id"), str) or not meta["id"]:
+        raise ValueError("record requires a nonempty _meta.id")
+    questions, expected = {}, {}
+    for qid, question in raw["questions"].items():
+        kind = question["type"]
+        if kind == "choice":
+            labels = list(question["criteria"])
+            try:
+                label = labels.index(question["label"])
+            except ValueError as exc:
+                raise ValueError(f"{meta['id']}/{qid}: label is not an option") from exc
+        elif kind == "noul":
+            if type(question["label"]) is not bool:
+                raise ValueError(f"{meta['id']}/{qid}: noul label must be a boolean")
+            labels, label = ["false", "true"], int(question["label"])
+        elif kind == "score":
+            labels = [str(i) for i in range(len(question["criteria"]))]
+            label = question["label"]
+            if type(label) is not int or not 0 <= label < len(labels):
+                raise ValueError(f"{meta['id']}/{qid}: score label is out of range")
+        else:
+            raise ValueError(f"unsupported question type: {kind}")
+        if len(labels) < 2 or len(set(labels)) != len(labels):
+            raise ValueError(f"{meta['id']}/{qid}: invalid option labels")
+        questions[qid] = {
+            key: copy.deepcopy(question[key])
+            for key in ("type", "instructions", "criteria")
+            if key in question
+        }
+        expected[qid] = {
+            "labels": labels,
+            "target": [float(i == label) for i in range(len(labels))],
+            "label": label,
+            "type": kind,
+            "task": question.get("src", meta["source"]),
+        }
+    if not questions:
+        raise ValueError(f"{meta['id']}: no questions")
+    record = {"state": copy.deepcopy(raw["state"]), "questions": questions}
+    for key in ("images", "options"):
+        if key in raw:
+            record[key] = copy.deepcopy(raw[key])
+    return {
+        "id": meta["id"],
+        "suite": suite,
+        "split": split,
+        "source": meta["source"],
+        "variant": meta.get("variant", "clean"),
+        "record": record,
+        "expected": expected,
+        "metadata": meta,
+    }
+
+
+def parse_partition(data: bytes, suite: str, split: str) -> list[dict[str, Any]]:
+    records, seen = [], set()
+    for number, line in enumerate(data.decode("utf-8").splitlines(), 1):
+        if not line.strip():
+            raise ValueError(f"{suite}/{split}:{number}: blank JSONL record")
+        row = normalize_record(json.loads(line), suite, split)
+        if row["id"] in seen:
+            raise ValueError(f"duplicate record id: {row['id']}")
+        seen.add(row["id"])
+        records.append(row)
+    return records
+
+
+def population_counts(records: list[dict[str, Any]]) -> dict[str, Any]:
+    clean = [row for row in records if row["variant"] == "clean"]
+    return {
+        "records": len(records),
+        "questions": sum(len(row["expected"]) for row in records),
+        "clean_records": len(clean),
+        "clean_questions": sum(len(row["expected"]) for row in clean),
+        "headline_questions": sum(
+            len(row["expected"]) for row in clean if row["source"] != "unknowable"
+        ),
+        "variants": dict(Counter(row["variant"] for row in records)),
+        "clean_sources": dict(Counter(row["source"] for row in clean)),
+        "question_types": dict(
+            Counter(q["type"] for row in records for q in row["expected"].values())
+        ),
+        "maximum_options": max(
+            (len(q["labels"]) for row in records for q in row["expected"].values()),
+            default=0,
+        ),
+    }
+
+
+def source_provenance(
+    records: list[dict[str, Any]], upstream_manifest: dict[str, Any]
+) -> dict[str, Any]:
+    """Point to underlying dataset licenses; do not relicense mixed source data."""
+    datasets = {}
+    for row in records:
+        meta = row["metadata"]
+        repo = meta.get("repo")
+        if not repo:
+            continue
+        revision = meta.get("revision")
+        key = (repo, revision)
+        external = upstream_manifest.get("external", {})
+        is_external = external.get("repo", "").endswith("/" + repo)
+        datasets[key] = {
+            "repository": repo,
+            "revision": revision,
+            "source_url": (
+                external["repo"]
+                if is_external
+                else f"https://huggingface.co/datasets/{repo}"
+            ),
+            "license": external.get("license")
+            if is_external
+            else "see upstream dataset",
+        }
+    return {
+        "kev_repository_license": "Apache-2.0",
+        "kev_license_url": f"{KEV_REPOSITORY}/blob/{KEV_COMMIT}/LICENSE",
+        "dataset_notice": (
+            "Public and downloadable does not mean all source datasets share Apache-2.0. "
+            "Their individual licenses and attribution terms continue to apply."
+        ),
+        "datasets": list(datasets.values()),
+        "external": upstream_manifest.get("external"),
+        "dataset_revisions_from_manifest": upstream_manifest.get(
+            "dataset_revisions", {}
+        ),
+    }

server/jev_adapter/benchmarks/jevbench.py

@@ -0,0 +1,292 @@
+"""Prepare JevBench's three pinned public tiers without exposing answer metadata.
+
+This is an independent format conversion. The public 231 questions are not the
+534-question leaderboard, whose remaining questions are private or untracked.
+"""
+
+from __future__ import annotations
+
+import copy
+import json
+import math
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+import httpx
+
+from jev_adapter.protocol import SystemOneRequest
+
+from .data import normalize_record, population_counts, require_sha256, sha256
+
+JEVBENCH_COMMIT = "fd51755eb0c0b546ca206d764faf3302feca913e"
+JEVBENCH_REPOSITORY = "https://github.com/fstandhartinger/jevbench"
+JEVBENCH_RAW = (
+    f"https://raw.githubusercontent.com/fstandhartinger/jevbench/{JEVBENCH_COMMIT}"
+)
+JEVBENCH_NOTICES = {
+    "LICENSE": "3e5beed774bb0bcbfb2fcf24ba9554212c6ae112937d308040989465ab0c5784",
+    "THIRD-PARTY.md": "396422e29055bba4073f4a9e5a7163cc724f13e970a34ef58e8ea1fb439b9b38",
+}
+
+
+@dataclass(frozen=True)
+class JevBenchSpec:
+    path: str
+    sha256: str
+    records: int
+    description: str
+
+
+JEVBENCH_SUITES = {
+    "jevbench-original": JevBenchSpec(
+        "datasets/public/original.jsonl",
+        "5c2414edb3006b8bfcb70fda433f0f9ca015759433849f8d3104328a1f7c4180",
+        72,
+        "Public original tier: 72 decisions in 36 paraphrase pairs.",
+    ),
+    "jevbench-easy": JevBenchSpec(
+        "datasets/public/easy.jsonl",
+        "231df3c2c8e88a1a8c137ebe85de96ba70fabd330849098ac7b3c52c70b7172b",
+        48,
+        "Public easy tier: 48 explicit facts, intents and tool selections.",
+    ),
+    "jevbench-hard": JevBenchSpec(
+        "datasets/public/hard.jsonl",
+        "89e9e6becb33ed88c1de7d42dcc87531b2fb64cfaef4e1986faf7c37b3f80ebb",
+        111,
+        "Public hard tier: 111 decisions; ten supply probability references.",
+    ),
+}
+
+
+def normalize_jevbench_record(raw: dict[str, Any], suite: str) -> dict[str, Any]:
+    """Keep native question/option order and canonical scoring label order."""
+    if raw.get("split") != "public":
+        raise ValueError("JevBench converter accepts only the public subset")
+    if not isinstance(raw.get("id"), str) or not raw["id"]:
+        raise ValueError("JevBench record requires a nonempty id")
+    labels = raw["labels"]
+    if (
+        not isinstance(labels, list)
+        or len(labels) < 2
+        or any(not isinstance(label, str) or not label for label in labels)
+        or len(set(labels)) != len(labels)
+    ):
+        raise ValueError(f"{raw['id']}: invalid canonical labels")
+    question = raw["question"]
+    kind, gold = question["type"], raw["expected"]
+    if kind == "choice":
+        if not isinstance(question["criteria"], dict) or set(labels) != set(
+            question["criteria"]
+        ):
+            raise ValueError(f"{raw['id']}: choice criteria/labels mismatch")
+        if not isinstance(gold, str) or gold not in labels:
+            raise ValueError(f"{raw['id']}: invalid choice expected label")
+        canonical_labels, canonical_gold = list(labels), gold
+    elif kind == "noul":
+        if labels != ["no", "yes"] or gold not in ("no", "yes"):
+            raise ValueError(f"{raw['id']}: noul requires no/yes canonical labels")
+        canonical_labels = ["false", "true"]
+        canonical_gold, gold = ("true" if gold == "yes" else "false"), gold == "yes"
+    elif kind == "score":
+        if not isinstance(question["criteria"], list) or labels != [
+            str(i) for i in range(len(question["criteria"]))
+        ]:
+            raise ValueError(f"{raw['id']}: score criteria/labels mismatch")
+        if type(gold) is not int or not 0 <= gold < len(labels):
+            raise ValueError(f"{raw['id']}: invalid score expected label")
+        canonical_labels, canonical_gold = list(labels), str(gold)
+    else:
+        raise ValueError(f"unsupported JevBench question type: {kind}")
+    provenance = copy.deepcopy(raw["provenance"])
+    reference = provenance.get("gold_probs")
+    if provenance.get("exclude_reason"):
+        raise ValueError(f"{raw['id']}: excluded upstream item is not evaluable")
+    family = raw["family"]
+    if not isinstance(family, str) or not family:
+        raise ValueError(f"{raw['id']}: missing task family")
+    # Only these three native question fields are allowed into the request.
+    clean_question = {
+        key: copy.deepcopy(question[key])
+        for key in ("type", "instructions", "criteria")
+        if key in question
+    }
+    clean_question["label"] = gold
+    row = normalize_record(
+        {
+            "state": raw["state"],
+            "questions": {"decision": clean_question},
+            "_meta": {
+                "id": raw["id"],
+                "source": family,
+                "variant": "clean",
+                "group_id": raw.get("group") or raw["id"],
+                "upstream_group": raw.get("group"),
+                "canonical_labels": list(labels),
+                "provenance": provenance,
+                "gold_policy": {
+                    "hard_label": "authored_reviewed_rubric",
+                    "reference_probs": (
+                        "countable_mathematical_probability"
+                        if reference is not None
+                        else None
+                    ),
+                    "argmax_tie_break": "lexicographic_label",
+                },
+            },
+        },
+        suite,
+        "public",
+    )
+    expected = row["expected"]["decision"]
+    expected["labels"] = canonical_labels
+    expected["label"] = canonical_labels.index(canonical_gold)
+    expected["target"] = [float(label == canonical_gold) for label in canonical_labels]
+    if reference is not None:
+        if not isinstance(reference, dict) or set(reference) != set(labels):
+            raise ValueError(f"{raw['id']}: reference probability labels mismatch")
+        probabilities = [reference[label] for label in labels]
+        if any(
+            isinstance(value, bool)
+            or not isinstance(value, (float, int))
+            or not math.isfinite(value)
+            or not 0 <= value <= 1
+            for value in probabilities
+        ) or not math.isclose(sum(probabilities), 1.0, rel_tol=0, abs_tol=1e-9):
+            raise ValueError(f"{raw['id']}: invalid reference probabilities")
+        expected["reference_probs"] = [float(value) for value in probabilities]
+    SystemOneRequest.model_validate({"model": "validation-only", **row["record"]})
+    return row
+
+
+def parse_jevbench(data: bytes, suite: str) -> list[dict[str, Any]]:
+    records, seen = [], set()
+    for number, line in enumerate(data.decode("utf-8").splitlines(), 1):
+        if not line.strip():
+            raise ValueError(f"{suite}/public:{number}: blank JSONL record")
+        row = normalize_jevbench_record(json.loads(line), suite)
+        if row["id"] in seen:
+            raise ValueError(f"duplicate record id: {row['id']}")
+        seen.add(row["id"])
+        records.append(row)
+    return records
+
+
+def prepare_jevbench(
+    suite: str,
+    output: Path,
+    *,
+    client: httpx.Client | None = None,
+    source_root: Path | None = None,
+    limit: int | None = None,
+) -> dict[str, Any]:
+    """Verify all source bytes before writing an immutable prepared public tier."""
+    if suite not in JEVBENCH_SUITES:
+        raise ValueError(f"unknown JevBench suite: {suite}")
+    if limit is not None and (type(limit) is not int or limit < 1):
+        raise ValueError("--limit must be positive")
+
+    def read_source(path: str) -> bytes:
+        if source_root is not None:
+            return (source_root / path).read_bytes()
+        if client is None:
+            raise ValueError("an HTTP client or --source-root is required")
+        response = client.get(f"{JEVBENCH_RAW}/{path}")
+        response.raise_for_status()
+        return response.content
+
+    spec = JEVBENCH_SUITES[suite]
+    raw_bytes = read_source(spec.path)
+    require_sha256(raw_bytes, spec.sha256, spec.path)
+    notices = {}
+    for path, digest in JEVBENCH_NOTICES.items():
+        content = read_source(path)
+        require_sha256(content, digest, path)
+        notices[path] = content
+    records = parse_jevbench(raw_bytes, suite)
+    counts = population_counts(records)
+    if counts["records"] != spec.records or counts["questions"] != spec.records:
+        raise ValueError(f"upstream record/question count mismatch: {suite}/public")
+    selected = records if limit is None else records[:limit]
+    serialized = "".join(
+        json.dumps(row, ensure_ascii=False, allow_nan=False) + "\n" for row in selected
+    ).encode("utf-8")
+    full = len(selected) == len(records)
+    manifest = {
+        "schema_version": 1,
+        "suite": suite,
+        "split": "public",
+        "description": spec.description,
+        "data_file": "public.jsonl",
+        "data_sha256": sha256(serialized),
+        "upstream": {
+            "repository": JEVBENCH_REPOSITORY,
+            "commit": JEVBENCH_COMMIT,
+            "path": spec.path,
+            "url": f"{JEVBENCH_RAW}/{spec.path}",
+            "sha256": spec.sha256,
+            "notice_sha256": dict(JEVBENCH_NOTICES),
+        },
+        "full_partition": counts,
+        "selected": population_counts(selected),
+        "selection": {
+            "method": "full" if full else "prefix",
+            "requested_limit": limit,
+            "is_full_partition": full,
+            "note": (
+                "Full frozen public tier; not the complete published leaderboard."
+                if full
+                else "Smoke subset only; not a full-tier result. Pairs may be incomplete."
+            ),
+        },
+        "protocol": {
+            "calibration_applied": False,
+            "training_data_downloaded": False,
+            "gold_labels_sent_to_model": False,
+            "headline_variant": "clean",
+            "exclude_from_headline_sources": [],
+            "locked_test": False,
+            "notes": [
+                "Community benchmark; not an official TypeSafe dataset release.",
+                "Public original/easy/hard have 72/48/111 questions; report separately.",
+                "The 534-item leaderboard contains private/untracked data not downloaded here.",
+                "Choice request criteria retain their native order; scoring labels retain canonical order.",
+                "Native accuracy uses argmax with lexicographic label tie-break, including Score.",
+                "Score also supports expected-value MAE; original has 36 paraphrase groups.",
+                "One-hot labels and ten explicit mathematical probability references are separate targets.",
+                "Provenance, rationale and gold probabilities are never sent in requests.",
+            ],
+        },
+        "provenance": {
+            "dataset_license": "MIT",
+            "license_url": f"{JEVBENCH_REPOSITORY}/blob/{JEVBENCH_COMMIT}/LICENSE",
+            "notices": list(JEVBENCH_NOTICES),
+            "reference_probability_questions": sum(
+                "reference_probs" in row["expected"]["decision"] for row in records
+            ),
+            "gold_policy": (
+                "Authored rubric labels reviewed before inference; hard items retain author "
+                "and review metadata. Explicit mathematical distributions are not teacher "
+                "model confidence or population frequency estimates."
+            ),
+        },
+    }
+    artifacts = {
+        "public.jsonl": serialized,
+        "public.manifest.json": (
+            json.dumps(manifest, ensure_ascii=False, indent=2, allow_nan=False) + "\n"
+        ).encode("utf-8"),
+        **notices,
+    }
+    directory = output / suite
+    for name, content in artifacts.items():
+        path = directory / name
+        if path.exists() and path.read_bytes() != content:
+            raise FileExistsError(
+                f"refusing to overwrite a different prepared artifact: {path}"
+            )
+    directory.mkdir(parents=True, exist_ok=True)
+    for name, content in artifacts.items():
+        (directory / name).write_bytes(content)
+    return manifest

server/jev_adapter/benchmarks/metrics.py

@@ -0,0 +1,326 @@
+"""Dependency-free metrics following KEV's frozen benchmark definitions.
+
+Reimplemented against jaredpalmer/kev commit
+4f8110a3f8620cc3a182ae9a708e4398492c4b1a, kev/benchmark.py,
+kev/evaluate.py and kev/contrastive.py (Apache-2.0). Differences are explicit:
+partial pairs are counted, and unknowable items never receive accuracy metrics.
+"""
+
+import math
+from collections import defaultdict
+from statistics import fmean
+
+
+def argmax(values):
+    return max(range(len(values)), key=values.__getitem__)
+
+
+def predicted_index(row):
+    if row.get("argmax_tie_break") == "lexicographic_label":
+        return min(range(len(row["p"])), key=lambda i: (-row["p"][i], row["keys"][i]))
+    return argmax(row["p"])
+
+
+def distribution(raw, labels):
+    if not isinstance(raw, dict) or set(raw) != set(labels):
+        raise ValueError("probability keys differ from requested labels")
+    p = [raw[key] for key in labels]
+    if any(
+        type(value) not in (int, float)
+        or not math.isfinite(value)
+        or not 0 <= value <= 1
+        for value in p
+    ):
+        raise ValueError("invalid probability value")
+    total = math.fsum(p)
+    if total <= 0 or abs(total - 1) > max(1e-5, len(labels) * 0.005 + 1e-8):
+        raise ValueError("invalid probability sum")
+    return [value / total for value in p], total
+
+
+def prediction_rows(item, response):
+    expected = item["expected"]
+    answers = response.get("answers")
+    if not isinstance(answers, dict) or set(answers) != set(expected):
+        raise ValueError("response question IDs differ from request")
+    rows = []
+    meta = item["metadata"]
+    for qid, target in expected.items():
+        answer = answers[qid]
+        kind = target["type"]
+        if not isinstance(answer, dict) or answer.get("type") != kind:
+            raise ValueError("response answer type differs from request")
+        if kind == "noul":
+            value = answer.get("noul")
+            if type(value) not in (int, float):
+                raise ValueError("invalid noul probability")
+            raw = {"true": value, "false": 1 - value}
+        else:
+            raw = answer.get("probabilities")
+        labels = target["labels"]
+        p, total = distribution(raw, labels)
+        rows.append(
+            {
+                "id": item["id"],
+                "suite": item["suite"],
+                "split": item["split"],
+                "question": qid,
+                "source": item["source"],
+                "task": target["task"],
+                "type": kind,
+                "variant": item["variant"],
+                "keys": labels,
+                "label": target["label"],
+                "p": p,
+                "group": meta.get("group_id", item["id"]),
+                "pair_id": meta.get("pair_id"),
+                "sibling": meta.get("sibling"),
+                "control_id": meta.get("control_id"),
+                "parent": meta.get("parent_id")
+                or (item["id"] if item["variant"] == "clean" else meta.get("group_id")),
+                "raw_probability_sum": total,
+                "zero_count": sum(value == 0 for value in p),
+                "reference_probs": target.get("reference_probs"),
+                "argmax_tie_break": meta.get("gold_policy", {}).get("argmax_tie_break"),
+            }
+        )
+    return rows
+
+
+def coverage_at_error(confidence, correct, budget):
+    # Stable ties deliberately match KEV. This empirical, retrospective cutoff
+    # is not a risk guarantee and must not be deployed as a fitted threshold.
+    order = sorted(range(len(confidence)), key=lambda i: -confidence[i])
+    wrong, accepted = 0, 0
+    for rank, index in enumerate(order, 1):
+        wrong += not correct[index]
+        if wrong <= budget * rank:
+            accepted = rank
+    return accepted / len(order)
+
+
+def metrics(rows):
+    if not rows:
+        return None
+    nll, correct, confidence, brier, mae, rps = [], [], [], [], [], []
+    for row in rows:
+        p, y = row["p"], row["label"]
+        nll.append(-math.log(max(p[y], 1e-9)))
+        correct.append(predicted_index(row) == y)
+        confidence.append(max(p))
+        brier.append(math.fsum((value - (i == y)) ** 2 for i, value in enumerate(p)))
+        if row["type"] == "score":
+            mae.append(abs(math.fsum(i * value for i, value in enumerate(p)) - y))
+            cumulative = 0.0
+            errors = []
+            for i, value in enumerate(p[:-1]):
+                cumulative += value
+                errors.append((cumulative - (i >= y)) ** 2)
+            rps.append(fmean(errors))
+    ece = 0.0
+    for index in range(10):
+        # Match numpy.linspace(0, 1, 11), including floating-point bin edges.
+        lo, hi = index * 0.1, (index + 1) * 0.1
+        bucket = [
+            i
+            for i, c in enumerate(confidence)
+            if lo <= c and (c < hi if index < 9 else c <= 1)
+        ]
+        if bucket:
+            ece += (
+                len(bucket)
+                / len(rows)
+                * abs(
+                    fmean(correct[i] for i in bucket)
+                    - fmean(confidence[i] for i in bucket)
+                )
+            )
+    high = [i for i, value in enumerate(confidence) if value >= 0.9]
+    result = {
+        "n": len(rows),
+        "acc": fmean(correct),
+        "nll": fmean(nll),
+        "brier": fmean(brier),
+        "ece": ece,
+        "mean_conf": fmean(confidence),
+        "confidence_bias": fmean(confidence) - fmean(correct),
+        "confident_error_rate": sum(not correct[i] for i in high) / len(rows),
+        "coverage_at_0_9": len(high) / len(rows),
+        "accuracy_at_0_9": fmean(correct[i] for i in high) if high else None,
+        "coverage_at_5pct_error": coverage_at_error(confidence, correct, 0.05),
+        "coverage_at_1pct_error": coverage_at_error(confidence, correct, 0.01),
+    }
+    if mae:
+        result.update(score_mae=fmean(mae), ranked_probability_score=fmean(rps))
+    return result
+
+
+def grouped_metrics(rows, key):
+    grouped = defaultdict(list)
+    for row in rows:
+        if row["source"] != "unknowable":
+            grouped[row[key]].append(row)
+    return {name: metrics(group) for name, group in sorted(grouped.items())}
+
+
+def paired_flip(rows):
+    pairs = defaultdict(dict)
+    for row in rows:
+        if row.get("pair_id"):
+            pair = pairs[row["pair_id"], row["question"]]
+            if row["sibling"] in pair:
+                raise ValueError("duplicate contrastive sibling")
+            pair[row["sibling"]] = row
+    complete = [pair for pair in pairs.values() if set(pair) == {"a", "b"}]
+
+    def truth(row):
+        return row["keys"][row["label"]]
+
+    def prediction(row):
+        return row["keys"][predicted_index(row)]
+
+    relevant = [p for p in complete if truth(p["a"]) != truth(p["b"])]
+    invariant = [p for p in complete if truth(p["a"]) == truth(p["b"])]
+
+    def both(group):
+        return (
+            fmean(all(prediction(r) == truth(r) for r in p.values()) for p in group)
+            if group
+            else None
+        )
+
+    return {
+        "pairs": len(relevant),
+        "incomplete_pairs": len(pairs) - len(complete),
+        "flip_rate": fmean(prediction(p["a"]) != prediction(p["b"]) for p in relevant)
+        if relevant
+        else None,
+        "both_correct_rate": both(relevant),
+        "invariant_pairs": len(invariant),
+        "invariant_both_correct_rate": both(invariant),
+        "invariance_rate": fmean(
+            prediction(p["a"]) == prediction(p["b"]) for p in invariant
+        )
+        if invariant
+        else None,
+    }
+
+
+def summarize(rows):
+    clean = [r for r in rows if r["variant"] == "clean"]
+    knowable = [r for r in clean if r["source"] != "unknowable"]
+    controls = {r["id"]: r for r in clean if r["source"] == "unknowable_control"}
+    unknown = [r for r in clean if r["source"] == "unknowable"]
+    unknowable = None
+    if unknown:
+        paired = [
+            (max(r["p"]), max(controls[r["control_id"]]["p"]))
+            for r in unknown
+            if r.get("control_id") in controls
+        ]
+        unknowable = {
+            "n": len(unknown),
+            "mean_max_p": fmean(max(r["p"]) for r in unknown),
+            "share_at_0_9": fmean(max(r["p"]) >= 0.9 for r in unknown),
+            "control_acc": (
+                fmean(predicted_index(r) == r["label"] for r in controls.values())
+                if controls
+                else None
+            ),
+            "control_mean_max_p": (
+                fmean(max(r["p"]) for r in controls.values()) if controls else None
+            ),
+            "control_share_at_0_9": (
+                fmean(max(r["p"]) >= 0.9 for r in controls.values())
+                if controls
+                else None
+            ),
+            "paired_confidence_drop": fmean(c - u for u, c in paired)
+            if paired
+            else None,
+            "share_less_confident_than_control": (
+                fmean(u < c for u, c in paired) if paired else None
+            ),
+        }
+    originals = {(r["id"], r["question"]): r for r in clean}
+    deltas, flips, missing = [], [], 0
+    for row in rows:
+        if row["variant"] != "permuted" or row["type"] != "choice":
+            continue
+        original = originals.get((row["parent"], row["question"]))
+        if original is None:
+            missing += 1
+            continue
+        aligned = [row["p"][row["keys"].index(key)] for key in original["keys"]]
+        deltas.append(max(abs(a - b) for a, b in zip(aligned, original["p"])))
+        flips.append(argmax(aligned) != argmax(original["p"]))
+    tasks = grouped_metrics(clean, "task")
+    return {
+        "clean": metrics(knowable),
+        "tasks": tasks,
+        "sources": grouped_metrics(clean, "source"),
+        "variants": grouped_metrics(rows, "variant"),
+        "macro_task_acc": fmean(t["acc"] for t in tasks.values()) if tasks else None,
+        "objective": -fmean(t["nll"] for t in tasks.values()) if tasks else None,
+        "paired_flip": paired_flip(clean),
+        "reference_distribution": reference_metrics(clean),
+        "unknowable": unknowable,
+        "permutation": {
+            "n": len(flips),
+            "missing_parents": missing,
+            "flip_rate": fmean(flips) if flips else None,
+            "mean_max_delta": fmean(deltas) if deltas else None,
+        },
+        "metric_policy": {
+            "nll_floor": 1e-9,
+            "ece_bins": 10,
+            "confidence_for_calibration": "maximum label probability",
+            "calibration_applied": False,
+            "unknown_accuracy_excluded": True,
+            "coverage_error_budget_is_retrospective": True,
+            "raw_sums_outside_1e_5": sum(
+                abs(r["raw_probability_sum"] - 1) > 1e-5 for r in rows
+            ),
+            "returned_zeros": sum(r["zero_count"] for r in rows),
+        },
+    }
+
+
+def reference_metrics(rows):
+    selected = [row for row in rows if row.get("reference_probs") is not None]
+    if not selected:
+        return None
+    squared, variation, kl = [], [], []
+    for row in selected:
+        p, q = row["p"], row["reference_probs"]
+        squared.append(math.fsum((a - b) ** 2 for a, b in zip(p, q)))
+        variation.append(0.5 * math.fsum(abs(a - b) for a, b in zip(p, q)))
+        kl.append(
+            math.fsum(b * math.log(b / max(a, 1e-9)) for a, b in zip(p, q) if b > 0)
+        )
+    return {
+        "n": len(selected),
+        "squared_l2": fmean(squared),
+        "total_variation": fmean(variation),
+        "kl_reference_to_model": fmean(kl),
+        "note": "Exact reference-distribution fidelity, separate from hard-label Brier/ECE.",
+    }
+
+
+def quantile(values, q):
+    if not values:
+        return None
+    ordered = sorted(values)
+    position = (len(ordered) - 1) * q
+    lo, hi = math.floor(position), math.ceil(position)
+    return ordered[lo] + (ordered[hi] - ordered[lo]) * (position - lo)
+
+
+def latency_summary(values):
+    return {
+        "n": len(values),
+        "mean": fmean(values) if values else None,
+        "p50": quantile(values, 0.5),
+        "p95": quantile(values, 0.95),
+        "p99": quantile(values, 0.99),
+    }

server/jev_adapter/benchmarks/prepare.py

@@ -0,0 +1,227 @@
+"""Download pinned KEV and JevBench evaluation data, without training data."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+import httpx
+
+from jev_adapter.protocol import SystemOneRequest
+
+from .data import (
+    DEFAULT_SUITES,
+    KEV_COMMIT,
+    KEV_RAW,
+    KEV_REPOSITORY,
+    SUITES,
+    parse_partition,
+    population_counts,
+    require_sha256,
+    sha256,
+    source_provenance,
+)
+from .jevbench import JEVBENCH_SUITES, prepare_jevbench
+
+
+def read_source(
+    path: str, client: httpx.Client | None, source_root: Path | None
+) -> bytes:
+    if source_root is not None:
+        return (source_root / path).read_bytes()
+    if client is None:
+        raise ValueError("an HTTP client or --source-root is required")
+    response = client.get(f"{KEV_RAW}/{path}")
+    response.raise_for_status()
+    return response.content
+
+
+def prepare_suite(
+    suite: str,
+    split: str,
+    output: Path,
+    *,
+    client: httpx.Client | None = None,
+    source_root: Path | None = None,
+    allow_test: bool = False,
+    limit: int | None = None,
+) -> dict:
+    if suite not in SUITES:
+        raise ValueError(f"unknown suite: {suite}")
+    if split not in ("development", "test"):
+        raise ValueError("only development and test are evaluation partitions")
+    if split == "test" and not allow_test:
+        raise ValueError(
+            "locked test requires --allow-test; use development for iteration"
+        )
+    if limit is not None and limit < 1:
+        raise ValueError("--limit must be positive")
+    spec = SUITES[suite]
+    manifest_bytes = read_source(f"{spec.path}/manifest.json", client, source_root)
+    require_sha256(manifest_bytes, spec.manifest_sha256, f"{suite}/manifest.json")
+    upstream = json.loads(manifest_bytes)
+    expected_file = upstream["files"][f"{split}.jsonl"]
+    raw_bytes = read_source(f"{spec.path}/{split}.jsonl", client, source_root)
+    require_sha256(raw_bytes, expected_file["sha256"], f"{suite}/{split}.jsonl")
+    records = parse_partition(raw_bytes, suite, split)
+    counts = population_counts(records)
+    if counts["records"] != expected_file["records"]:
+        raise ValueError(f"upstream record count mismatch: {suite}/{split}")
+    if (
+        "questions" in expected_file
+        and counts["questions"] != expected_file["questions"]
+    ):
+        raise ValueError(f"upstream question count mismatch: {suite}/{split}")
+    if not records:
+        raise ValueError(
+            f"{suite}/{split} is empty; this suite may be development-only"
+        )
+    selected = records if limit is None else records[:limit]
+    for row in selected:
+        SystemOneRequest.model_validate({"model": "validation-only", **row["record"]})
+    serialized = (
+        "".join(
+            json.dumps(row, ensure_ascii=False, allow_nan=False) + "\n"
+            for row in selected
+        )
+    ).encode("utf-8")
+    directory = output / suite
+    data_path = directory / f"{split}.jsonl"
+    manifest_path = directory / f"{split}.manifest.json"
+    upstream_path = directory / "upstream.manifest.json"
+    manifest = {
+        "schema_version": 1,
+        "suite": suite,
+        "split": split,
+        "description": spec.description,
+        "data_file": data_path.name,
+        "data_sha256": sha256(serialized),
+        "upstream": {
+            "repository": KEV_REPOSITORY,
+            "commit": KEV_COMMIT,
+            "path": f"{spec.path}/{split}.jsonl",
+            "url": f"{KEV_RAW}/{spec.path}/{split}.jsonl",
+            "sha256": sha256(raw_bytes),
+            "manifest_sha256": spec.manifest_sha256,
+        },
+        "full_partition": counts,
+        "selected": population_counts(selected),
+        "selection": {
+            "method": "full" if len(selected) == len(records) else "prefix",
+            "requested_limit": limit,
+            "is_full_partition": len(selected) == len(records),
+            "note": (
+                "Full frozen partition."
+                if len(selected) == len(records)
+                else "Smoke subset only; not a full-suite result. Pairs may be incomplete."
+            ),
+        },
+        "protocol": {
+            "calibration_applied": False,
+            "training_data_downloaded": False,
+            "gold_labels_sent_to_model": False,
+            "headline_variant": "clean",
+            "exclude_from_headline_sources": ["unknowable"],
+            "locked_test": split == "test",
+            "notes": list(spec.notes),
+            "upstream_context_policy": upstream.get("context"),
+        },
+        "provenance": source_provenance(records, upstream),
+    }
+    manifest_payload = (
+        json.dumps(manifest, ensure_ascii=False, indent=2) + "\n"
+    ).encode()
+    # Idempotent preparation is safe; refuse to replace any changed frozen artifact.
+    for path, content in (
+        (data_path, serialized),
+        (manifest_path, manifest_payload),
+        (upstream_path, manifest_bytes),
+    ):
+        if path.exists() and path.read_bytes() != content:
+            raise FileExistsError(
+                f"refusing to overwrite a different prepared artifact: {path}"
+            )
+    directory.mkdir(parents=True, exist_ok=True)
+    data_path.write_bytes(serialized)
+    manifest_path.write_bytes(manifest_payload)
+    upstream_path.write_bytes(manifest_bytes)
+    return manifest
+
+
+def main(argv: list[str] | None = None) -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument(
+        "--suite", action="append", choices=tuple(SUITES) + tuple(JEVBENCH_SUITES)
+    )
+    parser.add_argument(
+        "--split", choices=("development", "test"), default="development"
+    )
+    parser.add_argument("--allow-test", action="store_true")
+    parser.add_argument(
+        "--limit", type=int, help="deterministic prefix smoke subset per suite"
+    )
+    parser.add_argument(
+        "--source-root", type=Path, help="offline KEV checkout; same SHA256 checks"
+    )
+    parser.add_argument(
+        "--jevbench-source-root", type=Path, help="offline pinned JevBench checkout"
+    )
+    args = parser.parse_args(argv)
+    if args.split == "test" and not args.allow_test:
+        parser.error("--split test requires --allow-test")
+    if args.limit is not None and args.limit < 1:
+        parser.error("--limit must be positive")
+    suites = args.suite or (
+        DEFAULT_SUITES if args.split == "test" else (*DEFAULT_SUITES, *JEVBENCH_SUITES)
+    )
+    if args.split == "test" and any(suite in JEVBENCH_SUITES for suite in suites):
+        parser.error("JevBench supplies a public subset, not a locked test split")
+    with httpx.Client(timeout=60, follow_redirects=True) as client:
+        for suite in dict.fromkeys(suites):
+            if suite in JEVBENCH_SUITES:
+                manifest = prepare_jevbench(
+                    suite,
+                    args.output,
+                    client=client,
+                    source_root=args.jevbench_source_root,
+                    limit=args.limit,
+                )
+                print(
+                    json.dumps(
+                        {
+                            "suite": suite,
+                            "split": "public",
+                            "selected": manifest["selected"],
+                            "is_full_partition": manifest["selection"][
+                                "is_full_partition"
+                            ],
+                        }
+                    )
+                )
+                continue
+            manifest = prepare_suite(
+                suite,
+                args.split,
+                args.output,
+                client=client,
+                source_root=args.source_root,
+                allow_test=args.allow_test,
+                limit=args.limit,
+            )
+            print(
+                json.dumps(
+                    {
+                        "suite": suite,
+                        "split": args.split,
+                        "data": str(args.output / suite / f"{args.split}.jsonl"),
+                        "selected": manifest["selected"],
+                        "is_full_partition": manifest["selection"]["is_full_partition"],
+                    }
+                )
+            )
+
+
+if __name__ == "__main__":
+    main()

server/jev_adapter/benchmarks/run.py

@@ -0,0 +1,441 @@
+"""Evaluate frozen decisions against a running, untrained-model adapter.
+
+No retries, generation, calibration, truncation, or training. Latency includes
+adapter HTTP, tokenization, engine scheduling and forward work; it is NOT CUDA
+kernel time. Model load and warmup are excluded. Failed runs have no headline.
+"""
+
+import argparse
+import asyncio
+import hashlib
+import json
+import math
+import os
+import platform
+import random
+import sys
+import time
+from collections import defaultdict
+from datetime import UTC, datetime
+from pathlib import Path
+
+import httpx
+
+from jev_adapter.protocol import SystemOneRequest
+
+from .metrics import latency_summary, prediction_rows, summarize
+
+
+def digest(data):
+    return hashlib.sha256(data).hexdigest()
+
+
+def write_json(path, value):
+    path.write_text(
+        json.dumps(value, ensure_ascii=False, indent=2, allow_nan=False) + "\n"
+    )
+
+
+def load_data(paths, model, allow_test=False, limit=None):
+    items, seen, sources = [], set(), []
+    for path in paths:
+        raw = path.read_bytes()
+        loaded = [json.loads(line) for line in raw.splitlines() if line.strip()]
+        source = {
+            "path": str(path.resolve()),
+            "sha256": digest(raw),
+            "records": len(loaded),
+        }
+        manifest_path = path.with_suffix(".manifest.json")
+        if manifest_path.exists():
+            prepared = json.loads(manifest_path.read_text())
+            if prepared.get("data_sha256") != digest(raw):
+                raise ValueError(f"prepared dataset checksum mismatch: {path}")
+            if prepared.get("selected", {}).get("records") != len(loaded):
+                raise ValueError(f"prepared dataset count mismatch: {path}")
+            source["prepared_manifest"] = prepared
+        sources.append(source)
+        for item in loaded:
+            key = item["suite"], item["split"], item["id"]
+            if key in seen:
+                raise ValueError(f"duplicate input record: {key}")
+            seen.add(key)
+            if item["split"] == "test" and not allow_test:
+                raise ValueError(
+                    "test data requires --allow-test; use development first"
+                )
+            request = SystemOneRequest.model_validate(
+                {**item["record"], "model": model}
+            )
+            if set(request.questions) != set(item["expected"]):
+                raise ValueError(f"question/label mismatch: {key}")
+            for qid, expected in item["expected"].items():
+                labels, target, label = (
+                    expected["labels"],
+                    expected["target"],
+                    expected["label"],
+                )
+                if (
+                    len(labels) < 2
+                    or len(labels) != len(set(labels))
+                    or type(label) is not int
+                    or not 0 <= label < len(labels)
+                    or target != [int(i == label) for i in range(len(labels))]
+                ):
+                    raise ValueError(
+                        "this runner requires valid one-hot objective labels"
+                    )
+                question = request.questions[qid]
+                actual = (
+                    list(question.criteria)
+                    if question.type == "choice"
+                    else ["false", "true"]
+                    if question.type == "noul"
+                    else [str(i) for i in range(len(question.criteria))]
+                )
+                if set(labels) != set(actual) or expected["type"] != question.type:
+                    raise ValueError(
+                        f"expected labels/type differ from request: {key}/{qid}"
+                    )
+                reference = expected.get("reference_probs")
+                if reference is not None and (
+                    not isinstance(reference, list)
+                    or len(reference) != len(labels)
+                    or any(
+                        type(p) not in (int, float)
+                        or not math.isfinite(p)
+                        or not 0 <= p <= 1
+                        for p in reference
+                    )
+                    or abs(math.fsum(reference) - 1) > 1e-8
+                ):
+                    raise ValueError("invalid reference probability distribution")
+            items.append(item)
+    if limit is not None:
+        items = items[:limit]
+    if not items:
+        raise ValueError("no input records")
+    return items, sources
+
+
+async def engine_snapshot(client, base_url, cache_mode):
+    headers = {}
+    if key := os.environ.get("SGLANG_API_KEY"):
+        headers["Authorization"] = f"Bearer {key}"
+    results = {}
+    allowed = {
+        "model_path",
+        "model_type",
+        "architectures",
+        "served_model_name",
+        "revision",
+        "tokenizer_revision",
+        "dtype",
+        "quantization",
+        "tp_size",
+        "tp",
+        "version",
+        "context_length",
+        "max_total_tokens",
+        "max_running_requests",
+        "chunked_prefill_size",
+        "mem_fraction_static",
+        "disable_radix_cache",
+        "mm_preprocess_cache_size_mb",
+        "enable_prefix_mm_cache",
+        "enable_mm_global_cache",
+        "speculative_algorithm",
+        "is_generation",
+        "has_image_understanding",
+        "attention_backend",
+        "moe_runner_backend",
+    }
+    for name in ("model_info", "server_info"):
+        response = await client.get(base_url.rstrip("/") + "/" + name, headers=headers)
+        if response.status_code == 404:
+            response = await client.get(
+                base_url.rstrip("/") + "/get_" + name, headers=headers
+            )
+        response.raise_for_status()
+        value = response.json()
+        if not isinstance(value, dict):
+            raise TypeError(f"invalid engine {name}")
+        results[name] = {k: v for k, v in value.items() if k in allowed}
+    config = results["server_info"]
+    if config.get("speculative_algorithm") is not None:
+        raise ValueError("disable speculative decoding for this benchmark")
+    if cache_mode == "full-prefill":
+        if config.get("disable_radix_cache") is not True:
+            raise ValueError("full-prefill requires engine --disable-radix-cache")
+        if config.get("mm_preprocess_cache_size_mb") != 0:
+            raise ValueError("full-prefill requires --mm-preprocess-cache-size-mb 0")
+        if config.get("enable_prefix_mm_cache") or config.get("enable_mm_global_cache"):
+            raise ValueError("full-prefill requires multimodal feature caches disabled")
+    return results
+
+
+def checked_usage(response):
+    usage, meta = response.get("usage", {}), response.get("metadata", {})
+    if type(usage.get("output_tokens")) is not int or usage["output_tokens"] != 0:
+        raise ValueError("engine must return exactly zero output tokens")
+    if type(usage.get("input_tokens")) is not int or usage["input_tokens"] < 1:
+        raise ValueError("missing or invalid input-token accounting")
+    if type(meta.get("evaluations")) is not int or meta["evaluations"] < 1:
+        raise ValueError("missing forward evaluation count")
+    elapsed = meta.get("adapter_elapsed_ms")
+    if type(elapsed) not in (int, float) or not math.isfinite(elapsed) or elapsed < 0:
+        raise ValueError("missing or invalid adapter elapsed time")
+    if not isinstance(response.get("model"), str) or not response["model"]:
+        raise ValueError("missing served model identity")
+    return usage, meta
+
+
+def validate_launch(snapshot, launch):
+    model, engine = launch["model"], launch["engine"]
+    server, info = snapshot["server_info"], snapshot["model_info"]
+    for field, expected in {
+        "model_path": model["repo_id"],
+        "revision": model["revision"],
+        "dtype": model["dtype"],
+        "quantization": model["quantization"],
+    }.items():
+        if server.get(field) != expected:
+            raise ValueError(f"launch manifest {field} differs from running engine")
+    if info.get("model_path") != model["repo_id"]:
+        raise ValueError("launch manifest model differs from running engine model_info")
+    if not engine.get("revision") or not launch.get("profile"):
+        raise ValueError("launch manifest lacks pinned engine/profile identity")
+
+
+async def evaluate(args, client=None):
+    items, inputs = load_data(args.data, args.model, args.allow_test, args.limit)
+    args.output.mkdir(parents=True, exist_ok=False)
+    manifest = {
+        "created_at": datetime.now(UTC).isoformat(),
+        "status": "started",
+        "python": sys.version,
+        "platform": platform.platform(),
+        "requested_model": args.model,
+        "base_url": args.base_url,
+        "engine_url": args.engine_url,
+        "data": inputs,
+        "limit": args.limit,
+        "concurrency": args.concurrency,
+        "warmup_requests": args.warmup,
+        "repeats": args.repeats,
+        "seed": args.seed,
+        "cache_mode": args.cache_mode,
+        "options": {"temperature": 1.0, "permutations": 1, "enable_thinking": False},
+        "latency_scope": "client HTTP wall time, excluding queue before dispatch; NOT GPU kernel time",
+        "quality_repeats": "first measured repetition only",
+        "assistant_prefix": args.assistant_prefix,
+        "code_sha256": {
+            str(p.relative_to(Path(__file__).parents[1])): digest(p.read_bytes())
+            for p in Path(__file__).parents[1].rglob("*.py")
+        },
+    }
+    write_json(args.output / "manifest.json", manifest)
+    owned = client is None
+    if owned:
+        client = httpx.AsyncClient(
+            timeout=args.timeout, limits=httpx.Limits(max_connections=args.concurrency)
+        )
+    headers = {}
+    if key := os.environ.get("JEV_API_KEY"):
+        headers["Authorization"] = f"Bearer {key}"
+    results, failures = [], []
+    try:
+        manifest["engine"] = await engine_snapshot(
+            client, args.engine_url, args.cache_mode
+        )
+        expected_served = manifest["engine"]["model_info"].get("served_model_name")
+        if not expected_served:
+            raise ValueError("engine did not identify its served model")
+        if args.engine_manifest:
+            manifest["launch_manifest"] = json.loads(args.engine_manifest.read_text())
+            validate_launch(manifest["engine"], manifest["launch_manifest"])
+        write_json(args.output / "manifest.json", manifest)
+
+        async def call(item):
+            request = {
+                **item["record"],
+                "model": args.model,
+                "options": {
+                    "temperature": 1.0,
+                    "permutations": 1,
+                    "return_logprobs": True,
+                },
+            }
+            if args.assistant_prefix is not None:
+                request["options"]["assistant_prefix"] = args.assistant_prefix
+            started = time.perf_counter()
+            response = await client.post(
+                args.base_url.rstrip("/") + "/v1/systemone",
+                headers=headers,
+                json=request,
+            )
+            response.raise_for_status()
+            body = response.json()
+            elapsed = (time.perf_counter() - started) * 1000
+            usage, meta = checked_usage(body)
+            if body["model"] != expected_served:
+                raise ValueError("adapter response model differs from engine snapshot")
+            if meta["evaluations"] != len(item["record"]["questions"]):
+                raise ValueError("expected one forward evaluation per question")
+            rows = prediction_rows(item, body)
+            return {
+                "request_sha256": digest(json.dumps(request, sort_keys=True).encode()),
+                "id": item["id"],
+                "suite": item["suite"],
+                "split": item["split"],
+                "latency_ms": elapsed,
+                "adapter_ms": meta["adapter_elapsed_ms"],
+                "input_tokens": usage["input_tokens"],
+                "evaluations": meta["evaluations"],
+                "served_model": body["model"],
+                "response": body,
+                "rows": rows,
+            }
+
+        for i in range(args.warmup):
+            # Span the full input so warmup includes more than the first task.
+            item = items[(i * len(items) // max(args.warmup, 1)) % len(items)]
+            print(
+                f"Warmup {i + 1}/{args.warmup}: {item['suite']} {item['id']}",
+                flush=True,
+            )
+            await call(item)
+        jobs = [
+            (i, repeat) for repeat in range(args.repeats) for i in range(len(items))
+        ]
+        random.Random(args.seed).shuffle(jobs)
+        semaphore = asyncio.Semaphore(args.concurrency)
+        started = time.perf_counter()
+        with (args.output / "predictions.jsonl").open("w") as handle:
+
+            async def job(index, repeat):
+                async with semaphore:
+                    try:
+                        result = await call(items[index])
+                        result.update(index=index, repeat=repeat)
+                        results.append(result)
+                        handle.write(
+                            json.dumps(result, ensure_ascii=False, allow_nan=False)
+                            + "\n"
+                        )
+                        handle.flush()
+                    except (httpx.HTTPError, ValueError, KeyError, TypeError) as error:
+                        failures.append(
+                            {
+                                "index": index,
+                                "repeat": repeat,
+                                "id": items[index]["id"],
+                                "suite": items[index]["suite"],
+                                "error_type": type(error).__name__,
+                                "error": str(error),
+                            }
+                        )
+                    done = len(results) + len(failures)
+                    if done % 50 == 0 or done == len(jobs):
+                        print(
+                            f"{done}/{len(jobs)} requests; errors={len(failures)}",
+                            flush=True,
+                        )
+
+            await asyncio.gather(*(job(index, repeat) for index, repeat in jobs))
+        elapsed = time.perf_counter() - started
+        served = {r["served_model"] for r in results}
+        if len(served) > 1:
+            failures.append({"error": "served model changed during run"})
+        report = {
+            "status": "failed" if failures else "complete",
+            "requested_records": len(items),
+            "requested_questions": sum(len(i["expected"]) for i in items),
+            "attempted_requests": len(jobs),
+            "successful_requests": len(results),
+            "errors": len(failures),
+            "served_models": sorted(served),
+            "elapsed_measurement_s": elapsed,
+            "requests_per_second": len(results) / elapsed,
+            "questions_per_second": sum(r["evaluations"] for r in results) / elapsed,
+            "latency_ms": latency_summary([r["latency_ms"] for r in results]),
+            "adapter_ms": latency_summary([r["adapter_ms"] for r in results]),
+            "input_tokens_per_request": latency_summary(
+                [r["input_tokens"] for r in results]
+            ),
+            "suites": {},
+            "latency_scope": manifest["latency_scope"],
+            "partial_dataset": args.limit is not None
+            or any(
+                not source.get("prepared_manifest", {})
+                .get("selection", {})
+                .get("is_full_partition", False)
+                for source in inputs
+            ),
+        }
+        if not failures:
+            grouped, times = defaultdict(list), defaultdict(list)
+            for result in sorted(results, key=lambda r: (r["index"], r["repeat"])):
+                key = result["suite"] + "/" + result["split"]
+                times[key].append(result["latency_ms"])
+                if result["repeat"] == 0:
+                    grouped[key].extend(result["rows"])
+            report["suites"] = {
+                key: {**summarize(rows), "latency_ms": latency_summary(times[key])}
+                for key, rows in grouped.items()
+            }
+        write_json(args.output / "report.json", report)
+        if failures:
+            write_json(args.output / "failures.json", failures)
+        manifest.update(status=report["status"], served_models=sorted(served))
+        write_json(args.output / "manifest.json", manifest)
+        return report
+    except BaseException as error:
+        manifest.update(
+            status="failed", failure_type=type(error).__name__, failure=str(error)
+        )
+        write_json(args.output / "manifest.json", manifest)
+        raise
+    finally:
+        if owned:
+            await client.aclose()
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--data", type=Path, action="append", required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument("--base-url", default="http://127.0.0.1:30120")
+    parser.add_argument("--engine-url", default="http://127.0.0.1:30000")
+    parser.add_argument("--engine-manifest", type=Path)
+    parser.add_argument("--model", default="decision-model")
+    parser.add_argument("--concurrency", type=int, default=1)
+    parser.add_argument("--warmup", type=int, default=20)
+    parser.add_argument("--repeats", type=int, default=1)
+    parser.add_argument("--seed", type=int, default=42)
+    parser.add_argument("--limit", type=int)
+    parser.add_argument("--timeout", type=float, default=120)
+    parser.add_argument("--assistant-prefix")
+    parser.add_argument("--allow-test", action="store_true")
+    parser.add_argument(
+        "--cache-mode",
+        choices=["full-prefill", "server-default"],
+        default="full-prefill",
+    )
+    args = parser.parse_args()
+    if (
+        args.concurrency < 1
+        or args.repeats < 1
+        or args.warmup < 0
+        or (args.limit is not None and args.limit < 1)
+        or not math.isfinite(args.timeout)
+        or args.timeout <= 0
+    ):
+        parser.error("counts and timeout must be positive; warmup may be zero")
+    report = asyncio.run(evaluate(args))
+    print(json.dumps(report, ensure_ascii=False, indent=2))
+    raise SystemExit(0 if report["status"] == "complete" else 1)
+
+
+if __name__ == "__main__":
+    main()

server/tests/test_benchmark_data.py

@@ -0,0 +1,220 @@
+"""Check benchmark integrity and prevent labels from entering inference requests."""
+
+import copy
+import json
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+import httpx
+
+from jev_adapter.benchmarks.data import (
+    SuiteSpec,
+    normalize_record,
+    parse_partition,
+    population_counts,
+    sha256,
+)
+from jev_adapter.benchmarks.prepare import prepare_suite
+
+
+def fixture_record(name="example/1"):
+    return {
+        "state": {"message": "Needs a refund", "count": 4},
+        "questions": {
+            "queue": {
+                "type": "choice",
+                "instructions": "Choose the queue.",
+                "criteria": {"other": None, "billing": "Refunds"},
+                "label": "billing",
+                "src": "routing",
+            },
+            "urgent": {
+                "type": "noul",
+                "instructions": "Is the message urgent?",
+                "label": False,
+                "src": "urgency",
+            },
+            "priority": {
+                "type": "score",
+                "instructions": "Select priority.",
+                "criteria": ["low", "medium", "high"],
+                "label": 2,
+                "src": "priority",
+            },
+        },
+        "_meta": {
+            "id": name,
+            "source": "example",
+            "variant": "clean",
+            "group_id": "pair-1",
+            "pair_id": "pair-1",
+            "sibling": "a",
+        },
+    }
+
+
+def fake_source(records):
+    payload = ("".join(json.dumps(r) + "\n" for r in records)).encode()
+    manifest = json.dumps(
+        {
+            "files": {
+                name: {
+                    "sha256": sha256(payload),
+                    "records": len(records),
+                    "questions": sum(len(r["questions"]) for r in records),
+                }
+                for name in ("development.jsonl", "test.jsonl")
+            },
+            "dataset_revisions": {},
+        }
+    ).encode()
+    spec = SuiteSpec("evals/example", sha256(manifest), "Test fixture")
+    return spec, manifest, payload
+
+
+class TestBenchmarkNormalization(unittest.TestCase):
+    def test_all_question_types_preserve_order_and_hide_gold(self):
+        original = fixture_record()
+        before = copy.deepcopy(original)
+        row = normalize_record(original, "example", "development")
+        self.assertEqual(original, before)
+        self.assertEqual(
+            list(row["record"]["questions"]), ["queue", "urgent", "priority"]
+        )
+        self.assertNotIn("model", row["record"])
+        self.assertNotIn("_meta", row["record"])
+        for q in row["record"]["questions"].values():
+            self.assertNotIn("label", q)
+            self.assertNotIn("src", q)
+        self.assertEqual(row["expected"]["queue"]["labels"], ["other", "billing"])
+        self.assertEqual(row["expected"]["queue"]["target"], [0, 1])
+        self.assertEqual(row["expected"]["urgent"]["labels"], ["false", "true"])
+        self.assertEqual(row["expected"]["urgent"]["target"], [1, 0])
+        self.assertEqual(row["expected"]["priority"]["target"], [0, 0, 1])
+        self.assertIsNone(row["record"]["questions"]["queue"]["criteria"]["other"])
+        self.assertEqual(row["metadata"]["pair_id"], "pair-1")
+
+    def test_large_choice_space_is_not_truncated(self):
+        raw = fixture_record()
+        raw["questions"] = {
+            "intent": {
+                "type": "choice",
+                "instructions": "Which intent?",
+                "criteria": {f"intent_{i}": None for i in range(78)},
+                "label": "intent_77",
+            }
+        }
+        row = normalize_record(raw, "example", "development")
+        self.assertEqual(len(row["expected"]["intent"]["labels"]), 78)
+        self.assertEqual(row["expected"]["intent"]["label"], 77)
+
+    def test_duplicate_ids_and_malformed_targets_fail(self):
+        raw = fixture_record()
+        payload = (json.dumps(raw) + "\n") * 2
+        with self.assertRaisesRegex(ValueError, "duplicate record"):
+            parse_partition(payload.encode(), "example", "development")
+        for question, label in (
+            ("queue", "absent"),
+            ("urgent", "false"),
+            ("priority", 9),
+        ):
+            modified = copy.deepcopy(raw)
+            modified["questions"][question]["label"] = label
+            with self.subTest(question=question), self.assertRaises(ValueError):
+                normalize_record(modified, "example", "development")
+
+    def test_population_separates_perturbations_and_unknowable(self):
+        raws = [fixture_record(str(i)) for i in range(3)]
+        raws[1]["_meta"]["variant"] = "permuted"
+        raws[2]["_meta"]["source"] = "unknowable"
+        counts = population_counts(
+            [normalize_record(raw, "example", "development") for raw in raws]
+        )
+        self.assertEqual(counts["records"], 3)
+        self.assertEqual(counts["questions"], 9)
+        self.assertEqual(counts["clean_questions"], 6)
+        self.assertEqual(counts["headline_questions"], 3)
+
+
+class TestBenchmarkPreparation(unittest.TestCase):
+    def test_verified_download_subset_and_idempotency(self):
+        spec, manifest, payload = fake_source(
+            [fixture_record("one"), fixture_record("two")]
+        )
+        fetched = []
+
+        def transport(request):
+            fetched.append(request.url.path)
+            return httpx.Response(
+                200,
+                content=manifest
+                if request.url.path.endswith("manifest.json")
+                else payload,
+            )
+
+        with (
+            tempfile.TemporaryDirectory() as tmp,
+            patch.dict("jev_adapter.benchmarks.prepare.SUITES", {"example": spec}),
+            httpx.Client(transport=httpx.MockTransport(transport)) as client,
+        ):
+            out = Path(tmp)
+            result = prepare_suite(
+                "example", "development", out, client=client, limit=1
+            )
+            again = prepare_suite("example", "development", out, client=client, limit=1)
+            self.assertEqual(result, again)
+            self.assertFalse(result["selection"]["is_full_partition"])
+            self.assertEqual(result["full_partition"]["questions"], 6)
+            self.assertEqual(result["selected"]["questions"], 3)
+            self.assertEqual(
+                result["data_sha256"],
+                sha256((out / "example/development.jsonl").read_bytes()),
+            )
+            self.assertTrue(all("train" not in path for path in fetched))
+            with self.assertRaises(FileExistsError):
+                prepare_suite("example", "development", out, client=client)
+
+    def test_tampered_partition_rejected_without_writing(self):
+        spec, manifest, payload = fake_source([fixture_record()])
+
+        def transport(request):
+            return httpx.Response(
+                200,
+                content=manifest
+                if request.url.path.endswith("manifest.json")
+                else payload + b" ",
+            )
+
+        with (
+            tempfile.TemporaryDirectory() as tmp,
+            patch.dict("jev_adapter.benchmarks.prepare.SUITES", {"example": spec}),
+            httpx.Client(transport=httpx.MockTransport(transport)) as client,
+        ):
+            out = Path(tmp)
+            with self.assertRaisesRegex(ValueError, "SHA256 mismatch"):
+                prepare_suite("example", "development", out, client=client)
+            self.assertFalse((out / "example").exists())
+
+    def test_offline_source_has_same_integrity_and_test_gate(self):
+        spec, manifest, payload = fake_source([fixture_record()])
+        with (
+            tempfile.TemporaryDirectory() as tmp,
+            patch.dict("jev_adapter.benchmarks.prepare.SUITES", {"example": spec}),
+        ):
+            root = Path(tmp) / "source"
+            directory = root / "evals/example"
+            directory.mkdir(parents=True)
+            (directory / "manifest.json").write_bytes(manifest)
+            (directory / "test.jsonl").write_bytes(payload)
+            out = Path(tmp) / "out"
+            with self.assertRaisesRegex(ValueError, "locked test"):
+                prepare_suite("example", "test", out, source_root=root)
+            result = prepare_suite(
+                "example", "test", out, source_root=root, allow_test=True
+            )
+            self.assertTrue(result["protocol"]["locked_test"])
+            (directory / "manifest.json").write_bytes(manifest + b" ")
+            with self.assertRaisesRegex(ValueError, "SHA256 mismatch"):
+                prepare_suite("example", "test", out, source_root=root, allow_test=True)

server/tests/test_disconnect_cleanup.py

@@ -0,0 +1,150 @@
+"""A completed score must not wait forever for a cancelled disconnect poll."""
+
+import asyncio
+
+import pytest
+
+from jev_adapter.backend import AdapterError, ScoringResult
+from jev_adapter.protocol import SystemOneRequest
+from jev_adapter.server import create_app
+
+
+class ControlledBackend:
+    model = "test-model"
+
+    def __init__(self):
+        self.release = asyncio.Event()
+        self.entered = asyncio.Event()
+        self.cancelled = asyncio.Event()
+
+    def labels(self, count):
+        return tuple("AB"[:count]), tuple(range(count))
+
+    async def evaluate(self, prompt, images, labels, token_ids, assistant_prefix):
+        self.entered.set()
+        try:
+            await self.release.wait()
+        except asyncio.CancelledError:
+            self.cancelled.set()
+            raise
+        return ScoringResult((-0.1, -2.0), 10)
+
+
+class CancellationSwallowingRequest:
+    """Reproduce a request probe whose internal cancellation scope wins a race.
+
+    CancelledError can be suppressed within a probe, leaving its caller alive.
+    A real Request.is_disconnected uses a self-cancelling AnyIO CancelScope;
+    deliberately control that boundary here rather than rely on scheduler luck.
+    """
+
+    def __init__(self):
+        self.entered = asyncio.Event()
+        self.swallowed = asyncio.Event()
+        self.probe_task = None
+        self.calls = 0
+
+    async def is_disconnected(self):
+        self.probe_task = asyncio.current_task()
+        self.calls += 1
+        self.entered.set()
+        try:
+            await asyncio.Future()
+        except asyncio.CancelledError:
+            self.swallowed.set()
+            return False
+
+
+def endpoint_and_body(backend):
+    app = create_app(backend)
+    endpoint = next(
+        route.endpoint
+        for route in app.routes
+        if getattr(route, "path", None) == "/v1/systemone"
+    )
+    body = SystemOneRequest.model_validate(
+        {
+            "model": backend.model,
+            "state": "A billing question.",
+            "questions": {
+                "route": {
+                    "type": "choice",
+                    "instructions": "Choose the team.",
+                    "criteria": {"billing": None, "technical": None},
+                }
+            },
+        }
+    )
+    return endpoint, body
+
+
+@pytest.mark.asyncio
+async def test_completed_score_returns_when_disconnect_probe_swallows_cancel():
+    backend = ControlledBackend()
+    endpoint, body = endpoint_and_body(backend)
+    request = CancellationSwallowingRequest()
+    response = asyncio.create_task(endpoint(body, request))
+    try:
+        await asyncio.wait_for(request.entered.wait(), timeout=1)
+        await asyncio.wait_for(backend.entered.wait(), timeout=1)
+        backend.release.set()
+        await asyncio.wait_for(request.swallowed.wait(), timeout=1)
+        done, _ = await asyncio.wait({response}, timeout=0.1)
+        assert response in done, "completed inference is stuck cleaning up its watcher"
+        result = response.result()
+        assert result["answers"]["route"]["choice"] == "billing"
+        assert request.calls == 1
+        assert request.probe_task.done()
+    finally:
+        # Do not let a deliberately cancellation-resistant fake leak after red.
+        request.is_disconnected = disconnected_now
+        if request.probe_task is not None:
+            request.probe_task.cancel()
+        response.cancel()
+        await asyncio.gather(response, return_exceptions=True)
+
+
+async def disconnected_now():
+    return True
+
+
+@pytest.mark.asyncio
+async def test_client_disconnect_still_cancels_unfinished_inference():
+    backend = ControlledBackend()
+    endpoint, body = endpoint_and_body(backend)
+
+    class DisconnectedRequest:
+        async def is_disconnected(self):
+            await backend.entered.wait()
+            return True
+
+    with pytest.raises(AdapterError) as error:
+        await asyncio.wait_for(endpoint(body, DisconnectedRequest()), timeout=1)
+    assert error.value.code == "client_disconnected"
+    assert error.value.status == 499
+    assert backend.cancelled.is_set()
+
+
+@pytest.mark.asyncio
+async def test_cancelled_handler_joins_work_and_cancellation_resistant_watcher():
+    backend = ControlledBackend()
+    endpoint, body = endpoint_and_body(backend)
+    request = CancellationSwallowingRequest()
+    response = asyncio.create_task(endpoint(body, request))
+    try:
+        await asyncio.wait_for(request.entered.wait(), timeout=1)
+        await asyncio.wait_for(backend.entered.wait(), timeout=1)
+        response.cancel()
+        done, _ = await asyncio.wait({response}, timeout=0.1)
+        assert response in done, "cancelled handler did not release its child tasks"
+        with pytest.raises(asyncio.CancelledError):
+            response.result()
+        assert backend.cancelled.is_set()
+        assert request.swallowed.is_set()
+        assert request.probe_task.done()
+    finally:
+        request.is_disconnected = disconnected_now
+        if request.probe_task is not None:
+            request.probe_task.cancel()
+        response.cancel()
+        await asyncio.gather(response, return_exceptions=True)

server/tests/test_main.py

@@ -0,0 +1,368 @@
+"""CLI wiring tests for --prompt-wording (argparse choices/env default, and
+plumbing into SGLangBackend + create_app). uvicorn.run is monkeypatched so
+main() never actually binds a socket; SGLangBackend/create_app/NativeTokenizer
+are monkeypatched to capture their constructor arguments instead of talking to
+a real engine or loading real tokenizer files (transformers is an optional,
+not-installed-here dependency for this default venv)."""
+
+import sys
+
+import pytest
+
+import jev_adapter.__main__ as main_module
+import jev_adapter.native_tokenizer as native_tokenizer_module
+import jev_adapter.server as server_module
+import jev_adapter.sglang as sglang_module
+
+
+class Capture:
+    def __init__(self):
+        self.backend_kwargs = None
+        self.app_kwargs = None
+
+    def fake_backend(self, *args, **kwargs):
+        self.backend_kwargs = kwargs
+        return object()
+
+    def fake_create_app(self, backend, **kwargs):
+        self.app_kwargs = kwargs
+        return object()
+
+
+@pytest.fixture
+def capture(monkeypatch):
+    capture = Capture()
+    monkeypatch.setattr(sglang_module, "SGLangBackend", capture.fake_backend)
+    monkeypatch.setattr(server_module, "create_app", capture.fake_create_app)
+    monkeypatch.setattr("uvicorn.run", lambda app, **kwargs: None)
+    return capture
+
+
+def run_main(monkeypatch, argv, env=None):
+    monkeypatch.setattr(sys, "argv", ["jev-adapter", *argv])
+    for key in (
+        "JEV_PROMPT_WORDING",
+        "JEV_DEFAULT_TEMPERATURE",
+        "JEV_NATIVE_SYSTEM_PROMPT",
+    ):
+        monkeypatch.delenv(key, raising=False)
+    for key, value in (env or {}).items():
+        monkeypatch.setenv(key, value)
+    main_module.main()
+
+
+def test_prompt_wording_defaults_to_served(monkeypatch, capture):
+    run_main(monkeypatch, ["--model", "decision-model"])
+    assert capture.backend_kwargs["prompt_wording"] == "served"
+    assert capture.backend_kwargs["native_system_prompt"] is None
+    assert capture.app_kwargs["prompt_wording"] == "served"
+
+
+def test_prompt_wording_flag_selects_native(monkeypatch, capture):
+    run_main(monkeypatch, ["--model", "decision-model", "--prompt-wording", "native"])
+    assert capture.backend_kwargs["prompt_wording"] == "native"
+    assert capture.app_kwargs["prompt_wording"] == "native"
+
+
+def test_prompt_wording_env_var_sets_the_default(monkeypatch, capture):
+    run_main(
+        monkeypatch,
+        ["--model", "decision-model"],
+        env={"JEV_PROMPT_WORDING": "native"},
+    )
+    assert capture.backend_kwargs["prompt_wording"] == "native"
+    assert capture.app_kwargs["prompt_wording"] == "native"
+
+
+def test_explicit_flag_overrides_env_var(monkeypatch, capture):
+    run_main(
+        monkeypatch,
+        ["--model", "decision-model", "--prompt-wording", "served"],
+        env={"JEV_PROMPT_WORDING": "native"},
+    )
+    assert capture.backend_kwargs["prompt_wording"] == "served"
+
+
+def test_invalid_prompt_wording_choice_rejected(monkeypatch, capture):
+    monkeypatch.setattr(sys, "argv", ["jev-adapter", "--model", "m", "--prompt-wording", "bogus"])
+    with pytest.raises(SystemExit):
+        main_module.main()
+
+
+def test_native_wording_without_tokenizer_model_serves_text_only_and_warns(
+    monkeypatch, capture, caplog
+):
+    with caplog.at_level("WARNING"):
+        run_main(monkeypatch, ["--model", "decision-model", "--prompt-wording", "native"])
+    assert capture.backend_kwargs["native_system_prompt"] is None
+    assert any("native" in record.message for record in caplog.records)
+
+
+def test_native_wording_with_tokenizer_model_extracts_the_system_prompt(
+    monkeypatch, capture
+):
+    calls = []
+
+    class FakeNativeTokenizer:
+        @classmethod
+        def from_pretrained(cls, model, revision):
+            calls.append(("from_pretrained", model, revision))
+            return object()
+
+        @classmethod
+        def native_default_system_prompt(cls, model, revision):
+            calls.append(("native_default_system_prompt", model, revision))
+            return "Default system text."
+
+    monkeypatch.setattr(native_tokenizer_module, "NativeTokenizer", FakeNativeTokenizer)
+    run_main(
+        monkeypatch,
+        [
+            "--model",
+            "decision-model",
+            "--prompt-wording",
+            "native",
+            "--tokenizer-model",
+            "mistralai/Ministral-3-8B-Instruct-2512-BF16",
+            "--tokenizer-revision",
+            "f" * 40,
+        ],
+    )
+    assert calls == [
+        ("from_pretrained", "mistralai/Ministral-3-8B-Instruct-2512-BF16", "f" * 40),
+        (
+            "native_default_system_prompt",
+            "mistralai/Ministral-3-8B-Instruct-2512-BF16",
+            "f" * 40,
+        ),
+    ]
+    assert capture.backend_kwargs["native_system_prompt"] == "Default system text."
+
+
+def test_served_wording_with_tokenizer_model_never_extracts_a_system_prompt(
+    monkeypatch, capture
+):
+    calls = []
+
+    class FakeNativeTokenizer:
+        @classmethod
+        def from_pretrained(cls, model, revision):
+            calls.append("from_pretrained")
+            return object()
+
+        @classmethod
+        def native_default_system_prompt(cls, model, revision):
+            calls.append("native_default_system_prompt")
+            return "should not be reached"
+
+    monkeypatch.setattr(native_tokenizer_module, "NativeTokenizer", FakeNativeTokenizer)
+    run_main(
+        monkeypatch,
+        [
+            "--model",
+            "decision-model",
+            "--tokenizer-model",
+            "org/model",
+            "--tokenizer-revision",
+            "a" * 40,
+        ],
+    )
+    assert calls == ["from_pretrained"]
+    assert capture.backend_kwargs["prompt_wording"] == "served"
+    assert capture.backend_kwargs["native_system_prompt"] is None
+
+
+def test_native_system_prompt_defaults_to_auto(monkeypatch, capture):
+    calls = []
+
+    class FakeNativeTokenizer:
+        @classmethod
+        def from_pretrained(cls, model, revision):
+            return object()
+
+        @classmethod
+        def native_default_system_prompt(cls, model, revision):
+            calls.append((model, revision))
+            return "Default system text."
+
+    monkeypatch.setattr(native_tokenizer_module, "NativeTokenizer", FakeNativeTokenizer)
+    run_main(
+        monkeypatch,
+        [
+            "--model",
+            "decision-model",
+            "--prompt-wording",
+            "native",
+            "--tokenizer-model",
+            "org/model",
+            "--tokenizer-revision",
+            "a" * 40,
+        ],
+    )
+    assert calls == [("org/model", "a" * 40)]
+    assert capture.backend_kwargs["native_system_prompt"] == "Default system text."
+
+
+def test_native_system_prompt_none_skips_extraction_and_warning(
+    monkeypatch, capture, caplog
+):
+    calls = []
+
+    class FakeNativeTokenizer:
+        @classmethod
+        def from_pretrained(cls, model, revision):
+            return object()
+
+        @classmethod
+        def native_default_system_prompt(cls, model, revision):
+            calls.append((model, revision))
+            return "should not be reached"
+
+    monkeypatch.setattr(native_tokenizer_module, "NativeTokenizer", FakeNativeTokenizer)
+    with caplog.at_level("WARNING"):
+        run_main(
+            monkeypatch,
+            [
+                "--model",
+                "decision-model",
+                "--prompt-wording",
+                "native",
+                "--tokenizer-model",
+                "org/model",
+                "--tokenizer-revision",
+                "a" * 40,
+                "--native-system-prompt",
+                "none",
+            ],
+        )
+    assert calls == []
+    assert capture.backend_kwargs["native_system_prompt"] is None
+    assert not any("native" in record.message for record in caplog.records)
+
+
+def test_native_system_prompt_none_without_tokenizer_never_warns(
+    monkeypatch, capture, caplog
+):
+    with caplog.at_level("WARNING"):
+        run_main(
+            monkeypatch,
+            [
+                "--model",
+                "decision-model",
+                "--prompt-wording",
+                "native",
+                "--native-system-prompt",
+                "none",
+            ],
+        )
+    assert capture.backend_kwargs["native_system_prompt"] is None
+    assert caplog.records == []
+
+
+def test_native_system_prompt_env_var_sets_the_default(monkeypatch, capture, caplog):
+    with caplog.at_level("WARNING"):
+        run_main(
+            monkeypatch,
+            ["--model", "decision-model", "--prompt-wording", "native"],
+            env={"JEV_NATIVE_SYSTEM_PROMPT": "none"},
+        )
+    assert capture.backend_kwargs["native_system_prompt"] is None
+    assert caplog.records == []
+
+
+def test_native_system_prompt_explicit_flag_overrides_env_var(monkeypatch, capture):
+    calls = []
+
+    class FakeNativeTokenizer:
+        @classmethod
+        def from_pretrained(cls, model, revision):
+            return object()
+
+        @classmethod
+        def native_default_system_prompt(cls, model, revision):
+            calls.append((model, revision))
+            return "Default system text."
+
+    monkeypatch.setattr(native_tokenizer_module, "NativeTokenizer", FakeNativeTokenizer)
+    run_main(
+        monkeypatch,
+        [
+            "--model",
+            "decision-model",
+            "--prompt-wording",
+            "native",
+            "--tokenizer-model",
+            "org/model",
+            "--tokenizer-revision",
+            "a" * 40,
+            "--native-system-prompt",
+            "auto",
+        ],
+        env={"JEV_NATIVE_SYSTEM_PROMPT": "none"},
+    )
+    assert calls == [("org/model", "a" * 40)]
+    assert capture.backend_kwargs["native_system_prompt"] == "Default system text."
+
+
+def test_native_system_prompt_none_has_no_effect_on_served_wording(
+    monkeypatch, capture
+):
+    calls = []
+
+    class FakeNativeTokenizer:
+        @classmethod
+        def from_pretrained(cls, model, revision):
+            return object()
+
+        @classmethod
+        def native_default_system_prompt(cls, model, revision):
+            calls.append((model, revision))
+            return "should not be reached"
+
+    monkeypatch.setattr(native_tokenizer_module, "NativeTokenizer", FakeNativeTokenizer)
+    run_main(
+        monkeypatch,
+        [
+            "--model",
+            "decision-model",
+            "--tokenizer-model",
+            "org/model",
+            "--tokenizer-revision",
+            "a" * 40,
+            "--native-system-prompt",
+            "auto",
+        ],
+    )
+    assert calls == []
+    assert capture.backend_kwargs["prompt_wording"] == "served"
+    assert capture.backend_kwargs["native_system_prompt"] is None
+
+
+def test_invalid_native_system_prompt_choice_rejected(monkeypatch, capture):
+    monkeypatch.setattr(
+        sys,
+        "argv",
+        ["jev-adapter", "--model", "m", "--native-system-prompt", "bogus"],
+    )
+    with pytest.raises(SystemExit):
+        main_module.main()
+
+
+def test_native_system_prompt_startup_log_line(monkeypatch, capture, caplog):
+    with caplog.at_level("INFO"):
+        run_main(
+            monkeypatch,
+            [
+                "--model",
+                "decision-model",
+                "--prompt-wording",
+                "native",
+                "--native-system-prompt",
+                "none",
+            ],
+        )
+    info_records = [r for r in caplog.records if r.levelname == "INFO"]
+    assert len(info_records) == 1
+    message = info_records[0].getMessage()
+    assert "prompt_wording=native" in message
+    assert "native_system_prompt=none" in message

server/tests/test_service.py

@@ -0,0 +1,324 @@
+import asyncio
+import math
+import unittest
+
+from fastapi.testclient import TestClient
+
+from jev_adapter.__main__ import positive_temperature
+from jev_adapter.backend import AdapterError, ScoringResult
+from jev_adapter.protocol import SystemOneRequest, probabilities_from_logprobs
+from jev_adapter.server import create_app
+from jev_adapter.service import SystemOneService
+
+
+def payload():
+    return {
+        "model": "test-model",
+        "state": {"text": "결제가 두 번 되었어요. 환불해 주세요."},
+        "questions": {
+            "route": {
+                "type": "choice",
+                "instructions": "담당 부서",
+                "criteria": {"billing": "청구", "technical": "기술"},
+            },
+            "refund": {"type": "noul", "instructions": "환불 요청인가?"},
+            "urgency": {
+                "type": "score",
+                "instructions": "긴급도",
+                "criteria": ["낮음", "중간", "높음"],
+            },
+        },
+    }
+
+
+class FakeBackend:
+    model = "test-model"
+
+    def __init__(self, delay=0, failure=False):
+        self.calls = []
+        self.active = self.peak = self.cancelled = 0
+        self.started = self.closed = False
+        self.delay, self.failure = delay, failure
+
+    async def start(self):
+        self.started = True
+
+    async def close(self):
+        self.closed = True
+
+    def labels(self, count):
+        return tuple(chr(65 + i) for i in range(count)), tuple(range(65, 65 + count))
+
+    async def evaluate(self, prompt, images, labels, token_ids, assistant_prefix):
+        self.calls.append((prompt, images, labels, token_ids, assistant_prefix))
+        self.active += 1
+        self.peak = max(self.peak, self.active)
+        try:
+            if self.failure:
+                raise AdapterError(
+                    "backend_unavailable", "Engine unavailable.", status=502
+                )
+            await asyncio.sleep(self.delay)
+            return ScoringResult(
+                tuple(math.log(0.8 if i == 0 else 0.1) for i in range(len(labels))), 20
+            )
+        except asyncio.CancelledError:
+            self.cancelled += 1
+            raise
+        finally:
+            self.active -= 1
+
+
+class TestService(unittest.IsolatedAsyncioTestCase):
+    async def test_mixed_questions_keep_images_and_build_typed_response(self):
+        backend = FakeBackend()
+        body = payload()
+        body["images"] = ["data:image/png;base64,aGVsbG8="]
+        result = await SystemOneService(backend).score(
+            SystemOneRequest.model_validate(body)
+        )
+        self.assertEqual(result["answers"]["route"]["choice"], "billing")
+        self.assertAlmostEqual(result["answers"]["refund"]["noul"], 8 / 9)
+        self.assertAlmostEqual(result["answers"]["urgency"]["score"], 0.3)
+        self.assertEqual(result["usage"], {"input_tokens": 60, "output_tokens": 0})
+        self.assertEqual(result["metadata"]["evaluations"], 3)
+        for call in backend.calls:
+            self.assertEqual(call[1], body["images"])
+        self.assertNotIn("담당 부서", backend.calls[1][0])
+
+    async def test_rotations_preserve_canonical_option_mapping(self):
+        body = payload()
+        body["questions"] = {"route": body["questions"]["route"]}
+        body["options"] = {"permutations": 2, "return_logprobs": True}
+        result = await SystemOneService(FakeBackend()).score(
+            SystemOneRequest.model_validate(body)
+        )
+        route = result["answers"]["route"]
+        self.assertAlmostEqual(route["probabilities"]["billing"], 0.5)
+        self.assertAlmostEqual(route["confidence"], 0)
+        self.assertEqual(len(route["logprobs"]), 2)
+
+    async def test_concurrency_limit_applies_across_requests(self):
+        backend = FakeBackend(delay=0.01)
+        service = SystemOneService(backend, max_concurrency=2)
+        await asyncio.gather(
+            *(
+                service.score(SystemOneRequest.model_validate(payload()))
+                for _ in range(3)
+            )
+        )
+        self.assertEqual(len(backend.calls), 9)
+        self.assertEqual(backend.peak, 2)
+
+    async def test_cancellation_closes_all_pending_work(self):
+        backend = FakeBackend(delay=10)
+        task = asyncio.create_task(
+            SystemOneService(backend, max_concurrency=2).score(
+                SystemOneRequest.model_validate(payload())
+            )
+        )
+        while backend.active != 2:
+            await asyncio.sleep(0)
+        task.cancel()
+        with self.assertRaises(asyncio.CancelledError):
+            await task
+        self.assertEqual(backend.cancelled, 2)
+        self.assertEqual(backend.active, 0)
+        self.assertEqual(len(backend.calls), 2)
+
+    async def test_wrong_model_rejected_before_inference(self):
+        backend = FakeBackend()
+        body = payload()
+        body["model"] = "wrong-model"
+        with self.assertRaises(AdapterError) as error:
+            await SystemOneService(backend).score(SystemOneRequest.model_validate(body))
+        self.assertEqual(error.exception.status, 404)
+        self.assertEqual(backend.calls, [])
+
+
+class TestDefaultTemperature(unittest.IsolatedAsyncioTestCase):
+    """A server-side default fills an omitted options.temperature only."""
+
+    vector = tuple(math.log(0.8 if i == 0 else 0.1) for i in range(3))
+
+    async def score(self, body, **service_options):
+        service = SystemOneService(FakeBackend(), **service_options)
+        return await service.score(SystemOneRequest.model_validate(body))
+
+    def route_probabilities(self, result):
+        return list(result["answers"]["urgency"]["probabilities"].values())
+
+    async def test_unset_default_keeps_request_default_of_one(self):
+        result = await self.score(payload())
+        self.assertEqual(result["metadata"]["temperature"], 1.0)
+        expected = probabilities_from_logprobs(self.vector, 1.0)
+        for actual, wanted in zip(self.route_probabilities(result), expected):
+            self.assertAlmostEqual(actual, wanted)
+
+    async def test_omitted_temperature_takes_server_default(self):
+        for body in (payload(), {**payload(), "options": {"permutations": 1}}):
+            result = await self.score(body, default_temperature=2.4)
+            self.assertEqual(result["metadata"]["temperature"], 2.4)
+            expected = probabilities_from_logprobs(self.vector, 2.4)
+            for actual, wanted in zip(self.route_probabilities(result), expected):
+                self.assertAlmostEqual(actual, wanted)
+            self.assertNotAlmostEqual(
+                self.route_probabilities(result)[0],
+                probabilities_from_logprobs(self.vector, 1.0)[0],
+            )
+
+    async def test_explicit_request_temperature_wins_over_default(self):
+        body = {**payload(), "options": {"temperature": 1.0}}
+        result = await self.score(body, default_temperature=2.4)
+        self.assertEqual(result["metadata"]["temperature"], 1.0)
+        expected = probabilities_from_logprobs(self.vector, 1.0)
+        for actual, wanted in zip(self.route_probabilities(result), expected):
+            self.assertAlmostEqual(actual, wanted)
+        body = {**payload(), "options": {"temperature": 3.0}}
+        result = await self.score(body, default_temperature=2.4)
+        self.assertEqual(result["metadata"]["temperature"], 3.0)
+
+    async def test_disabled_scaling_ignores_default(self):
+        body = {**payload(), "options": {"temperature_scaling": False}}
+        result = await self.score(body, default_temperature=2.4)
+        self.assertEqual(result["metadata"]["temperature"], 1.0)
+        expected = probabilities_from_logprobs(self.vector, 1.0)
+        for actual, wanted in zip(self.route_probabilities(result), expected):
+            self.assertAlmostEqual(actual, wanted)
+
+    async def test_default_does_not_mutate_the_request(self):
+        request = SystemOneRequest.model_validate(payload())
+        service = SystemOneService(FakeBackend(), default_temperature=2.4)
+        await service.score(request)
+        self.assertEqual(request.options.temperature, 1.0)
+        self.assertNotIn("temperature", request.options.model_fields_set)
+        self.assertEqual(service.effective_options(request).temperature, 2.4)
+
+    def test_invalid_default_temperature_rejected_at_construction(self):
+        for value in (0, -1.0, float("inf"), float("nan"), True, "2.4", None):
+            with self.assertRaises((ValueError, TypeError)):
+                SystemOneService(FakeBackend(), default_temperature=value)
+        self.assertEqual(
+            SystemOneService(FakeBackend(), default_temperature=2).default_temperature,
+            2.0,
+        )
+
+    def test_cli_default_temperature_type(self):
+        self.assertEqual(positive_temperature("2.4"), 2.4)
+        self.assertEqual(positive_temperature("1"), 1.0)
+        import argparse
+
+        for text in ("0", "-1", "inf", "nan", "abc", ""):
+            with self.assertRaises(argparse.ArgumentTypeError):
+                positive_temperature(text)
+
+
+class TestPromptWording(unittest.IsolatedAsyncioTestCase):
+    """--prompt-wording is a server-wide setting (CLI/env, not per-request);
+    the default keeps today's served text unchanged."""
+
+    async def test_default_served_wording_is_unchanged(self):
+        backend = FakeBackend()
+        body = payload()
+        body["questions"] = {"route": body["questions"]["route"]}
+        await SystemOneService(backend).score(SystemOneRequest.model_validate(body))
+        prompt = backend.calls[0][0]
+        self.assertTrue(prompt.startswith("Context:\n"))
+
+    async def test_native_wording_is_plumbed_into_build_prompt(self):
+        backend = FakeBackend()
+        body = payload()
+        body["questions"] = {"route": body["questions"]["route"]}
+        service = SystemOneService(backend, prompt_wording="native")
+        await service.score(SystemOneRequest.model_validate(body))
+        prompt = backend.calls[0][0]
+        self.assertTrue(prompt.startswith("Read the state and question."))
+        self.assertIn("\n\nState:\n", prompt)
+        self.assertNotIn("Context:\n", prompt)
+
+    async def test_native_wording_requires_canonical_az_labels(self):
+        # FakeBackend.labels() already returns canonical A, B, C, ... so this
+        # documents the happy path; build_prompt itself enforces the
+        # requirement (see TestNativePromptWording in test_protocol.py) when a
+        # backend's labels are not canonical.
+        backend = FakeBackend()
+        labels, _ = backend.labels(3)
+        self.assertEqual(labels, ("A", "B", "C"))
+
+    def test_invalid_prompt_wording_rejected_at_construction(self):
+        for value in ("", "SERVED", "native ", None, 1):
+            with self.assertRaises(ValueError):
+                SystemOneService(FakeBackend(), prompt_wording=value)
+
+
+class TestHTTP(unittest.TestCase):
+    def test_standalone_app_lifecycle_schema_and_alias(self):
+        backend = FakeBackend()
+        with TestClient(create_app(backend)) as client:
+            self.assertTrue(backend.started)
+            self.assertEqual(client.get("/health").json(), {"status": "ok"})
+            self.assertEqual(
+                client.get("/v1/models").json()["data"][0]["id"], backend.model
+            )
+            body = payload()
+            body["model"] = "jev-latest"
+            response = client.post("/v1/systemone", json=body)
+            self.assertEqual(response.status_code, 200)
+            self.assertEqual(response.json()["model"], backend.model)
+            self.assertEqual(
+                set(response.json()["answers"]), {"route", "refund", "urgency"}
+            )
+            bad = client.post("/v1/systemone", json={"model": backend.model})
+            self.assertEqual(bad.status_code, 422)
+            self.assertEqual(bad.json()["error"]["code"], "invalid_request")
+        self.assertTrue(backend.closed)
+
+    def test_app_default_temperature_applies_when_request_omits_it(self):
+        with TestClient(create_app(FakeBackend(), default_temperature=2.4)) as client:
+            response = client.post("/v1/systemone", json=payload())
+            self.assertEqual(response.status_code, 200)
+            self.assertEqual(response.json()["metadata"]["temperature"], 2.4)
+            body = {**payload(), "options": {"temperature": 1.5}}
+            response = client.post("/v1/systemone", json=body)
+            self.assertEqual(response.json()["metadata"]["temperature"], 1.5)
+        with self.assertRaises(ValueError):
+            create_app(FakeBackend(), default_temperature=0)
+
+    def test_app_prompt_wording_native_reaches_the_backend_over_http(self):
+        backend = FakeBackend()
+        with TestClient(create_app(backend, prompt_wording="native")) as client:
+            body = payload()
+            body["questions"] = {"route": body["questions"]["route"]}
+            response = client.post("/v1/systemone", json=body)
+            self.assertEqual(response.status_code, 200)
+            self.assertTrue(backend.calls[0][0].startswith("Read the state and question."))
+
+    def test_auth_guards_inference_and_model_discovery(self):
+        backend = FakeBackend()
+        with TestClient(create_app(backend, api_key="test-secret")) as client:
+            self.assertEqual(client.get("/v1/models").status_code, 401)
+            self.assertEqual(
+                client.post("/v1/systemone", json=payload()).status_code, 401
+            )
+            self.assertEqual(
+                client.post(
+                    "/v1/systemone",
+                    json=payload(),
+                    headers={b"Authorization": b"Bearer caf\xe9"},
+                ).status_code,
+                401,
+            )
+            self.assertEqual(backend.calls, [])
+            response = client.post(
+                "/v1/systemone",
+                json=payload(),
+                headers={"Authorization": "Bearer test-secret"},
+            )
+            self.assertEqual(response.status_code, 200)
+
+    def test_upstream_failure_is_not_returned_as_probabilities(self):
+        with TestClient(create_app(FakeBackend(failure=True))) as client:
+            response = client.post("/v1/systemone", json=payload())
+            self.assertEqual(response.status_code, 502)
+            self.assertEqual(response.json()["error"]["code"], "backend_unavailable")
+            self.assertNotIn("answers", response.json())

server/tests/test_sglang.py

@@ -0,0 +1,409 @@
+import copy
+import json
+
+import httpx
+import pytest
+
+from jev_adapter.backend import AdapterError
+from jev_adapter.sglang import SGLangBackend
+
+
+class Engine:
+    def __init__(self):
+        self.calls = []
+        self.info = {
+            "served_model_name": "qwen",
+            "model_path": "org/qwen",
+            "is_generation": True,
+            "has_image_understanding": True,
+            "model_type": "qwen3_5",
+            "architectures": ["Qwen3_5ForConditionalGeneration"],
+        }
+        self.config = {"speculative_algorithm": None, "skip_tokenizer_init": False}
+        self.mutate_result = lambda result: result
+        self.invalid_boundary = False
+        self.lossy_roundtrip = False
+        self.special_label = None
+        self.fail_path = None
+        self.failure = None
+
+    @staticmethod
+    def encode(text):
+        return [ord(char) + 1000 for char in text]
+
+    @staticmethod
+    def decode(tokens):
+        return "".join(chr(token - 1000) for token in tokens)
+
+    def __call__(self, request):
+        path = request.url.path
+        body = json.loads(request.content) if request.content else None
+        self.calls.append((path, body, request))
+        if path == self.fail_path:
+            if isinstance(self.failure, Exception):
+                raise self.failure
+            return self.failure
+        if path == "/model_info":
+            return httpx.Response(200, json=self.info)
+        if path == "/server_info":
+            return httpx.Response(200, json=self.config)
+        if path == "/v1/tokenize":
+            if "messages" in body:
+                content = body["messages"][0]["content"]
+                if isinstance(content, list):
+                    content = "<image>" * (len(content) - 1) + content[-1]["text"]
+                tokens = self.encode("<user>" + content + "</user><assistant>")
+            else:
+                assert body["add_special_tokens"] is False
+                tokens = [self.encode(text) for text in body["prompt"]]
+                if body["prompt"][0].startswith("<user>"):
+                    if self.invalid_boundary:
+                        tokens[-1][-1] += 1
+                    if self.lossy_roundtrip:
+                        tokens[0][-1] += 1
+            return httpx.Response(200, json={"tokens": tokens})
+        if path == "/v1/detokenize":
+            tokens = body["tokens"]
+            if isinstance(tokens[0], list):
+                assert body["skip_special_tokens"] is True
+                text = [self.decode(ids) for ids in tokens]
+                text = ["" if value == self.special_label else value for value in text]
+            else:
+                assert body["skip_special_tokens"] is False
+                text = self.decode(tokens)
+            return httpx.Response(200, json={"text": text})
+        if path == "/generate":
+            result = {
+                "text": "",
+                "meta_info": {
+                    "completion_tokens": 0,
+                    "prompt_tokens": 71,
+                    # Return labels out of order to ensure alignment by token ID.
+                    "output_token_ids_logprobs": [
+                        [
+                            [-index - 0.5, token, None]
+                            for index, token in reversed(
+                                list(enumerate(body["token_ids_logprob"]))
+                            )
+                        ]
+                    ],
+                },
+            }
+            return httpx.Response(200, json=self.mutate_result(result))
+        raise AssertionError(f"Unexpected route: {path}")
+
+
+async def make_backend(engine=None, **kwargs):
+    engine = engine or Engine()
+    client = httpx.AsyncClient(transport=httpx.MockTransport(engine))
+    backend = SGLangBackend("http://engine", "qwen", client=client, **kwargs)
+    await backend.start()
+    return backend, engine, client
+
+
+@pytest.mark.asyncio
+async def test_public_http_contract_and_zero_decode():
+    backend, engine, client = await make_backend(api_key="private-key")
+    try:
+        labels, tokens = backend.labels(3)
+        result = await backend.evaluate("pick an answer", [], labels, tokens, None)
+        assert result.logprobs == (-0.5, -1.5, -2.5)
+        assert result.input_tokens == 71
+        path, body, _ = engine.calls[-1]
+        assert path == "/generate"
+        assert "text" not in body and "image_data" not in body
+        assert body["input_ids"] == engine.encode(
+            "<user>pick an answer</user><assistant>"
+        )
+        assert body["sampling_params"] == {
+            "max_new_tokens": 0,
+            "temperature": 1,
+            "top_p": 1,
+            "top_k": -1,
+        }
+        assert body["token_ids_logprob"] == list(tokens)
+        assert body["logprob_start_len"] == -1
+        assert body["return_logprob"] is True
+        assert body["top_logprobs_num"] == 0
+        assert body["return_text_in_logprobs"] is False
+        assert body["stream"] is False
+        assert body["rid"].startswith("jev-")
+        assert all(
+            request.headers["Authorization"] == "Bearer private-key"
+            for _, _, request in engine.calls
+        )
+        assert engine.calls[-4][1]["chat_template_kwargs"] == {"enable_thinking": False}
+    finally:
+        await client.aclose()
+
+
+def test_prompt_wording_rejects_unknown_value():
+    with pytest.raises(ValueError):
+        SGLangBackend("http://engine", "qwen", prompt_wording="bogus")
+
+
+@pytest.mark.asyncio
+async def test_native_wording_prepends_system_message_over_http():
+    backend, engine, client = await make_backend(
+        prompt_wording="native", native_system_prompt="Default system text."
+    )
+    try:
+        labels, tokens = backend.labels(3)
+        await backend.evaluate("pick an answer", [], labels, tokens, None)
+        # The messages-based /v1/tokenize call, three calls before /generate
+        # (detokenize, boundary-check tokenize, generate follow it).
+        path, body, _ = engine.calls[-4]
+        assert path == "/v1/tokenize"
+        assert body["messages"] == [
+            {"role": "system", "content": "Default system text."},
+            {"role": "user", "content": "pick an answer"},
+        ]
+    finally:
+        await client.aclose()
+
+
+@pytest.mark.asyncio
+async def test_served_wording_never_sends_a_system_message_even_if_configured():
+    # native_system_prompt is only honored when prompt_wording == "native";
+    # the default ("served") must stay byte-identical to today's behaviour.
+    backend, engine, client = await make_backend(
+        prompt_wording="served", native_system_prompt="Should be ignored."
+    )
+    try:
+        labels, tokens = backend.labels(2)
+        await backend.evaluate("look", [], labels, tokens, None)
+        path, body, _ = engine.calls[-4]
+        assert path == "/v1/tokenize"
+        assert body["messages"] == [{"role": "user", "content": "look"}]
+    finally:
+        await client.aclose()
+
+
+@pytest.mark.asyncio
+async def test_native_wording_passes_system_prompt_to_native_tokenizer():
+    class RecordingNativeTokenizer:
+        def __init__(self):
+            self.calls = []
+
+        def prepare(self, prompt, labels, token_ids, assistant_prefix, system_prompt=None):
+            self.calls.append((prompt, labels, token_ids, assistant_prefix, system_prompt))
+            return [1, 2, 3]
+
+    native_tokenizer = RecordingNativeTokenizer()
+    engine = Engine()
+    client = httpx.AsyncClient(transport=httpx.MockTransport(engine))
+    backend = SGLangBackend(
+        "http://engine",
+        "qwen",
+        client=client,
+        native_tokenizer=native_tokenizer,
+        prompt_wording="native",
+        native_system_prompt="Default system text.",
+    )
+    try:
+        await backend.start()
+        labels, tokens = backend.labels(2)
+        await backend.evaluate("pick", [], labels, tokens, "Answer:")
+        assert native_tokenizer.calls == [
+            ("pick", labels, tokens, "Answer:", "Default system text.")
+        ]
+    finally:
+        await client.aclose()
+
+
+@pytest.mark.asyncio
+async def test_images_remain_in_native_request_with_template_and_prefix():
+    backend, engine, client = await make_backend()
+    try:
+        images = ["data:image/png;base64,YQ==", "https://example.com/image.png"]
+        labels, tokens = backend.labels(2)
+        await backend.evaluate("look", images, labels, tokens, "Answer: ")
+        body = engine.calls[-1][1]
+        assert body["image_data"] == images
+        assert body["text"] == "<user><image><image>look</user><assistant>Answer: "
+        assert "input_ids" not in body
+        content = engine.calls[-4][1]["messages"][0]["content"]
+        assert content == [
+            {"type": "image_url", "image_url": {"url": images[0]}},
+            {"type": "image_url", "image_url": {"url": images[1]}},
+            {"type": "text", "text": "look"},
+        ]
+    finally:
+        await client.aclose()
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+    "field,value,code",
+    [
+        ("has_image_understanding", False, "images_not_supported"),
+        ("model_type", "kimi_k3", "image_model_not_supported"),
+    ],
+)
+async def test_image_capability_rejections(field, value, code):
+    engine = Engine()
+    engine.info[field] = value
+    engine.info["architectures"] = []
+    backend, engine, client = await make_backend(engine)
+    try:
+        labels, tokens = backend.labels(2)
+        with pytest.raises(AdapterError, match="image|Image") as error:
+            await backend.evaluate(
+                "look", ["https://example.com/a.png"], labels, tokens, None
+            )
+        assert error.value.code == code
+        assert not any(path == "/generate" for path, _, _ in engine.calls)
+    finally:
+        await client.aclose()
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+    "flag,code",
+    [
+        ("invalid_boundary", "invalid_label_boundary"),
+        ("lossy_roundtrip", "unsupported_tokenizer_roundtrip"),
+    ],
+)
+async def test_rejects_unsafe_tokenization_before_inference(flag, code):
+    backend, engine, client = await make_backend()
+    try:
+        setattr(engine, flag, True)
+        labels, tokens = backend.labels(2)
+        with pytest.raises(AdapterError) as error:
+            await backend.evaluate("look", [], labels, tokens, None)
+        assert error.value.code == code
+        assert not any(path == "/generate" for path, _, _ in engine.calls)
+    finally:
+        await client.aclose()
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+    "mutation",
+    [
+        "decoded",
+        "missing_label",
+        "duplicate_label",
+        "nan",
+        "positive",
+        "missing_usage",
+        "bad_usage",
+        "empty_positions",
+        "multiple_positions",
+    ],
+)
+async def test_rejects_unusable_engine_results(mutation):
+    backend, engine, client = await make_backend()
+
+    def mutate(original):
+        result = copy.deepcopy(original)
+        meta = result["meta_info"]
+        entries = meta["output_token_ids_logprobs"][0]
+        if mutation == "decoded":
+            meta["completion_tokens"] = 1
+        elif mutation == "missing_label":
+            entries.pop()
+        elif mutation == "duplicate_label":
+            entries.append(entries[0])
+        elif mutation == "nan":
+            # JSON null is also rejected; transport JSON forbids NaN values.
+            entries[0][0] = None
+        elif mutation == "positive":
+            entries[0][0] = 1.0
+        elif mutation == "missing_usage":
+            del meta["completion_tokens"]
+        elif mutation == "bad_usage":
+            meta["prompt_tokens"] = True
+        elif mutation == "empty_positions":
+            meta["output_token_ids_logprobs"] = []
+        elif mutation == "multiple_positions":
+            meta["output_token_ids_logprobs"].append(entries)
+        return result
+
+    engine.mutate_result = mutate
+    try:
+        labels, tokens = backend.labels(2)
+        with pytest.raises(AdapterError) as error:
+            await backend.evaluate("look", [], labels, tokens, None)
+        assert error.value.status == 502
+        assert error.value.code == "invalid_engine_response"
+    finally:
+        await client.aclose()
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+    "failure,status,code",
+    [
+        (httpx.ReadTimeout("secret upstream location"), 504, "engine_timeout"),
+        (httpx.ConnectError("secret upstream location"), 502, "engine_unavailable"),
+        (httpx.Response(401, text="secret-key"), 502, "engine_http_error"),
+        (httpx.Response(422, text="sensitive prompt"), 422, "engine_rejected_request"),
+        (httpx.Response(200, text="not-json secret"), 502, "invalid_engine_response"),
+    ],
+)
+async def test_transport_failures_are_sanitized(failure, status, code):
+    backend, engine, client = await make_backend()
+    engine.fail_path, engine.failure = "/generate", failure
+    try:
+        labels, tokens = backend.labels(2)
+        with pytest.raises(AdapterError) as error:
+            await backend.evaluate("look", [], labels, tokens, None)
+        assert error.value.status == status
+        assert error.value.code == code
+        assert "secret" not in str(error.value) and "sensitive" not in str(error.value)
+    finally:
+        await client.aclose()
+
+
+@pytest.mark.asyncio
+async def test_special_labels_removed_and_start_idempotent():
+    engine = Engine()
+    engine.special_label = "A"
+    backend, engine, client = await make_backend(engine)
+    try:
+        labels, _ = backend.labels(2)
+        assert labels == ("B", "C")
+        count = len(engine.calls)
+        await backend.start()
+        assert len(engine.calls) == count
+        with pytest.raises(AdapterError):
+            backend.labels(255)
+    finally:
+        await client.aclose()
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+    "config,code",
+    [
+        ({"speculative_algorithm": "EAGLE"}, "speculation_not_supported"),
+        ({"skip_tokenizer_init": True}, "tokenizer_unavailable"),
+    ],
+)
+async def test_startup_validates_engine_configuration(config, code):
+    engine = Engine()
+    engine.config.update(config)
+    async with httpx.AsyncClient(transport=httpx.MockTransport(engine)) as client:
+        backend = SGLangBackend("http://engine", "qwen", client=client)
+        with pytest.raises(AdapterError) as error:
+            await backend.start()
+        assert error.value.code == code
+
+
+@pytest.mark.asyncio
+async def test_nonfinite_raw_json_logprob_rejected():
+    backend, engine, client = await make_backend()
+    engine.fail_path = "/generate"
+    engine.failure = httpx.Response(
+        200,
+        text='{"meta_info":{"completion_tokens":0,"prompt_tokens":1,"output_token_ids_logprobs":[[[NaN,1065],[-1.0,1066]]]}}',
+    )
+    try:
+        labels, tokens = backend.labels(2)
+        with pytest.raises(AdapterError) as error:
+            await backend.evaluate("look", [], labels, tokens, None)
+        assert error.value.code == "invalid_engine_response"
+    finally:
+        await client.aclose()

tekken.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:600bb27946565481ecf51ba8aee252e49b9a68507866080ac9c30185bb312843
+size 16753784

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d5f6046775b112f0e2d456ee9dba450684ab964fe5c4e231599bdc6773028135
+size 17078128