license: cc-by-nc-4.0
base_model: Qwen/Qwen3.5-9B
library_name: peft
pipeline_tag: text-classification
tags:
- jev
- jev-torch
- decision-model
- calibration
- lora
- distillation
datasets:
- SargeDev/jev-distill-corpus-v3
persly-jev-9b
An unofficial JEV-style decision model trained with jev-torch. It reads a state, a question and a list of options, runs one forward pass, and returns a calibrated probability for every option.
Usage
pip install git+https://github.com/smha1012/jev-torch.git
from jev import JEVPredictor
jev = JEVPredictor("persly/persly-jev-9b")
jev.predict(
kind="noul",
state="The canary shows p99 latency up 40% after the deploy.",
question="Should the rollout be paused?",
options=["false", "true"],
)
Question kinds: noul (exactly 2 options, false-like then true-like), choice (2–16 options),
score (ordered levels 0–5).
Training
- Training data: jev_distill (https://huggingface.co/datasets/SargeDev/jev-distill-corpus-v3 @ fc99c6357a9f89f7512c4a987314352addead049)
- Base model:
Qwen/Qwen3.5-9B(frozen) + LoRA r=16, α=32 + 24-slot fp32 decision head - Loss:
1·kl + 0.5·rps[score] - Optimization: 4750 steps × 128 rows, LoRA lr 0.0001, head lr 0.0002
- Temperatures: noul 1.008, choice 1.000, score 1.008
Evaluation
Agreement with the teacher
On the 25,376 rows of test_set_30k labelled by TypeSafe Jev 1.13, the same rows the autotrust/JEV cards
report on. These measure how closely the model copies Jev, not accuracy on ground truth (see JevBench below).
| Metric | This model | autotrust JEV-9B | autotrust JEV-27B |
|---|---|---|---|
| Mean KL to teacher (lower is better) | 0.0184 | 0.0190 | 0.0170 |
| Choice top-1 agreement, teacher-labelled rows | 90.1% | 90.2% | – |
| Choice top-1 agreement, all choice rows | 89.9% | 89.8% | 90.3% |
| ECE vs teacher probabilities | 0.0009 | 0.0007 | 0.0009 |
Per kind: choice 90.1%, noul 95.8%, score 87.9% top-1 agreement.
JevBench (Leanmcp v0.1)
Accuracy and calibration on 6,516 public benchmark cases with gold answers, next to TypeSafe Jev 1.13.0 on the same cases: 74.1% vs 84.9% accuracy (87.3% of Jev's), ECE 0.029 vs 0.020.
Limitations
- Distilled, not RLCD. The model learns to reproduce TypeSafe Jev's output distributions (supervised KL + RPS) and is calibrated afterwards with temperature scaling. It does not reproduce Jev's own training method or architecture, which are unpublished, and cannot exceed the teacher.
- Knowledge comes from the backbone. Gaps to Jev on JevBench are largest on knowledge-heavy questions (MMLU-Pro −24 points, MedQA −15) and did not shrink with training.
- Long inputs. Agent trajectories longer than the 1,024-token training context trail Jev by ~14 points.
- At most 16 options per choice question; trained mostly on short, synthetic English scenarios. Validate on your own domain before relying on it.
License and attribution
Weights: CC BY-NC 4.0, non-commercial use only. Most training labels are outputs of the closed TypeSafe Jev 1.13 model, so the weights are not offered for commercial use. For other uses, contact the author.
The code that trains and loads the model, jev-torch, is Apache-2.0. This is an independent project, not affiliated with TypeSafe AI or autotrust.