Text Classification
Transformers
Safetensors
English
qwen3_5_text
text-generation
system-one
system-two
blocks-of-experts
typed-decisions
decision-model
calibrated-probabilities
knowledge-distillation
jev
noul
choice
score
lora
qwen3_5
dual-head
vllm
multimodal
vision
computer-use
robotics
Eval Results (legacy)
Instructions to use autotrust/AutoTrust-Decider-Mini-9B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use autotrust/AutoTrust-Decider-Mini-9B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="autotrust/AutoTrust-Decider-Mini-9B")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("autotrust/AutoTrust-Decider-Mini-9B") model = AutoModelForCausalLM.from_pretrained("autotrust/AutoTrust-Decider-Mini-9B", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Model card: repository renamed to autotrust/JEV-9B; all ids, paths and commands updated
Browse files
README.md
CHANGED
|
@@ -29,7 +29,7 @@ metrics:
|
|
| 29 |
- brier
|
| 30 |
- ece
|
| 31 |
model-index:
|
| 32 |
-
- name: autotrust/JEV (student of TypeSafe Jev 1.13)
|
| 33 |
results:
|
| 34 |
- task:
|
| 35 |
type: text-classification
|
|
@@ -62,17 +62,17 @@ model-index:
|
|
| 62 |
value: 0.954
|
| 63 |
---
|
| 64 |
|
| 65 |
-
# autotrust/JEV — an open-weights student of TypeSafe Jev 1.13 (Qwen3.5-9B, calibrated *System One* decisions)
|
| 66 |
|
| 67 |
> **Two different models, two different organisations — please read the names carefully.**
|
| 68 |
> **TypeSafe Jev 1.13** is the hosted, closed model made by TypeSafe AI; it is the *teacher* whose public
|
| 69 |
-
> outputs this model was trained on. **autotrust/JEV** (this repository) is an independent open-weights
|
| 70 |
> *student* built by autotrust from the Apache-2.0 corpus
|
| 71 |
> [`SargeDev/jev-distill-corpus-v3`](https://huggingface.co/datasets/SargeDev/jev-distill-corpus-v3).
|
| 72 |
-
> autotrust/JEV is not affiliated with, endorsed by, or a product of TypeSafe AI. Throughout this card the
|
| 73 |
-
> teacher is always written *TypeSafe Jev 1.13* and this model is always written *autotrust/JEV*.
|
| 74 |
|
| 75 |
-
**One forward pass in, a calibrated probability distribution out.** autotrust/JEV answers typed questions
|
| 76 |
(`noul` yes/no · `choice` over 2–16 options · `score` on a 0–5 scale) about any text or JSON state.
|
| 77 |
No JSON parsing, no prompt engineering — it was distilled from the full output distributions of
|
| 78 |
**TypeSafe Jev 1.13** onto a pretrained **Qwen3.5-9B** backbone. The bundle ships **two heads on one
|
|
@@ -81,14 +81,14 @@ generation). The backbone weights on disk are **bit-identical to Qwen3.5-9B**; t
|
|
| 81 |
separate 154 MB LoRA that is only applied for decisions — so generation quality is exactly the base model's.
|
| 82 |
|
| 83 |
**Model family** — same recipe, same API, same two-head packaging:
|
| 84 |
-
**autotrust/JEV** (this repository, Qwen3.5-9B, 18 GB) · [autotrust/JEV-27B](https://huggingface.co/autotrust/JEV-27B)
|
| 85 |
(Qwen3.8-27B, 54 GB: KL 0.019, choice 90.3 %, out-of-distribution KL 0.104, HumanEval 78.0 %).
|
| 86 |
|
| 87 |
## Headline results
|
| 88 |
|
| 89 |
-
Held-out test set: 29,955 questions, 53 domains, all three question types. Numbers are for **autotrust/JEV**.
|
| 90 |
|
| 91 |
-
| What is measured | autotrust/JEV | How to read it |
|
| 92 |
|---|---|---|
|
| 93 |
| **How closely it reproduces the teacher's probabilities** — mean KL divergence, lower is better, 0 = identical | **0.021** | The student's probability distributions are nearly indistinguishable from TypeSafe Jev 1.13's |
|
| 94 |
| **Yes/no questions** (`noul`) — AUROC, higher is better | **0.996** | Ranks true vs. false almost perfectly |
|
|
@@ -121,7 +121,7 @@ financial tweets, under three kinds of pressure: more options, near-miss options
|
|
| 121 |
are human gold labels, none of this data is in our training set, and TypeSafe Jev 1.13's results are published with
|
| 122 |
it. We re-ran the same items, up to our 16-option limit.
|
| 123 |
|
| 124 |
-
| | TypeSafe Jev 1.13 (published) | **autotrust/JEV**
|
| 125 |
|---|---|---|---|
|
| 126 |
| Accuracy with 2 / 4 / 8 / 16 options (800 items, 4 domains) | 0.890 / 0.801 / 0.782 / 0.769 | **0.868 / 0.774 / 0.735 / 0.694** | 0.876 / 0.784 / 0.767 / 0.740 |
|
| 127 |
| 16 options — CLINC / DBpedia / GoEmotions / MTOP | 0.945 / 0.900 / 0.470 / 0.760 | **0.875 / 0.855 / 0.325 / 0.720** | 0.930 / 0.885 / 0.415 / 0.730 |
|
|
@@ -141,14 +141,14 @@ seeded differently, so compare the aggregates, not individual items.
|
|
| 141 |
The expected answers were written by hand before the model was run. These are illustrations (≈ 110 decisions), not a
|
| 142 |
benchmark.
|
| 143 |
|
| 144 |
-
| task | autotrust/JEV
|
| 145 |
|---|---|---|
|
| 146 |
| Topic of 19 HN front-page stories (10 options) + "is it about AI?" | **38 / 38** | 38 / 38 |
|
| 147 |
| 12 comments from a heated HN thread: "insults or attacks someone?" + "what is it mainly doing?" (6 options) | **22 / 24** | 23 / 24 |
|
| 148 |
| 10 V2EX hot posts in **Chinese**: "contains a referral / invite code?" + "promotes a product or paid offer?" | **18 / 19** | 19 / 19 |
|
| 149 |
| Community use cases: code-rule checks in the style of `adhere`, injection filtering, ticket routing, phishing, code-review diffs, urgency scores | **14 / 15** | 15 / 15 |
|
| 150 |
|
| 151 |
-
| input | question | autotrust/JEV
|
| 152 |
|---|---|---|
|
| 153 |
| HN: "Two-tier encryption in the UK" | topic (10 options) | security & privacy · 0.87 |
|
| 154 |
| HN: "Using LLMs to trace alchemical knowledge and decode 17th century letters" | about AI? | P(true) = 0.88 |
|
|
@@ -187,9 +187,9 @@ decision head. `adapter_vllm/` contains the backbone LoRA plus the 24-slot decis
|
|
| 187 |
### 1 — Start the server (OpenAI-compatible)
|
| 188 |
|
| 189 |
```bash
|
| 190 |
-
hf download autotrust/JEV --local-dir JEV
|
| 191 |
-
vllm serve JEV --served-model-name autotrust/JEV \
|
| 192 |
-
--enable-lora --max-lora-rank 32 --lora-modules jev-decision=JEV/adapter_vllm \
|
| 193 |
--logprobs-mode processed_logprobs --max-model-len 4096
|
| 194 |
```
|
| 195 |
|
|
@@ -201,7 +201,7 @@ state (see notes below).
|
|
| 201 |
|
| 202 |
```bash
|
| 203 |
curl -s localhost:8000/v1/chat/completions -H 'content-type: application/json' -d '{
|
| 204 |
-
"model": "autotrust/JEV",
|
| 205 |
"messages": [{"role": "user", "content": "In one sentence, what is safety stock?"}],
|
| 206 |
"max_tokens": 60, "chat_template_kwargs": {"enable_thinking": false}}'
|
| 207 |
```
|
|
@@ -212,7 +212,7 @@ curl -s localhost:8000/v1/chat/completions -H 'content-type: application/json' -
|
|
| 212 |
import json, math, requests
|
| 213 |
from huggingface_hub import hf_hub_download
|
| 214 |
|
| 215 |
-
REPO, URL = "autotrust/JEV", "http://localhost:8000"
|
| 216 |
dh = json.load(open(hf_hub_download(REPO, "adapter_vllm/decision_head.json"))) # bias + verbalizer token ids
|
| 217 |
T = json.load(open(hf_hub_download(REPO, "calibration.json")))["per_kind"] # per-kind temperatures
|
| 218 |
|
|
@@ -248,8 +248,8 @@ that vLLM applies cancels out, so the result is exactly the decision head's cali
|
|
| 248 |
from vllm import LLM, SamplingParams
|
| 249 |
from vllm.lora.request import LoRARequest
|
| 250 |
|
| 251 |
-
llm = LLM("JEV", enable_lora=True, max_lora_rank=32, logprobs_mode="processed_logprobs", max_model_len=4096)
|
| 252 |
-
decision = LoRARequest("jev-decision", 1, "JEV/adapter_vllm")
|
| 253 |
|
| 254 |
gen = llm.generate(["..."], SamplingParams(temperature=0.0, max_tokens=256)) # base model, no LoRA
|
| 255 |
dec = llm.generate([prompt], [SamplingParams(max_tokens=1, temperature=1.0, # decision head
|
|
@@ -345,7 +345,7 @@ from peft import PeftModel
|
|
| 345 |
from safetensors.torch import load_file
|
| 346 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 347 |
|
| 348 |
-
repo = "autotrust/JEV"
|
| 349 |
tok = AutoTokenizer.from_pretrained(repo)
|
| 350 |
base = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda") # == Qwen3.5-9B text model
|
| 351 |
|
|
@@ -386,7 +386,7 @@ inside `with model.disable_adapter():`.
|
|
| 386 |
## Why start from a pretrained LLM
|
| 387 |
|
| 388 |
A System One judge is "just a classifier", so one could train a small encoder from scratch or fine-tune a
|
| 389 |
-
BERT-class model. autotrust/JEV starts from a pretrained 9 B LLM instead, and that choice — not the 3 GPU-hours of
|
| 390 |
distillation — is where most of the result above comes from:
|
| 391 |
|
| 392 |
1. **Pretraining does most of the work; distillation only sharpens.** The 24-slot head is *initialised from
|
|
@@ -402,7 +402,7 @@ distillation — is where most of the result above comes from:
|
|
| 402 |
after the first 64 k rows validation KL was already below 0.10.
|
| 403 |
|
| 404 |
3. **Transfer to unseen task families.** On the Open-Jev *OOD* split (game-state and workflow-policy
|
| 405 |
-
decisions never seen in training) autotrust/JEV reaches KL 0.234 and top-1 0.918 — a pretrained LLM reads the
|
| 406 |
*content* of a new task instead of matching surface patterns of the 53 training domains.
|
| 407 |
|
| 408 |
4. **Reads real, long, structured states.** States in the corpus are prose, JSON game states, policy
|
|
@@ -425,7 +425,7 @@ distillation — is where most of the result above comes from:
|
|
| 425 |
|
| 426 |
### Additional metrics (`test_set_30k`, temperature applied)
|
| 427 |
|
| 428 |
-
| metric | autotrust/JEV |
|
| 429 |
|---|---|
|
| 430 |
| `noul` Brier score against the teacher's probability (lower is better) | 0.0015 |
|
| 431 |
| `score` ranked probability score (lower is better) | 0.0085 |
|
|
@@ -440,7 +440,7 @@ distillation — is where most of the result above comes from:
|
|
| 440 |
| weights | pass@1 | note |
|
| 441 |
|---|---|---|
|
| 442 |
| Qwen3.5-9B (base) | 70.7 % (116/164) | same loader and protocol as below |
|
| 443 |
-
| **autotrust/JEV — generation path (backbone + `lm_head`, adapter off)** | **70.7 % (116/164)** | all 164 completions byte-identical to the base model |
|
| 444 |
| LoRA folded into the backbone + base `lm_head` (*not shipped*) | 61.6 % (101/164) | why the adapter is kept separate |
|
| 445 |
|
| 446 |
### Per source × primitive (`test_set_30k`, temperature applied)
|
|
@@ -466,7 +466,7 @@ Validation KL on a fixed 4 k-row subset; `test_set_30k` metrics after calibratio
|
|
| 466 |
|
| 467 |
| stage | rows seen | val KL | t30k KL | choice top-1 | score MAE | noul AUROC | ECE |
|
| 468 |
|---|---|---|---|---|---|---|---|
|
| 469 |
-
| **autotrust/JEV v0.8.0 — released weights (4,750 steps ≈ 0.93 epoch, LR annealed to ≈ 0.07×)** | 608 k | **0.019** | **0.0210** | **0.898** | **0.103** | **0.996** | **0.0007** |
|
| 470 |
| v0.7.0 (step 2000 + 500-step LR cool-down) | 320 k | 0.026 | 0.0276 | 0.884 | 0.119 | 0.994 | 0.0014 |
|
| 471 |
| step 2000 | 256 k | 0.0325 | 0.037 | 0.865 | 0.143 | 0.992 | 0.004 |
|
| 472 |
| step 1500 | 192 k | 0.038 | 0.040 | 0.861 | 0.151 | 0.991 | 0.0025 |
|
|
@@ -495,9 +495,9 @@ another 24 % and added 1.4 points of choice agreement. Checkpoints taken at high
|
|
| 495 |
|
| 496 |
## Limitations
|
| 497 |
|
| 498 |
-
* **autotrust/JEV mirrors TypeSafe Jev 1.13, including its mistakes.** This is a distillation, not an independent judge: where
|
| 499 |
-
the teacher was wrong or uncalibrated, so is autotrust/JEV. Published evaluations of the teacher show it is unreliable
|
| 500 |
-
for multi-hop reasoning, arithmetic, dates, counting, and adversarial inputs; autotrust/JEV inherits all of that. It
|
| 501 |
cannot generate text or explanations *as a decision model*; the generation head is simply the unmodified base
|
| 502 |
model and knows nothing about the decisions it is packaged with.
|
| 503 |
* **It inherits the teacher's blind spots, and at 9 B adds some of its own.** On fresh inputs it shoves a poker spot
|
|
@@ -507,7 +507,7 @@ another 24 % and added 1.4 points of choice agreement. Checkpoints taken at high
|
|
| 507 |
* **Choice agreement is capped by teacher ambiguity.** The teacher's `choice` distributions are soft (median
|
| 508 |
top-1 probability 0.70). On the 14 % of rows where the teacher's top-2 options are within 0.1 of each other,
|
| 509 |
argmax agreement is near chance for *any* faithful mimic (0.46 where the gap is < 0.05). On teacher-decisive
|
| 510 |
-
rows agreement is 0.954, and autotrust/JEV's argmax captures 97.7 % of the teacher probability mass a perfect mimic
|
| 511 |
could (0.693 vs 0.709).
|
| 512 |
* **Fixed option sets.** `noul` and `score` accept only their canonical options; `choice` accepts 2–16
|
| 513 |
options. Inputs longer than 1,024 tokens are truncated (state only, head 60 % / tail 40 %) at serving unless
|
|
@@ -515,7 +515,7 @@ another 24 % and added 1.4 points of choice agreement. Checkpoints taken at high
|
|
| 515 |
* **English-centric.** The corpus is English; multilingual behaviour is inherited from the backbone but was not
|
| 516 |
measured.
|
| 517 |
* **Placeholder labels in the corpus.** The `yuri_v1` memory-relevance stream is 100 % exact-uniform
|
| 518 |
-
`[0.5, 0.5]` — those rows teach nothing about relevance. autotrust/JEV outputs ≈ 0.5 on them by design; do not use it
|
| 519 |
for memory-relevance scoring without further training.
|
| 520 |
* **Not for high-stakes decisions.** Use confidence gating: act automatically only above a threshold you
|
| 521 |
validated on your own data, and route the rest to a stronger model or a human.
|
|
@@ -541,14 +541,14 @@ reports/ evaluation reports: test-set evaluation,
|
|
| 541 |
|
| 542 |
Weights: **Apache-2.0** (base model `Qwen/Qwen3.5-9B` is Apache-2.0; training corpus
|
| 543 |
`SargeDev/jev-distill-corpus-v3` is Apache-2.0, its `openjev_v2` stream additionally CC0). The System One
|
| 544 |
-
framing and the `noul` / `choice` / `score` primitives originate with TypeSafe AI's Jev; autotrust/JEV is an
|
| 545 |
independent student model trained on public data and shares no weights, code or affiliation with TypeSafe AI.
|
| 546 |
|
| 547 |
```bibtex
|
| 548 |
@misc{autotrust_jev_2026,
|
| 549 |
-
title = {autotrust/JEV: an open-weights student of TypeSafe Jev 1.13 for calibrated System One decisions (Qwen3.5-9B)},
|
| 550 |
author = {autotrust},
|
| 551 |
year = {2026},
|
| 552 |
-
url = {https://huggingface.co/autotrust/JEV}
|
| 553 |
}
|
| 554 |
```
|
|
|
|
| 29 |
- brier
|
| 30 |
- ece
|
| 31 |
model-index:
|
| 32 |
+
- name: autotrust/JEV-9B (student of TypeSafe Jev 1.13)
|
| 33 |
results:
|
| 34 |
- task:
|
| 35 |
type: text-classification
|
|
|
|
| 62 |
value: 0.954
|
| 63 |
---
|
| 64 |
|
| 65 |
+
# autotrust/JEV-9B — an open-weights student of TypeSafe Jev 1.13 (Qwen3.5-9B, calibrated *System One* decisions)
|
| 66 |
|
| 67 |
> **Two different models, two different organisations — please read the names carefully.**
|
| 68 |
> **TypeSafe Jev 1.13** is the hosted, closed model made by TypeSafe AI; it is the *teacher* whose public
|
| 69 |
+
> outputs this model was trained on. **autotrust/JEV-9B** (this repository) is an independent open-weights
|
| 70 |
> *student* built by autotrust from the Apache-2.0 corpus
|
| 71 |
> [`SargeDev/jev-distill-corpus-v3`](https://huggingface.co/datasets/SargeDev/jev-distill-corpus-v3).
|
| 72 |
+
> autotrust/JEV-9B is not affiliated with, endorsed by, or a product of TypeSafe AI. Throughout this card the
|
| 73 |
+
> teacher is always written *TypeSafe Jev 1.13* and this model is always written *autotrust/JEV-9B*.
|
| 74 |
|
| 75 |
+
**One forward pass in, a calibrated probability distribution out.** autotrust/JEV-9B answers typed questions
|
| 76 |
(`noul` yes/no · `choice` over 2–16 options · `score` on a 0–5 scale) about any text or JSON state.
|
| 77 |
No JSON parsing, no prompt engineering — it was distilled from the full output distributions of
|
| 78 |
**TypeSafe Jev 1.13** onto a pretrained **Qwen3.5-9B** backbone. The bundle ships **two heads on one
|
|
|
|
| 81 |
separate 154 MB LoRA that is only applied for decisions — so generation quality is exactly the base model's.
|
| 82 |
|
| 83 |
**Model family** — same recipe, same API, same two-head packaging:
|
| 84 |
+
**autotrust/JEV-9B** (this repository, Qwen3.5-9B, 18 GB) · [autotrust/JEV-27B](https://huggingface.co/autotrust/JEV-27B)
|
| 85 |
(Qwen3.8-27B, 54 GB: KL 0.019, choice 90.3 %, out-of-distribution KL 0.104, HumanEval 78.0 %).
|
| 86 |
|
| 87 |
## Headline results
|
| 88 |
|
| 89 |
+
Held-out test set: 29,955 questions, 53 domains, all three question types. Numbers are for **autotrust/JEV-9B**.
|
| 90 |
|
| 91 |
+
| What is measured | autotrust/JEV-9B | How to read it |
|
| 92 |
|---|---|---|
|
| 93 |
| **How closely it reproduces the teacher's probabilities** — mean KL divergence, lower is better, 0 = identical | **0.021** | The student's probability distributions are nearly indistinguishable from TypeSafe Jev 1.13's |
|
| 94 |
| **Yes/no questions** (`noul`) — AUROC, higher is better | **0.996** | Ranks true vs. false almost perfectly |
|
|
|
|
| 121 |
are human gold labels, none of this data is in our training set, and TypeSafe Jev 1.13's results are published with
|
| 122 |
it. We re-ran the same items, up to our 16-option limit.
|
| 123 |
|
| 124 |
+
| | TypeSafe Jev 1.13 (published) | **autotrust/JEV-9B** | autotrust/JEV-27B |
|
| 125 |
|---|---|---|---|
|
| 126 |
| Accuracy with 2 / 4 / 8 / 16 options (800 items, 4 domains) | 0.890 / 0.801 / 0.782 / 0.769 | **0.868 / 0.774 / 0.735 / 0.694** | 0.876 / 0.784 / 0.767 / 0.740 |
|
| 127 |
| 16 options — CLINC / DBpedia / GoEmotions / MTOP | 0.945 / 0.900 / 0.470 / 0.760 | **0.875 / 0.855 / 0.325 / 0.720** | 0.930 / 0.885 / 0.415 / 0.730 |
|
|
|
|
| 141 |
The expected answers were written by hand before the model was run. These are illustrations (≈ 110 decisions), not a
|
| 142 |
benchmark.
|
| 143 |
|
| 144 |
+
| task | autotrust/JEV-9B | autotrust/JEV-27B |
|
| 145 |
|---|---|---|
|
| 146 |
| Topic of 19 HN front-page stories (10 options) + "is it about AI?" | **38 / 38** | 38 / 38 |
|
| 147 |
| 12 comments from a heated HN thread: "insults or attacks someone?" + "what is it mainly doing?" (6 options) | **22 / 24** | 23 / 24 |
|
| 148 |
| 10 V2EX hot posts in **Chinese**: "contains a referral / invite code?" + "promotes a product or paid offer?" | **18 / 19** | 19 / 19 |
|
| 149 |
| Community use cases: code-rule checks in the style of `adhere`, injection filtering, ticket routing, phishing, code-review diffs, urgency scores | **14 / 15** | 15 / 15 |
|
| 150 |
|
| 151 |
+
| input | question | autotrust/JEV-9B |
|
| 152 |
|---|---|---|
|
| 153 |
| HN: "Two-tier encryption in the UK" | topic (10 options) | security & privacy · 0.87 |
|
| 154 |
| HN: "Using LLMs to trace alchemical knowledge and decode 17th century letters" | about AI? | P(true) = 0.88 |
|
|
|
|
| 187 |
### 1 — Start the server (OpenAI-compatible)
|
| 188 |
|
| 189 |
```bash
|
| 190 |
+
hf download autotrust/JEV-9B --local-dir JEV-9B # ~18 GB
|
| 191 |
+
vllm serve JEV-9B --served-model-name autotrust/JEV-9B \
|
| 192 |
+
--enable-lora --max-lora-rank 32 --lora-modules jev-decision=JEV-9B/adapter_vllm \
|
| 193 |
--logprobs-mode processed_logprobs --max-model-len 4096
|
| 194 |
```
|
| 195 |
|
|
|
|
| 201 |
|
| 202 |
```bash
|
| 203 |
curl -s localhost:8000/v1/chat/completions -H 'content-type: application/json' -d '{
|
| 204 |
+
"model": "autotrust/JEV-9B",
|
| 205 |
"messages": [{"role": "user", "content": "In one sentence, what is safety stock?"}],
|
| 206 |
"max_tokens": 60, "chat_template_kwargs": {"enable_thinking": false}}'
|
| 207 |
```
|
|
|
|
| 212 |
import json, math, requests
|
| 213 |
from huggingface_hub import hf_hub_download
|
| 214 |
|
| 215 |
+
REPO, URL = "autotrust/JEV-9B", "http://localhost:8000"
|
| 216 |
dh = json.load(open(hf_hub_download(REPO, "adapter_vllm/decision_head.json"))) # bias + verbalizer token ids
|
| 217 |
T = json.load(open(hf_hub_download(REPO, "calibration.json")))["per_kind"] # per-kind temperatures
|
| 218 |
|
|
|
|
| 248 |
from vllm import LLM, SamplingParams
|
| 249 |
from vllm.lora.request import LoRARequest
|
| 250 |
|
| 251 |
+
llm = LLM("JEV-9B", enable_lora=True, max_lora_rank=32, logprobs_mode="processed_logprobs", max_model_len=4096)
|
| 252 |
+
decision = LoRARequest("jev-decision", 1, "JEV-9B/adapter_vllm")
|
| 253 |
|
| 254 |
gen = llm.generate(["..."], SamplingParams(temperature=0.0, max_tokens=256)) # base model, no LoRA
|
| 255 |
dec = llm.generate([prompt], [SamplingParams(max_tokens=1, temperature=1.0, # decision head
|
|
|
|
| 345 |
from safetensors.torch import load_file
|
| 346 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 347 |
|
| 348 |
+
repo = "autotrust/JEV-9B"
|
| 349 |
tok = AutoTokenizer.from_pretrained(repo)
|
| 350 |
base = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda") # == Qwen3.5-9B text model
|
| 351 |
|
|
|
|
| 386 |
## Why start from a pretrained LLM
|
| 387 |
|
| 388 |
A System One judge is "just a classifier", so one could train a small encoder from scratch or fine-tune a
|
| 389 |
+
BERT-class model. autotrust/JEV-9B starts from a pretrained 9 B LLM instead, and that choice — not the 3 GPU-hours of
|
| 390 |
distillation — is where most of the result above comes from:
|
| 391 |
|
| 392 |
1. **Pretraining does most of the work; distillation only sharpens.** The 24-slot head is *initialised from
|
|
|
|
| 402 |
after the first 64 k rows validation KL was already below 0.10.
|
| 403 |
|
| 404 |
3. **Transfer to unseen task families.** On the Open-Jev *OOD* split (game-state and workflow-policy
|
| 405 |
+
decisions never seen in training) autotrust/JEV-9B reaches KL 0.234 and top-1 0.918 — a pretrained LLM reads the
|
| 406 |
*content* of a new task instead of matching surface patterns of the 53 training domains.
|
| 407 |
|
| 408 |
4. **Reads real, long, structured states.** States in the corpus are prose, JSON game states, policy
|
|
|
|
| 425 |
|
| 426 |
### Additional metrics (`test_set_30k`, temperature applied)
|
| 427 |
|
| 428 |
+
| metric | autotrust/JEV-9B |
|
| 429 |
|---|---|
|
| 430 |
| `noul` Brier score against the teacher's probability (lower is better) | 0.0015 |
|
| 431 |
| `score` ranked probability score (lower is better) | 0.0085 |
|
|
|
|
| 440 |
| weights | pass@1 | note |
|
| 441 |
|---|---|---|
|
| 442 |
| Qwen3.5-9B (base) | 70.7 % (116/164) | same loader and protocol as below |
|
| 443 |
+
| **autotrust/JEV-9B — generation path (backbone + `lm_head`, adapter off)** | **70.7 % (116/164)** | all 164 completions byte-identical to the base model |
|
| 444 |
| LoRA folded into the backbone + base `lm_head` (*not shipped*) | 61.6 % (101/164) | why the adapter is kept separate |
|
| 445 |
|
| 446 |
### Per source × primitive (`test_set_30k`, temperature applied)
|
|
|
|
| 466 |
|
| 467 |
| stage | rows seen | val KL | t30k KL | choice top-1 | score MAE | noul AUROC | ECE |
|
| 468 |
|---|---|---|---|---|---|---|---|
|
| 469 |
+
| **autotrust/JEV-9B v0.8.0 — released weights (4,750 steps ≈ 0.93 epoch, LR annealed to ≈ 0.07×)** | 608 k | **0.019** | **0.0210** | **0.898** | **0.103** | **0.996** | **0.0007** |
|
| 470 |
| v0.7.0 (step 2000 + 500-step LR cool-down) | 320 k | 0.026 | 0.0276 | 0.884 | 0.119 | 0.994 | 0.0014 |
|
| 471 |
| step 2000 | 256 k | 0.0325 | 0.037 | 0.865 | 0.143 | 0.992 | 0.004 |
|
| 472 |
| step 1500 | 192 k | 0.038 | 0.040 | 0.861 | 0.151 | 0.991 | 0.0025 |
|
|
|
|
| 495 |
|
| 496 |
## Limitations
|
| 497 |
|
| 498 |
+
* **autotrust/JEV-9B mirrors TypeSafe Jev 1.13, including its mistakes.** This is a distillation, not an independent judge: where
|
| 499 |
+
the teacher was wrong or uncalibrated, so is autotrust/JEV-9B. Published evaluations of the teacher show it is unreliable
|
| 500 |
+
for multi-hop reasoning, arithmetic, dates, counting, and adversarial inputs; autotrust/JEV-9B inherits all of that. It
|
| 501 |
cannot generate text or explanations *as a decision model*; the generation head is simply the unmodified base
|
| 502 |
model and knows nothing about the decisions it is packaged with.
|
| 503 |
* **It inherits the teacher's blind spots, and at 9 B adds some of its own.** On fresh inputs it shoves a poker spot
|
|
|
|
| 507 |
* **Choice agreement is capped by teacher ambiguity.** The teacher's `choice` distributions are soft (median
|
| 508 |
top-1 probability 0.70). On the 14 % of rows where the teacher's top-2 options are within 0.1 of each other,
|
| 509 |
argmax agreement is near chance for *any* faithful mimic (0.46 where the gap is < 0.05). On teacher-decisive
|
| 510 |
+
rows agreement is 0.954, and autotrust/JEV-9B's argmax captures 97.7 % of the teacher probability mass a perfect mimic
|
| 511 |
could (0.693 vs 0.709).
|
| 512 |
* **Fixed option sets.** `noul` and `score` accept only their canonical options; `choice` accepts 2–16
|
| 513 |
options. Inputs longer than 1,024 tokens are truncated (state only, head 60 % / tail 40 %) at serving unless
|
|
|
|
| 515 |
* **English-centric.** The corpus is English; multilingual behaviour is inherited from the backbone but was not
|
| 516 |
measured.
|
| 517 |
* **Placeholder labels in the corpus.** The `yuri_v1` memory-relevance stream is 100 % exact-uniform
|
| 518 |
+
`[0.5, 0.5]` — those rows teach nothing about relevance. autotrust/JEV-9B outputs ≈ 0.5 on them by design; do not use it
|
| 519 |
for memory-relevance scoring without further training.
|
| 520 |
* **Not for high-stakes decisions.** Use confidence gating: act automatically only above a threshold you
|
| 521 |
validated on your own data, and route the rest to a stronger model or a human.
|
|
|
|
| 541 |
|
| 542 |
Weights: **Apache-2.0** (base model `Qwen/Qwen3.5-9B` is Apache-2.0; training corpus
|
| 543 |
`SargeDev/jev-distill-corpus-v3` is Apache-2.0, its `openjev_v2` stream additionally CC0). The System One
|
| 544 |
+
framing and the `noul` / `choice` / `score` primitives originate with TypeSafe AI's Jev; autotrust/JEV-9B is an
|
| 545 |
independent student model trained on public data and shares no weights, code or affiliation with TypeSafe AI.
|
| 546 |
|
| 547 |
```bibtex
|
| 548 |
@misc{autotrust_jev_2026,
|
| 549 |
+
title = {autotrust/JEV-9B: an open-weights student of TypeSafe Jev 1.13 for calibrated System One decisions (Qwen3.5-9B)},
|
| 550 |
author = {autotrust},
|
| 551 |
year = {2026},
|
| 552 |
+
url = {https://huggingface.co/autotrust/JEV-9B}
|
| 553 |
}
|
| 554 |
```
|