JPT-4B

A fast decision model. Give it a situation (text, optionally with images) and typed questions. For every option it returns a probability from one forward pass: no generated explanation and no reasoning tokens. The softmax temperature is fit once, globally, on a held-out split — never per benchmark. Calibration varies by task (ECE 0.02–0.19 in the tables below), so read the probabilities as well-ordered confidence, not exact frequencies everywhere.

It playing 2048, Snake and Tetris: each move is one choice question answered in one forward pass (30 moves each, seed 0, served by SGLang, recorded headless):

2048	Snake	Tetris

• Question types: choice (pick one of 2–255 options), score (a level on an ordered scale) and noul (yes/no).
• One pass per question: the answer is read from the next-token distribution over the option labels. There is never a thinking phase, so latency is one prefill.
• Images: the vision tower of Qwen3.5-4B is unchanged, so screenshots, photos and video frames work next to the text state.

Built on Qwen/Qwen3.5-4B: a LoRA fine-tune merged into full weights.

Results

JevBench v1.4

JevBench v1.4.0 scores systems on 231 public items (frozen since v1.2), a private judge tier and held-out hard items, and 308 sealed items. Only the benchmark maintainer can run the sealed items, so JPT-4B has no official v1.4 score yet. The column we can reproduce is public accuracy (all 231 public items, the public_accuracy field of the v1.4 results). Our runner gives JevK5 0.861 there against its official 0.853.

System	Params	Public accuracy (231)	Sealed accuracy (308)	v1.4 score
JPT-4B	4B	0.879	pending	pending
Jev 1.13.0 (TypeSafe AI, API)	closed	0.866	0.367	63.3
JevK5 v0.2.0	27B	0.853	0.331	62.0
Winnow-12B Q8	12B	0.857	0.331	55.6
Hopper	—	0.823	0.341	59.4
reflex 4B	4B	0.792	0.282	54.0
local-jev Qwen3.5-4B	4B	0.805	0.260	46.8
SemIf, formerly OpenJev (TheoLeeCJ, Qwen3.5-4B)	4B	0.810	0.263	47.7
openjev 4B v5 (AlexWortega)¹	4B	0.814	—	—
decider-35b-a3b	35B-A3B	0.831	0.315	41.2
kev 4B (research preview)	4B	0.662	0.224	36.1
Qwen3.5-4B, same prompt, zero-shot (our run)	4B	0.740	—	—

Other systems' rows are copied from results/v1.4/jevbench-v1.4-results.json at jevbench commit 2fa63fa (2026-09-23). ¹ Not in the v1.4 results. The number is its own card's report on the 231 public items with the benchmark's harness (hard 0.622, standard 0.986, easy 1.000).

Head-to-head with the NeoHorse-Jev-4B comparison

The same items, selection rules and metrics as the text table on the NeoHorse-Jev-4B card (results dated 2026-09-24). Only JPT-4B was run by us; every other number is copied from that card.

Model	Params	JevBench (family-macro)	Kev (clean acc.)	OpenJev text, 18 tasks	Mean of the three
JPT-4B	4B	88.84	79.69	69.20	79.25
NeoHorse-Jev-4B	4B	75.73	81.92	56.75	71.47
Open-Jev-9B	9B	77.13	77.87	63.75	72.92
Kev-4B	4B	73.71	81.47	52.82	69.33
Laya English	—	55.82	61.30	37.24	51.45

What differs from their table, and per-task numbers

• OpenJev is 18 tasks, not 19. GSM8K nli_rerank@4 needs NeoHorse's frozen candidate pool, which is not published, so it is left out. The other models' 18-task means are recomputed here from the per-task numbers on their card; they are not on the card.
• Nimble, VitaminC and MASSIVE are not run. Their frozen subset IDs are not published, so we cannot guarantee the same items. For the same reason there is no six-group AVG for JPT-4B.
• GPQA. OpenJev lists the correct answer first. That doesn't matter for per-option scorers, but JPT-4B reads all four options at once, so we shuffle each item's options with a fixed seed. This can only lower our score.
• Kev. 177 transcripts whose turns use the role customer are sent as {"conversation": [...]}, because Qwen's chat template rejects that role.
• Kev comes from jaredpalmer/kev at 30c619b. OpenJev's task loaders are imported unchanged from AlexWortega/openjev at 552759d.

OpenJev task	JPT-4B	NeoHorse-Jev-4B	Open-Jev-9B	Kev-4B
scitail	79.21	87.02	79.16	84.81
anli r1 / r2 / r3	71.90 / 60.00 / 63.08	67.20 / 56.20 / 53.42	74.00 / 66.30 / 59.42	65.60 / 54.30 / 52.25
wanli	65.10	65.74	67.10	63.50
control	67.20	65.96	67.58	64.35
MNLI matched / mismatched	86.52 / 86.71	88.95 / 89.46	80.64 / 80.54	89.17 / 89.35
arc_easy / arc_challenge	97.47 / 92.83	88.93 / 78.50	95.71 / 87.29	75.42 / 66.89
winogrande	68.59	60.69	66.30	58.33
gsm8k_mc4 / mc10	65.50 / 68.84	41.77 / 21.83	52.69 / 35.71	38.59 / 20.77
gpqa / gpqa_fewshot	40.40 / 38.89	34.34 / 34.34	37.37 / 39.90	33.84 / 34.85
chess	51.80	22.00	52.40	19.20
hellaswag	84.66	34.95	54.09	18.92
mmlu / mmlu_fewshot	72.37 / 71.08	59.19 / 60.20	66.80 / 65.05	52.29 / 57.50

Kev suite	JPT-4B	NeoHorse-Jev-4B	Kev-4B
decision-v7 dev / test	82.67 / 80.58	86.23 / 86.58	87.18 / 87.08
transfer-v4 dev / test	80.18 / 84.15	81.71 / 84.60	79.73 / 83.69
transfer-v9 dev / test	74.86 / 75.72	75.53 / 76.86	74.76 / 76.39

Decision Index

Decision Index 0.2 (2026-09-24), the full frozen suite (162,841 requests, 40 benchmarks, chance-corrected), run ourselves through llm2jev over SGLang and submitted as apolinario/decision-index#7:

Model	Params	Decision Index
JPT-4B	4B	40.33
Hopper	—	36.71
Decider 4B	4B	36.58
JevK5	27B	36.31
Kev 4B	4B	31.31

Full run and scores.json: kirp/decision-index-results-jpt-4b (gated: it carries the suite's GPQA/HLE item text).

Other benchmarks

Benchmark	What it tests	JPT-4B	Qwen3.5-4B (same prompt, zero-shot)
JevBench public hard tier (111)	hardest general decisions	0.784 (ECE 0.068, Brier 0.318)	0.595
Typed decisions test (2,000)	in-distribution typed decisions	0.796 (ECE 0.160, Brier 0.332)	0.596 (ECE 0.171, Brier 0.559)
ANLI r1 / r3	adversarial natural-language inference	0.697 / 0.613	0.660 / 0.513
Banking77 / MASSIVE (en / de / zh)	intent classification, incl. multilingual	0.757 / 0.857 / 0.833 / 0.837	0.663 / 0.733 / 0.670 / 0.703
EnvBench v0.1 public / held-out (skill, 0–100)	sequential decisions in game/puzzle envs (2048, Snake, chess, …)	47.7 / 47.0	—
ScreenSpot-v2 set-of-marks (300, images)	GUI element grounding from a screenshot	0.923 (ECE 0.029)	0.903 (ECE 0.058)
Screen2Words match (300, images)	screenshot summarization	0.927 (ECE 0.037)	0.887 (ECE 0.030)
ERQA (400, images)	embodied/robotics visual reasoning	0.455 (ECE 0.150)	0.463 (ECE 0.100)

Numbers are accuracy, with ECE (10 bins) and Brier where shown, at the fitted temperature T = 1.036. The banking/MASSIVE/typed rows are in-distribution: their train splits are in the training mix, their test items are not. Image rows are zero-shot: no image or robotics data was trained on (ERQA is 0.8 points below the base model).

JevBench public hard tier by family (n):

Family	JPT-4B	Qwen3.5-4B zero-shot
long_policy (19)	0.74	0.47
judge_hard (17)	0.76	0.71
multi_hop (18)	0.83	0.56
temporal_numeric (15)	0.40	0.33
probability (10)	0.90	0.50
trap / adversarial / routing_hard (19)	1.00	0.95
ambiguous (7)	0.86	0.71
tradeoff (6)	0.83	0.33

Option order: reversing the options of the 139 public choice questions changes 8 verdicts; accuracy is 0.892 either way.

Quick start

llm2jev (>= 0.6.1) in front of the weights, reading option probabilities off the running engine. Two things any real model server is — an engine, and a client that talks to it:

python -m sglang.launch_server --model-path kirp/jpt-4b --port 30000 \
  --context-length 32768 --mamba-scheduler-strategy extra_buffer &  # Qwen3.5's DeltaNet layers need this flag
llm2jev --model kirp/jpt-4b --backend sglang --url http://127.0.0.1:30000 --port 8080 --temperature 1.036

vLLM instead: vllm serve kirp/jpt-4b --max-logprobs 256 --return-tokens-as-token-ids --enable-scale-out --port 8000 then llm2jev --model kirp/jpt-4b --backend vllm --url http://127.0.0.1:8000 --port 8080 --temperature 1.036 — the three vLLM flags are required, not optional: without them every request comes back a plain HTTP 400 with no hint why.

No GPU / just trying it out, no engine, no clone — llm2jev runs the model itself:

pip install "llm2jev[hf,vision]"
llm2jev --model kirp/jpt-4b --backend hf --port 8080 --temperature 1.036

This path serializes requests (one forward at a time in-process — concurrent calls are safe, just not parallel); sglang/vllm hand batching to the engine and scale with concurrent load, which is what our own eval and serving use — hf is for a quick check, not for traffic.

Tested with SGLang 0.5.9. Its torch 2.9.1 pins cuDNN 9.10, which SGLang refuses to run on, so install cuDNN 9.15+ over it: pip install "sglang==0.5.9" && pip install "nvidia-cudnn-cu12>=9.15". --temperature 1.036 is the temperature fitted on the calibration split; leaving it at 1.0 changes calibration slightly, never the ranking.

import requests
r = requests.post("http://127.0.0.1:8080/v1/systemone", json={
    "state": "Refund policy: full refund within 30 days of purchase; 50% until day 60; none after.\n"
             "Order 1182 was bought on 3 March and returned on 20 April.",
    "questions": {
        "refund": {"type": "choice", "instructions": "What refund does order 1182 get?",
                   "criteria": {"full": "Full refund", "half": "50% refund", "none": "No refund"}},
        "late":   {"type": "noul", "instructions": "Was the return made after day 30?",
                   "criteria": {"true": "Yes", "false": "No"}}}})
print(r.json()["answers"])   # each answer has the per-option probabilities

With an image, make the state a chat message list and put image parts before the text:

state = [{"role": "user", "content": [
    {"type": "image", "image": "https://example.com/screen.png"},
    {"type": "text", "text": "Task: open the settings page. Numbered boxes mark clickable elements."}]}]
questions = {"click": {"type": "choice", "instructions": "Which box should be clicked?",
                       "criteria": {"1": None, "2": None, "3": None, "4": None, "5": None}}}

Training

• Method: LoRA (r=16) on every attention, DeltaNet and MLP projection of the language model, merged into full weights. The vision tower is untouched. The loss is the multi-class Brier score over the option labels, on llm2jev's chat prompt with thinking disabled: the same prompt the model is served with.
• Data (49,221 questions in 32,835 records, all converted to typed decisions; one epoch over two option-shuffled copies):
  • public classification, NLI, QA, preference and safety datasets;
  • long legal and contract documents (ContractNLI, MAUD, LegalBench, ConditionalQA, ShARC);
  • table and numeric reasoning (TAT-QA, MultiHiertt);
  • multi-hop QA (MuSiQue, BEIR);
  • agent and tool traces (Mind2Web, AgentTraj, ToolACE);
  • community typed-decision sets;
  • oracle-labelled rollouts from 20 small game and puzzle environments;
  • programmatically generated rule-arithmetic items (dates, time zones, day counts, caps; labels computed by code);
  • 1,289 long policy / contract / regulation documents (11–17k characters, 4,471 questions) with amendments, exceptions, precedence rules and constrained trade-offs (ranked rules, scarce-resource allocation, authority limits), written by an LLM (GPT-6 Luna) with no JevBench item shown to it.
• Held out: no item from JevBench, EnvBench's held-out seeds, the Decision Index frozen suite or our typed test split was used in training. Documents were checked for 8-gram overlap with JevBench.

Limitations

• It answers from the evidence it is given. Multi-step arithmetic and date computation are still its weakest area: it has no reasoning phase by design.
• Accepts up to 255 options. Training covered up to 77 (Banking77); beyond that, quality is not established.
• English first. Other languages were seen only through a few multilingual classification sets.
• Image questions work zero-shot through the base vision tower. No image data was used in fine-tuning.
• Weak at Minesweeper-style belief reasoning: in our demo loop it reveals cells in reading order and loses within a few moves, even though the game is in its training mix.

License

CC BY-NC 4.0. The weights derive from Qwen3.5-4B (Apache-2.0), but some training datasets allow only non-commercial or research use, so the model is released for non-commercial use.

---

JPT-4B is an independent open model that implements a typed-decision interface (noul, choice and score questions answered with probabilities). It is not affiliated with, endorsed by or derived from TypeSafe AI or its Jev model, and it was not trained on Jev outputs.