▶ Watch it play, live · PyTorch + CUDA (main repo) · GGUF (llama.cpp) · Which version? · Quickstart · Images · Benchmarks

qev-0.8b-MLX: the tiny local decision model, for Apple silicon

qev-0.8b for Apple silicon with MLX.

weights	folder	size	argmax agreement with the CUDA reference¹
bf16	repo root	1.4 GB	99.3%
8-bit (affine, group size 64)	q8/	0.78 GB	99.0%

¹ Measured on the first release's weights; the conversion and runner are unchanged. A 4-bit build is not provided: it changes about 6% of the argmax answers, which is too much for calibrated probabilities.

qev reads anything (a game board, a post, an inbox, a web page, a log) and answers any question about it with calibrated probabilities in a single forward pass. Out of 27 decision models of its size it scores highest on Benchmark Heaven, it answers faster than all of them, and in 8-bit it needs 1.3 GB of RAM on a Mac.

What you're watching. These are screen recordings of the live demo, with nothing staged. One copy of qev-0.8b on one GPU runs all of these at once, in real time: 🐍 Snake, 🧱 Tetris, 👻 Pac-Man, 🔢 2048, 🏓 Breakout and 🔨 Whac-A-Mole. Tetris, Pac-Man, 2048, Breakout and Whac-A-Mole were held out of training. 🧹 LinkedIn slop screening: bait, humblebrag, made-up story, buzzwords and substance, five questions per post. 🛡️ A prompt-injection firewall for an agent's inputs. 🧭 An agent tool router, 🎧 support triage, ⭐ review ratings and 📥 inbox sorting: six emails and 18 questions in one request. For the games, the game code writes down what each move would do ("it hits the wall and dies", "clears 2 lines"), and qev answers one yes/no question per move. The most likely "yes" is played. qev does no search and no lookahead; every move is one forward pass of about 4 ms.

At a glance

	qev-0.8b
parameters	0.8B (full fine-tune of Qwen3.5-0.8B-Base)
Benchmark Heaven	77.1%: first of 27 decision models around 1B
latency	3.5 ms p50 per request on 1× L40S: fastest of 27 · 109 ms on an M4 Pro Mac (MLX)
memory	1.7 GB weights in bf16, 0.8 GB in 8-bit · 1.3 GB RAM on a Mac in 8-bit
also first on	JevExtra (72.0%, agent and safety skills), JevSpatial (74.2%, games and space), Rules-7 (63.7%, tied)
output	calibrated probabilities for choice, noul (yes / no) and score questions, up to 255 options each
images	image input (zero-shot) in both builds: <image:NAME> in the state, at most 196 tokens per image by default · JevVision 0.757 pooled in bf16 and 8-bit, as the transformers reference (0.758) · Images
formats	MLX bf16 and 8-bit for Apple silicon (this repo), PyTorch + fast CUDA engine, GGUF (also stock llama-server at /v1/systemone)
license	Apache-2.0

qev does not generate text. Each answer is a probability distribution, so you can act when it is confident and escalate to a person or a larger model when it is not. Nothing has to leave your machine.

Which version should I use?

Where you run it	Use	Text, one request (p50)	Images, one image per request (batch 1)
NVIDIA GPU, server or Python (recommended for production)	labqoat/qev-0.8b: pip install -r requirements.txt -r requirements-cuda.txt, then python qev_server.py or QEV.from_pretrained("labqoat/qev-0.8b", images=True) (the fast CUDA engine)	3.5 ms (L40S, benchmark items); 5.7 ms on longer dev requests	18–32 ms (L40S, 224×224 to 768×768)
Mac (Apple silicon)	labqoat/qev-0.8b-MLX: qev_mlx.py	109 ms (M4 Pro, bf16, dev requests; 57 ms up to 512 tokens)	not measured on a Mac yet
llama.cpp anywhere: CPU, Metal, CUDA, Vulkan	labqoat/qev-0.8b-GGUF: llama-server -hf labqoat/qev-0.8b-GGUF:Q8_0 for text, llama-server -hf labqoat/qev-0.8b-GGUF:vision-Q8_0 for text plus images	about 17 ms (llama-server, M4 Pro, Metal, a short one-question request)	25 ms (llama-server, L40S CUDA, 448×448, one question); not measured on a Mac
Anything else (Apple MPS or CPU with PyTorch)	labqoat/qev-0.8b: qev.py (transformers backend; pip install -r requirements.txt) and qev_image.py	not measured	61–69 ms (qev_image.py on L40S CUDA); MPS / CPU not measured

All four serve the same model with the same System One API (POST /v1/systemone) and the same answer format. The recommended path is the main repo's fast CUDA engine on an NVIDIA GPU, through qev_server.py or qev.py.

Recommended encoding. Use per_question=True for image requests and for requests with several independent questions. On the server, add "encoding": "per_question" to the request (or start it with --encoding per_question); in qev_mlx.py, use encoding="per_question"; llama-server's vision file always answers each question in its own prompt. On JevVision v0, per_question raises the fast engine from 0.757 to 0.783 pooled accuracy (ECE 0.115 to 0.105; one L40S, same run). MLX and GGUF measure 0.785 with ECE about 0.10. On text, it raises JevBench coherence from 64.5 to 71.2 at about the same accuracy. The cost is more compute per question. On the fast engine, a text request computes the shared state once (7.9 ms against 5.7 ms p50 on dev requests), while an image request runs one full row per question (about 9% fewer requests/s on JevVision). Single-question requests get the same answers in both encodings.

Benchmarks: 27 decision models around 1B

We ran qev-0.8b against 26 other decision models around 1B parameters. Every model got the same items and scoring, ran on one NVIDIA L40S, and used its own published inference recipe.

Accuracies are in %. ECE, latency and memory are lower-is-better. Bold marks the best value in each column.

model	params	VRAM (GB, est.)	p50 ms	BH	BH ECE ↓	JevBench	coherence	suite	JevExtra	JevSpatial	Rules-7
qev-0.8b	0.8B	2.2	3.5	77.1	0.063	85.1	71.2	61.7	72.0	74.2	63.7
decision-2b (flymy-ai)	2B	5.6	81.3	76.2	0.042	86.0	61.8	60.9	64.1	69.5	58.1
decider-2b	1.9B	4.3	7.2	75.8	0.065	85.9	72.5	70.7	58.8	–	–
EXAONE-4.0-1.2B-JEV	1.2B	3.1	27.8	68.8	0.097	83.9	67.9	67.3	64.0	70.0	58.8
malkuth-2b	2B	4.3	46.2	68.8	0.131	84.8	70.8	66.6	63.6	69.9	63.7
classone (Gemma-4-E2B)	~5B	10.7	56.5	66.7	0.260	82.1	66.6	48.2	52.5	63.4	48.1
jul-2b	2B	5.5	88.9	65.8	0.059	83.3	68.4	57.8	56.4	66.8	58.0
Open-Jev-2B	1.9B	4.3	191.7	64.5	0.127	84.9	76.2	59.9	57.6	72.2	59.7
decision-fast	0.6B	1.7	58.9	63.6	0.106	73.0	54.3	52.0	52.0	48.5	47.1
Tacet Sonata¹	144M	0.8	20.5	62.3	0.148	64.9	45.5	31.6	36.1	50.6	35.4

The other 17 models

model	params	VRAM (GB, est.)	p50 ms	BH	BH ECE ↓	JevBench	coherence	suite	JevExtra	JevSpatial	Rules-7
smalljev-2.5b¹	2.5B	5.6	68.8	61.5	0.142	71.2	61.1	45.6	46.2	53.7	49.3
jevos-v5 (feder-cr), partial	~0.9B	1.2	–	61.0	–	84.1	–	49.9	–	–	–
Quyet-1.0-Small¹	328M	1.8	19.3	59.7	0.066	71.3	56.8	43.8	42.4	48.4	45.4
watt-flash-0.1	140M	0.8	15.7	59.7	0.073	67.1	61.9	48.8	39.1	51.3	36.3
Ghafiri Qwen3.5-0.8B	0.8B	2.0	44.6	59.3	0.083	78.1	61.7	41.6	41.2	62.1	40.1
WaterSheep	~150M	1.1	18.9	59.3	0.127	73.5	57.9	56.4	50.8	57.9	41.0
lev-350m	361M	1.2	24.1	58.4	0.094	66.9	65.5	47.5	44.5	–	–
deem-0.8b¹	0.8B	2.0	54.6	58.0	0.240	75.0	59.3	42.9	48.3	51.1	46.8
Laya	421M	1.3	24.4	58.0	0.095	70.7	56.8	53.7	43.0	47.8	44.1
openjev-verdict¹	151M	1.1	13.5	57.6	0.081	63.9	59.7	32.8	37.9	38.4	34.7
Quyet-1.0-Tiny¹	183M	1.2	17.5	57.1	0.097	67.7	54.9	42.4	46.9	52.6	47.1
Qwen3-1.7B (raw)¹	1.7B	4.6	27.8	55.8	0.411	64.5	70.7	39.8	48.7	52.4	41.3
SimpleJev (Qwen3.5-0.8B)	0.8B	2.2	82.9	55.4	0.183	55.6	46.6	47.6	45.6	54.1	36.1
Open-Jev-DeBERTa¹	435M	2.2	34.4	55.0	0.091	79.0	53.3	38.4	41.5	54.2	42.0
verdict-small	118M	1.0	8.6	53.7	0.115	50.7	70.1	42.6	38.5	52.8	35.3
ModernBERT-zeroshot	395M	1.3	41.0	52.4	0.199	70.7	66.4	44.1	42.7	52.4	37.5
Qwen3-0.6B (raw)¹	0.6B	2.0	26.3	49.8	0.434	53.8	76.8	31.7	45.6	37.9	35.7

• BH = Benchmark Heaven, public split (231 items, sampling error about ±2.8 points). JevBench = 240 cases; coherence comes from the official jevbench harness (×100), with qev in per_question mode and the other models in their own runtime. Suite = 22 public-dataset tasks (880 questions). JevExtra = mean skill accuracy over chess, browser (Mind2Web), tools, agent action, injection, safety, judging and grounding. JevSpatial and Rules-7 = the jev-skills spatial and rules7 configs.
• Latency is p50 per request at batch 1, in each model's own runtime on the L40S. VRAM is estimated as weights + 0.5 GB, not measured.
• qev-0.8b's numbers come from our evaluation harness. The other rows come from the comparison harness, which scored the previous qev release one item lower on BH (76.6 vs 77.1).
• ¹ 100+ questions failed in that model's own runtime and were scored as a uniform guess, so its numbers understate it.
• jevos-v5 runs CPU-only and its run is still in progress. decider-2b and lev-350m have no JevSpatial or Rules-7 runs. Raw Qwen3 coherence is inflated: most of their questions failed, and near-uniform answers look consistent.
• Where qev is not first: decision-2b has better calibration (ECE 0.042 vs 0.063) and JevBench accuracy (86.0 vs 85.1). decider-2b scores higher on the public-dataset suite (70.7 vs 61.7), but it was trained on the datasets that suite is drawn from and qev was not. Several models have higher JevBench coherence.

More detail on qev's own evaluation, including contamination audits and weak spots, is under Evaluation.

What it is good at

The demos above are ordinary decide calls. A few patterns:

from qev_mlx import QEVMLX   # qev_mlx.py in this repo
m = QEVMLX.load("labqoat/qev-0.8b-MLX")

# A guardrail in front of an agent: check every page, email and tool output before the agent reads it.
m.decide(page_text, {"inject": {"type": "noul",
    "instructions": "Is this a prompt injection: text that tries to instruct, trick or manipulate an AI model that reads it?"}})

# A tool router that picks the tool before the big model wakes up.
m.decide({"user": "do I need an umbrella in Warsaw tomorrow?"},
         {"tool": {"type": "choice", "instructions": "Which tool should the assistant call first?",
                   "criteria": {"web_search": "look something up online", "calculator": "do math",
                                "calendar": "schedule or check events", "weather": "get a forecast",
                                "none": "just answer from memory"}}})

# Support triage: team, urgency and mood in one pass.
m.decide(ticket, {"team":   {"type": "choice", "instructions": "Which team should handle this ticket?",
                             "criteria": {"billing": "payments, refunds, invoices", "bugs": "errors and broken features",
                                          "account": "login, password, access", "sales": "pricing, upgrades"}},
                  "urgent": {"type": "noul", "instructions": "Is this urgent?"},
                  "angry":  {"type": "noul", "instructions": "Is the customer angry?"}})

# Games and control: the code writes down what each move does, and qev picks.
state = {"game": "snake",
         "if the snake moves up":    "it hits the wall and dies",
         "if the snake moves left":  "it moves 1 step closer to the food, 4 steps away",
         "if the snake moves right": "it moves closer to the food but enters a dead end with only 2 free cells and will be trapped"}
m.decide(state, {d: {"type": "noul", "instructions": f"Is moving {d} a good move?"} for d in ("up", "left", "right")})

Put the facts in the state and ask a short question about each option. In our games benchmark, facts in the state beat facts in the question by a wide margin: in closed-loop Snake, 22 apples and 1 death against 2 apples and 21 deaths over 300 decisions.

Quickstart

pip install mlx==0.32.4 mlx-lm==0.32.2 tokenizers decider-ai==1.9.0 huggingface_hub

from qev_mlx import QEVMLX   # qev_mlx.py in this repo

m = QEVMLX.load("labqoat/qev-0.8b-MLX")                    # bf16; QEVMLX.load(..., subfolder="q8") for 8-bit
m.decide({"ticket": "My card was charged twice for the same order, please refund one."},
         {"dept":   {"type": "choice", "instructions": "Which team should handle this?",
                     "criteria": {"billing": "payments and refunds", "tech": "bugs", "sales": None}},
          "urgent": {"type": "noul", "instructions": "Is this urgent?"}})
# {"dept": {"type": "choice", "choice": "billing", ..., "probabilities": {"billing": 0.9505, ...}},
#  "urgent": {"type": "noul", "noul": 0.1645}}

m = QEVMLX.load("labqoat/qev-0.8b-MLX", encoding="per_question", cache_mb=1024)

python qev_mlx.py --state '{"ticket": "Refund please"}' \
  --questions '{"urgent": {"type": "noul", "instructions": "Is this urgent?"}}'

The model is mlx-lm's Qwen3.5 (mlx_lm.utils.load_model). qev does not generate text, so mlx_lm.generate is not the way to use it. qev_mlx.py runs one forward pass and reads the answers:

1. The request becomes decider's state-first System One prompt row.
2. The hidden state at each Answer: ( slot is multiplied by the rows of the option-letter tokens in the tied embedding (qev_config.json lists them).
3. Each question's answer is the softmax over its options, tempered per question type (choice 1.7, noul 0.9, score 1.25).
4. The answers use decider's /v1/systemone format.

Options

• encoding="per_question": one row per question, for higher consistency across the questions of a request. The "Context:\n" + state prefix is computed once: mlx-lm's cache keeps the Gated DeltaNet conv and recurrent states and the attention K / V. The question suffixes then run as one batch from copies of that state. This is equal to running each row on its own up to bf16 round-off (99.7% argmax agreement).
• cache_mb=N: an LRU of those prefix states, keyed by the hash of the prefix tokens. A request on an already-seen state, such as the same document with other questions, runs only its question part. Answers are bit-identical to the uncached path. One state takes about 25 MB.
• compile=True (the default): mx.compile of the joint forward per 64-token length bucket. The median is about 13% lower; each new bucket compiles once, in about 0.4 s.

HTTP

qev_mlx.py is a Python library and a command-line tool; it has no HTTP server. To serve qev over HTTP on a Mac with the System One API (POST /v1/systemone), use stock llama-server with qev-0.8b-GGUF (Metal; the :vision-Q8_0 file for images), or qev_server.py from the main repo (transformers on Apple MPS).

Image input

qev can also read images, in both builds. qev_mlx.py runs the vision tower and merger that qev inherited from Qwen3.5-0.8B-Base (unchanged: qev's training touched only the text layers), ported to MLX in qev_mlx_vision.py with its weights in vision.safetensors (bf16, 0.2 GB; the 8-bit build in q8/ uses the same bf16 vision weights). Put <image:NAME> in the state where the image belongs and pass the image under that name:

pip install mlx==0.32.4 mlx-lm==0.32.2 tokenizers decider-ai==1.9.0 huggingface_hub pillow numpy

from qev_mlx import QEVMLX

m = QEVMLX.load("labqoat/qev-0.8b-MLX")                    # or QEVMLX.load(..., subfolder="q8")
m.decide({"ticket": "A customer sent this screenshot: <image:shot>"},
         {"error":  {"type": "noul", "instructions": "Does the screenshot show an error dialog?"},
          "screen": {"type": "choice", "instructions": "Which screen is this?",
                     "criteria": {"checkout": None, "login": None, "settings": None}}},
         images={"shot": "screenshot.png"})               # a path or a PIL image
m.decide_batch([(state, questions, images), ...])        # images are optional per request

python qev_mlx.py --state 'Customer screenshot: <image:shot>' --image shot=screenshot.png \
  --questions '{"error": {"type": "noul", "instructions": "Does the screenshot show an error dialog?"}}'

It works as qev_image.py in the main repo does:

• An image the state does not mention is appended at the end of the state; several <image:...> markers work.
• Each image is resized (aspect kept, each side a multiple of 32) to at most max_pixels, 448×448 by default, and becomes <|vision_start|>, one <|image_pad|> per 32×32 pixels, <|vision_end|>: at most 196 tokens per image (QEVMLX.load(..., max_pixels=640*640) allows up to 400). m.image_tokens(image) gives the count.
• The vision embeddings take the place of the pad tokens, the attention layers get Qwen3.5's 3D (M-RoPE) positions, and the answers are read at the same Answer: ( slots with the same letters and temperatures as text requests.
• A request without images takes the text path unchanged: the same answers as before images were added (0 flips on 50 text requests, joint and per_question, bf16 and 8-bit).
• Image requests use the same fast path as text requests: an mx.compiled forward per 64-token length bucket; decide_batch runs image requests together (batch_size, default 8, sorted by length), and the images of a batch go through the vision tower together (images of one size as one compiled batch); per_question and cache_mb share the image-bearing "Context:\n" + state prefix like text does (the prefix-cache key includes the images' content, not only the tokens); and an LRU of image embeddings (image_cache=64 images) skips the vision tower for an image it has already seen.

JevVision v0 (zero-shot). We measured image support on JevVision v0, our internal image decision benchmark: 1,306 items in 17 skills (charts, scenes of shapes, boards, documents, app screens, and public VQAv2, GQA, CLEVR and ScienceQA evaluation splits), each a state with one image and typed questions with exact gold answers. The benchmark is described in the main repo; it is not released. Every model below gets the same items, prompts, readout and temperatures; pooled is the mean over the 17 skills.

family	transformers (qev_image.py)	MLX bf16	MLX 8-bit
charts (4 skills)	0.881	0.877	0.883
scenes (3)	0.838	0.837	0.839
documents (2)	0.868	0.870	0.868
app screens (1)	0.833	0.841	0.838
boards (3)	0.519	0.517	0.514
public splits (4)	0.679	0.678	0.676
pooled (17 skills)	0.758	0.757	0.757
pooled ECE (lower is better)	0.113	0.111	0.113
top answer equal to transformers		98.7%	98.4%

The MLX builds score as the transformers reference does: every family is within 0.01, and 98% of the 2,371 top answers are the same (the rest are near-ties that bf16 round-off tips either way). The MLX columns were run through decide_batch (batches of 8). With encoding="per_question" (one row per question, which qev_image.py does not offer) MLX bf16 scores 0.785 pooled, ECE 0.100. Each skill has 70–100 items, so a single skill's accuracy is uncertain by about ±0.03–0.05. qev is strong on charts, scenes, documents and app screens and weak on boards (tic-tac-toe, connect-four, chess) and on counting real objects; see the main repo for the comparison with the base models. These numbers were measured with MLX's Linux CUDA build on the same files.

Speed: Linux numbers only, not Apple silicon. Image latency on a Mac has not been measured yet. The table below is MLX's Linux CUDA build (mlx[cuda12]==0.32.4) on one L40S. On that backend mlx-lm runs the Gated DeltaNet layers as a per-token loop (its fused kernel is Metal-only), so time grows with the number of tokens and is far above what a Mac gives. Use the table to compare the paths with each other, not to predict Mac latency.

Linux, mlx[cuda12], L40S, bf16	value
one image request, batch 1, p50: 224×224 (49 image tokens)	490 ms
448×448 (196 image tokens)	877 ms
768×768 (576 image tokens, max_pixels=768*768)	2,047 ms
throughput, 128 benchmark requests: one at a time	1.0 requests/s
decide_batch, batch 8 / batch 16	5.5 / 6.6 requests/s
text requests, p50 over 30 requests, joint / per_question: before images were added	622 / 639 ms; 638 / 646 ms
the same with this runner	655 / 663 ms; 603 / 609 ms

The two text rows come from two runs with the runners in opposite order (before first, then after first). The text code path is the same in both runners, and whichever ran first was faster, so text speed is unchanged. A repeated image (the embedding cache) saves little on this backend because the vision tower is a small part of the time.

Questions and answers

A request has a state (a string, an object or an array) and one or more questions, each of one type:

type	what it asks	answer fields
choice	pick one of the named options (criteria: {name: description or null})	choice, confidence, x_p_max, certainty, probabilities
noul	a yes / no question (instructions, optional criteria {"true": ..., "false": ...})	noul: the probability of true
score	a rating on an ordered scale (criteria: list of levels, low to high)	score (expected level index), confidence, x_p_max, certainty, legend, probabilities

The fields follow decider's format_answer, the /v1/systemone wire format:

• confidence is TypeSafe's confidence. For a choice, it is the top probability rescaled so that uniform = 0. For a score, it is 1 minus the expected distance from the top level, relative to a uniform distribution.
• x_p_max is the top probability.
• certainty is 1 minus the normalised entropy.

A string state is used as it is. An object or array is rendered as JSON with array indices annotated, as decider's server does. A question can have up to 255 options.

How a request is answered

1. The request becomes a prompt row in decider's state-first System One layout: Context: and the state, then each question with its lettered options, ending in an Answer: ( slot.
2. One forward pass reads the logits of the option letter tokens at each answer slot. Only the letters' rows of the tied embedding are used.
3. Each question's answer is the softmax over its own options, sharpened per question type with temperatures fitted on a held-out calibration split: choice 1.7, noul 0.9, score 1.25 (p -> p^(1/t), renormalised). The temperatures and the letter tokens are in qev_config.json.

Encodings

• joint (default): one row per request, with every question behind one copy of the state. This is the cheapest encoding.
• per_question (per_question=True): one row per question, each with its own copy of the state. Each answer then does not depend on which other questions were asked, or in what order. This raises JevBench coherence from 64.5 to 71.2 at about the same accuracy, and on JevVision v0 it raises image accuracy from 0.757 to 0.783 (ECE 0.115 to 0.105) on the fast engine. Its cost depends on the runner. On qev.py's fast engine, a text request computes the shared state once and then runs each question's part. An image request runs one full row per question. The transformers backend runs one full row per question. The MLX runner in qev-0.8b-MLX computes the shared state once, so there it costs about the same as joint.

The two encodings give the same answers for single-question requests. For multi-question requests where consistency across questions matters, use per_question.

In this repo, choose the encoding with QEVMLX.load(..., encoding="per_question"). The shared state is computed once, so per_question costs about the same as joint here.

Evaluation

All numbers come from one harness, with the same items and scoring for every model. The temperatures were fitted on our own calibration split. No benchmark data was used for training, tuning or calibration.

• Benchmark Heaven (BH): the public split, 231 items. The sampling error of BH accuracy is about ±2.8 points.
• JevBench: 240 cases, 1,938 gold questions. Coherence is scored by the official jevbench harness, ×100.
• Jev suite: 22 public-dataset tasks, 880 questions.
• Pooled: the mean of BH, JevBench and suite accuracy.
• JevExtra: our own skills benchmark: chess, browser, tools, agent action, prompt injection, safety, judging and grounding. The headline is the mean skill accuracy.
  • qev's indirect prompt-injection items share text with training rows (16-gram audit). The table includes them, like every other item.
• JevSpatial: our games and spatial benchmark, 2,040 items: game moves with the facts in the question and in the state, rules in the question, coordinates and small grids. Breakout, Whac-A-Mole, Knife Hit, 2048, Tetris and Pac-Man are never used in training.
• JevosTasks: seven policy and rule tasks built like the ones other System One models report (admission, rental, agent rules, authority, fraud points; ShARC dev; patent phrase similarity test), 3,440 items. The five generated domains are never used in training.

Where two values are given, they are joint / per_question. The two are equal on BH and the suite, which have one question per request.

benchmark	metric	qev-0.8b (joint / per_question)	Qwen3.5-0.8B-Base (zero-shot)	decider-0.8b
Benchmark Heaven	accuracy	0.771	0.528	0.658
Benchmark Heaven	Brier (lower is better)	0.337	0.569	0.430
Benchmark Heaven	ECE (lower is better)	0.063	0.089	0.061
JevBench	accuracy	0.851 / 0.849	0.439	0.822
JevBench	coherence	64.5 / 71.2	51.6	69.6
Jev suite	accuracy	0.617	0.392	0.682
pooled (BH, JevBench, suite)	accuracy	0.746 / 0.745	0.453	0.721
JevExtra	mean skill accuracy	0.720	0.442	0.581
JevSpatial	accuracy	0.742
JevosTasks	accuracy	0.637

Contamination audit. None of the training data shares any 16-token sequence with a BH or JevBench item. In the Jev suite, 2 of 22,773 test-pool items reach 50% 16-gram coverage. The training data was also screened by meaning against every test item, using embeddings with a threshold set from near copies.

Where it is weakest:

• JevBench score questions (4-level scales): accuracy 0.39. Most errors are off by one level, and predictions lean toward the middle levels.
• Long and hard BH items: accuracy 0.51 on long items and 0.57 on the hard tier.
• The suite's wide-label and graded tasks: go_emotions, helpsteer2, stsb.
• Abstaining in agent decisions, when no tool fits or information is missing.

Runtime fidelity.

• On the 902-request dev set used for release checks (1,782 questions), qev.py on an NVIDIA L40S agrees with our CUDA reference path on 99.7% of argmax answers, with mean absolute probability difference 0.003.
• On Apple MPS (bf16) and on CPU (fp32), over the first 50 requests (103 questions), the agreement is 99.0%: one near-tie question flips. The mean difference is 0.003.
• The MLX and GGUF builds agree with the reference on 99.0–99.8% of argmax answers (see their repos).
• These agreement figures were measured on the first release's weights; the architecture, code and quantisation are unchanged.

This repo. qev_mlx.py agrees with the CUDA reference on 99.3% (bf16) and 99.0% (8-bit) of argmax answers. per_question with the shared state prefix agrees with running each row on its own on 99.7%.

Speed

qev_mlx.py (M4 Pro, 24 GB, bf16)

Steady-state batch-1 latency over 200 dev requests (mean 746 tokens, median 417).

setting	p50 latency
single request, compiled (default)	109 ms (57 ms for requests up to 512 tokens)
single request, per_question with a shared state prefix	128 ms
repeated state, prefix cache hit (joint)	115 ms; a 900-token state with a new question: 18.5 ms

Memory is 2.0 GB RSS and 2.6 GB Metal for bf16 (1.3 / 2.0 GB for 8-bit), plus about 25 MB per cached state. Prefill is compute-bound on the Mac GPU, so 8-bit saves memory, not time.

The CUDA engine in the main repo, for comparison

Measured on our dev requests: mean 746 tokens, median 417, about 2 typed questions each. These are longer than the benchmark items behind the 3.5 ms p50 in the 27-model comparison above, hence the higher figure here. CUDA: one NVIDIA L40S with our optimised engine (packed variable-length batches, the flash-linear-attention Gated DeltaNet kernels, CUDA graphs for small batches). Apple: an M4 Pro (24 GB) with MLX, bf16, compiled.

setting	hardware	p50 latency	throughput
single request, CUDA graphs	1x L40S	5.7 ms	~100 req/s, ~200 decisions/s
packed batch (32k tokens, ~43 requests)	1x L40S	~370 ms	~117 req/s, ~230 decisions/s
single request, MLX bf16	Apple M4 Pro	109 ms	~6 req/s, ~11 decisions/s

per_question (shared prefix): 7.9 ms p50 on the L40S. MLX prefix-cache hit: 18.5 ms.

The CUDA rows are what qev.py gives by default on an NVIDIA GPU: its fast backend (qev_fast/), after pip install -r requirements.txt -r requirements-cuda.txt. In a fresh environment installed from those two files alone (torch 2.14, flash-linear-attention 0.5.2, transformers 5.20; no vLLM), one L40S measured:

• 5.6 ms p50 and 19.5 ms p90 per request, 9.7 ms mean;
• about 105 requests/s one request at a time, and about 97 requests/s in packed batches (117 with vLLM's FlashAttention-2 build installed);
• 99.8% argmax agreement with our evaluation path on 1,782 dev questions.

Start-up: about 25 s until ready when the kernel cache is warm, about 2 min on the very first start (kernel compilation). The common CUDA graphs are captured at start; the rest are captured in the background after the model is ready.

The model is 0.8B parameters: 1.7 GB in bf16, 0.8 GB in 8-bit.

Training

qev-0.8b is a full fine-tune of Qwen3.5-0.8B-Base (all text layers and the final norm; the tied embeddings are the base's). It was trained on our own decision data, in stages:

1. Base fine-tune: two epochs, about 1.8B tokens, over a decision mix of 1.28M requests (learning rate 5e-6; DiLoCo over 16 workers).
2. Anneal on a newer data release (about 142M tokens).
3. Coherence anneal: about 52M tokens, with a consistency loss on twin sets built from our own probability laws: invariance to rephrasing and reordering, complements, partitions, bounds, monotonicity and conditioning. No benchmark code or items were used.
4. A further round on newer data with coherence twins (about 104M tokens).
5. Weight soups of the trunk tensors to keep calibration while adding consistency.
6. A gap pass (150M tokens) over data for weak skills, BH-style long hard decisions, hard yes / no and grounding negatives, order-invariance twins, and replay.
7. A 50/50 weight soup of the pre-gap and post-gap models (the first release).
8. A games, rules and spatial pass (109M tokens with replay; 27M new): code-generated game-move decisions with the facts in the question or in the state, rules written in the question, and coordinate / grid relations. The labels are computed by code; the benchmark-only games and domains above are excluded, and every row was screened against the benchmark items.
9. The release: a 50/50 weight soup of the model from step 8 and the first release. The per-type temperatures were then refit on the calibration split.

Training data and licenses

The decision data comes from public datasets and data generated for the project. Each training row records its source, its license and how its label was made. Labels are of these kinds:

label kind	share of rows
the source dataset's own label	about 49%
computed exactly by a program	37%
a writer model's label confirmed by two independent solver models	7%
a model's rating or soft distribution	about 7%

By license, for the decision dataset release the training mix was drawn from (2.2M requests):

• Permissive licenses, about two thirds of the rows: CC-BY-4.0 (about 35% in total, including NVIDIA's Nemotron agentic data, reasoning-core's procedural pile, FLD and ProofWriter), CC0-1.0 / public domain (about 17%, including the CFPB complaint database and NVD), Apache-2.0 (about 8%), MIT (about 5%, including QA sets under mixed MIT / CC-BY / Apache terms) and BSD-3-Clause (2.7%). A further 1.8% is under per-source upstream terms or ODC-BY.
• Share-alike sources, about 10.5%:
  • CC-BY-SA-4.0 from MTOP (1.6%), the Schema-Guided Dialogue dataset (1.1%), SQuAD-derived QNLI (1.1%) and HotpotQA (0.2%), plus other CC-BY-SA-4.0 sources (4.3%);
  • DBpedia (CC-BY-SA-3.0, 1.1%);
  • Bitext customer-support data (CDLA-Sharing-1.0, 1.0%).
• Generated for the project, about 17%:
  • computed by programs (6.3%);
  • written by an LLM and checked by two blind solver models (8.7%);
  • other generated rows released under CC0 (2.0%).
• Model-made labels: a further 1.2% carries ratings made by a language model, under MIT.
• Research-use releases: about 0.7% comes from the LeWiDi research release, whose component datasets carry their own terms, including research-use-only terms for one of them.

Later training stages added program-generated and writer/verifier-generated rows of the same kinds. The model is released under Apache-2.0. Whether share-alike or research-use terms on a part of the training data affect a given use of the weights is for the user to assess.

Limitations

• Ordinal scores are the weakest type. Treat score outputs as soft signals.
• Long and hard inputs. Accuracy drops on long states (several thousand tokens) and on deliberately hard items. The model reads the request in one pass and does no reasoning.
• Coherence is improved, not guaranteed. Complements and marginalisation across related questions, and unit restatements such as "90 min" vs "1.5 h", are still often violated. Use per_question for multi-question requests, and do not assume that answers to related questions are mutually consistent.
• Abstention. The model rarely chooses "cannot answer" or "no tool call" options, even when it should. Give it an explicit option and check its probability rather than relying on the argmax.
• Calibration is fitted, not universal. On data far from the calibration split, probabilities can be over- or under-confident. Recalibrate on your own labelled data when probabilities drive decisions.
• Language and domain. The data is mostly English business, operations and agent-log decisions. Other languages and domains have not been evaluated.
• Options. A question can have at most 255 options. Option names matter, because the model reads them.
• Images are zero-shot. qev was trained on text only and reads images through the base model's vision tower. It is weak on board positions (connect-four and chess are near chance, tic-tac-toe 0.56) and on counting real objects (CLEVR 0.39), and it was tested on one image per request; videos are not supported. Image requests cost more tokens than a short text state (up to 196 per image by default), and their speed on a Mac has not been measured (the Linux numbers above are not representative of Apple silicon).
• Inherited behaviour. The model inherits biases of the base model and of its data, part of which was labelled by larger models. Do not use it as the only decision maker for consequential decisions about people (medical, legal, credit, employment).

Files

file	what it is
model.safetensors, model.safetensors.index.json, config.json	the weights in bf16 and mlx-lm's Qwen3.5 config
q8/	the same model with 8-bit affine weights (group size 64): model.safetensors, model.safetensors.index.json, config.json, tokenizer.json, tokenizer_config.json, chat_template.jinja, qev_config.json, vision.safetensors
tokenizer.json, tokenizer_config.json, chat_template.jinja	the tokenizer
qev_config.json	temperatures, the 255 option-letter tokens, the prompt and readout description
qev_mlx.py	the MLX runner (text and images)
qev_mlx_vision.py	the vision tower, merger and M-RoPE text forward in MLX, used by qev_mlx.py for images
vision.safetensors, q8/vision.safetensors	the vision tower and merger in bf16 (the same file in both builds)
preprocessor_config.json	the base model's image processor settings (mean / std 0.5, patch 16, merge 2), for reference
SHA256SUMS	checksums
.gitattributes	Git LFS settings

Provenance

The base is Qwen/Qwen3.5-0.8B-Base at revision dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68. qev-0.8b is the release named "A1" (checkpoint SE50) in our internal reports.

The MLX weights are qev-0.8b's weights in mlx-lm's Qwen3.5 layout (mlx_lm.utils.load_model loads them); q8/ holds the same model with 8-bit affine weights, group size 64. vision.safetensors holds the base model's 153 vision tensors (tower and merger) from the qev-0.8b checkpoint, values unchanged, in bf16 in the layout of qev_mlx_vision.py.

License

Apache-2.0, as is the base model, Qwen3.5-0.8B-Base. See "Training data and licenses" for the data.