Manchego

A 4B model for typed decisions. One forward pass, a probability for every option.

Give Manchego a state (text or JSON), a question and a closed set of options. It returns a probability for each one: pick an option (choice, up to 255 options), a yes/no condition (noul), or an ordered level (score). It does not generate text. Qwen3.5-4B + LoRA (merged here), trained only on computed labels and original human annotations.

Benchmark (our runs)	Manchego v2.1	Rank of 28, our runs	Best other open ~4B
JevBench, 231 public decisions: our run, not an official JevBench score (JevBench-informed, see below)	0.805	3=	JevK5 0.861, resolved ahead
Held-out task types, 27 NI tasks (source-clean for Manchego)	0.701	5	decider-4b 0.717, not resolved
Public8, 1,600 real-text rows	0.732	15	decider-4b 0.800, trained on 7 of the 8 datasets

No open ~4B model is ahead on all three. None is resolved ahead on the held-out task types, or on Public8 datasets it did not train on.

About the JevBench numbers. Every JevBench figure on this card, in tables and charts, is our own run of the benchmark's 231 public decisions through each model's shipped code. It is not an official JevBench score or rank: the official score is measured by the JevBench maintainers on public, held-out and sealed decisions and combines intelligence, calibration, speed and cost.

The JevBench step (+0.030) is inside its interval, and that row is JevBench-informed (see Read before relying on it); the held-out task types step (+0.024) is resolved.

Quick start

Not a pipeline("text-classification") model: that pipeline would attach an untrained classification head. Read the option-letter logits as below.

# pip install "transformers>=5.17" torch      (tested: transformers 5.17.0, torch 2.10)
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("oraculumai/Manchego", revision="v2.1")
model = AutoModelForCausalLM.from_pretrained("oraculumai/Manchego", revision="v2.1", dtype=torch.bfloat16,
                                             device_map="auto").eval()

def decide(question, state, options):  # options: {value: description or None}, at most 26
    L = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"[: len(options)]
    menu = "\n".join(f"{c} = {v} — {d}" if d else f"{c} = {v}" for c, (v, d) in zip(L, options.items()))
    user = f"{question}\n\nState:\n{state}\n\nOptions:\n{menu}\n\nReply with only the letter of the best option."
    text = tok.apply_chat_template([{"role": "user", "content": user}], tokenize=False,
                                   add_generation_prompt=True, enable_thinking=False)
    ids = tok(text, return_tensors="pt", add_special_tokens=False).to(model.device)
    with torch.no_grad():
        logits = model(**ids).logits[0, -1].float()
    probs = torch.softmax(logits[[tok.encode(c, add_special_tokens=False)[0] for c in L]], 0)
    return dict(zip(options, probs.tolist()))

print(decide("Is this message spam?", "WIN a free cruise! Reply YES now.", {"yes": None, "no": None}))

Other formats: Manchego-MLX-8bit, Manchego-MLX-4bit. The 27–255-option prompt, the training recipe, every result and every weakness: DETAILS.md. Training code is not public; the prompt contract (contract_v2.py) and the aggregates behind the Manchego-only charts (eval/) are in this repository; other models' accuracies are the values printed on the field charts.

Benchmarks

Against Kev. Level with Kev-9B everywhere except the four Public8 datasets Kev trains on.

Every model on JevBench and Public8. Most of the Public8 leaders trained on, or cannot be cleared of, Public8's source data (purple).

JevBench by tier. Level with the best open ~4B on the easy and standard tiers; the whole gap is the hard tier.

Task definitions absent from its training data (sealed set). v2.1 removes v2's overconfidence (calibration error 0.153 to 0.040); its accuracy is below the untrained base on this set (0.695 against 0.723, not resolved).

The project's own held-out tasks. Where training bought the most.

Public8, dataset by dataset.

openjev's evaluators.

Calibration. Better than v2 everywhere except held-out real text, where v2 was already well calibrated.

Formats. MLX 8-bit stays within one decision of bf16 on the 80 task rows; MLX 4-bit loses six of those 80 rows (7.5 points).

Intended use and limits

• For: typed decisions over options your software supplies (routing, policy checks, triage, graded judgments), where you read the probabilities.
• Not for: unreviewed high-stakes decisions (medical, legal, financial, safety) without a person in the loop; open-ended text generation; images (not measured).
• Limits: up to 26 options with the short prompt, up to 255 with the state-first prompt; trained on prompts up to about 9,000 tokens (longer inputs are untested); English. Machine-readable: manchego_config.json.

Read before relying on it

• JevBench-informed. Our training families were designed from an earlier version's per-family JevBench hard-tier scores (the published spec only, no item text). JevK5 leads there: hard tier 0.739 against our 0.613.
• Lineage. Every training label is computed or human-annotated; no model wrote a label. Two caveats: an early audit used Jev (TypeSafe AI's hosted decision service) to flag phrases in two of our data pools, and 86 phrases were dropped; and the first stage's registered checkpoint selection read one development group whose targets came from Jev (a rule excluding that group selects the same checkpoint).
• Held-out tasks. The accuracy gain over the base (+0.024 on 27 held-out tasks, resolved) comes from the first training stage; v2.1's second stage improved its probabilities, not its accuracy.

How the comparison was run

Every other model ran through its own pinned code and prompt on a disposable cloud A100; Manchego, v2 and the untrained base were scored on an RTX 5090. Identical rows everywhere. Intervals are paired 95% bootstraps. Deviations: jeb-4b ran in bf16 because its FP8 setting cannot run on an A100, and refusals count as wrong for every model (JevK5 refuses the 104 held-out-task rows with more than 16 options, APUS-OpenJev-v1-4B 104 held-out-task rows, Kahn1-Qwen2.5-3B 36 JevBench rows, metask-jev-4b-policy-mix one). Model authors: if we ran yours wrong, please open a discussion on this repository.

Full-field charts

Versions

This repository holds Manchego v2.1 (2026-09-21), tagged v2.1. Earlier versions are not published.

Files

Accuracy (cross-entropy) on 80 held-out task rows and 200 human-labelled real-text rows, short prompt, one pass; the untrained base scores 0.650 and 0.690.

format	repository	size	80 task rows: accuracy (CE)	200 real-text rows: accuracy (CE)
bf16 (Transformers)	Manchego	9.3 GB	0.950 (0.177)	0.905 (0.280)
MLX 8-bit	Manchego-MLX-8bit	4.5 GB	0.938 (0.180)	0.910 (0.279)
MLX 4-bit	Manchego-MLX-4bit	2.4 GB	0.875 (0.288)	0.900 (0.290)

MLX 4-bit loses six of the 80 task rows (7.5 points). No GGUF build is published.

Also here: LICENSE (Apache-2.0), NOTICE, manchego_config.json (the decision contract and its limits), contract_v2.py (the state-first prompt for 27–255 options), and eval/ (the numbers behind the Manchego-only charts).

Attribution and licences

Base model: Qwen3.5-4B (Apache-2.0). Real-text training corpora, each used under its own licence with its human labels: WANLI (Liu, Swayamdipta, Smith, Choi, 2022; CC BY 4.0); MultiNLI (Williams, Nangia, Bowman, 2018; most of the corpus under the OANC licence; in the fiction genre Seven Swords is CC BY-SA 3.0, Living History and Password Incorrect are CC BY 3.0, the remaining works are in the US public domain); Bitext customer support dataset (Bitext Innovations; CDLA-Sharing-1.0, whose terms place no restriction on results computed from the data; the rows themselves are not redistributed here); banking77 (Casanueva, Temčinas, Gerz, Henderson, Vulić, 2020; CC BY 4.0). Super-NaturalInstructions (Wang, Mishra, et al., 2022; the collection is Apache-2.0; each task's instances carry their upstream dataset's licence): the 101 stage-two tasks come from 42 upstream datasets, every one of them listed with its licence and reference in NATURAL_TASKS_ATTRIBUTION.md in this repository (CC BY 4.0 / 3.0 sources include ATOMIC, CaSiNo, CLS, bAbI and others named there). One task (task289_gigaword_summarization, 89 rows) takes its text from the English Gigaword corpus, which the Linguistic Data Consortium (LDC) distributes under its own licence; the task listing records MIT. Doom states were recorded from ViZDoom (MIT) scenarios with Freedoom assets (BSD-3). Evaluation sets: JevBench, TheoLeeCJ/openjev (MIT), the eight-dataset suite from logan-markewich/jeff (MIT), Super-NaturalInstructions, each dataset under its own licence. Illustration: the Manchego mascot, an AI-generated image (ChatGPT image generation) supplied by the project's author. The interface follows TypeSafe AI's System One contract; Manchego is an independent project, not affiliated with or endorsed by TypeSafe AI.