mentat-sys1-v0.3

Decisions with probabilities.

A 4B model for choosing actions, using tools and making structured decisions. Give it a state and typed questions; it returns answer probabilities in one forward pass per question, without generating answer text.

Qwen3.5-4B · Text / JSON · 16K context · LoRA adapter + decision readout

GitHub · Code & accelerated inference · Team · THU-IDM · Demos · Quick start · Results · Documentation

See it in action

Recorded games and completed workflows. Click a demo to watch the enlarged replay.

Mario Jump over obstacles and reach the goal.	Football Play against JevBall’s built-in AI team.	Doom Battle seven built-in Doom bots.
Shopping Compare prices and delivery options, then place an order.	Workspace Invite a teammate with the requested permissions.	Chess Play against a greedy chess bot.

The model receives structured text/JSON states and selects actions.

_{Controllers provide Mario lookahead, Doom aiming/navigation, football physics/off-ball AI, and chess legal/tactical facts. Shopping and Workspace use local mock sites. Game clips show selected highlights.}

Results

Measurement	mentat-sys1-v0.3	Jev, reported
Intern-Decision — average accuracy (%) ↑	89.59%	88.74%
JevBench — correct answers (public set) ↑	204/231	199/231
Typed Decisions — Brier score ↓	0.05403	0.148
Decision Index 0.2.1 ↑	52.07*	57.91
Latency (ms) ↓	21.47 ms	150 ms†

_{Jev benchmark results: StartLux results.}

_{* DI: 149,816 requests, 37 tasks; HLE unmeasured (zero contribution, original weights); abstentions count as wrong.}

_{† mentat: one question, accelerated runtime, warm A800, 600 localhost HTTP requests. Jev: TypeSafe’s official 150 ms reference, question count unspecified.}

Quick start

Python 3.12 and a CUDA GPU:

hf download yunqu/mentat-sys1-v0.3 --revision v0.3 --local-dir mentat-v0.3
hf download Qwen/Qwen3.5-4B \
  --revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a \
  --local-dir qwen-base

pip install -r mentat-v0.3/requirements.txt
python mentat-v0.3/infer.py --base-model qwen-base --input mentat-v0.3/example.json

Supports named choices (choice), yes/no (noul) and ordered scores (score). Request / response example and HTTP API.

License

Adapter and bundled code: Apache-2.0. The separately downloaded Qwen base retains its upstream license. NOTICE.