jbilcke/vllm-decision-2.0-mlx

Decision 2.0 MLX

Choose among your own options, locally on Apple silicon.

This collection ports the Decision 2.0 family to MLX: compact Qwen text backbones with a trained head that scores a caller-supplied menu. Give the model a context, a question and candidate descriptions; it returns normalized candidate scores, a Boolean score or an expected ordinal level.

Each bundle includes the trained decision head, exact prompt/tokenizer contract, local loader and pinned conversion provenance. Mixed-precision and quantized backbones provide different storage choices while keeping the complete decision head in FP32. This is a community conversion of the upstream releases; no new training or fine-tuning was performed.

Models at a glance

Variant	Text backbone	Source parameters, including head	Maximum serialized input	Bundle directories
Kai 0.6B	Qwen3	597,103,104	8,192 tokens	kai/mixed, kai/q4, kai/q8
Eos 0.8B	Qwen3.5 text	753,446,208	16,384 tokens	eos/mixed, eos/q4
Sol 2B	Qwen3.5 text	1,883,930,944	16,384 tokens	sol/mixed, sol/q4
Nox 4B	Qwen3.5 text	4,208,383,488	16,384 tokens	nox/q4

Consult models.json for the published inventory, source revisions, checksums and completed validation records. Nox is available as q4 with a separate staged-reference validation method, described below.

mixed retains source BF16 linear matrices and FP32 other tensors. q4 and q8 use affine quantization with groups of 64 for supported backbone Linear and Embedding modules. The candidate/query LayerNorms, bilinear projections and exact-GELU MLP decision head remain FP32 in every bundle. Download size excludes activation memory, resident mappings, runtime buffers and allocator overhead; it is not a peak-RAM estimate.

Bundle	Model weight bytes	Complete bundle download bytes
kai/mixed	1,507,647,373	1,519,139,286
kai/q4	349,525,513	361,017,558
kai/q8	647,517,969	659,010,017
eos/mixed	2,018,595,501	2,038,652,078
eos/q4	445,125,255	465,181,964
sol/mixed	4,790,330,229	4,810,387,780
sol/q4	1,100,706,850	1,120,764,537
nox/q4	2,419,415,448	2,439,473,912

Complete bundle bytes sum the published model-directory files, including its README and MLX_MANIFEST.json; they exclude collection-level metadata and locally generated caches. One GB is 1,000,000,000 bytes; one GiB is 1,073,741,824 bytes. These file sizes and the process-memory measurements below measure different things.

Download one model

Use an Apple silicon Mac with a visible Metal device. The conversion/runtime was developed with Python 3.14, MLX/MLX-LM 0.31.1 and macOS 27. Dependencies are pinned inside each bundle. Other Python/OS combinations need their own validation.

Download only the selected bundle and collection metadata:

python3 -m venv .venv
. .venv/bin/activate
python -m pip install huggingface_hub

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="jbilcke/vllm-decision-2.0-mlx",
    local_dir="./decision2-mlx",
    allow_patterns=[
        "kai/q8/*", "README.md", "LICENSE", "THIRD_PARTY_NOTICES.md",
        "models.json", "SHA256SUMS",
    ],
    # For reproducibility, set revision to a published collection commit.
)

Replace kai/q8 with a directory listed in models.json. The download is an explicit installation step; subsequent inference reads the installed directory and has no cloud fallback or telemetry.

Run a local decision

cd decision2-mlx/kai/q8
python -m pip install -r requirements.txt

import json
from model import predict

result = predict(".", {
    "task_type": "choice",
    "state": "Mara walks to the stove and fries an egg.",
    "instructions": "Select the activity explicitly performed in the context.",
    "options": [
        {"key": "cooking", "description": "Preparing food"},
        {"key": "fishing", "description": "Catching fish"},
        {"key": "none", "description": "No explicit activity"},
    ],
})
print(json.dumps(result, indent=2, ensure_ascii=False))

predict returns logits, a key-to-probability mapping, a typed answer, temperature, the input token count and the token-ID SHA-256. It loads and verifies the bundle for this one request. Applications handling repeated requests should retain a loaded model and serialize access rather than repeatedly loading weights. The bundled load_bundle, encode and answer functions expose that lower-level path.

The included Metal preflight runs before MLX imports. If macOS exposes no Metal device, it stops with a readable error instead of allowing MLX's native import to abort. Use a normal local terminal or an execution environment with Metal access; this check does not grant device permissions.

Input and answer contract

Task	Ordered options	result["answer"]
choice	2–255 unique nonempty string keys; caller-defined descriptions	Chosen key, normalized-entropy confidence and candidate probabilities
noul	Exactly the keys false and true, each with a description	{"type": "noul", "noul": P(true)}
score	2–10 levels with keys "0" through "L-1"	Expected zero-indexed level, confidence, probabilities and a level legend

state and instructions accept finite JSON text, objects or arrays. Instructions cannot be an empty string. Option descriptions accept text, objects or arrays; choice also permits null. Option order is retained, including the first option winning an exact tie. Score keys identify numerical levels even when callers reorder the option list. Kai's pinned five-level score correction is applied before softmax and is included in its bundles.

The temperature is 1.0 in the provided predict API. The lower-level answer function accepts a finite positive temperature. Probabilities are model scores, not independently calibrated correctness probabilities. Product confidence is 1 - H(p) / log(K), measuring how concentrated the distribution is; a confident answer can still be wrong. Noul's public answer exposes P(true) directly.

The formatter separately tokenizes the context/question prefix, each option segment and the final query suffix, without adding special tokens or applying a chat template. The trained head reads each option's endpoint and the final query hidden state. Tokenizing a concatenated prompt or using chat formatting changes the model's contract. The complete serialized input must fit the variant's limit; overflow is rejected without trimming context or candidate options.

Format and compatibility

Each model directory is self-contained:

kai/q8/
  model.safetensors
  config.json
  backbone_config.json
  tokenizer.json
  tokenizer_config.json
  score_bias.json              # Kai only
  model.py
  metal_check.py
  requirements.txt
  MLX_MANIFEST.json
  UPSTREAM_MANIFEST.json
  README.md
  LICENSE
  THIRD_PARTY_NOTICES.md

The local loader checks the required inventory, pinned source identity, configuration, tensor shapes/dtypes and file hashes. It retains the original backbone configuration and upstream manifest, and rejects unsupported formats or missing score corrections. A self-described checksum detects file corruption; applications should additionally pin their trusted collection revision or bundle manifest hash.

These are custom decision readout packages. A generic mlx_lm.generate, Transformers chat pipeline or root-level AutoModel invocation does not load the trained head or implement this API. MLX-LM supplies the Qwen hidden-state backbones; the included model.py supplies the decision architecture. Qwen3.5 normalization offsets and depthwise-convolution axes are converted explicitly, and the decision-only gated-delta prefill preserves the pinned source's L2 normalization epsilon.

A native Swift package is included under swift/; it reads the verified local MLX bundles directly. Core ML and Core AI remain separate conversion paths. Physical iPhone, iPad and Apple TV inference, long-context behavior, batching and device-specific latency/memory remain outside the supplied Mac functional checks.

Conversion validation

The published validation/ reports record the source revision, environment, exact token hashes, logits and softmax comparisons on original synthetic fixtures. For Kai/Eos/Sol, the gate compares actual upstream CPU FP32 inference with an MLX FP32 mathematical baseline, then evaluates each stored mixed/quantized bundle. FP32 acceptance limits are maximum absolute logit error ≤0.01 and maximum absolute probability error ≤0.002. Quantized exports are reported separately because quantization can change scores or the winning option.

The executed CPU FP32 and MLX comparisons used five original short fixtures: explicit action, hypothetical/negated dialogue, Boolean, five-level Score and Unicode/structured JSON. Their segmented token IDs and candidate/query positions matched exactly. The following maximum errors cover those five cases, not the full supported context or a held-out task benchmark.

Variant	Executed precisions	FP32 max logit / probability error	Quantized maximum probability drift	Report
Kai	FP32 baseline, mixed, q4, q8	Passed: 0.000004291535 / 0.0000001968333	q4: 0.1906578; q8: 0.005756557	Kai report
Eos	FP32 baseline, mixed, q4	Passed: 0.000008821487 / 0.0000001812316	q4: 0.02863582	Eos report
Sol	FP32 baseline, mixed, q4	Passed: 0.000003814697 / 0.0000001368104	q4: 0.009483443	Sol report
Nox	q4 against staged source reference	Full FP32 MLX baseline not run	q4: 0.02571482	Nox report

Mixed storage matched the FP32 baseline errors shown above. Every evaluated export retained the upstream winner on all five fixtures. The largest q4 logit errors were 0.8965921 (Kai), 2.176143 (Eos) and 1.493322 (Sol); unchanged winners on these cases do not make quantization lossless. The Qwen3.5 decision prefill's L2 epsilon correction is supported by these actual CPU comparisons; Eos's corrected FP32 baseline maximum logit error was 0.000008821487. Additional diagnostic operator runs remain visible in its report.

Reports were captured on an Apple M4 MacBook Air with 24 GiB unified memory, macOS 27, Torch 2.14.1, Transformers 5.18.0 and MLX/MLX-LM 0.31.1. See each report for complete dependency versions and hashes binding the evaluated exports. Their single-pass times are development-host observations with unrelated activity, not a controlled latency or memory comparison.

Numerical fidelity on a few short fixtures measures conversion correctness. It does not reproduce the source model's published task benchmarks or establish application task accuracy, evidence admission, calibrated confidence or safe automatic actions. Hosts must validate supplied indices, source evidence, permissions and deadlines before applying a decision. Any published timing here must identify preparation, first response, later responses and whole-process peak RSS separately; development-host observations are not controlled benchmarks.

Local decision benchmark

The local authored benchmark contains 22 unique cases, covering supplied choices, Boolean decisions, ordinal levels, explicit facts, contradictions and permitted compatible alternatives. Each case was run three times with unchanged text, rules, option descriptions and order: 66 predictions per path, not 66 independent examples. The frozen corpus SHA-256 is d9393119fa0281126d188f886c40d22ab8e3f47fb000b652c9d36acf4c59e2f0. The public aggregate summary contains the measurements below. The corpus and reports containing individual case text are not distributed.

Runs on 4 October 2026 used an Apple M4 MacBook Air with 24 GiB unified memory and macOS 27. Models ran sequentially in fresh processes, with coordinated builds, conversions and other inference idle. Unrelated user activity and thermal state were not instrumented. All four paths produced 66 valid predictions; each case's answer and admission status were identical across its three trials.

Product path	Correct unique cases	Raw correct / 66	Admitted / 66	Correct among admitted
Kai q8, Python MLX GPU	19/22	57/66	30/66	27/30
Eos q4, Python MLX GPU	21/22	63/66	66/66	63/66
Sol q4, Python MLX GPU	22/22	66/66	45/66	45/45
GLiNER 2.5 w8, Swift/Core ML CPU+GPU	18/22	54/66	57/66	54/57

Decision 2.0's diagnostic admission requires both normalized-entropy confidence and selected-option probability ≥0.6. For Noul, the benchmark derives entropy from the actual Boolean distribution; its native answer still exposes P(true). GLiNER retains its recorded native-confidence ≥0.6 diagnostic. These confidence semantics differ and are uncalibrated. Admission here is a grading filter, not permission to apply an action or override explicit evidence/conflict guards.

Product path	Preparation (ms)	First prediction (ms)	Later-trial median (ms)	Peak process RSS (bytes)	MLX peak allocation (bytes)
Kai q8	2,898.202291	80.198542	71.268292	1,165,410,304	886,278,408
Eos q4	2,188.516292	189.271041	99.117480	1,042,661,376	774,544,704
Sol q4	2,888.251709	218.895709	215.162167	1,716,109,312	1,539,423,640
GLiNER 2.5 w8	54,127.430458	15,492.721792	1,712.164521	6,163,120,128	—

Preparation includes imports, integrity verification, weight loading and materialization, separately from prediction. The Python prediction timer includes encoding, tensor creation, completed MLX execution and answer normalization. The first response includes process-cold compilation behavior; shared filesystem caches may already be warm. Later-trial medians use the 44 predictions from the second and third complete passes, with no cached answers, batching, shared-context acceleration, omitted evidence or shortened menus.

Whole-process peak RSS includes the interpreter/framework, resident mappings, allocators and activations. MLX peak allocation is a separate allocator measure; neither is model-only or retained RAM. Python MLX GPU and native Swift/Core ML CPU+GPU use different formatter, loading, compute and queue implementations. The Python path has no per-prediction deadline in this experiment; the native path uses a 30-second scheduler deadline. Their latency difference is a measured product-path observation, not an architecture-only speed claim or the measured performance of the native Swift Decision 2.0 path.

Sol q4 is the strongest tested quality candidate on this small corpus when raw correctness and avoiding wrong admissions take priority: all 22 raw choices were correct, and all 15 admitted unique cases were correct. Seven correct raw choices fell below the unchanged admission policy. This supports further testing; it does not establish broad task accuracy. The diagnostic admission floor remains a separate application policy; creative Choice can use a valid argmax at any confidence.

Eos q4 combines a lower observed median with greater coverage, but its sole unique error was confidently admitted: it chose one numerical value despite explicit unresolved evidence for two conflicting values. Entropy confidence was 0.898124 and selected probability 0.977654. Deterministic conflict guards remain necessary. Kai q8 was fastest here and admitted one wrong unique case.

The corpus is authored and small; repeated trials measure repeatability rather than general task accuracy. These results do not establish calibrated confidence, long-context behavior, physical mobile/TV acceptance or native Swift performance. The standalone Swift package below has its own build and functional validation.

Nox 4B q4

The largest family member is available in nox/q4: 2.44 GB for the complete bundle, including 2.42 GB of model weights. The export uses 4-bit affine quantization with groups of 64 for supported backbone matrices and retains the trained decision head in FP32. It is a conversion of the released, materialized 4,208,383,488-parameter checkpoint; no adapter merge or new training is required.

Nox uses a distinct, memory-conscious validation path. Its pinned upstream CPU reference keeps the original BF16 linear matrices in file-backed source storage and casts one linear operation's weights to FP32 at a time. The forward operators and decision head compute in FP32. This modifies upstream loading and is recorded as experimental-source-storage-dtypes-cpu-fp32-linear-casts-v1; it is not an unmodified full-FP32 load or an MLX FP32 baseline. The staged method reproduced the saved full-FP32 reference exactly on all five fixtures for each of Kai, Eos and Sol, including candidate/query hidden states. Their independent proof reports are included as staged Kai, staged Eos and staged Sol.

The separately executed Nox staged reference and Nox q4 comparison cover the same five short fixtures. Segmented token IDs, candidate endpoints and the final query position matched exactly. Nox q4 retained 5/5 winning options, with maximum absolute logit drift 0.3787954 and maximum absolute probability drift 0.02571482 against that reference. The report binds the actual evaluated runtime, tokenizer, configuration, weights, staged source and smaller-model proofs by SHA-256. No full-FP32 Nox MLX baseline was executed.

These are conversion and quantization checks, not a task-quality benchmark. Nox has not yet been evaluated on the 22-case decision benchmark above. The staged CPU-reference process reached 8,096,268,288 bytes of whole-process peak RSS; this is a reference-validation measurement, not the memory requirement for q4 inference. The 2.44 GB download does not include activations, framework/allocator overhead or context-dependent runtime buffers. Its source limit of 16,384 tokens does not establish full-context performance or physical mobile-device acceptance.

Reproduction and provenance

Collection-level scripts reproduce pinned downloads, conversion and validation. Runtime installation requires only a chosen bundle; conversion additionally requires the reference dependencies in requirements.txt at the collection root.

python -m pip install -r requirements.txt
python download.py kai
python convert.py kai --bits 0
python convert.py kai --bits 4
python convert.py kai --bits 8
python validate.py kai --bits 0 4 8

The reproduction scripts write local outputs to models/decision2-kai-mlx-{mixed,q4,q8}. Public collection paths such as kai/q8 refer to the same complete bundle format. Repeat only the published and verified precision combinations for Eos/Sol. For Nox, use the separate staged-reference path below; the ordinary full-FP32 reference path can exceed available RAM.

python download.py nox
python convert.py nox --bits 4
python staged_source_reference.py nox \
    --source .cache/upstream/nox --fixtures fixtures.json \
    --output validation/staged-nox.json --execute \
    --smaller-proof validation/staged-kai.json validation/staged-eos.json validation/staged-sol.json \
    --max-rss-bytes 11000000000
python validate_staged_export.py \
    --reference validation/staged-nox.json \
    --smaller-proof validation/staged-kai.json validation/staged-eos.json validation/staged-sol.json

Run the CPU reference and MLX q4 evaluation in separate processes. The RSS check stops further work when the observed process peak exceeds the chosen limit; it cannot prevent an individual allocation or guarantee that another machine has enough free memory. Keep the unchanged fixtures and included proof reports when reproducing this validation.

Variant	Pinned upstream revision
Kai	cd49ea3813fd8ba0928a9a23ef6c9a0f2f0cd764
Eos	3594047d69f476f1d01cf84c593e213fc3a4dfe0
Sol	64235bef55dad29387dd16da7c90e038bf2f0972
Nox	25e8f67d1b486c647222df3aac640d2d5d736bbe

models.json is the collection catalog; SHA256SUMS and per-bundle MLX_MANIFEST.json provide the exact release inventory. Validation reports should bind to the released model/runtime/config/tokenizer hashes. Preserve those files when copying a model directory.

Native Swift library and examples

This release includes a standalone Swift package in swift/. Import Decision2Kit to run the trained decision head directly with MLX: supplied-option Choice, Boolean probability and expected ordinal Score. The native loader supports the verified Kai q8, Eos q4, Sol q4 and Nox q4 bundles. It verifies each pinned manifest and every local file before model preparation.

Native inference requires Swift 6.2+, macOS 27+, Apple silicon and a physical Metal 4-capable device. The package checks the actual device before creating MLX arrays. Mobile, Intel, Linux and Simulator inference are unavailable; portable data types and CPU contracts remain separate from that hardware gate.

Install the source and your chosen model explicitly. These commands download only the Swift subtree and Sol q4, respectively:

python -m pip install huggingface_hub

hf download jbilcke/vllm-decision-2.0-mlx \
  --include "swift/**" --local-dir ./decision2-source

hf download jbilcke/vllm-decision-2.0-mlx \
  --revision bc4dd9af2efd09039ff9b3f2c7f198da8f84836f \
  --include "sol/q4/*" --local-dir ./decision2-models

The source package is decision2-source/swift, and the model directory is decision2-models/sol/q4. Add the source directory as a local Swift package in Xcode or SwiftPM; the model collection's root is not a Swift package. The first Swift build resolves pinned open-source dependencies. Model preparation and inference then use local files only, with no Python subprocess, model download, remote inference or telemetry.

Three executable examples use fresh requests for support-ticket routing, creative theme selection and preference scoring:

swift run --package-path ./decision2-source/swift -c release route-support-ticket \
  --bundle "$PWD/decision2-models/sol/q4" --variant sol

swift run --package-path ./decision2-source/swift -c release choose-creative-theme \
  --bundle "$PWD/decision2-models/sol/q4" --variant sol

swift run --package-path ./decision2-source/swift -c release score-preference \
  --bundle "$PWD/decision2-models/sol/q4" --variant sol

For Nox, download nox/q4/*, point --bundle at its complete local directory and pass --variant nox. Eos uses eos/q4/eos; Kai uses kai/q8/kai. Each command prints actual ordered probabilities, the selected supplied key, normalized-entropy confidence and token count; Score also prints the expected zero-indexed level. A Choice result has no confidence cutoff: creative applications can use its highest-scoring supplied option while retaining the confidence for inspection. Invalid output shapes and unknown keys are rejected.

The exact segmented tokenizer, learned FP32 head and Kai Score offsets are preserved. Input overflow is rejected without shortening the context or menu. A process-wide bounded serial worker retains running GPU work until it finishes, including after cancellation or a deadline. Download size remains distinct from working memory. See the Swift README for the public API, explicit opt-in model tests, supported limits and attribution.

The Release library and all three executables compiled on a supported Mac. Seventeen package tests passed, including actual Nox inference for these three fresh examples; their CLI commands and help/error exits also passed. The functional smoke record contains the real outputs and compiled core-source hashes. These are functional checks, separate from general accuracy or controlled latency measurements.

Original library and example code is Apache-2.0. Adapted MLX Swift LM portions retain their MIT notices in Swift third-party notices, with the package's LICENSE and NOTICE included. Model weights and dependency code retain their original licenses.

Sources and license

The source Kai, Eos, Sol and Nox weights, prompt/readout contract and Qwen backbone weights retain Apache License 2.0. Their complete license and original component provenance remain included. Format conversion does not replace upstream attribution or grant additional rights.

MLX and MLX-LM are MIT-licensed Apple projects. The adapted Qwen3.5 prefill retains its Apple copyright and MIT notice in THIRD_PARTY_NOTICES.md. See those notices for runtime and reference dependency attribution. This is a community conversion, with no endorsement by the source authors or Apple implied. Cite the original Decision 2.0 release and relevant Qwen research when using these derived weights.