jbilcke/vllm-decision-3.0-mlx

Decision 3.0 MLX

Answer your own questions about text, images and video, locally on Apple silicon.

This community conversion ports the six models in the Decision 3.0 collection to MLX, following the local bundle, provenance, validation and native Swift approach of Decision 2.0 MLX. No training or fine-tuning is performed.

Decision 3.0 is a code-readout decision model. It reads a complete prompt for each question and scores up to 255 answer codes at the final hidden state. The shared state and optional media can support several caller-defined questions in one request. Choice selects a supplied key; Noul returns P(true); Score returns an expected ordinal level. The complete trained readout remains FP32 in every export.

This is a different architecture and prompt contract from Decision 2.0. The Qwen3.5 softmax-attention layers are bidirectional, while Gated DeltaNet retains its causal scan. The port honors every source layer type, including d3-edge's irregular schedule, source normalization offsets, convolution layouts, L2 epsilon and three-axis multimodal RoPE. The exact source chat template opens the assistant turn with thinking disabled. Generic mlx_lm.generate does not implement this model.

Model inventory

The completed export inventory and individual validation status are recorded in models.json. Names below identify upstream variants; file size is distinct from peak runtime memory.

Variant	Upstream approximate size	Local bundle layout
d3-edge	0.6B	edge/q4, edge/q8, edge/mixed
d3-lite	0.9B	lite/q4, lite/q8, lite/mixed
d3-nano	2B	nano/q4, nano/q8, nano/mixed
d3-mini	5B	mini/q4
d3-flash	8B	flash/q4
d3	26B	d3/q4

mixed stores source BF16 matrices and FP32 other tensors. q4 and q8 use affine quantization with groups of 64 on supported language and vision Linear/Embedding matrices. Positional embeddings, convolution and matrices whose input axis is not divisible by 64 are excluded. Flash and d3 retain their vision MLP down-projection matrices in BF16. Small norms, biases and the 255-way code readout stay FP32. The port computes prompt hidden states in FP32. Its numerical baseline promotes the source checkpoint to FP32; this differs from the official runtime's default BF16 execution. Quantization can change probabilities and decisions; consult the executed reports.

Bundle	Weight bytes	Complete bundle bytes
d3/q4	16,512,842,312	16,535,869,012
edge/mixed	1,192,688,072	1,215,706,567
edge/q4	379,999,816	403,018,541
edge/q8	675,534,664	698,553,389
flash/q4	5,448,434,504	5,471,455,083
lite/mixed	1,710,680,264	1,733,699,407
lite/q4	542,793,992	565,813,365
lite/q8	967,500,296	990,519,669
mini/q4	2,852,726,984	2,875,747,042
nano/mixed	4,433,543,880	4,456,562,829
nano/q4	1,397,015,432	1,420,034,612
nano/q8	2,501,232,392	2,524,251,572

Explicit installation

Use an Apple silicon Mac with a visible Metal device and a Python environment matching the supplied requirements. The development environment uses Python 3.14, MLX/MLX-LM 0.31.1, Transformers 5.17.0 and macOS 27. Other combinations require validation. Inference uses installed local files and has no cloud fallback or automatic download.

Prepare a download environment:

python3 -m venv .venv
. .venv/bin/activate
python -m pip install huggingface_hub==1.7.1

from huggingface_hub import snapshot_download
snapshot_download(
    repo_id="jbilcke/vllm-decision-3.0-mlx",
    local_dir="./decision3-mlx",
    allow_patterns=["edge/q4/*", "README.md", "models.json", "LICENSE", "SHA256SUMS", "infer.py", "example-request.json"],
    # Pin revision to the trusted published collection commit.
)

cd decision3-mlx/edge/q4
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements.txt

The preflight checks Metal availability before importing MLX, so unavailable GPU access yields a readable error. PyTorch and Torchvision support the pinned Transformers media processor and the separate source-reference tools; the decision and vision forward passes execute through MLX.

Local text, image and video decisions

from model import system_one

questions = {
    "route": {
        "type": "choice",
        "instructions": "Which team should handle this request?",
        "criteria": {
            "returns": "Refunds, replacements and damaged deliveries",
            "billing": "Payments, invoices and charges",
            "technical": "Product setup and faults",
        },
    },
    "receipt": {"type": "noul", "instructions": "Does the customer have a receipt?"},
    "urgency": {
        "type": "score", "instructions": "How urgent is the request?",
        "criteria": ["Routine", "Soon", "Today"],
    },
}
result = system_one(
    ".", state="The blender arrived cracked. The customer has a receipt.",
    questions=questions,
)
print(result["answers"])

# Optional installed local media; each question sees every supplied item.
image_result = system_one(
    ".", state="An attached receipt.", images=["/absolute/path/receipt.png"],
    questions={"payment": {"type": "choice", "instructions": "How was it paid?",
                           "criteria": {"card": None, "cash": None}}},
)
video_result = system_one(
    ".", state="A test camera clip.", videos=["/absolute/path/clip.mp4"],
    questions={"motion": {"type": "noul", "instructions": "Does an object move?"}},
)

Repeated applications should retain the model and serialize access:

from model import load_bundle, run
model, config, contract = load_bundle(".")
result = run(model, config, contract, state="A request.", questions=questions)

The collection's CLI also accepts a JSON request file:

cd ../..
python infer.py edge/q4 example-request.json

Input and answer contract

Question	Criteria	Answer
choice	1–255 unique nonempty keys, ordered descriptions or null	Supplied key, ordered probabilities, normalized-entropy confidence
noul	Optional descriptions for false and true	{"type":"noul","noul":P(true)}
score	2–10 ordered criteria in a list	Expected zero-indexed level, probabilities, confidence and legend

Structured state/instructions/descriptions use the upstream JSON serialization and insertion order. Empty state and missing instructions use source defaults. Each invalid question returns a typed per-question error; valid questions remain runnable. The local runtime cap is 32,768 serialized tokens per question, including expanded media tokens. This is a local work limit, not a demonstrated full-context performance result. Overflow rejects that question without truncating its context or options.

Temperature is the source default 1.0. Confidence measures the concentration of option scores, not measured correctness or calibration. A confident answer may be wrong. Ties select the first supplied option. Hosts remain responsible for evidence and authorization before applying decisions.

The Python path runs one unpadded complete forward pass per question and does not maintain generation or cross-request KV caches. It currently computes vision features per question. Local files and bounded base64/PIL/frame inputs are supported; remote media URLs are rejected so inference cannot fetch implicitly. Image processing uses up to 1.6MP; videos use 2fps, at most 32 sampled frames, up to 0.2MP per frame and the source total video token cap. Local decoder limits are 8MB/16MP per image and 32MB/300s/8.3MP source frames per video. Video timestamps and multimodal position IDs follow the source processor.

Provenance and validation

Every bundle is self-contained with the runtime, exact tokenizer/processor configurations, original source configuration, source decision configuration, Apache-2.0 LICENSE and NOTICE, original manifest, conversion environment and SHA-256 inventory. pins.json binds the six source revisions and manifests. The loader validates source identity, required inventory, sizes, hashes, architecture and tensors before inference. A self-described checksum detects corruption; products should additionally pin a trusted collection revision or bundle manifest hash.

The source downloader is explicit, resumable and verifies Hugging Face blob hashes. The converter quantizes rows in chunks and spools tensors to assemble a standard safetensors file, avoiding full FP32 copies of the largest source model. These are bounded development workflows, not guarantees about total system or GPU memory.

Executed numerical comparisons, fixtures, source identity and precision-specific drift are in validation/. The validation tools separate source reference and MLX evaluation into fresh processes so their allocations do not overlap. The reports identify full FP32 and staged file-backed references separately. Short synthetic conversion checks establish operator fidelity only for those cases; they do not reproduce upstream task-quality benchmarks, prove calibrated confidence, validate the maximum context or certify mobile/TV performance. Development-run timings are observations with unrelated user activity, not controlled product latency benchmarks.

Executed on 2026-10-14: all 12 published bundles were compared against exact saved source inputs on seven fixtures each (five text questions, one synthetic image and one synthetic video). Each report binds the final bundle manifest hash. The table shows maximum absolute differences from the FP32 source reference; probability differences use a 0–1 scale. Winner agreement is agreement with the source output, not correctness against fixture labels.

Bundle	Max raw logit difference	Max probability difference	Source winner agreement
edge/mixed	2.09808e-05	3.58851e-06	7/7
edge/q4	1.74092	0.208298	7/7
edge/q8	0.0521561	0.0147794	7/7
lite/mixed	8.58307e-05	3.22875e-05	7/7
lite/q4	0.943799	0.0519961	6/7
lite/q8	0.0517387	0.00430001	7/7
nano/mixed	0.000545979	5.99021e-05	7/7
nano/q4	0.688751	0.0187109	7/7
nano/q8	0.085042	0.0133523	7/7
mini/q4	0.536719	0.0168118	7/7
flash/q4	0.556782	0.118278	7/7
d3/q4	0.566178	0.129445	7/7

The unquantized FP32 MLX baselines for edge, lite and nano passed the stated operator-parity limits, with maximum probability differences of 3.59e-6, 3.23e-5 and 5.99e-5 respectively. Their file-backed staged source references reproduced the full source references exactly on all seven fixtures. Mini, flash and d3 use that staged source procedure; a full unquantized MLX baseline was not run for those larger models.

Lite q4 changed the selected answer on the video fixture. Its source chose still and q4 chose right; this remains a conversion discrepancy even though the synthetic square moved right. Other exports retained the source winners in these fixtures. Edge q4 also showed a probability change of 0.2083, so matching winners alone do not imply close probabilities. These measured quantization effects are published without a lossless-parity claim.

The edge source dtype comparison separately checks promoted FP32 against the official BF16 default: maximum probability difference 0.009203 and winner agreement 7/7. This is the only official-BF16 comparison performed. The local contract suite passed 37 tests, and the tiny original-model versus staged-reference test passed. The documented edge CLI request and the largest d3 q4 text/image/video smoke also executed successfully.

Native Swift package

swift/ provides Decision3Kit and three executable examples using verified installed bundles. It supports the text Choice/Noul/Score contract on macOS and reads the same quantized language weights and FP32 readout directly. Native image/video inference is not implemented. Compilation and opt-in inference tests are recorded separately from Python numerical checks. Physical iPhone/iPad/Apple TV inference and device-specific memory/latency require separate validation.

The final edge q4 bundle passed 12 native Release tests with zero failures or skips, five exact upstream tokenization cases, and three fresh executable examples. Native/Python parity passed all five text cases with identical tokens, option order and selected keys; maximum raw logit difference was 9.54e-6 and maximum probability difference 1.16e-6. Other native variants have not been exercised.

Reproduce an export

Run these commands from a full development checkout of this repository. The minimal single-bundle installation above contains inference files only.

python download.py edge
python convert.py edge --bits 4
python validate.py edge --reference-only
python validate.py edge --evaluate-only --bits 4
python -m unittest test_contract

The source runtime is executed only by the explicit reference-validation command. Normal inference runs the bundled local port and Transformers' installed processor without remote custom code.

License and attribution

Original collection code is Apache-2.0. Upstream Decision 3.0 and Qwen3.5 model weights retain Apache-2.0 and their original NOTICE and provenance. MLX and MLX-LM are MIT-licensed Apple projects; adapted vision portions retain mlx-vlm's Prince Canuma MIT notice. See THIRD_PARTY_NOTICES.md and the full notices included in vision.py and swift/. This community conversion implies no endorsement by the upstream authors or Apple.