squad-0.8b-MLX

A team of models. On your device. In milliseconds. ▶ Live demo (previous version)

squad-0.8b for Apple silicon, built from its October 2026 revision 58c22ba: the new trunk, a 1.1 MB router and four 2-bit LoRA experts (games, e2e, multilingual, documents), a MoQLE (Mixture of Quantized LoRA Experts). Same System One API, routing and answers as the CUDA version. Method and limitations: squad-0.8b.

Use

pip install mlx==0.32.4 mlx-lm==0.32.2 tokenizers decider-ai==1.9.0 huggingface_hub safetensors pillow numpy
hf download labqoat/squad-0.8b-MLX --local-dir squad-0.8b-MLX && cd squad-0.8b-MLX

from qev_mlx import QEVMLX   # qev_mlx.py in this repo

m = QEVMLX.load("labqoat/squad-0.8b-MLX")                 # bf16 trunk + router + 4 experts
m = QEVMLX.load("labqoat/squad-0.8b-MLX", subfolder="q8") # 8-bit trunk, the same experts
m = QEVMLX.load("labqoat/squad-0.8b-MLX", slots=False)    # the trunk alone
m.decide(state, questions)                                # images: "<image:NAME>" in the state + images={...}
m.last_slots                                              # the expert per prompt row (0 = trunk, 1..4)

python qev_mlx_server.py                     # POST http://127.0.0.1:8017/v1/systemone (prefix cache, batching, images)
python qev_mlx_server.py --subfolder q8      # the 8-bit trunk
python qev_mlx_server.py --no-slots          # the trunk alone

The response carries "routing": {"slots": [...], "names": [...]}, e.g. ["games"], ["multilingual"] or ["none"].

Routing is the CUDA runtime's: top-1 of {trunk, games, e2e, multilingual, documents} on the router's logits, and an expert listed in router.min_p_slots (manifest.json; e2e: 0.35) is taken only when the router gives it at least that probability, else the trunk answers. With encoding="per_question" every question of a request takes the route of the request's joint prompt. The router's 6 layers are the first 6 layers of the request's own forward (joint encoding), so text requests pay nothing extra. The experts stay 2-bit in memory (slots_2bit.safetensors, mx.quantized_matmul).

Results

Measured on the CUDA release (same weights; from the squad-0.8b card):

	squad-0.8b (previous)	squad-0.8b (this revision)	basal-omni-pico
Benchmark Heaven (public, 231)	77.1	76.6	75.8
JevExtra	71.3	71.7	64.6
JevSpatial	75.8	79.7	68.4
Rules-7	63.8	74.1	69.5
JevBench accuracy	84.9	86.4	84.0
JevBench coherence	70.6	76.4	74.2
KLEJ (Polish, 7 tasks, 500 test items each)	70.9 (trunk only)	73.4	75.3
games_bench (9 unseen games, closed loop; 0 = random, 100 = planner)	11.3	10.4	7.5
e2e on 8 real web apps (61 tests × 3 runs)	10 / 183	34 / 183	–

Checked against the CUDA reference (qev.QEV.from_pretrained(..., specialists=True) at 58c22ba, L40S) on 310 requests (387 questions: Benchmark Heaven public 231, game observations 30, browser steps 25, hand-written multilingual and document requests 24), on an M4 Pro:

	same top answer	max |Δp| (median / p95 / worst request)	same expert chosen
bf16, joint	100.0% of 387	0.002 / 0.024 / 0.146	99.7% (309 / 310)
bf16, per_question	100.0%	0.003 / 0.023 / 0.067	99.7%
q8, joint	99.7%	0.003 / 0.035 / 0.236	99.4%
q8, per_question	99.5%	0.003 / 0.043 / 0.487	99.4%

Experts chosen by the CUDA reference on these requests: trunk 260, games 27, multilingual 17, e2e 5, documents 1. Every route that differs is a browser step whose e2e probability sits on the 0.35 threshold (0.347–0.352 here), where bf16 round-off decides; the larger |Δp| outliers are requests on a different side of that threshold.

Speed on an M4 Pro (24 GB), one request at a time, after warm-up, the same 310 requests:

	median	p95
bf16, joint	54 ms	1.3 s (the longest Benchmark Heaven prompts, 3–3.7k tokens)
bf16, per_question	51 ms	0.6 s
q8, joint	61 ms	0.8 s
qev_mlx_server.py, new state, warm (prefix cache on)	110–135 ms	–

Short requests (30–70 tokens of state) take 34–53 ms; long browser states (380 tokens) about 330 ms. For reference, the CUDA release on one L40S: 4.4 ms median on the same requests. Memory: about 2–2.5 GB for the bf16 model (mx.set_cache_limit keeps the MLX buffer cache small on a shared machine).

Image input was not re-evaluated for this revision: the trunk changed, the vision tower (vision.safetensors) is the same file as before.

Files

model.safetensors, vision.safetensors, configs and tokenizer (bf16; q8/ holds the 8-bit trunk, affine, group size 64) · manifest.json, slots_2bit.safetensors (10.9 MB, 4 experts), router.safetensors (1.1 MB): the experts, used by the root and by q8/ · qev_mlx.py, qev_mlx_server.py, qev_mlx_slots.py (converts the CUDA repo's specialists/ folder: python qev_mlx_slots.py SPECIALISTS OUT), qev_mlx_vision.py · SHA256SUMS.

Built from labqoat/squad-0.8b@58c22ba: the trunk is its model.safetensors in mlx-lm's Qwen3.5 layout (mlx-lm's own sanitize; the unchanged embeddings and final norm are bit-identical to the previous MLX build); the experts are its specialists/ rows re-packed for MLX, codes and per-row scales unchanged (max |dequantised difference| 1.5e-8).

License: Apache-2.0.