Tess-4-27B โ€” MLX 8-bit, with a working MTP head

An MLX conversion of migtissera/Tess-4-27B (Qwen3.6-27B base, dense, vision-capable) quantized to 8-bit (group size 64, affine), with the native multi-token-prediction head restored as a BF16 sidecar โ€” which, at the time of upload, no other public MLX conversion of the Qwen3.6 family ships.

A 4-bit build with the same MTP sidecar is at Tess-4-27B-MLX-Q4 (16.1 GB).

Why the MTP sidecar matters

mlx-lm's qwen3_5 sanitize() unconditionally drops mtp.* tensors during conversion, so every community MLX checkpoint of this family carries mtp_num_hidden_layers: 1 in its config while containing zero MTP weights. Check your model.safetensors.index.json, not the config โ€” the config will tell you the head is there when it is not.

This repo re-extracts the head from the original BF16 release and ships it at mtp/weights.safetensors (15 tensors, BF16 โ€” quantizing the head collapses acceptance to ~0%).

Measured with vllm-mlx 0.4.0 (--enable-mtp, greedy envelope) on an M5 Max: 78% draft acceptance, +19.5% decode throughput with acceptance-verified speculative decoding โ€” no output change vs. non-speculative greedy.

An honest caveat. MTP is real and it works, but it is worth less than it sounds. On the same model and workload, omlx decodes at 18.4 tok/s with no speculative decoding at all, versus vllm-mlx's 19.4 tok/s with MTP (and ~17.3 without). A better runtime recovered nearly the whole gain on its own. Take the head because it is free; do not architect around it.

Benchmarks, and what you gain by choosing 8-bit over 4-bit

Same harness, same 3500-token budget, same machine (M5 Max). The Q4 build is the same conversion at 4 bits, so this is a clean read on what the extra 14 GB buys you:

Axis Q8 (this repo) Q4 ฮ”
HumanEval+ pass@1 (50-problem subset) 0.90 0.82 +0.08
MMLU-Pro-style reasoning (40q, 10 options) 0.80 0.85 โˆ’0.05
Tool-calling (20 scenarios) 0.85 0.85 โ€”
Browser action-selection (20 simulated pages) 0.95 0.95 โ€”
RULER-style long-context (8k/16k/32k) 0.80 0.73 +0.07
Deterministic VQA (10 items) 1.00 1.00 โ€”
My hard screenshot test (fine-print OCR) 2/2 2/2 โ€”
Decode, single stream 17.3 tok/s 28.3 tok/s โˆ’64%
On disk 30.4 GB 16.1 GB +14.3 GB

The honest summary: 8-bit buys you ~8 points of coding and ~7 of long-context. It buys you nothing on reasoning, tool-use, browser action-selection or vision โ€” those survive 4-bit intact. And it costs you 40% of your decode speed and 14 GB.

Take Q8 if you lean on long-context recall or code generation, and you have the memory. Take Q4 if your workload is agentic (tool calls, short-to-medium turns, vision), or you are on a 32 GB machine โ€” there it is close to free.

Small subsets: treat as directional, not as leaderboard numbers. The reasoning axis showing Q4 ahead is well within the noise of a 40-question set โ€” read it as "no loss", not "Q4 is smarter".

Usage

# omlx (recommended โ€” faster engine, and the only one I trust with large images)
omlx serve --model-dir <dir-containing-this-model>

# vllm-mlx (if you want the MTP head to actually engage)
vllm-mlx serve <this-repo> --mllm --enable-mtp --mtp-num-draft-tokens 1
# MTP engages only on exactly-greedy requests: temperature=0, top_p=1, top_k=0, min_p=0

Anything that loads standard MLX safetensors will run this model; engines without MTP support simply ignore the mtp/ sidecar.

Known ecosystem traps this conversion accounts for

  • Do not load MLX-format Qwen3.6 checkpoints with mlx-vlm 0.6.4 โ€” it re-applies a +1.0 RMSNorm shift to already-converted weights and produces deterministic garbage, with no error and no warning. Use โ‰ฅ0.6.5 or โ‰ค0.6.3.
  • Do not serve large images through vllm-mlx 0.4.0's batched MLLM path โ€” it silently drops image context and answers anyway, confidently and with fabricated detail (0/2 on a hard screenshot test while inventing numbers; omlx scored 2/2). Serve vision without continuous batching, or via omlx.
  • Do not benchmark this model with a short generation cap. It is a reasoning model: a 768-token cap does not truncate its answer, it truncates its thinking, so it emits nothing and scores near zero on problems it can solve. Budget โ‰ฅ3500 tokens.

Provenance & credits

  • Base model: migtissera/Tess-4-27B by Migel Tissera โ€” post-trained on long-context agentic traces atop Qwen/Qwen3.6-27B. All model capabilities are theirs; this repo is packaging.
  • Conversion: mlx_vlm convert (8-bit, gs64) + BF16 MTP re-attachment.
  • License: Apache-2.0, inherited from the base model.

Converted and benchmarked as part of a local inference stack for a personal Automated Agentic Software Factory โ€” something I'm building solo and will make publicly available after its limited-alpha phase. Full benchmark data, harnesses and the conversion recipe: https://huggingface.co/datasets/studioburnside/mlx-local-inference-benchmarks.

Questions, results and corrections all welcome in Discussions.

Downloads last month
21
Safetensors
Model size
27B params
Tensor type
U32
ยท
BF16
ยท
MLX
Hardware compatibility
Log In to add your hardware

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for studioburnside/Tess-4-27B-MLX-Q8

Base model

Qwen/Qwen3.6-27B
Quantized
(36)
this model