Tess-4-27B โ€” MLX 4-bit, with a working MTP head

A 4-bit MLX conversion of migtissera/Tess-4-27B (Qwen3.6-27B base, dense, vision-capable) โ€” group size 64, affine, 4.695 bits per weight, 16.1 GB โ€” with the native multi-token-prediction head restored as a BF16 sidecar.

Built from the original BF16 release, not by re-quantizing an 8-bit checkpoint, so there is no compounded quantization error.

Sized for a 32 GB Mac. The 8-bit build is at Tess-4-27B-MLX-Q8 (30.4 GB).

What 4-bit actually costs (measured, not guessed)

Same harness, same 3500-token budget, same machine (M5 Max):

Axis Q8 Q4 ฮ”
HumanEval+ pass@1 (50-problem subset) 0.90 0.82 โˆ’0.08
MMLU-Pro-style reasoning (40q, 10 options) 0.80 0.85 +0.05
Tool-calling (20 scenarios) 0.85 0.85 โ€”
Browser action-selection (20 simulated pages) 0.95 0.95 โ€”
RULER-style long-context (8k/16k/32k) 0.80 0.73 โˆ’0.07
Deterministic VQA (10 items) 1.00 1.00 โ€”
My hard screenshot test 2/2 2/2 โ€”
Decode, single stream 17.3 tok/s 28.3 tok/s +64%

The honest summary: reasoning, tool-use, browser action-selection and vision survive 4-bit intact. Coding loses ~8 points and long-context loses ~7 โ€” the latter is the consistent casualty of quantization in this family, because error compounds across a long attention span and short tasks hide it entirely. In exchange you get 1.64ร— the decode speed and half the memory.

If your workload is agentic (tool calls, short-to-medium turns, vision), Q4 is close to free. If it leans on long-context recall, pay for Q8.

โš ๏ธ A warning for anyone benchmarking this. My first Q4 run scored 0.36 on code and looked like a catastrophic quantization cliff. It was a 768-token generation cap. For a reasoning model that does not truncate the answer โ€” it truncates the thinking, so the model never emits code at all. The longest generation ended mid-sentence: "Or simply: return string.swapcase() I'll". Re-run with a 3500-token budget: 0.82. If a quantization result looks like a cliff, check your token budget before you believe it.

The MTP sidecar

mlx-lm's qwen3_5 sanitize() drops mtp.* tensors during conversion, so every community MLX checkpoint of this family declares mtp_num_hidden_layers: 1 while containing zero MTP weights. Check model.safetensors.index.json, not the config.

This repo re-extracts the head from the original BF16 release and ships it at mtp/weights.safetensors (15 tensors, kept BF16 โ€” quantizing the head collapses draft acceptance to ~0%, so it stays BF16 even in a 4-bit build).

Acceptance on a 4-bit base is lower than on 8-bit (the head was fitted against BF16 weights), and MTP is worth less than it sounds regardless: on the same model, omlx decodes at 18.4 tok/s with no speculative decoding versus vllm-mlx's 19.4 with it. Take the head because it is free; do not architect around it.

Usage

# omlx (recommended โ€” faster engine, and the only one I trust with large images)
omlx serve --model-dir <dir-containing-this-model>

# vllm-mlx (if you want the MTP head to engage)
vllm-mlx serve <this-repo> --mllm --enable-mtp --mtp-num-draft-tokens 1
# MTP engages only on exactly-greedy requests: temperature=0, top_p=1, top_k=0, min_p=0

Known ecosystem traps

  • Do not load MLX-format Qwen3.6 checkpoints with mlx-vlm 0.6.4 โ€” it re-applies a +1.0 RMSNorm shift to already-converted weights and produces deterministic garbage, silently. Use โ‰ฅ0.6.5 or โ‰ค0.6.3.
  • Do not serve large images through vllm-mlx 0.4.0's batched MLLM path โ€” it drops image context and answers anyway, with fabricated detail. Use omlx, or disable continuous batching.
  • Do not benchmark with a short generation cap. See the warning above.

Provenance & credits

  • Base model: migtissera/Tess-4-27B by Migel Tissera, post-trained atop Qwen/Qwen3.6-27B. All model capabilities are theirs; this repo is packaging.
  • Conversion: mlx_vlm convert (4-bit, gs64) from the BF16 source + BF16 MTP re-attachment.
  • License: Apache-2.0, inherited from the base model.

Converted and benchmarked as part of a local inference stack for a personal Automated Agentic Software Factory โ€” something I'm building solo and will make publicly available after its limited-alpha phase. Full benchmark data, harnesses and the conversion recipe: https://huggingface.co/datasets/studioburnside/mlx-local-inference-benchmarks.

Questions, results and corrections all welcome in Discussions.

Downloads last month
22
Safetensors
Model size
27B params
Tensor type
U32
ยท
BF16
ยท
MLX
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for studioburnside/Tess-4-27B-MLX-Q4

Base model

Qwen/Qwen3.6-27B
Quantized
(36)
this model