Qwen3.6-27B optimized for MLX.

  • A mixed-precision quant that balances speed, memory, and accuracy.
  • 4-bit baseline with important layers at 6, 8, and BF16.
  • This quant supports image input and requires a vision-capable server. Non-vision version here.

Usage

# Start server at http://localhost:8080/v1/chat/completions
uvx --from mlx-vlm \
  mlx_vlm.server \
  --host 127.0.0.1 \
  --port 8080 \
  --model spicyneuron/Qwen3.6-27B-MLX-5.7bit-vision

Benchmarks

metric unsloth/Qwen3.6-27B-UD-MLX-4bit mlx-community/Qwen3.6-27B-OptiQ-4bit 5.7 bit (this model)
bpw 7.516 5.575 5.679
base memory 23.534 17.457 17.781
peak memory (1024/512) 27.085 20.633 20.966
prompt tok/s (1024) 420.712 ยฑ 0.129 428.184 ยฑ 0.165 422.521 ยฑ 0.948
gen tok/s (512) 24.759 ยฑ 0.025 31.521 ยฑ 0.030 30.460 ยฑ 0.106
kl mean 0.031 ยฑ 0.003 0.044 ยฑ 0.004 0.027 ยฑ 0.002
kl p95 0.107 ยฑ 0.003 0.164 ยฑ 0.004 0.103 ยฑ 0.002
perplexity* 4.560 ยฑ 0.026 4.850 ยฑ 0.020 4.872 ยฑ 0.029
hellaswag 0.552 ยฑ 0.011 0.552 ยฑ 0.011 0.556 ยฑ 0.011

Unsloth's "4bit" actually averages 7.5 per weight even after excluding the vision tower. This quant is smaller, matches in KL divergence and Hellaswag, and has significantly faster token generation.

OptiQ lands around the same size. This quant is slightly slower but slightly better on KLD (measured against this dataset).

* Perplexity on this model seems to swing a ton based on number of samples, so treat this as a noisy result.

Tested on a Mac Studio M3 Ultra with:

mlx_lm.convert --hf-path Qwen/Qwen3.6-35B-A3B --mlx-path ./mlx && mlx_lm.kld --baseline-model ./mlx
mlx_lm.perplexity --sequence-length 1024 --seed 123
mlx_lm.benchmark --prompt-tokens 1024 --generation-tokens 512 --num-trials 5
mlx_lm.evaluate --tasks hellaswag --seed 123 --num-shots 0 --limit 2000

Required PRs:

Methodology

Quantized with a mlx-vlm fork. MLX quantization options differ than llama.cpp, but the principles are the same:

  • Sensitive layers like MoE routing, attention, and output embeddings get higher precision
  • More tolerant layers like MoE experts get lower precision
Downloads last month
17
Safetensors
Model size
27B params
Tensor type
U32
ยท
BF16
ยท
MLX
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for spicyneuron/Qwen3.6-27B-MLX-5.7bit-vision

Base model

Qwen/Qwen3.6-27B
Quantized
(759)
this model