spicyneuron/Qwen3.5-9B-MLX-5.6bit-vision

Qwen3.5 9B optimized to run on Mac.

• A mixed-precision quant that balances speed, memory, and accuracy.
• 4-bit baseline with important layers at 8 and BF16.

Usage

# Start server at http://localhost:8080/chat/completions
uvx --from mlx-lm --with torchvision \
  mlx_vlm.server \
  --host 127.0.0.1 \
  --port 8080 \
  --model spicyneuron/Qwen3.5-9B-MLX-5.6bit-vision

Benchmarks

metric	this model
bpw	5.554
base memory	5.789
peak memory (1024/512)	7.446
prompt tok/s (1024)	1481.661 ± 6.709
gen tok/s (512)	91.086 ± 0.101
kl mean	0.032 ± 0.002
kl p95	0.069 ± 0.002
perplexity	3.739 ± 0.018
winogrande	0.660 ± 0.021

Tested on a Mac Studio M3 Ultra. KL divergence is approximate, based on top_k not full logits. Here's the code.

mlx_lm.kld --baseline-model path/to/mlx-full-precision
mlx_lm.perplexity --sequence-length 2048 --seed 123
mlx_lm.benchmark --prompt-tokens 1024 --generation-tokens 512 --num-trials 5
mlx_lm.evaluate --tasks winogrande --seed 123 --num-shots 0 --limit 500

Methodology

Quantized with a mlx-lm fork, drawing inspiration from Unsloth/AesSedai/ubergarm style mixed-precision GGUFs. MLX quantization options differ from llama.cpp, but the principles are the same:

• Sensitive layers like MoE routing, attention, and output embeddings get higher precision
• More tolerant layers like MoE experts get lower precision