Qwen/Qwen3.5-4B optimized for MLX.

  • A mixed-precision quant that balances speed, memory, and accuracy.
  • 4-bit baseline with important layers at higher precision.
  • This quant supports image input and requires a vision-capable server.

Usage

# Start server at http://localhost:8080/v1/chat/completions
uvx --from mlx-vlm mlx_vlm.server \ 
  --host 127.0.0.1 \
  --port 8080 \
  --model spicyneuron/Qwen3.5-4B-MLX-6.8bit-vision

Benchmarks

metric mlx-community/Qwen3.5-4B-MLX-4bit mlx-community/Qwen3.5-4B-OptiQ-4bit 6.8 bit (this model)
bpw 4.503 6.219 6.750
base memory 2.205 3.045 3.305
peak memory (1024/512) 3.625 4.541 4.814
prompt tok/s (1024) 2674.115 卤 3.462 2661.445 卤 1.504 2589.364 卤 13.109
gen tok/s (512) 166.004 卤 0.312 135.779 卤 0.031 127.137 卤 0.121
kl mean 0.083 卤 0.004 0.041 卤 0.002 0.007 卤 0.001
kl p95 0.180 卤 0.004 0.100 卤 0.002 0.020 卤 0.001
perplexity 5.053 卤 0.035 4.932 卤 0.034 4.921 卤 0.034

Tested on a Mac Studio M3 Ultra with:

mlx_lm.convert --hf-path ... --mlx-path ./mlx && mlx_lm.kld --baseline-model ./mlx
mlx_lm.perplexity --sequence-length 512 --seed 123
mlx_lm.benchmark --prompt-tokens 1024 --generation-tokens 512 --num-trials 5

Required PRs:

Methodology

Quantized with a mlx-vlm fork. MLX quantization options differ than llama.cpp, but the principles are the same:

  • Sensitive layers like MoE routing, attention, and output embeddings get higher precision
  • More tolerant layers like MoE experts get lower precision
Downloads last month
42
Safetensors
Model size
5B params
Tensor type
U32
路
BF16
路
F32
路
MLX
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 馃檵 Ask for provider support

Model tree for spicyneuron/Qwen3.5-4B-MLX-6.8bit-vision

Finetuned
Qwen/Qwen3.5-4B
Quantized
(684)
this model