vLLM optimized version for 26B-A4B model
The current announcement notes that:
Compressed Tensors (w4a16): QAT checkpoints serialized in the compressed-tensors format for native, optimized inference with vLLM. Available for Gemma 4 E2B, E4B, 12B, and 31B.
Thanks for sharing them! Will there also be a release for the Mixture-of-Experts 26B-A4B-it model?
No, there won't be a release for Gemma 4 26B-A4B due to excessive quality loss with 4-bit quantization. Instead, you should use --quantization int8_per_channel_weight_only.
See the vLLM Recipe card for Gemma 4:
Quantized Models (QAT W4A16)
Google provides official QAT (Quantization-Aware Training) W4A16 checkpoints for the Gemma 4 dense model family. These use 4-bit integer weights with 16-bit activations (group_size=32) in
compressed-tensorsformat, delivering significant memory savings and throughput improvements with minimal quality loss.Available Checkpoints
Base Model QAT W4A16 Checkpoint Memory Savings google/gemma-4-E2B-it google/gemma-4-E2B-it-qat-w4a16-ct ~26% (9.8→7.3 GB) google/gemma-4-E4B-it google/gemma-4-E4B-it-qat-w4a16-ct ~36% (15.2→9.8 GB) google/gemma-4-12B-it google/gemma-4-12B-it-qat-w4a16-ct ~64% (22.8→8.3 GB) google/gemma-4-31B-it google/gemma-4-31B-it-qat-w4a16-ct ~66% (59.0→19.8 GB) ℹ️ Note
The 26B-A4B MoE model is not included — its small expert dimensions (704) cause excessive quality loss with 4-bit quantization. For the MoE model, use--quantization int8_per_channel_weight_only(online, no checkpoint needed) which provides ~47% memory savings with negligible quality impact.