Post
106
We trained 4-bit versions of Qwen3.5, Qwen3.8, Gemma 4 and Muse Glimmer with QUASAR, our quantization-aware training method. They stay closer to the original BF16 models than other 4-bit releases, including Google's and NVIDIA's.
How it works: the original BF16 model is the teacher, and the 4-bit model learns from the teacher's own responses. The result is a standard checkpoint (NVFP4 for vLLM, Q4_0 GGUF for llama.cpp / Ollama / LM Studio), so no custom kernels and no conversion step.
KL to BF16, QUASAR vs. the reference build (lower is better):
- Muse Glimmer 30B NVFP4: 0.0183 vs 0.0300 (NVIDIA)
- Muse Glimmer 30B NVFP4 W4A4: 0.0530 vs 0.0664 (Red Hat)
- Muse Glimmer 30B Q4_0 GGUF: 0.0213 vs 0.0228 (Meta Q4_K_M)
- Gemma 4 12B W4A16: 0.034 vs 0.044 (Google QAT)
- Gemma 4 12B Q4_0 GGUF: 0.048 vs 0.069 (Google QAT)
- Gemma 4 E4B W4A16: 0.022 vs 0.064 (Google QAT)
- Gemma 4 E4B Q4_0 GGUF: 0.044 vs 0.088 (Google QAT)
- Qwen3.5-4B NVFP4: 0.0181 vs 0.0419 (cyankiwi AWQ)
- Qwen3.5-4B NVFP4 W4A4: 0.046 vs 0.075 (cosmicproc)
- Qwen3.5-4B Q4_0 GGUF: 0.235 vs 0.284 (bartowski)
Highlights:
- Muse Glimmer 30B matches or beats BF16 on GPQA-Diamond, MMLU-Pro and AIME'25.
- Qwen3.8-27B has every linear layer in NVFP4 and scores 90.9 on GPQA-Diamond (BF16: 91.4), at 19.7 GB.
- Gemma 4 E4B and 12B beat Google's QAT on every benchmark we ran.
Models: QUASAR-QAT/all-quasar-models-6aab09fd2913ec22df0d5f15
Paper: QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction (2608.13966)
Code: https://github.com/vincentcounathe/quasar-qat
Project page: https://quasar-qat.github.io/
Feedback welcome!
How it works: the original BF16 model is the teacher, and the 4-bit model learns from the teacher's own responses. The result is a standard checkpoint (NVFP4 for vLLM, Q4_0 GGUF for llama.cpp / Ollama / LM Studio), so no custom kernels and no conversion step.
KL to BF16, QUASAR vs. the reference build (lower is better):
- Muse Glimmer 30B NVFP4: 0.0183 vs 0.0300 (NVIDIA)
- Muse Glimmer 30B NVFP4 W4A4: 0.0530 vs 0.0664 (Red Hat)
- Muse Glimmer 30B Q4_0 GGUF: 0.0213 vs 0.0228 (Meta Q4_K_M)
- Gemma 4 12B W4A16: 0.034 vs 0.044 (Google QAT)
- Gemma 4 12B Q4_0 GGUF: 0.048 vs 0.069 (Google QAT)
- Gemma 4 E4B W4A16: 0.022 vs 0.064 (Google QAT)
- Gemma 4 E4B Q4_0 GGUF: 0.044 vs 0.088 (Google QAT)
- Qwen3.5-4B NVFP4: 0.0181 vs 0.0419 (cyankiwi AWQ)
- Qwen3.5-4B NVFP4 W4A4: 0.046 vs 0.075 (cosmicproc)
- Qwen3.5-4B Q4_0 GGUF: 0.235 vs 0.284 (bartowski)
Highlights:
- Muse Glimmer 30B matches or beats BF16 on GPQA-Diamond, MMLU-Pro and AIME'25.
- Qwen3.8-27B has every linear layer in NVFP4 and scores 90.9 on GPQA-Diamond (BF16: 91.4), at 19.7 GB.
- Gemma 4 E4B and 12B beat Google's QAT on every benchmark we ran.
Models: QUASAR-QAT/all-quasar-models-6aab09fd2913ec22df0d5f15
Paper: QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction (2608.13966)
Code: https://github.com/vincentcounathe/quasar-qat
Project page: https://quasar-qat.github.io/
Feedback welcome!
- 1 reply