Hugging Face's logo

google
/
gemma-4-E4B-it

Any-to-Any
Transformers
Safetensors
gemma4
image-text-to-text
Eval Results
Model card Files
xet
 Community
55

BF16 vs 4bit QAT benchmark on RTX 5060 Ti (vLLM 0.29): 32 min runtime and almost zero quality loss in 4bit

#55
by Mausbiber - opened
Discussion
Mausbiber
•
edited 1 day ago

Hi everyone,

I ran a direct comparison between the unquantized BF16 google/gemma-4-E4B-it (14.9 GiB weights) and the 4bit QAT version (xbill9/gemma-4-E4B-it-qat-q4_0-w4a16-ct-text-ple4, 4.7 GiB weights) on 2x NVIDIA RTX 5060 Ti (32 GB VRAM, vLLM 0.29.0, 131k context).

Unlike standard post training quantization on small models, Google's QAT holds up remarkably well in 4bit while nearly doubling speed:

  • Throughput (vllm bench serve): 219.2 tok/s (4bit QAT) vs 118.5 tok/s (BF16)
  • Thinking On Overall: 44.3% in 32 minutes (BF16) vs 43.1% in 21 minutes (4bit QAT)
  • IFBench (Thinking On): 40.70% (4bit QAT) vs 40.12% (BF16)
  • IFEval (Thinking On): 84.47% (4bit QAT) vs 87.25% (BF16)
  • Thinking Off Overall: 21.2% (4bit QAT) vs 19.6% (BF16), with 23.73% on SciCode for the 4bit QAT checkpoint

Finishing the entire 3 benchmark reasoning suite (1,841 prompts) in just 21 to 32 minutes makes E4B the most time efficient model in our entire test.

Full interactive table:
https://llmbenchmarks.io/local-gpu-benchmarks/
Mausbiber changed discussion title from Full local GPU benchmark results (Score, real runtime & time-efficiency, Thinking ON vs OFF) to BF16 vs 4bit QAT benchmark on RTX 5060 Ti (vLLM 0.29): 32 min runtime and almost zero quality loss in 4bit

Sign up or log in to comment