axteam/gemma-4-31B-it-NVFP4

gemma-4-31B-it · NVFP4 (W4A4, FP4)

4-bit quantization of google/gemma-4-31B-it (commit 842da37), produced for efficient vLLM serving. Quantized by axteam.

Base model	google/gemma-4-31B-it (commit 842da37)
Method	NVFP4 (W4A4, FP4)
Format	compressed-tensors nvfp4-pack-quantized
Weights / activations	W4A4 — FP4 (E2M1) weights group 16 (FP8 E4M3 scale + per-tensor FP32 global scale), FP4 activations
Architecture	Gemma 4 · Dense
Size on disk	~20.4 GB
Calibration	256 samples (Korean-heavy; -it includes 32 self-generated thinking-mode answers)
License	Gemma 4 / Apache-2.0

Quantization details

• Scheme — W4A4 — FP4 (E2M1) weights group 16 (FP8 E4M3 scale + per-tensor FP32 global scale), FP4 activations, stored as compressed-tensors nvfp4-pack-quantized.
• Kept in BF16 — vision tower, vision projection, token embeddings (tied with lm_head), and all norms. These stay unquantized to preserve quality — about 6% of all parameters in this build.
• Tokenizer / chat_template / processor_config / generation_config are byte-identical to the original, including Google's July-2026 corrected chat_template. config.json is the original plus a quantization_config block.

Quantization method — PTQ (AWQ / RTN) vs QAT

This model is a post-training quantization (PTQ): the released BF16 weights are quantized after training, with no retraining. How close each build stays to BF16 is reported in the measured sections where available.

• RTN (round-to-nearest) — weights are simply rounded to the quantization grid with per-group scales; no calibration data. Fastest, but outliers aren't compensated, so 4-bit fidelity is the lowest. Used only as an internal baseline here, not published.
• AWQ (activation-aware weight quantization) — a small calibration set finds the weight channels that multiply the largest activations; those salient channels are protected by per-channel scaling (the scale is folded into adjacent layers, e.g. LayerNorm → q/k/v) before rounding. Still PTQ (no gradient, seconds–minutes), and far better than RTN at 4-bit.
• QAT (quantization-aware training) — the model is fine-tuned with quantization simulated in the forward pass, so the weights adapt to quantization error via gradient descent. Best low-bit fidelity, but needs the full training stack, data and GPU hours. Google ships separate Gemma QAT checkpoints; this repo is not one of them.

Rough low-bit fidelity: QAT ≥ AWQ > RTN. Cost: QAT ≫ AWQ > RTN. This build: AWQ — a calibration-based PTQ (256-sample calibration, no gradient).

Benchmarks (measured)

Every column was measured with the same harness and settings — lm-evaluation-harness on vLLM, greedy, few-shot completion (no chat template). Percentages are recovery vs. the BF16 original.

Benchmark	BF16 (original)	FP8-dynamic	AWQ-8bit	NVFP4A16	NVFP4 (this repo)	AWQ-4bit
MMLU (5-shot)	56.46	56.07 (99.3%)	56.41 (99.9%)	58.32 (103.3%)	58.51 (103.6%)	57.12 (101.2%)
GSM8K (5-shot, strict)	91.81	91.89 (100.1%)	91.66 (99.8%)	91.96 (100.2%)	91.74 (99.9%)	92.34 (100.6%)
ARC-Challenge (0-shot)	57.59	58.36 (101.3%)	57.68 (100.1%)	57.76 (100.3%)	56.91 (98.8%)	57.17 (99.3%)
Winogrande (5-shot)	66.22	66.14 (99.9%)	65.82 (99.4%)	65.59 (99.0%)	64.96 (98.1%)	66.77 (100.8%)
KMMLU (5-shot, direct)	25.02	23.89 (95.5%)	25.21 (100.8%)	26.55 (106.1%)	26.78 (107.0%)	25.02 (100.0%)
HAE-RAE (0-shot)	76.81	75.25 (98.0%)	76.81 (100.0%)	77.09 (100.4%)	77.09 (100.4%)	78.09 (101.7%)
Average	62.32	61.93 (99.4%)	62.27 (99.9%)	62.88 (100.9%)	62.66 (100.6%)	62.75 (100.7%)

Evaluation (measured)

Measured against the BF16 original on an internal mixed English/Korean set. Quantization variant: AWQ attention.

Metric	This model	BF16 original
Perplexity (lower is better)	1.2326	1.1120 (+10.8%)
Top-1 next-token agreement	92.4%	99.4%

These numbers measure fidelity to the base model, not absolute task scores.

Comparison with other public quantizations

Same eval set and harness for every row — closer to the BF16 original is better.

Model	Perplexity	Top-1
BF16 original (upper bound)	1.1120	99.4%
This model (axteam)	1.2326	92.4%
RedHatAI/gemma-4-31B-it-NVFP4	1.2753	91.4%

On our harness this model is on par with or ahead of the public same-format quantizations above. It also ships Google's corrected July-2026 chat_template; some earlier community builds were based on the older template.

Requirements

Runtime	vLLM — the architecture is in vLLM ≥ 0.19.0; tested here with vLLM 0.32.0.
vLLM install	vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately).
GPU	Full FP4 (W4A4) speed on Blackwell — B200/GB200 (sm100), B300 (sm103), RTX 50xx / RTX PRO 6000 (sm120). Turing–Hopper (sm75–90) fall back to weight-only FP4 (slower, vLLM warns). V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.)
Memory	Weights: 20.4 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use --tensor-parallel-size N when the weights do not fit one GPU.
Transformers (without vLLM)	transformers>=5.5.0 (5.5.2+ recommended; not run here for 31B).
trust_remote_code	not needed

Usage (vLLM)

vllm serve axteam/gemma-4-31B-it-NVFP4

from vllm import LLM, SamplingParams

llm = LLM(model="axteam/gemma-4-31B-it-NVFP4")
params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=1024)
out = llm.chat(
    [{"role": "user", "content": "Write a short joke about saving RAM."}],
    params,
)
print(out[0].outputs[0].text)

Recommended sampling (per Google): temperature=1.0, top_p=0.95, top_k=64. Thinking mode is off by default; turn it on with chat_template_kwargs={"enable_thinking": True} (Python llm.chat(..., chat_template_kwargs=...) or the request body), and add --reasoning-parser gemma4 to vllm serve to return the reasoning separately. Full speed needs Blackwell FP4 tensor cores (e.g. B200, RTX PRO 6000); older GPUs run it through slower fallback kernels.

Reducing KV-cache memory (optional, serve-time)

The KV cache is unquantized (BF16) here — the standard for quantized checkpoints (RedHatAI, NVIDIA, etc. ship the same way). KV-cache precision is a runtime choice, independent of the weight quantization above. For long context or large batches you can roughly halve KV-cache memory with FP8:

vllm serve axteam/gemma-4-31B-it-NVFP4 --kv-cache-dtype fp8

--kv-cache-dtype accepts auto (default, BF16), fp8 / fp8_e4m3, fp8_e5m2. For a GGUF build the llama.cpp equivalent is --cache-type-k q8_0 --cache-type-v q8_0.

License

This is a quantized derivative of Google's Gemma 4. Use is governed by the Gemma 4 license. All original weights, tokenizer, and chat template belong to Google DeepMind.

---

한국어 요약

google/gemma-4-31B-it의 NVFP4 (W4A4) 양자화 빌드 · 제작 axteam

개요

항목	내용
원본 모델	google/gemma-4-31B-it (커밋 842da37)
양자화 방식	NVFP4 (W4A4, 가중치·활성화 FP4)
저장 형식	compressed-tensors nvfp4-pack-quantized
모델 구조	Gemma 4 · Dense
용량	약 20.4 GB
보정 데이터	샘플 256개 (한국어 위주, 원본 모델이 스스로 생성한 thinking 모드 답변 32개 포함)

양자화 상세

• 가중치: FP4(E2M1), 그룹 크기 16 (그룹별 FP8(E4M3) 스케일 + 텐서별 FP32 전역 스케일)
• 활성화: 실행 시 FP4 (Blackwell FP4 텐서코어에서 가속)
• 양자화 절차: PTQ(학습 후 양자화, 재학습 없음), AWQ 방식 — 보정 데이터로 활성화가 큰 가중치 채널을 찾아 채널별 스케일로 보호한 뒤 FP4로 반올림
• BF16 유지: 비전 타워, 비전 투영층, 토큰 임베딩(LM 헤드와 공유), 모든 정규화층 — 전체 파라미터의 약 6%
• 원본 파일: 토크나이저·chat template(Google의 2026년 7월 수정본)·processor·generation config를 바이트 단위로 그대로 사용, config.json은 원본에 quantization_config 블록만 추가

품질 (실측)

• 원본 대비 충실도: PPL 1.2326 (원본 1.1120, +10.8%, 낮을수록 원본에 가까움), 다음 토큰 top-1 일치율 92.4% (원본 자체 기준값 99.4%) — 내부 영어·한국어 혼합 평가셋, 원본(BF16)과 비교 측정
• 다른 공개 빌드와 비교: 같은 형식의 RedHatAI/gemma-4-31B-it-NVFP4 PPL 1.2753 · top-1 91.4%, 이 빌드 PPL 1.2326 · top-1 92.4% (같은 평가셋·같은 평가 코드) — 이 빌드가 동등 이상, Google의 2026년 7월 수정 chat template 포함 (일부 이전 커뮤니티 빌드는 수정 전 템플릿 기반)
• 벤치마크 평균: 62.66 (원본 62.32, 100.6%) — MMLU·GSM8K·ARC-Challenge·Winogrande·KMMLU·HAE-RAE 6종 평균, lm-evaluation-harness + vLLM, chat template 없는 few-shot·greedy, 모든 빌드를 같은 설정으로 측정
• 한국어 벤치마크: KMMLU 26.78 (원본 25.02, 107.0%) · HAE-RAE 77.09 (원본 76.81, 100.4%)

요구사항

• 실행 엔진: vLLM 0.19.0 이상(이 구조를 지원하는 첫 버전), 이 빌드는 vLLM 0.32.0에서 실행 확인
• vLLM 0.32 설치 조건: Python 3.11~3.14, torch 2.13(CUDA 13.0 빌드, CUDA 13 지원 드라이버 필요), transformers 5.16.1 이상 5.20 미만, compressed-tensors 0.17.0 — transformers·compressed-tensors는 vLLM이 버전을 고정하므로 compressed-tensors를 따로 업그레이드하지 말 것 (이 체크포인트는 0.19.0으로 저장됐지만 0.17.0에서도 로드됨)
• GPU: Blackwell(B200·GB200 sm100, B300 sm103, RTX 50xx·RTX PRO 6000 sm120)에서 FP4(W4A4) 연산 가속, V100(sm70) 미지원 — TuringHopper(sm7590)는 가중치 전용 FP4로 자동 전환되어 속도 저하, vLLM 경고 출력 (GPU별 지원 여부는 vLLM 0.32 커널 선택 코드 기준, 실제 실행은 B200에서만 확인)
• 메모리: 가중치 20.4 GB + KV 캐시(컨텍스트 길이 × 동시 요청 수에 비례) + 활성화·CUDA graph용 수 GB — GPU 한 장에 다 들어가지 않으면 --tensor-parallel-size N으로 분산
• Transformers로 직접 사용: transformers 5.5.0 이상 (5.5.2 이상 권장) — 이 모델은 Transformers로 직접 실행해 보지 않음
• trust_remote_code: 불필요

사용법

• 서버 실행: vllm serve axteam/gemma-4-31B-it-NVFP4
• Python: LLM(model="axteam/gemma-4-31B-it-NVFP4").chat(...) — 전체 예시는 위 영어 본문 'Usage (vLLM)' 절 참고
• 권장 샘플링: temperature=1.0, top_p=0.95, top_k=64 (Google 권장값)
• thinking 모드: 기본 꺼짐 — 켜려면 chat_template_kwargs={"enable_thinking": True} 전달(Python은 llm.chat() 인자, HTTP는 요청 본문의 "chat_template_kwargs": {"enable_thinking": true}), 추론 과정을 답변과 분리해 받으려면 vllm serve에 --reasoning-parser gemma4 추가
• KV 캐시 절감(선택): vllm serve에 --kv-cache-dtype fp8 추가 — 기본 BF16인 KV 캐시를 FP8로 저장해 KV 캐시 메모리를 약 절반으로 줄임 (긴 컨텍스트·동시 요청이 많을 때 유리, 가중치 양자화와 별개인 실행 옵션)

라이선스

• 라이선스: Apache-2.0 (Gemma 4 라이선스 안내) — 원본 모델의 라이선스를 그대로 따름
• 권리: 원본 가중치·토크나이저·chat template은 Google DeepMind 소유, 이 리포는 양자화 파생물