Instructions to use axteam/gemma-4-E2B-it-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use axteam/gemma-4-E2B-it-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="axteam/gemma-4-E2B-it-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("axteam/gemma-4-E2B-it-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("axteam/gemma-4-E2B-it-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use axteam/gemma-4-E2B-it-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "axteam/gemma-4-E2B-it-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/gemma-4-E2B-it-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/axteam/gemma-4-E2B-it-NVFP4

SGLang

How to use axteam/gemma-4-E2B-it-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "axteam/gemma-4-E2B-it-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/gemma-4-E2B-it-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "axteam/gemma-4-E2B-it-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/gemma-4-E2B-it-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use axteam/gemma-4-E2B-it-NVFP4 with Docker Model Runner:
```
docker model run hf.co/axteam/gemma-4-E2B-it-NVFP4
```

gemma-4-E2B-it · NVFP4 (W4A4, FP4)

4-bit quantization of google/gemma-4-E2B-it, produced for efficient vLLM serving. Quantized by axteam.


Base model	`google/gemma-4-E2B-it`
Method	NVFP4 (W4A4, FP4)
Format	compressed-tensors `nvfp4-pack-quantized`
Weights / activations	W4A4 — FP4 (E2M1) weights group 16 (FP8 E4M3 scale + per-tensor FP32 global scale), FP4 activations
Architecture	Gemma 4 · Dense
Size on disk	~7.6 GB
Calibration	224 samples (Korean-heavy; no thinking-mode answers — none were generated for this model)
License	Gemma 4 / Apache-2.0

Quantization details

Scheme — W4A4 — FP4 (E2M1) weights group 16 (FP8 E4M3 scale + per-tensor FP32 global scale), FP4 activations, stored as compressed-tensors nvfp4-pack-quantized.
Kept in BF16 — vision tower, audio tower, vision/audio projections, token embeddings (tied with lm_head), the per-layer embedding table (embed_tokens_per_layer), and all norms. These stay unquantized to preserve quality — about 64% of all parameters in this build.
Tokenizer / chat_template / processor_config / generation_config are byte-identical to the original, including Google's July-2026 corrected chat_template. config.json is the original plus a quantization_config block.

Quantization method — PTQ (AWQ / RTN) vs QAT

This model is a post-training quantization (PTQ): the released BF16 weights are quantized after training, with no retraining. How close each build stays to BF16 is reported in the measured sections where available.

RTN (round-to-nearest) — weights are simply rounded to the quantization grid with per-group scales; no calibration data. Fastest, but outliers aren't compensated, so 4-bit fidelity is the lowest. Used only as an internal baseline here, not published.
AWQ (activation-aware weight quantization) — a small calibration set finds the weight channels that multiply the largest activations; those salient channels are protected by per-channel scaling (the scale is folded into adjacent layers, e.g. LayerNorm → q/k/v) before rounding. Still PTQ (no gradient, seconds–minutes), and far better than RTN at 4-bit.
QAT (quantization-aware training) — the model is fine-tuned with quantization simulated in the forward pass, so the weights adapt to quantization error via gradient descent. Best low-bit fidelity, but needs the full training stack, data and GPU hours. Google ships separate Gemma QAT checkpoints; this repo is not one of them.

Rough low-bit fidelity: QAT ≥ AWQ > RTN. Cost: QAT ≫ AWQ > RTN. This build: AWQ — a calibration-based PTQ (224-sample calibration, no gradient).

Benchmarks (measured)

Every column was measured with the same harness and settings — lm-evaluation-harness on vLLM, greedy, few-shot completion (no chat template). Percentages are recovery vs. the BF16 original.

Benchmark	BF16 (original)	FP8-dynamic	AWQ-8bit	NVFP4A16	NVFP4 (this repo)	AWQ-4bit
MMLU (5-shot)	55.97	55.49 (99.1%)	56.27 (100.5%)	53.40 (95.4%)	50.83 (90.8%)	53.82 (96.2%)
GSM8K (5-shot, strict)	51.63	48.82 (94.6%)	53.45 (103.5%)	46.47 (90.0%)	38.29 (74.2%)	42.53 (82.4%)
ARC-Challenge (0-shot)	49.57	48.04 (96.9%)	49.32 (99.5%)	50.68 (102.2%)	46.33 (93.5%)	48.63 (98.1%)
Winogrande (5-shot)	62.59	60.77 (97.1%)	62.59 (100.0%)	62.83 (100.4%)	59.67 (95.3%)	60.46 (96.6%)
KMMLU (5-shot, direct)	40.72	40.70 (100.0%)	40.71 (100.0%)	40.12 (98.5%)	38.71 (95.1%)	39.50 (97.0%)
HAE-RAE (0-shot)	62.14	62.05 (99.9%)	62.42 (100.4%)	59.76 (96.2%)	56.55 (91.0%)	59.67 (96.0%)
Average	53.77	52.65 (97.9%)	54.13 (100.7%)	52.21 (97.1%)	48.40 (90.0%)	50.77 (94.4%)

⚠️ Note: average recovery is 90.0% — noticeable degradation at this precision; prefer an 8-bit build if quality matters most.

Evaluation (measured)

Measured against the BF16 original on an internal mixed English/Korean set. Quantization variant: AWQ MLP-only (KV-shared layers).

Metric	This model	BF16 original
Perplexity (lower is better)	1.5860	1.1833 (+34.0%)
Top-1 next-token agreement	84.2%	99.0%

These numbers measure fidelity to the base model, not absolute task scores.

Requirements


Runtime	vLLM — the architecture is in vLLM ≥ 0.19.0; tested here with vLLM 0.32.0.
vLLM install	vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately).
GPU	Full FP4 (W4A4) speed on Blackwell — B200/GB200 (sm100), B300 (sm103), RTX 50xx / RTX PRO 6000 (sm120). Turing–Hopper (sm75–90) fall back to weight-only FP4 (slower, vLLM warns). V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.)
Memory	Weights: 7.6 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use `--tensor-parallel-size N` when the weights do not fit one GPU.
Transformers (without vLLM)	`transformers>=5.5.2` (5.5.0/5.5.1 cannot run the KV-shared layers).
trust_remote_code	not needed

Usage (vLLM)

vllm serve axteam/gemma-4-E2B-it-NVFP4

from vllm import LLM, SamplingParams

llm = LLM(model="axteam/gemma-4-E2B-it-NVFP4")
params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=1024)
out = llm.chat(
    [{"role": "user", "content": "Write a short joke about saving RAM."}],
    params,
)
print(out[0].outputs[0].text)

Recommended sampling (per Google): temperature=1.0, top_p=0.95, top_k=64. Full speed needs Blackwell FP4 tensor cores (e.g. B200, RTX PRO 6000); older GPUs run it through slower fallback kernels.

Reducing KV-cache memory (optional, serve-time)

The KV cache is unquantized (BF16) here — the standard for quantized checkpoints (RedHatAI, NVIDIA, etc. ship the same way). KV-cache precision is a runtime choice, independent of the weight quantization above. For long context or large batches you can roughly halve KV-cache memory with FP8:

vllm serve axteam/gemma-4-E2B-it-NVFP4 --kv-cache-dtype fp8

--kv-cache-dtype accepts auto (default, BF16), fp8 / fp8_e4m3, fp8_e5m2. For a GGUF build the llama.cpp equivalent is --cache-type-k q8_0 --cache-type-v q8_0.

License

This is a quantized derivative of Google's Gemma 4. Use is governed by the Gemma 4 license. All original weights, tokenizer, and chat template belong to Google DeepMind.

한국어 요약

google/gemma-4-E2B-it 를 NVFP4 (W4A4, FP4) 로 양자화한 서빙용(4-bit) 모델입니다. 작성: axteam.

방식 — 가중치·활성화 모두 FP4(W4A4). Blackwell(B200 등) FP4 텐서코어에서 제 속도. 활성화도 FP4라 품질 손실이 가장 큼 — 품질 우선이면 NVFP4A16 권장. 형식은 compressed-tensors nvfp4-pack-quantized.
BF16 유지 — 비전 타워·오디오 타워·비전/오디오 투영·토큰 임베딩(lm_head와 묶임)·층별 임베딩 표(embed_tokens_per_layer)·노름 (이 빌드 전체 파라미터의 약 64%).
토크나이저·chat_template·processor·generation_config 는 원본과 바이트까지 동일 (구글이 2026-07에 고친 chat_template 그대로). config.json 은 원본 + quantization_config.
양자화 방식(PTQ vs QAT) — 학습 후(PTQ) 방식이라 재학습 없이 원본 가중치를 저비트로 바꿉니다. RTN=보정 없이 반올림만(가장 빠르나 이상치 보상이 안 돼 4비트 품질 최저, 내부 기준용·비공개) · AWQ=소량 보정으로 큰 활성화와 곱해지는 중요 채널을 채널 스케일로 보호한 뒤 반올림(경사 없음, 4비트서 RTN보다 훨씬 좋음) · QAT=양자화를 흉내 내며 재학습해 가중치가 오차에 적응(저비트 최상이나 학습 스택·데이터·GPU 필요, 구글 Gemma QAT 체크포인트는 별도 — 본 레포는 QAT 아님). 저비트 품질 ≈ QAT ≥ AWQ > RTN. 본 빌드: AWQ — 보정 기반 PTQ(224문항 보정, 경사 없음).
품질(실측) — BF16 원본 대비 perplexity 1.5860(원본 1.1833, +34.0%), top-1 일치도 84.2%(원본 99.0%). 내부 영/한 평가셋에서 측정. 변형: AWQ MLP-only (KV-shared layers).
벤치마크(실측) — MMLU · GSM8K · ARC-Challenge · Winogrande · KMMLU · HAE-RAE 평균 BF16 53.77 → 이 모델 48.40 (회복률 90.0%). lm-evaluation-harness · vLLM, 모든 형식 같은 설정.
⚠️ 회복률 90.0% — 이 정밀도에서 열화가 눈에 띔. 품질 우선이면 8비트 권장.
요구사항 — vLLM ≥0.19.0 (0.32.0 에서 실측); GPU FP4(W4A4) 제 속도는 Blackwell(B200·B300·RTX 50xx/PRO 6000). Turing~Hopper 는 가중치 전용 FP4 로 자동 전환(느림); 가중치 7.6 GB + KV 캐시; Transformers 직접 사용: transformers ≥5.5.2.
사용 — vllm serve axteam/gemma-4-E2B-it-NVFP4. 권장 샘플링 temperature=1.0, top_p=0.95, top_k=64.
KV 캐시 절감(선택, 서빙 시) — 기본은 BF16 KV(양자화 체크포인트 표준, RedHat·NVIDIA 동일). 가중치 양자화와 무관한 런타임 선택. 긴 문맥·대배치에선 vllm serve … --kv-cache-dtype fp8 로 KV 메모리 절반. (GGUF라면 llama.cpp --cache-type-k q8_0 --cache-type-v q8_0)
라이선스 — 구글 Gemma 4 파생물. Gemma 4 라이선스 적용. 원본 가중치·토크나이저·chat_template 권리는 Google DeepMind.