Qwen3-VL-8B-Instruct-nvfp4

NVFP4 quantization of Qwen/Qwen3-VL-8B-Instruct, produced with llm-compressor 0.13.0 (compressed-tensors 0.18.0) and verified end to end on NVIDIA H100 80 GB with vLLM 0.26.0 — text and image prompts.

What is quantized

Scheme	NVFP4
Quantized	all 252 language-model linears, attention included
Kept in bf16	vision tower, multimodal projector, lm_head, embeddings
Calibration	512 sequences x 2048 tokens from HuggingFaceH4/ultrachat_200k
Size on disk	7.04 GiB
Format	HF format (config.json + sharded safetensors).

Measured on H100 (vLLM 0.26.0, lm-eval 0.4.12)

variant	size	GSM8K (5-shot, strict)	ARC-Challenge (acc_norm)
bf16 source	–	84.31	60.92
FP8 (this repo's sibling)	9.87 GiB	84.38 (+0.1 pp)	61.01 (+0.1 pp)
NVFP4	7.04 GiB	78.85 (-5.5 pp)	60.24 (-0.7 pp)

GSM8K n=1319, ARC-Challenge n=1172, greedy, served through the OpenAI completions API.

Kernels

On H100 vLLM serves this through MarlinNvFp4LinearKernel — 4-bit weights in HBM, bf16 GEMM. The input_global_scale tensors are carried for Blackwell, where the same checkpoint runs on real FP4 tensor cores with no changes. To preview that numerical behaviour on Hopper, serve with --linear-backend emulation (slow, but it rounds activations through FP4 exactly as SM100 would).

Serving

vllm serve mbehr90/Qwen3-VL-8B-Instruct-nvfp4 --max-model-len 32768 --gpu-memory-utilization 0.9

The quantization recipe is in recipe.yaml.