Gemma 4 MTP Assistant — NVFP4 (MSE scaling) for Blackwell

The speculative-decoding draft companion to gemma-4-31B-it-qat-NVFP4-mse-Blackwell, re-quantized with MSE-based weight scaling. It replaces my earlier absmax version the same way the main model does: per-block scales picked by reconstruction error instead of the block's largest value, via ModelOpt 0.44.0 (--qformat nvfp4_mse). The MTP handshake layers (pre_projection, post_projection) stay in high precision — quantizing those breaks the draft's connection to the backbone's KV states.

Which vLLM you're on decides which draft you need

vLLM 0.20.x (legacy --speculative-model flags): this NVFP4 draft works. That's the setup the serving example below uses.
vLLM ≥ 0.21 (speculative-config with method: gemma4_mtp): the MTP proposer loads the draft without applying its quant config, so a packed FP4 draft dies with a tensor-size error. Use the unquantized BF16 assistant (google/gemma-4-31B-it-qat-q4_0-unquantized-assistant, 927 MB) there instead. It's small enough that the quantization barely matters for VRAM.

Measured with the MSE main model as target on an RTX 6000 Blackwell (300-token generations, temp 0.2): 23.0 tok/s without a draft, 52.1 tok/s with, ~54% token acceptance (78% at position 0).

Serving (vLLM 0.20.x)

vllm serve melcheikh/gemma-4-31B-it-qat-NVFP4-mse-Blackwell \
    --quantization modelopt_fp4 \
    --kv-cache-dtype fp8 \
    --speculative-model melcheikh/gemma-4-31B-it-qat-assistant-NVFP4-mse-Blackwell \
    --num-speculative-tokens 3