DeepSeek-V4-Pro-NVFP4

Model Overview

• Model Architecture: DeepseekV4ForCausalLM
  • Input: Text
  • Output: Text
• Supported Hardware Microarchitecture: AMD MI355 / MI350 / MI300 (emulation)
• ROCm: 7.2.3
• PyTorch: 2.11.0
• Transformers: 5.13.1
• Operating System(s): Linux
• Inference Engine: vLLM / SGLang
• Model Optimizer: AMD-Quark (v0.12.0)
  • Quantized layers:
    • Router experts: NVFP4
    • shared_experts, attn: FP8-E4M3 per-block

Model Quantization

The model was quantized from deepseek-ai/DeepSeek-V4-Pro with experts quantized to MXFP4, and shared_experts and attn quantized to FP8. Using AMD Quark, we re-quantized both experts and shared_experts to NVFP4 while keeping attn in FP8.

Quantization script

The end-to-end recipe lives in the Quark examples: examples/torch/language_modeling/llm_ptq/deepseek_v4/nvfp4, and is driven by run_pipeline.sh. The quantization scope is controlled by EXCLUDE_LAYERS.

export EXCLUDE_LAYERS="*attn* *ffn.gate mtp* *shared_experts*" \
export SRC=deepseek-ai/DeepSeek-V4-Pro
export OUT=amd/DeepSeek-V4-Pro-NVFP4

bash run_pipeline.sh

Deployment

Use with vLLM

This model can be deployed efficiently using the vLLM backend. SGLang is also supported.

Evaluation

The model was evaluated on GSM8K benchmarks.

Accuracy

Benchmark	DeepSeek-V4-Pro (BF16)	DeepSeek-V4-Pro-NVFP4	Recovery
GSM8K (flexible-extract)	95.38	94.77	99.36%

Reproduction

The GSM8K result was obtained using the lm-evaluation-harness framework, based on the Docker image rocm/vllm-dev:nightly_main_20260714.

Install the lm-eval (Version: 0.4.12) in container first.

pip install lm-eval[api]

Launching server

VLLM_ROCM_USE_AITER=1 \
VLLM_ROCM_USE_AITER_MOE=1 \
vllm serve amd/DeepSeek-V4-Pro-NVFP4 \
  --host localhost \
  --port 8001 \
  --dtype auto \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 8 \
  --max-num-seqs 512 \
  --max-num-batched-tokens 8192 \
  --distributed-executor-backend mp \
  --trust-remote-code \
  --gpu-memory-utilization 0.9 \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --compilation-config '{"mode": 3, "cudagraph_mode": "FULL_DECODE_ONLY"}'

Evaluating model in a new terminal

lm_eval \
  --model local-completions \
  --model_args model=amd/DeepSeek-V4-Pro-NVFP4,tokenizer=amd/DeepSeek-V4-Pro-NVFP4,base_url=http://127.0.0.1:8001/v1/completions,num_concurrent=32,max_retries=10,max_gen_toks=2048,timeout=60000 \
  --batch_size auto \
  --tasks gsm8k \
  --num_fewshot 8 \
  --output_path . \
  2>&1 | tee -a eval.log

License

This model is a quantized derivative of deepseek-ai/DeepSeek-V4-Pro and is distributed under the same license as the source model: the MIT License. A copy of the upstream LICENSE is included in this repository.

Modifications Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. AMD has modified the model weights of the MoE expert layers by quantizing them to NVFP4 with AMD Quark; the modifications are provided under the same MIT License and are not subject to any separate or different license.