intel-ai/DeepSeek-V3.2-NVFP4-AutoRound

DeepSeek-V3.2 AutoRound MXFP8 with NVFP4 MoE Experts

Model Details

This checkpoint is a mixed-precision quantized version of DeepSeek-V3.2, generated with Intel AutoRound 0.17.0.

The general quantization configuration is MXFP8 W8A8 with group size 32 and dynamic activation scaling. MoE expert layers use NVFP4. The indexer modules are retained at 16-bit precision. Please follow the original model's license and usage terms.

Property	Value
Base model	deepseek-ai/DeepSeek-V3.2
Quantization	Mixed MXFP8 and NVFP4
General weight / activation format	MXFP8 W8A8
General group size	32
MoE expert quantization	NVFP4
Activation scaling	Dynamic
Indexer precision	16-bit
Quantization method	AutoRound
AutoRound version	0.17.0
Packing format	auto_round:llm_compressor

How to Use

Install the vLLM plugin from the main branch of Intel Neural Compressor into the same Python environment as vLLM. The plugin must be installed to register the AutoRound NVFP4 format with vLLM, including when activation QDQ is disabled.

git clone https://github.com/intel/neural-compressor.git
cd neural-compressor
python -m pip install --no-build-isolation -e benchmark/vllm-qdq-plugin/

VLLM_QDQ=1 vllm serve /home/xinhe/models/deepseek-ai/DeepSeek-V3.2-NVFP4-AutoRound/DeepSeek-V3.2-mxfp-w8g32 \
  --served-model-name dsv32 \
  --port 8001 \
  --tensor-parallel-size 4 \
  --max-model-len 102400 \
  --max-num-seqs 512 \
  --max-num-batched-tokens 20000 \
  --enable-chunked-prefill \
  --gpu-memory-utilization 0.95 \
  --trust-remote-code \
  --tokenizer-mode deepseek_v32 \
  --tool-call-parser deepseek_v32 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v3 \
  --kernel-config.enable_flashinfer_autotune=False

The server uses DeepSeek-V3.2's tokenizer, reasoning parser, and tool-call format. FP8 KV cache is a runtime option and is separate from the checkpoint's weight and activation quantization. For GPUs without native NVFP4 support, use the plugin's supported QDQ path with VLLM_QDQ=1 if supported by your vLLM and GPU configuration.

Evaluation

The table reports the supplied benchmark scores. The FP8 reference is the original model.

Benchmark	FP8 reference	Intel (NVFP4)	Ratio
AIME 2025	92.22	86.67	93.98%
GPQA Diamond	84.17	83.84	99.60%
GSM8K	97.45	97.37	99.92%
LiveCodeBench v6	80.39	81.52	101.41%
PIQA	94.29	94.67	100.41%

The evaluation generation settings enable DeepSeek reasoning. Example EvalScope command:

evalscope eval \
  --model dsv32 \
  --eval-type openai_api \
  --api-key EMPTY \
  --timeout 36000 \
  --datasets aime25 gpqa_diamond gsm8k live_code_bench piqa \
  --dataset-args '{"live_code_bench":{"subset_list":["v6"]}}' \
  --generation-config '{"temperature":1.0,"top_p":0.95,"n":1,"extra_body":{"chat_template_kwargs":{"enable_thinking":true,"reasoning_effort":"max"}},"max_tokens":64000}' \
  --eval-batch-size 256 \
  --api-url http://127.0.0.1:8001/v1

For meaningful comparisons, keep the benchmark versions, dataset subsets, generation settings, KV-cache configuration, and evaluation harness consistent.

Ethical Considerations and Limitations

The model may produce factually incorrect, biased, or otherwise harmful output. Do not rely on it as the sole source of factual information. Perform appropriate safety, quality, and performance testing before deployment, and follow the original model's terms and applicable laws.

Citation

Please cite the original DeepSeek-V3.2 work and the AutoRound paper when using this checkpoint:

@misc{deepseekai2025deepseekv32,
  title={DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models},
  author={DeepSeek-AI},
  year={2025}
}

@article{cheng2023optimize,
  title={Optimize weight rounding via signed gradient descent for the quantization of llms},
  author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi},
  journal={arXiv preprint arXiv:2309.05516},
  year={2023}
}

Signed-off-by: He, Xin xin3.he@intel.com