Qwen3-Next-80B-A3B-Instruct-NVFP4

Description

This is a 4-bit floating-point (NVFP4) quantization of Qwen/Qwen3-Next-80B-A3B-Instruct, the non-thinking (instruct) model of the Qwen3-Next series: an 80B-parameter, high-sparsity Mixture-of-Experts model with about 3B active parameters per token and a hybrid Gated DeltaNet / Gated Attention architecture. Please see the original Qwen model page for the model's capabilities, benchmarks, intended use and limitations. All credit for the model itself goes to the Qwen team.

The weights were converted from the original BF16 checkpoint with llm-compressor to shrink the model to roughly 46 GB (43 GiB) on disk so it fits on a single NVIDIA GB10 (Blackwell, unified memory) system. The architecture, tokenizer and chat template are unchanged. This is a community conversion, not an official Qwen release, and no accuracy benchmarks are published here; compare against the BF16 model on your own workload before relying on it.

Quantization details

• Base model: Qwen/Qwen3-Next-80B-A3B-Instruct (Apache-2.0)
• Format: compressed-tensors, nvfp4-pack-quantized (4-bit float weights, group size 16, FP8 E4M3 scales)
• Scheme: llm-compressor QuantizationModifier, scheme: nvfp4, targets: [Linear] (see recipe.yaml)
• Left unquantized: lm_head, mlp.shared_expert_gate, linear_attn.in_proj_ba, and the other modules listed under quantization_config.ignore in config.json
• Files: 12 safetensors shards
• Max position embeddings (config): 262144

Serving with vLLM (Docker Compose)

This is a sanitized copy of the compose file used to serve the model. It targets an NVIDIA GB10 system with nvcr.io/nvidia/vllm:26.08-py3 (vLLM 0.27.1). vLLM detects the quantization from config.json, so no --quantization flag is needed. Replace the host paths with your own and adjust the memory settings to your hardware.

services:
  vllm:
    image: nvcr.io/nvidia/vllm:26.08-py3
    container_name: qwen3-next-80b-a3b-instruct-nvfp4
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility
      - VLLM_NVFP4_GEMM_BACKEND=marlin
      - VLLM_USE_FLASHINFER=0
    volumes:
      - /path/to/Qwen3-Next-80B-A3B-Instruct-NVFP4:/model
      - /path/to/vllm-cache:/root/.cache/vllm
    ports:
      - "8000:8000"
    runtime: nvidia
    restart: "no"
    entrypoint: ["vllm"]
    command: ["serve", "/model",
              "--served-model-name", "qwen-Qwen3-Next-80B-A3B-Instruct-NVFP4",
              "--host", "0.0.0.0",
              "--port", "8000",
              "--trust-remote-code",
              "--max-model-len", "131072",
              "--max-num-batched-tokens", "24576",
              "--gpu-memory-utilization", "0.30",
              "--kv-cache-dtype", "fp8",
              "--kv-cache-memory-bytes", "22548578304",
              "--enable-prefix-caching",
              "--enable-chunked-prefill",
              "--max-num-seqs", "16",
              "--enable-auto-tool-choice",
              "--tool-call-parser", "hermes",
              "--no-enable-log-requests"]

Notes on the settings:

• VLLM_NVFP4_GEMM_BACKEND=marlin and VLLM_USE_FLASHINFER=0 are what this GB10 deployment used; other hardware or vLLM versions may not need them.
• --kv-cache-memory-bytes fixes the KV cache at 21 GiB (fp8), which vLLM reported as roughly 13x concurrency at the full 131072-token context on this deployment. When it is set, it takes precedence over --gpu-memory-utilization.
• This is the Instruct (non-thinking) model, so no --reasoning-parser is configured.

License

Apache-2.0, inherited from the base model.