nvidia

/

Qwen3.8-Flash-Next-FP4-Local

Like 401

Follow

NVIDIA 75.1k

Hi all! I got nvidia/Qwen3.8-Flash-Next-FP4-Local serving on SGLang on a single RTX PRO 6000 Blackwell (SM120). Sharing the setup and numbers for anyone who'd rather run it on upstream SGLang than on the vLLM Docker image from the model card.

What it takes

SGLang main plus three open pull requests:

• #41624: the NVFP4 PLE table and the MXFP8 layers (by @L14nY1Wang)
• #46153: loads the NVFP4 experts with CSF-compressed scales by decoding them to plain E4M3 at load time, so every existing NVFP4 MoE backend runs unchanged (no new kernels). It also loads the vision tower in BF16.
• #46128: reads each shard sequentially before the multi-thread loader maps it. The full ~95 GiB load went from 8:58 to 2:03 on my disk (load times vary between runs).

All three are needed. Without #41624, the server stops with a cuMemHostAlloc out-of-memory error in allocate_ple_host_table, however much memory is free. A branch with all three stacked (merge conflict resolved): https://github.com/AndreasBis/sglang/tree/test/fp4-local-e2e

FlashInfer's prebuilt SM120 kernels are also needed: flashinfer install-jit-cache-wheel --mode minimal --sm sm120f. Without them, FlashInfer starts compiling only after the PLE table is pinned, and the build fails with Ninja build failed.

Command

SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1 \
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,pinned_max_round_threshold_mb:1024 \
sglang serve \
  --model-path nvidia/Qwen3.8-Flash-Next-FP4-Local \
  --served-model-name Qwen3.8-Flash-Next-FP4-Local \
  --host 127.0.0.1 \
  --port 8000 \
  --device cuda \
  --random-seed 42 \
  --enable-multimodal \
  --stream-interval 1 \
  --tokenizer-worker-num 1 \
  --detokenizer-worker-num 1 \
  --mem-fraction-static 0.95 \
  --context-length 229376 \
  --kv-cache-dtype fp8_e4m3 \
  --max-running-requests 16 \
  --prefill-max-requests 16 \
  --max-queued-requests 16 \
  --chunked-prefill-size 4096 \
  --max-prefill-tokens 32768 \
  --page-size 64 \
  --radix-eviction-policy lru \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --cuda-graph-max-bs-decode 16 \
  --cuda-graph-max-bs-prefill 4096 \
  --enable-cudagraph-gc \
  --sampling-backend flashinfer \
  --mm-attention-backend flashinfer_cudnn \
  --prefill-attention-backend flashinfer \
  --decode-attention-backend flashinfer \
  --linear-attn-decode-backend flashinfer \
  --linear-attn-prefill-backend flashinfer \
  --mamba-backend flashinfer \
  --mamba-ssm-dtype bfloat16 \
  --mamba-radix-cache-strategy extra_buffer_lazy \
  --mamba-track-interval 256 \
  --max-mamba-cache-size 16 \
  --fp8-gemm-backend cutlass \
  --fp4-gemm-backend flashinfer_cutl
  --moe-runner-backend flashinfer_cutlass \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --speculative-moe-runner-backend f
  --ple-offload-embedding \
  --preferred-sampling-params '{"tem, "top_k": 20, "min_p": 0.0,"presence_penalty": 0.0, "repetition

No --quantization flag is needed: the checkpoint resolves to modelopt_mixed. Speculative decoding uses the model's built-in MTP layers (3 draft steps). I also set the attention, linear-attention and Mamba backends to flashinfer.

Accuracy

GSM8K: 97.80% (all 1,319 questions, single run). SGLang's cookbook lists 96.9–97.8% for NVFP4 exports.

Compared with RadixArk/Qwen3.8-Flash-Next-NVFP4 on the same SGLang setup

• about 10% higher throughput
• about twice the KV cache pool
• ~25 GiB less system memory (the PLE table is 26.8 GiB, against 47.7 GiB)

In daily use it feels as capable as the 126 GiB checkpoint; beyond GSM8K I haven't measured it.

Speed against the model card's vLLM recipe

local-inference-lab/vllm:release-10152026, same client, prompts and limits: FP8 KV cache, 4096-token prefill chunks, 256 output tokens. Each cell is SGLang / vLLM.

| Prompt | Concurrency | Output tok/s | TTFT (ms) | TPOT (ms) |
| 1K | 1 | 217 / 221 | 125 / 204 | 4.2 / 3.8 |
| 1K | 2 | 296 / 318 | 148 / 234 | 5.9 / 5.3 |
| 1K | 4 | 337 / 407 | 157 / 240 | 8.7 / 8.6 |
| 2K | 1 | 170 / 214 | 211 / 223 | 5.1 / 3.9 |
| 2K | 2 | 295 / 327 | 236 / 244 | 5.5 / 5.1 |
| 2K | 4 | 408 / 415 | 247 / 253 | 8.4 / 8.4 |
| 4K | 1 | 179 / 198 | 387 / 361 | 4.1 / 3.8 |
| 4K | 2 | 241 / 271 | 410 / 395 | 6.3 / 5.6 |
| 4K | 4 | 318 / 330 | 430 / 434 | 10.0 / 9.7 |
| 8K | 1 | 148 / 167 | 770 / 711 | 3.9 / 3.6 |
| 8K | 2 | 189 / 205 | 804 / 800 | 6.8 / 6.7 |
| 8K | 4 | 216 / 231 | 818 / 838 | 14.0 / 13.3 |
| 24K | 1 | 92 / 107 | 2288 / 2075 | 3.7 / 3.2 |
| 24K | 2 | 97 / 107 | 2754 / 2401 | 8.3 / 10.3 |
| 24K | 4 | 100 / 106 | 5209 / 3038 | 18.2 / 25.3 |
| 192K | 1 | 22 / 25 | 10711* / 9310* | 3.8 / 3.3 |
| 192K | 2 | 16 / 18 | 21448 / 19133 | 44.6 / 56.2 |
| 192K | 4 | 12 / 15 | 52438 / 43857 | 127.2 / 84.7 |

• Throughput: vLLM is 2–27% higher in every row.
• TTFT: SGLang is lower at 1K, about even from 2K to 8K, and higher from 24K up.
• TPOT: vLLM is lower or equal in most rows; SGLang is lower at 24K with concurrency 2 and 4, and at 192K with concurrency 2.

* The client's warm-up prompt repeats in each run and is served from the prefix cache, so the 192K concurrency-1 TTFT (2 prompts) and the tok/s at 24K and 192K are optimistic for both engines.

Feedback on the PRs, and test reports from other Blackwell setups, are very welcome. Thanks to NVIDIA for releasing this checkpoint!