SecureAI-SE/Qwen3.8-Flash-Next-NVFP4-PLE-INT3-FP8

Qwen3.8-Flash-Next — NVFP4 experts + INT3 PLE + FP8 dense path

Single-GPU build of Qwen3.8-Flash-Next targeting one GB10 (119 GiB unified memory) with 128k context.

⚠️ Requires a patched vLLM

Qwen3.8-Flash-Next is upstream as of vLLM 0.28.x, but INT3 PLE is not — that part is custom to this checkpoint.

git clone https://github.com/vllm-project/vllm.git && cd vllm
git checkout a69e75b9b            # or any 0.28.1+ revision
git am patches/qwen4_exp-int3-ple-support.patch
git am patches/reasoning-effort-budget.patch   # optional: reasoning_effort -> hard budget
# then build: TORCH_CUDA_ARCH_LIST=<your arch> uv pip install --no-build-isolation -e .

Verified on vLLM 0.28.1rc1 (Model loading took 89.26 GiB, BLHNC KV layout, coherent generation, 128k context at 4.12x concurrency).

Serving

# --- environment ------------------------------------------------------------
export VLLM_KV_CACHE_LAYOUT=BLHNC   # REQUIRED. 12 full-attn + 36 GDN layers have
                                    # different page sizes ([100352, 3211264]);
                                    # the default LBNHC layout cannot express that
                                    # and startup dies with:
                                    #   ValueError: QSA cache must be
                                    #   [pages, page_size, 1, head_dim]
export MAX_JOBS=2                   # bound the startup JIT. Unbounded ninja spawns
export NVCC_THREADS=1               # ~20 nvcc (~60 GB) beside 89 GiB of weights
                                    # and the process is SIGKILLed (exit 137).
export VLLM_REASONING_EFFORT_BUDGETS='{"minimal": 512, "low": 1024, "medium": 4096, "high": 16384}'
                                    # hard thinking cap per reasoning_effort
                                    # (needs reasoning-effort-budget.patch)

# --- baseline (no speculative decoding) -------------------------------------
vllm serve /path/to/checkpoint \
  --served-model-name qwen38-flash \
  --host 127.0.0.1 --port 8001 \
  --max-model-len 131072 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.85 \
  --language-model-only \
  --mm-processor-cache-gb 0 \
  --enable-prefix-caching \
  --moe_backend flashinfer_cutlass \
  --kernel-config '{"enable_flashinfer_autotune": false, "enable_jit_warmup": false, "enable_cutedsl_warmup": false}' \
  --trust-remote-code \
  --reasoning-parser qwen3 \
  --reasoning-config '{"reasoning_end_str": "\n\nConsidering the limited time by the user, I have to give the solution based on the thinking directly now.\n</think>\n\n"}' \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml

Add --enforce-eager for the first start on a cold compile cache. Without it, torch.compile runs inside profile_run(), and its autotuning allocations inflate torch_peak — a high-water mark that is never reset — so vLLM computes a negative KV budget and aborts:

Available KV cache memory: -10.38 GiB
ValueError: No available memory for the cache blocks.

The second start reuses the warm cache and succeeds; CUDA graphs then cost only 0.66 GiB and add ~7% decode.

# --- with speculative decoding (MTP) ----------------------------------------
cp model.safetensors.index.mtp.json model.safetensors.index.json

vllm serve /path/to/checkpoint \
  ... same flags as above ... \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3,"disable_eagle_block_drop":true}'

disable_eagle_block_drop is required with prefix caching (vLLM 0.28.1+). Without it vLLM cannot identify the draft model's KV group, flags every group as a draft group — Mamba groups included — and prefix-cache reuse is silently disabled:

WARNING kv_cache_utils.py: ... every group -- including Mamba groups [0,1,2,3] --
will be treated as a draft group ... prefix-cache reuse across requests will be
disabled ...

Reasoning budget

reasoning_effort on its own is only a system-prompt hint, and this model ignores it often enough to ramble even on low. A hard cap needs vLLM's thinking_token_budget, which forces the end of thinking once the budget is spent:

curl http://127.0.0.1:8001/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "qwen38-flash",
  "messages": [{"role": "user", "content": "..."}],
  "thinking_token_budget": 1024
}'

• --reasoning-config above is required for a clean cutoff. With the default, vLLM forces a bare </think> mid-thought and the model keeps reasoning in the answer. The transition sentence gives it an explicit "answer now" before </think>.
• With patches/reasoning-effort-budget.patch, reasoning_effort (top-level or in chat_template_kwargs) maps to a hard budget via VLLM_REASONING_EFFORT_BUDGETS. An explicit thinking_token_budget always wins. Requests without an effort default to xhigh and are not capped.
• The chat template tells the model its budget, so it usually finishes on its own before the cutoff. It accepts minimal/low/medium/high/xhigh/max (minimal→low, high/max→xhigh) and includes the Unsloth fixes from the Qwen3.8-27B template (developer role, merged system messages, stricter tool arguments).

Measured on GB10 (100 GSM8K questions, each at both efforts, 4 concurrent, max_tokens 16384):

	low (cap 1024)	xhigh (uncapped)
correct	99/100	97/100
time per answer, mean / p90	17.1 s / 25.9 s	30.1 s / 40.6 s
tokens per answer, mean / max	271 / 968	478 / 11060

low never hit its cap. xhigh costs the same on a typical question but has a long tail of overthinking (11k tokens, 11.6 min, wrong answer on one question).

If you port the INT PLE code: dequant is q * scale + zero_point, where the zero point is the row minimum in value space. (q - zero_point) * scale loads and answers coherently but drops GSM8K to ~0.88 and top-1 vs BF16 to ~0.73, with answers that run past max_tokens.

Notes

• --kv-cache-dtype fp8 is rejected: Qwen4Exp QSA requires a BF16 main KV cache. QSA's sparse-attention kernels read the KV cache directly and are BF16-only.
• --kv-cache-memory can raise the KV pool well past what --gpu-memory-utilization picks (vLLM prints the suggested value at startup), but on unified memory setting it to the "fully utilize" figure leaves the OS nothing: avail=0, swap engaged, NVRM: Out of memory in dmesg. Leave several GiB of headroom.
• Attention block size resolves to 1568 to match the mamba page, so prefix-cache matching is 1568-token granular.

Composition

component	format	notes
MoE experts (512/layer, top-10)	NVFP4 (4.5 bit)	as shipped by RadixArk, unmodified
PLE embeddings	INT3 asym per-row, MSE-optimal clip	3.20 bits, rel-err 0.1854
GDN linear_attn (36 layers)	FP8 E4M3, per-output-channel	4.17 → 2.10 GB
QSA self_attn + indexer (12 layers)	FP8 E4M3, per-output-channel	1.23 → 0.62 GB
shared experts	FP8 E4M3, per-output-channel	0.47 → 0.24 GB
routers, norms	BF16	never quantized — routing errors are discrete and non-averaging
hyperconnections, lm_head, embeddings	BF16	the fork hardcodes quant_config=None for hyperconnections; ParallelLMHead has no quantized path

Activations are quantized dynamically per token, so no calibration data is required.

Measured (GB10 / DGX Spark, 119 GiB unified, --max-model-len 131072)

All figures single-stream decode unless stated. "MTP" rows additionally require the MTP draft head (see below).

config	vLLM	decode	notes
BF16-dense reference	0.26.1	14.5 tok/s	the checkpoint this replaces
this build, eager	0.26.1	23.7	42.0 ms/token
this build + CUDA graphs	0.26.1	25.5	graphs cost 0.66 GiB
this build, eager	0.28.1	20.6	upstream traded decode for prefill
+ MTP, 1 speculative token	0.26.1	27.9	79.1% acceptance
+ MTP, 3 speculative tokens	0.28.1	31.7	2.89 accepted/forward
+ MTP-3, at 100k context	0.28.1	~45	repetitive text; treat as optimistic

Decode is roughly flat in context (33 tok/s at 61 tokens, 30.6 at 8k, 24.6 at 32k) — QSA sparse attention does not degrade with sequence length.

	0.26.1	0.28.1
prefill (2048 in)	1,628 tok/s	3,392 tok/s
resident weights	89.44 GiB	89.26 GiB
KV cache	410,128 tok	539,558 tok
128k concurrency	3.13x	4.12x

125k-token prompts verified end to end. Prefix caching with MTP enabled requires disable_eagle_block_drop (vLLM 0.28.1+): 98.67% hit rate on a repeated 60k prefix, 33x faster warm, with a fixed ~801-token trailing loss.

Speculative decoding (MTP)

The MTP draft head IS included: model-mtp-00001-of-00001.safetensors (31 tensors, 5.2 GB, BF16) plus model.safetensors.index.mtp.json. The default index omits it, because the main model must not load MTP weights.

To enable:

cp model.safetensors.index.mtp.json model.safetensors.index.json
vllm serve <path> \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3,"disable_eagle_block_drop":true}' \
  --enable-prefix-caching --kv-cache-dtype auto

disable_eagle_block_drop is required (vLLM 0.28.1+). Without it, vLLM flags every KV cache group as a draft group — including the Mamba groups — and prefix-cache reuse is silently disabled entirely.

Measured: 2.89 tokens accepted per forward pass; per-position acceptance 0.519 / 0.375 / 0.245. Costs 5.05 GiB of weights, which comes out of the KV pool.

Quality gate vs the BF16-dense reference

Both arms identical except for the FP8 dense path: same INT3 PLE, same NVFP4 experts, same hardware, same 200 GSM8K questions, same 40 teacher-forced sequences.

	BF16 dense	this build
GSM8K (n=200)	0.9300	0.9450
stop_rate	0.9300	0.9500

Delta +1.5pp against a 95% CI of 4.74pp: not statistically resolvable.

Token-level KL, teacher-forced over 8,044 positions on identical token sequences:

median KL	0.001467
mean KL	0.069232
p99 KL	1.285925
top-1 agreement	0.9432

Not yet validated

• CUDA graphs measured (+7.6%, 25.5 tok/s) but not quality-checked separately.
• MTP speculative decoding: untested on this build.
• Contexts beyond 131,072 (the config allows 262,144).
• The INT3 PLE choice itself was never validated against an INT4 near-reference.

Serving flags that matter: VLLM_KV_CACHE_LAYOUT=BLHNC (hybrid page sizes), --language-model-only, --enable-prefix-caching. --kv-cache-dtype fp8 is rejected: Qwen4Exp QSA requires a BF16 main KV cache.