Publish one-Spark runtime recipe and benchmark card

7179765 verified about 2 months ago

4.89 kB

---
pipeline_tag: image-text-to-text
base_model:
- nvidia/Qwen3.8-Flash-Next-NVFP4
license: other
license_name: nvidia-open-model-license
library_name: vllm
tags:
- nvidia
- dgx-spark
- gb10
- qwen3.8
- qwen4-exp
- modelopt
- nvfp4
- fp8
- speculative-decoding
- vllm
---
# Qwen3.8-Flash-Next NVFP4 + Dense FP8 — One DGX Spark

A tested one-DGX-Spark deployment of [`nvidia/Qwen3.8-Flash-Next-NVFP4`](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4) with targeted in-loader FP8 conversion of the remaining large BF16 dense projections, native MTP K=3, piecewise CUDA graphs, prefix caching, and C6 scheduling.

> **47–53 tok/s short code · 37–43 tok/s long-context/thinking · 60 tok/s structured · 969K-token BF16 KV pool · 72/72 BENCHY low-effort cases across C1–C6**

## Release status

**This page describes a runtime recipe; it does not yet contain transformed model weights.** The dense projections are converted to FP8 during model loading. A stock-vLLM safetensors export is a separate planned artifact and is not claimed here.

The reproducible runtime, overlays, benchmark evidence, and setup scripts live in the companion GitHub repository: https://github.com/gitcommit90/qwen38-flash-next-nvfp4-one-spark.

## What changed

NVIDIA's checkpoint stores the main routed experts in NVFP4 but leaves attention, Gated DeltaNet, `lm_head`, and other dense layers in BF16. Profiling on GB10 showed approximately 67% of decode kernel time in BF16 dense GEMVs.

The runtime recipe converts only the large decode-critical projections:

- `linear_attn.in_proj_qkvz`
- `linear_attn.out_proj`
- `self_attn.qkv_proj`
- `self_attn.o_proj`
- `lm_head`

Small hyper-connection, shared-expert, gate, indexer, and auxiliary matrices remain BF16 because FP8 kernel overhead made them slower on this device. The selected kernel is Marlin FP8 W8A16: FP8 weights with BF16 activations.

## Results

Measured on one NVIDIA DGX Spark / GB10, TP1, 262,144-token model context, BF16 KV, MTP K=3, C6 scheduler capacity:

| Workload | Result |
|---|---:|
| Short-code decode | **47–53 tok/s** |
| Decode at 15K context | **37.2–37.8 tok/s** |
| Thinking-on code | **39–43 tok/s** |
| Open prose | **33.7 tok/s** |
| Structured output | **60.2 tok/s** |
| Cold prefill, 8–64K | **~1,980–2,082 tok/s** |
| Warm cached 8K TTFT | **0.185–0.189 s** |
| KV capacity | **969,224 tokens / ~3.70 × 262K** |

The observed K3 agentic loop reached 74–81 decode tok/s, but shorter generated tool calls favored that figure. The robust comparison is **3.95 emitted tokens per target-model step at K3 versus 2.96 at K2 (+33%)**.

### BENCHY agent evaluation

A 24-run C1–C6 × effort matrix was executed without retries, edits, or overwritten passes: 288 total cases.

| Reasoning effort | Outcome across C1–C6 |
|---|---:|
| **Low** | **72 correct / 0 wrong / 0 unfinished** |
| Medium | 65 correct / 0 wrong / 7 unfinished |
| XHigh | 60 correct / 3 wrong / 9 unfinished |
| None | 13 correct / 59 wrong / 0 unfinished |

Recommended default for agent workloads: **`reasoning_effort: low`**.
## Quality characterization
Teacher-forced target-logit comparison against the NVIDIA checkpoint with BF16 dense layers, 30 texts / 2,019 positions:
- Top-1 agreement: **0.886**
- Mean KL divergence: **0.040**
- Margin >2 nats preserved: **1,013/1,014**
No fine-tuning was performed. Native MTP speculation verifies candidate tokens against the transformed target.
## Runtime configuration
- Source checkpoint revision tested: `fab0aecb760cec45227f6656abcaafa11abca87a`
- vLLM base: `8a728663c1c3eeace834a95f5654fa653cc1998c`
- ExLlamaV3 extension: `e648f1a131365aae15920073e761a3fa5a527654`
- Dense FP8 groups: `gdn,attn,lm_head`
- Dense kernel: Marlin FP8 W8A16
- MTP: K=3, EAGLE trailing-block drop disabled
- CUDA graphs: piecewise
- BF16 KV allocation: 28 GiB
- Prefix caching: enabled
- Async scheduling: disabled
- Maximum sequences: 6
## Intended use
Local text, coding, tool-use, and agent workloads on one DGX Spark where decode latency, usable concurrency, and large BF16 KV capacity are more important than retaining byte-identical BF16 dense weights.
## Limitations
- This page currently has no standalone dense-FP8 checkpoint; use the companion runtime recipe.
- The FP8 transformation changes target numerics relative to NVIDIA's original mixed checkpoint.
- Full 250K cache branching/retrieval qualification remains pending.
- Vision was not included in the performance/quality gate reported here.
- Results are hardware-, workload-, runtime-, and sampling-specific.
- The runtime overlays are pinned to one vLLM base and must be ported deliberately to newer versions.
## License
The runtime code is Apache-2.0. Upstream model weights remain governed by the NVIDIA Open Model License and applicable Qwen terms. This repository does not redistribute those weights.

	---
	pipeline_tag: image-text-to-text
	base_model:
	- nvidia/Qwen3.8-Flash-Next-NVFP4
	license: other
	license_name: nvidia-open-model-license
	library_name: vllm
	tags:
	- nvidia
	- dgx-spark
	- gb10
	- qwen3.8
	- qwen4-exp
	- modelopt
	- nvfp4
	- fp8
	- speculative-decoding
	- vllm
	---

	# Qwen3.8-Flash-Next NVFP4 + Dense FP8 — One DGX Spark

	A tested one-DGX-Spark deployment of [`nvidia/Qwen3.8-Flash-Next-NVFP4`](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4) with targeted in-loader FP8 conversion of the remaining large BF16 dense projections, native MTP K=3, piecewise CUDA graphs, prefix caching, and C6 scheduling.

	> 47–53 tok/s short code · 37–43 tok/s long-context/thinking · 60 tok/s structured · 969K-token BF16 KV pool · 72/72 BENCHY low-effort cases across C1–C6

	## Release status

	This page describes a runtime recipe; it does not yet contain transformed model weights. The dense projections are converted to FP8 during model loading. A stock-vLLM safetensors export is a separate planned artifact and is not claimed here.

	The reproducible runtime, overlays, benchmark evidence, and setup scripts live in the companion GitHub repository: https://github.com/gitcommit90/qwen38-flash-next-nvfp4-one-spark.

	## What changed

	NVIDIA's checkpoint stores the main routed experts in NVFP4 but leaves attention, Gated DeltaNet, `lm_head`, and other dense layers in BF16. Profiling on GB10 showed approximately 67% of decode kernel time in BF16 dense GEMVs.

	The runtime recipe converts only the large decode-critical projections:

	- `linear_attn.in_proj_qkvz`
	- `linear_attn.out_proj`
	- `self_attn.qkv_proj`
	- `self_attn.o_proj`
	- `lm_head`

	Small hyper-connection, shared-expert, gate, indexer, and auxiliary matrices remain BF16 because FP8 kernel overhead made them slower on this device. The selected kernel is Marlin FP8 W8A16: FP8 weights with BF16 activations.

	## Results

	Measured on one NVIDIA DGX Spark / GB10, TP1, 262,144-token model context, BF16 KV, MTP K=3, C6 scheduler capacity:

	\| Workload \| Result \|
	\|---\|---:\|
	\| Short-code decode \| 47–53 tok/s \|
	\| Decode at 15K context \| 37.2–37.8 tok/s \|
	\| Thinking-on code \| 39–43 tok/s \|
	\| Open prose \| 33.7 tok/s \|
	\| Structured output \| 60.2 tok/s \|
	\| Cold prefill, 8–64K \| ~1,980–2,082 tok/s \|
	\| Warm cached 8K TTFT \| 0.185–0.189 s \|
	\| KV capacity \| 969,224 tokens / ~3.70 × 262K \|

	The observed K3 agentic loop reached 74–81 decode tok/s, but shorter generated tool calls favored that figure. The robust comparison is 3.95 emitted tokens per target-model step at K3 versus 2.96 at K2 (+33%).

	### BENCHY agent evaluation

	A 24-run C1–C6 × effort matrix was executed without retries, edits, or overwritten passes: 288 total cases.

	\| Reasoning effort \| Outcome across C1–C6 \|
	\|---\|---:\|
	\| Low \| 72 correct / 0 wrong / 0 unfinished \|
	\| Medium \| 65 correct / 0 wrong / 7 unfinished \|
	\| XHigh \| 60 correct / 3 wrong / 9 unfinished \|
	\| None \| 13 correct / 59 wrong / 0 unfinished \|

	Recommended default for agent workloads: `reasoning_effort: low`.

	## Quality characterization

	Teacher-forced target-logit comparison against the NVIDIA checkpoint with BF16 dense layers, 30 texts / 2,019 positions:

	- Top-1 agreement: 0.886
	- Mean KL divergence: 0.040
	- Margin >2 nats preserved: 1,013/1,014

	No fine-tuning was performed. Native MTP speculation verifies candidate tokens against the transformed target.

	## Runtime configuration

	- Source checkpoint revision tested: `fab0aecb760cec45227f6656abcaafa11abca87a`
	- vLLM base: `8a728663c1c3eeace834a95f5654fa653cc1998c`
	- ExLlamaV3 extension: `e648f1a131365aae15920073e761a3fa5a527654`
	- Dense FP8 groups: `gdn,attn,lm_head`
	- Dense kernel: Marlin FP8 W8A16
	- MTP: K=3, EAGLE trailing-block drop disabled
	- CUDA graphs: piecewise
	- BF16 KV allocation: 28 GiB
	- Prefix caching: enabled
	- Async scheduling: disabled
	- Maximum sequences: 6

	## Intended use

	Local text, coding, tool-use, and agent workloads on one DGX Spark where decode latency, usable concurrency, and large BF16 KV capacity are more important than retaining byte-identical BF16 dense weights.

	## Limitations

	- This page currently has no standalone dense-FP8 checkpoint; use the companion runtime recipe.
	- The FP8 transformation changes target numerics relative to NVIDIA's original mixed checkpoint.
	- Full 250K cache branching/retrieval qualification remains pending.
	- Vision was not included in the performance/quality gate reported here.
	- Results are hardware-, workload-, runtime-, and sampling-specific.
	- The runtime overlays are pinned to one vLLM base and must be ported deliberately to newer versions.

	## License

	The runtime code is Apache-2.0. Upstream model weights remain governed by the NVIDIA Open Model License and applicable Qwen terms. This repository does not redistribute those weights.