|
Download UPSTREAM_README.md from isilicon/mango-dflash2: direct link, hf CLI and curl.
- Browser
- Download file 5.4 kB
-
https://huggingface.co/isilicon/mango-dflash2/resolve/main/UPSTREAM_README.md
- Command line
-
hf download hf://isilicon/mango-dflash2/UPSTREAM_README.md
-
curl -L -o UPSTREAM_README.md https://huggingface.co/isilicon/mango-dflash2/resolve/main/UPSTREAM_README.md
5.4 kB
| license: apache-2.0 | |
| library_name: transformers | |
| pipeline_tag: text-generation | |
| base_model: | |
| - Qwen/Qwen3.8-27B | |
| inference: false | |
| tags: | |
| - dflash2 | |
| - speculative-decoding | |
| - block-diffusion | |
| - draft-model | |
| - sglang | |
| - vllm | |
| # Qwen3.8-27B-DFlash2 | |
| [Blog](https://inco.ai/blog/dflash2/) | [GitHub](https://github.com/z-lab/dflash) | |
| This repository contains the DFlash 2 draft model for | |
| [`Qwen/Qwen3.8-27B`](https://huggingface.co/Qwen/Qwen3.8-27B). | |
| It is not a standalone language model: it runs inside a speculative | |
| decoding server and drafts tokens for the target model to verify. This repository is a mirror | |
| of [`incoai/Qwen3.8-27B-DFlash2`](https://huggingface.co/incoai/Qwen3.8-27B-DFlash2). | |
| DFlash 2 is a block-diffusion drafter for speculative decoding. It predicts | |
| a whole block of tokens in a single pass and keeps the top candidates at | |
| every position. A lightweight selector then traces one coherent path through them. | |
| Two-tap dynamic convolutions in the backbone keep the draft from decaying | |
| toward the end of the block. Decoding is lossless: greedy output | |
| matches the target model exactly, and sampling preserves its distribution. | |
| <div align="center"> | |
| <img src="assets/dflash2-figure.png" alt="DFlash 2: parallel block drafting with a candidate path selector" width="100%"> | |
| </div> | |
| ## Quick Start | |
| Serve with [SGLang](https://github.com/sgl-project/sglang): | |
| ```bash | |
| pip install "sglang[all] @ git+https://github.com/sgl-project/sglang.git#subdirectory=python" | |
| python -m sglang.launch_server \ | |
| --model-path Qwen/Qwen3.8-27B \ | |
| --speculative-algorithm DFLASH \ | |
| --speculative-draft-model-path incoai/Qwen3.8-27B-DFlash2 \ | |
| --speculative-num-draft-tokens 8 | |
| ``` | |
| Or with [vLLM](https://github.com/vllm-project/vllm): | |
| ```bash | |
| pip install -U "vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/52816/head" | |
| vllm serve Qwen/Qwen3.8-27B \ | |
| --speculative-config '{ | |
| "method": "dflash", | |
| "model": "incoai/Qwen3.8-27B-DFlash2", | |
| "num_speculative_tokens": 7 | |
| }' | |
| ``` | |
| See the [blog post](https://inco.ai/blog/dflash2/) for other engines and more details. | |
| ## Evaluation | |
| - Runtime: SGLang on one NVIDIA H200, with FlashAttention 3 for target and draft attention | |
| - Speculation block size: 8 (7 draft tokens per verification step) | |
| - Sampling: Qwen3.8's officially recommended parameters (temperature 1.0, top-p 0.95, top-k 20), with `xhigh` reasoning effort | |
| - Maximum new tokens: 4096 | |
| - Prompts: benchmark formatting from [`z-lab/dflash`](https://github.com/z-lab/dflash) | |
| We compare autoregressive decoding, Qwen3.8's built-in seven-token MTP, | |
| a community DSpark drafter | |
| ([`RadixArk/Qwen3.8-27B-DSpark`](https://huggingface.co/RadixArk/Qwen3.8-27B-DSpark)), | |
| and DFlash 2. All speculative methods propose seven draft tokens per | |
| verification step. | |
| ### Acceptance Length | |
| Acceptance length is the per-request mean of completion tokens divided by verification steps. | |
| Higher is better. | |
| | Task | MTP | DSpark | DFlash 2 | | |
| | :--- | ---: | ---: | ---: | | |
| | GSM8K | 5.02 | 4.36 | **5.46** | | |
| | MATH-500 | 4.72 | 3.92 | **5.28** | | |
| | HumanEval | 3.91 | 3.30 | **4.39** | | |
| | MBPP | 3.99 | 3.51 | **4.79** | | |
| | MT-Bench | 3.74 | 3.01 | **4.10** | | |
| ### Throughput | |
| Throughput is total output tokens divided by end-to-end wall time. | |
| Each cell shows `output tok/s (speedup vs. autoregressive)`. | |
| #### Concurrency 1 | |
| | Task | Autoregressive | MTP | DSpark | DFlash 2 | | |
| | :--- | ---: | ---: | ---: | ---: | | |
| | GSM8K | 68.9 | 178.5 (2.59×) | 185.3 (2.69×) | **236.1 (3.43×)** | | |
| | MATH-500 | 69.0 | 172.8 (2.51×) | 174.5 (2.53×) | **230.7 (3.34×)** | | |
| | HumanEval | 69.0 | 151.9 (2.20×) | 159.9 (2.32×) | **214.6 (3.11×)** | | |
| | MBPP | 69.0 | 153.1 (2.22×) | 163.3 (2.37×) | **226.9 (3.29×)** | | |
| | MT-Bench | 68.9 | 134.9 (1.96×) | 137.6 (2.00×) | **184.0 (2.67×)** | | |
| #### Concurrency 8 | |
| | Task | Autoregressive | MTP | DSpark | DFlash 2 | | |
| | :--- | ---: | ---: | ---: | ---: | | |
| | GSM8K | 467.2 | 1,022.1 (2.19×) | 1,040.8 (2.23×) | **1,328.7 (2.84×)** | | |
| | MATH-500 | 480.0 | 1,023.5 (2.13×) | 1,025.8 (2.14×) | **1,368.3 (2.85×)** | | |
| | HumanEval | 483.4 | 934.2 (1.93×) | 956.5 (1.98×) | **1,291.5 (2.67×)** | | |
| | MBPP | 478.0 | 938.1 (1.96×) | 974.1 (2.04×) | **1,328.0 (2.78×)** | | |
| | MT-Bench | 480.5 | 835.2 (1.74×) | 802.3 (1.67×) | **1,090.2 (2.27×)** | | |
| #### Concurrency 32 | |
| | Task | Autoregressive | MTP | DSpark | DFlash 2 | | |
| | :--- | ---: | ---: | ---: | ---: | | |
| | GSM8K | 1,329.8 | 1,381.1 (1.04×) | 1,506.5 (1.13×) | **1,922.5 (1.45×)** | | |
| | MATH-500 | 1,505.8 | 1,415.6 (0.94×) | 1,429.0 (0.95×) | **1,951.8 (1.30×)** | | |
| | HumanEval | 1,546.5 | 1,296.8 (0.84×) | 1,330.1 (0.86×) | **1,799.0 (1.16×)** | | |
| | MBPP | 1,507.7 | 1,314.9 (0.87×) | 1,361.3 (0.90×) | **1,886.8 (1.25×)** | | |
| | MT-Bench | 1,507.4 | 1,159.7 (0.77×) | 1,115.5 (0.74×) | **1,525.3 (1.01×)** | | |
| ## Citation | |
| If you find DFlash 2 useful, please cite: | |
| ```bibtex | |
| @misc{inco2026dflash2, | |
| title = {{DFlash 2: Keep Drafting Parallel}}, | |
| author = {{Inco AI}}, | |
| year = {2026}, | |
| month = {August}, | |
| url = {https://inco.ai/blog/dflash2/} | |
| } | |
| ``` | |
| Please also cite the original DFlash paper: | |
| ```bibtex | |
| @inproceedings{chen2026dflash, | |
| title = {{DFlash: Block Diffusion for Flash Speculative Decoding}}, | |
| author = {Chen, Jian and Liang, Yesheng and Liu, Zhijian}, | |
| booktitle = {International Conference on Machine Learning (ICML)}, | |
| year = {2026} | |
| } | |
| ``` | |