Qwen3.5-9B-MMFlash / README.md
FSCCS's picture nielsr's picture
nielsr HF Staff
Add pipeline tag and links to paper, project page and code (#1)
d41c280
|
Raw History Blame Contribute Delete
9.83 kB
---
base_model:
- Qwen/Qwen3.5-9B
license: apache-2.0
pipeline_tag: image-text-to-text
tags:
- mmflash
- dflash
- speculative-decoding
- speculative-decoding-draft
- block-diffusion
- draft-model
- multimodal
- sglang
- qwen
- qwen3.5
inference: false
---
<div align="center">
<svg xmlns="http://www.w3.org/2000/svg" width="440" height="99" viewBox="0 0 532 120" role="img" aria-label="MMFlash"><defs><linearGradient id="mmflash-logo-mm" x1="142.0" y1="0" x2="298.3" y2="0" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="#6366F1"/><stop offset="1" stop-color="#8B5CF6"/></linearGradient></defs><g transform="translate(4 4) scale(0.875000)"><defs><linearGradient id="mmflash-logo-bg" x1="0" y1="0" x2="128" y2="128" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="#4F46E5"/><stop offset="1" stop-color="#7C3AED"/></linearGradient><linearGradient id="mmflash-logo-bolt" x1="64" y1="25.0" x2="64" y2="103.0" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="#FDE047"/><stop offset="0.5" stop-color="#FBBF24"/><stop offset="1" stop-color="#F97316"/></linearGradient></defs><path d="M40 0 H88 C120.8 0 128 7.2 128 40 V88 C128 120.8 120.8 128 88 128 H40 C7.2 128 0 120.8 0 88 V40 C0 7.2 7.2 0 40 0 Z" fill="url(#mmflash-logo-bg)"/><g fill="none" stroke="#FFFFFF" stroke-opacity="0.94" stroke-width="6.5" stroke-linecap="round" stroke-linejoin="round"><path d="M22 42 V30 Q22 22 30 22 H42"/><path d="M86 22 H98 Q106 22 106 30 V42"/><path d="M106 86 V98 Q106 106 98 106 H86"/><path d="M42 106 H30 Q22 106 22 98 V86"/></g><path d="M77.06 28.50 L41.31 74.00 L62.44 74.00 L54.31 106.50 L91.69 57.75 L70.56 57.75 Z" fill="#1E1B4B" fill-opacity="0.32" stroke="#1E1B4B" stroke-opacity="0.32" stroke-width="3" stroke-linejoin="round"/><path d="M74.56 25.00 L38.81 70.50 L59.94 70.50 L51.81 103.00 L89.19 54.25 L68.06 54.25 Z" fill="url(#mmflash-logo-bolt)" stroke="url(#mmflash-logo-bolt)" stroke-width="3" stroke-linejoin="round"/></g><path fill="url(#mmflash-logo-mm)" d="M142 90.62V27.84H164.19L177.09 60.95H178.81L191.62 27.84H214.24V90.62H198.42V37.04L200.91 37.21L184.74 77.97H170.21L153.95 37.21L156.62 37.04V90.62H142ZM226.11 90.62V27.84H248.3L261.2 60.95H262.92L275.73 27.84H298.35V90.62H282.52V37.04L285.02 37.21L268.85 77.97H254.32L238.06 37.21L240.73 37.04V90.62H226.11Z"/><path fill="currentColor" d="M310.22 90.62V27.84H326.21V90.62H310.22ZM324.49 66.28V53.21H347.8V66.28H324.49ZM324.49 40.91V27.84H348.74V40.91H324.49ZM358.03 90.62V27.84H373.68V90.62H358.03ZM352.7 39.19V27.84H373.68V39.19H352.7ZM413.07 90.62V76.68H410.49V61.81Q410.49 58.54 408.99 56.95Q407.48 55.36 404.04 55.36Q402.41 55.36 399.23 55.44Q396.04 55.53 392.52 55.7Q388.99 55.87 386.07 56.04V42.97Q388.13 42.8 390.97 42.63Q393.81 42.46 396.86 42.37Q399.91 42.28 402.49 42.28Q409.8 42.28 414.92 44.43Q420.04 46.58 422.75 50.88Q425.46 55.18 425.46 61.81V90.62H413.07ZM398.02 91.82Q393.03 91.82 389.16 90.01Q385.29 88.21 383.1 84.81Q380.91 81.41 380.91 76.68Q380.91 71.35 383.75 68.04Q386.58 64.73 391.57 63.18Q396.56 61.63 403.01 61.63H412.21V69.72H403.01Q399.83 69.72 398.11 71.31Q396.39 72.9 396.39 75.57Q396.39 78.15 398.11 79.74Q399.83 81.33 403.01 81.33Q405.07 81.33 406.66 80.6Q408.26 79.87 409.29 78.06Q410.32 76.25 410.49 73.07L414.1 76.51Q413.67 81.41 411.74 84.81Q409.8 88.21 406.41 90.01Q403.01 91.82 398.02 91.82ZM453.92 92.16Q443.6 92.16 437.67 87.91Q431.73 83.65 431.3 75.91H445.06Q445.41 78.06 447.73 79.61Q450.05 81.16 454.27 81.16Q457.28 81.16 459.47 80.25Q461.66 79.35 461.66 77.29Q461.66 75.48 459.94 74.32Q458.22 73.16 453.66 72.73L450.4 72.38Q441.28 71.44 436.98 67.4Q432.68 63.35 432.68 57.25Q432.68 52 435.26 48.61Q437.84 45.21 442.4 43.49Q446.96 41.77 452.8 41.77Q462.35 41.77 468.2 45.94Q474.05 50.11 474.39 57.94H460.63Q460.29 55.79 458.31 54.28Q456.33 52.78 452.63 52.78Q449.97 52.78 448.29 53.72Q446.61 54.67 446.61 56.39Q446.61 58.19 448.2 59.05Q449.79 59.91 453.32 60.34L456.59 60.69Q465.7 61.63 470.65 65.72Q475.59 69.8 475.59 76.43Q475.59 81.33 472.97 84.85Q470.35 88.38 465.49 90.27Q460.63 92.16 453.92 92.16ZM482.56 90.62V27.84H498.21V63.61H496.15Q496.15 56.39 497.95 51.49Q499.76 46.58 503.24 44.09Q506.73 41.6 511.89 41.6H512.57Q520.31 41.6 524.4 47.19Q528.48 52.78 528.48 63.61V90.62H512.83V63.01Q512.83 59.74 510.85 57.64Q508.88 55.53 505.61 55.53Q502.34 55.53 500.28 57.64Q498.21 59.74 498.21 63.18V90.62H482.56Z"/></svg>
<br>
</div>
# Qwen3.5-9B-MMFlash
MMFlash draft model for speculative decoding of [`Qwen/Qwen3.5-9B`](https://huggingface.co/Qwen/Qwen3.5-9B) on image and text inputs. It is not a standalone language model; it is paired with the target model in a speculative-decoding server (SGLang `DFLASH`).
MMFlash builds on [DFlash](https://github.com/z-lab/dflash) block-diffusion drafting: the draft reads hidden states from five layers of the target and proposes a block of 16 tokens in one forward pass, and the target verifies the block in one forward pass. MMFlash keeps the DFlash architecture and is trained on multimodal data with a loss weighted by per-token visual-grounding (visual-KL) scores.
This checkpoint is presented in the paper [MMFlash: Faster Speculative Decoding for Vision-Language Models](https://huggingface.co/papers/2611.18482). Check out the [project page](https://fscdc.github.io/MMFlash/) and the [GitHub repository](https://github.com/fscdc/MMFlash) for code, training scripts and evaluation details.
<div align="center">
<video autoplay muted loop playsinline controls width="100%">
<source src="https://huggingface.co/FSCCS/Qwen3.5-9B-MMFlash/resolve/main/assets/demo.mp4" type="video/mp4">
</video>
</div>
<p align="center"><sub>Qwen3.5-9B on a MathVista question, recorded in real time on one NVIDIA H200 with SGLang (greedy decoding).</sub></p>
> [!TIP]
> Need a drafter for another model? Email us at [fscnkucs@gmail.com](mailto:fscnkucs@gmail.com) and we can help train one. We will keep updating these releases, and a Qwen4 checkpoint is planned.
## Model details
| | |
|---|---|
| Target model | [`Qwen/Qwen3.5-9B`](https://huggingface.co/Qwen/Qwen3.5-9B) |
| Parameters | 1.05B (bf16) |
| Block size | 16 |
| Target hidden states | layers 1, 8, 15, 22, 29 (of 32) |
## Usage
### SGLang
```bash
SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 python -m sglang.launch_server \
--model-path Qwen/Qwen3.5-9B \
--speculative-algorithm DFLASH \
--speculative-draft-model-path FSCCS/Qwen3.5-9B-MMFlash \
--speculative-dflash-block-size 16 \
--attention-backend fa3 \
--mm-attention-backend sdpa \
--disable-radix-cache \
--reasoning-parser qwen3 \
--trust-remote-code
```
### vLLM
```bash
vllm serve Qwen/Qwen3.5-9B \
--dtype bfloat16 \
--max-model-len 32768 \
--attention-backend FLASH_ATTN \
--reasoning-parser qwen3 \
--speculative-config '{"method": "dflash", "model": "FSCCS/Qwen3.5-9B-MMFlash", "num_speculative_tokens": 15}'
```
Tested with vLLM 0.32.0, which serves DFlash drafts natively. `num_speculative_tokens` is the block size minus one.
## Results
Each cell is **τ / speedup**: τ is the average acceptance length (tokens accepted per verification step), and the speedup is the decoding throughput over Qwen3.5-9B without speculative decoding.
### Temperature 0
| Method | ChartQA | CharXiv | MMStar | MMBench | DynaMath | MathVista | MathVerse | Avg. |
|---|---|---|---|---|---|---|---|---|
| MTP (3) | 3.32 / 1.83× | 3.28 / 1.87× | 3.01 / 1.73× | 2.88 / 1.57× | 3.38 / 1.80× | 3.26 / 1.80× | 3.24 / 1.80× | 3.20 / 1.77× |
| MTP (7) | 4.66 / 1.88× | 4.60 / 1.94× | 3.91 / 1.66× | 3.55 / 1.41× | 4.74 / 1.77× | 4.52 / 1.78× | 4.43 / 1.77× | 4.34 / 1.74× |
| MTP (15) | 5.27 / 1.38× | 5.27 / 1.47× | 4.20 / 1.18× | 3.75 / 0.99× | 5.25 / 1.30× | 5.08 / 1.32× | 4.90 / 1.30× | 4.81 / 1.27× |
| EAGLE-3 (16) | 3.93 / 1.81× | 3.85 / 1.93× | 3.06 / 1.55× | 2.72 / 1.32× | 3.60 / 1.76× | 3.50 / 1.63× | 3.48 / 1.75× | 3.45 / 1.68× |
| DFlash-z (16) | 5.22 / 3.03× | 5.27 / 3.15× | 3.93 / 2.46× | 3.67 / 2.11× | 4.96 / 2.75× | 4.77 / 2.75× | 4.52 / 2.67× | 4.62 / 2.70× |
| DFlash (16) | 5.59 / 3.20× | 5.17 / 3.09× | 4.15 / 2.46× | 3.79 / 2.20× | 5.62 / 2.71× | 5.38 / 2.83× | 4.92 / 2.79× | 4.95 / 2.75× |
| **MMFlash (16)** | **6.19 / 3.57×** | **5.67 / 3.42×** | **4.58 / 2.67×** | **4.20 / 2.39×** | **6.44 / 3.02×** | **6.49 / 3.17×** | **5.60 / 3.10×** | **5.60 / 3.05×** |
Values in parentheses are the number of draft steps for MTP, the draft tree size for EAGLE-3, and the block size for DFlash and MMFlash. MTP is Qwen3.5-9B's built-in multi-token-prediction head. DFlash-z is the public [`z-lab/Qwen3.5-9B-DFlash`](https://huggingface.co/z-lab/Qwen3.5-9B-DFlash) draft. DFlash is our DFlash draft trained on the same data without the visual-KL weighting.
### Concurrency (temperature 0)
MMFlash at 1, 8 and 32 requests in flight, on a single NVIDIA H200. Each cell is the decode throughput in tokens/s, with the speedup over Qwen3.5-9B alone at the same concurrency in parentheses.
| Concurrency | ChartQA | CharXiv | MMStar | MMBench | DynaMath | MathVista | MathVerse | Avg. |
|---|---|---|---|---|---|---|---|---|
| 1 | 620 (3.57×) | 582 (3.42×) | 459 (2.67×) | 421 (2.39×) | 519 (3.02×) | 544 (3.17×) | 530 (3.10×) | 525 (3.05×) |
| 8 | 2,438 (2.33×) | 2,534 (2.29×) | 1,964 (1.85×) | 1,869 (1.69×) | 2,541 (2.20×) | 1,675 (1.84×) | 2,435 (2.13×) | 2,208 (2.05×) |
| 32 | 3,685 (1.89×) | 3,771 (1.53×) | 2,814 (1.14×) | 2,599 (1.07×) | 3,792 (1.42×) | 2,170 (1.28×) | 3,326 (1.16×) | 3,165 (1.36×) |
## Citation
```bibtex
@article{feng2026mmflash,
title = {MMFlash: Faster Speculative Decoding for Vision-Language Models},
author = {Feng, Sicheng and Chen, Zigeng and Wang, Song and Fang, Gongfan and Wang, Xinchao},
journal = {arXiv preprint arXiv:2611.18482},
year = {2026}
}
```