README.md

metadata

license: apache-2.0
base_model:
  - Qwen/Qwen3.5-9B
inference: false
tags:
  - mmflash
  - dflash
  - speculative-decoding
  - speculative-decoding-draft
  - block-diffusion
  - draft-model
  - multimodal
  - sglang
  - qwen
  - qwen3.5

Qwen3.5-9B-MMFlash

MMFlash draft model for speculative decoding of Qwen/Qwen3.5-9B on image and text inputs. It is not a standalone language model; it is paired with the target model in a speculative-decoding server (SGLang DFLASH).

MMFlash builds on DFlash block-diffusion drafting: the draft reads hidden states from five layers of the target and proposes a block of 16 tokens in one forward pass, and the target verifies the block in one forward pass. MMFlash keeps the DFlash architecture and is trained on multimodal data with a loss weighted by per-token visual-grounding (visual-KL) scores.

_{Qwen3.5-9B on a MathVista question, recorded in real time on one NVIDIA H200 with SGLang (greedy decoding).}

Need a drafter for another model? Email us at fscnkucs@gmail.com and we can help train one. We will keep updating these releases, and a Qwen4 checkpoint is planned.

Model details

Target model	Qwen/Qwen3.5-9B
Parameters	1.05B (bf16)
Block size	16
Target hidden states	layers 1, 8, 15, 22, 29 (of 32)

Usage

SGLang

SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 python -m sglang.launch_server \
  --model-path Qwen/Qwen3.5-9B \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path FSCCS/Qwen3.5-9B-MMFlash \
  --speculative-dflash-block-size 16 \
  --attention-backend fa3 \
  --mm-attention-backend sdpa \
  --disable-radix-cache \
  --reasoning-parser qwen3 \
  --trust-remote-code

vLLM

vllm serve Qwen/Qwen3.5-9B \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --attention-backend FLASH_ATTN \
  --reasoning-parser qwen3 \
  --speculative-config '{"method": "dflash", "model": "FSCCS/Qwen3.5-9B-MMFlash", "num_speculative_tokens": 15}'

Tested with vLLM 0.32.0, which serves DFlash drafts natively. num_speculative_tokens is the block size minus one.

Results

Each cell is τ / speedup: τ is the average acceptance length (tokens accepted per verification step), and the speedup is the decoding throughput over Qwen3.5-9B without speculative decoding.

Temperature 0

Method	ChartQA	CharXiv	MMStar	MMBench	DynaMath	MathVista	MathVerse	Avg.
MTP (3)	3.32 / 1.83×	3.28 / 1.87×	3.01 / 1.73×	2.88 / 1.57×	3.38 / 1.80×	3.26 / 1.80×	3.24 / 1.80×	3.20 / 1.77×
MTP (7)	4.66 / 1.88×	4.60 / 1.94×	3.91 / 1.66×	3.55 / 1.41×	4.74 / 1.77×	4.52 / 1.78×	4.43 / 1.77×	4.34 / 1.74×
MTP (15)	5.27 / 1.38×	5.27 / 1.47×	4.20 / 1.18×	3.75 / 0.99×	5.25 / 1.30×	5.08 / 1.32×	4.90 / 1.30×	4.81 / 1.27×
EAGLE-3 (16)	3.93 / 1.81×	3.85 / 1.93×	3.06 / 1.55×	2.72 / 1.32×	3.60 / 1.76×	3.50 / 1.63×	3.48 / 1.75×	3.45 / 1.68×
DFlash-z (16)	5.22 / 3.03×	5.27 / 3.15×	3.93 / 2.46×	3.67 / 2.11×	4.96 / 2.75×	4.77 / 2.75×	4.52 / 2.67×	4.62 / 2.70×
DFlash (16)	5.59 / 3.20×	5.17 / 3.09×	4.15 / 2.46×	3.79 / 2.20×	5.62 / 2.71×	5.38 / 2.83×	4.92 / 2.79×	4.95 / 2.75×
MMFlash (16)	6.19 / 3.57×	5.67 / 3.42×	4.58 / 2.67×	4.20 / 2.39×	6.44 / 3.02×	6.49 / 3.17×	5.60 / 3.10×	5.60 / 3.05×

Values in parentheses are the number of draft steps for MTP, the draft tree size for EAGLE-3, and the block size for DFlash and MMFlash. MTP is Qwen3.5-9B's built-in multi-token-prediction head. DFlash-z is the public z-lab/Qwen3.5-9B-DFlash draft. DFlash is our DFlash draft trained on the same data without the visual-KL weighting.

Concurrency (temperature 0)

MMFlash at 1, 8 and 32 requests in flight, on a single NVIDIA H200. Each cell is the decode throughput in tokens/s, with the speedup over Qwen3.5-9B alone at the same concurrency in parentheses.

Concurrency	ChartQA	CharXiv	MMStar	MMBench	DynaMath	MathVista	MathVerse	Avg.
1	620 (3.57×)	582 (3.42×)	459 (2.67×)	421 (2.39×)	519 (3.02×)	544 (3.17×)	530 (3.10×)	525 (3.05×)
8	2,438 (2.33×)	2,534 (2.29×)	1,964 (1.85×)	1,869 (1.69×)	2,541 (2.20×)	1,675 (1.84×)	2,435 (2.13×)	2,208 (2.05×)
32	3,685 (1.89×)	3,771 (1.53×)	2,814 (1.14×)	2,599 (1.07×)	3,792 (1.42×)	2,170 (1.28×)	3,326 (1.16×)	3,165 (1.36×)

Citation

Coming soon.