Gemma4-E4B-it-MMFlash

MMFlash draft model for speculative decoding of google/gemma-4-E4B-it on image and text inputs. It is not a standalone language model; it is paired with the target model in a speculative-decoding server (SGLang DFLASH).

MMFlash builds on DFlash block-diffusion drafting: the draft reads hidden states from five layers of the target and proposes a block of 16 tokens in one forward pass, and the target verifies the block in one forward pass. MMFlash keeps the DFlash architecture and is trained on multimodal data with a loss weighted by per-token visual-grounding (visual-KL) scores.

_{Qwen3.5-9B on a MathVista question, recorded in real time on one NVIDIA H200 with SGLang (greedy decoding).}

Need a drafter for another model? Email us at fscnkucs@gmail.com and we can help train one. We will keep updating these releases, and a Qwen4 checkpoint is planned.

Model details

Target model	google/gemma-4-E4B-it
Parameters	537M (bf16)
Block size	16
Target hidden states	layers 1, 10, 20, 30, 39 (of 42)

Usage

SGLang

SGLANG_GEMMA_OUT_OF_PLACE_POSITION_MUTATION=1 \
SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 \
python -m sglang.launch_server \
  --model-path google/gemma-4-E4B-it \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path FSCCS/Gemma4-E4B-it-MMFlash \
  --speculative-dflash-block-size 16 \
  --attention-backend triton \
  --mm-attention-backend sdpa \
  --mem-fraction-static 0.7 \
  --reasoning-parser gemma4 \
  --trust-remote-code

vLLM

vllm serve google/gemma-4-E4B-it \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --reasoning-parser gemma4 \
  --speculative-config '{"method": "dflash", "model": "FSCCS/Gemma4-E4B-it-MMFlash", "num_speculative_tokens": 15}'

Tested with vLLM 0.32.0, which serves DFlash drafts natively. num_speculative_tokens is the block size minus one. vLLM does not have the gemma4-spec changes, so τ is lower than in the results below (ChartQA: 5.71 vs 6.39).

Results

Each cell is τ / speedup: τ is the average acceptance length (tokens accepted per verification step), and the speedup is the decoding throughput over Gemma 4 E4B-it without speculative decoding.

Temperature 0

Method	ChartQA	MMStar	DynaMath
MTP (5, official)	3.81 / 1.78×	3.19 / 1.48×	4.04 / 1.89×
MTP (15)	4.73 / 1.66×	3.74 / 1.33×	5.17 / 1.75×
EAGLE-3 (16)	4.87 / 2.27×	4.04 / 1.50×	6.23 / 1.90×
MMFlash (16)	6.39 / 2.87×	4.83 / 2.10×	7.15 / 3.02×

Values in parentheses are the number of draft steps for MTP, the draft tree size for EAGLE-3, and the block size for MMFlash. MTP (5, official) is Google's released drafter, google/gemma-4-E4B-it-assistant, run for 5 steps; MTP (15) runs it for 15 steps.

Concurrency (temperature 0)

MMFlash at 1, 8 and 32 requests in flight, on a single NVIDIA H200. Each cell is the decode throughput in tokens/s, with the speedup over Gemma 4 E4B-it alone at the same concurrency in parentheses.

Concurrency	ChartQA	CharXiv	MMStar	MMBench	DynaMath	MathVista	MathVerse	Avg.
1	521 (2.87×)	448 (2.50×)	377 (2.10×)	363 (1.99×)	543 (3.02×)	454 (2.53×)	436 (2.45×)	449 (2.49×)
8	2,819 (2.42×)	2,491 (2.16×)	2,129 (1.89×)	1,986 (1.75×)	3,068 (2.60×)	2,391 (2.25×)	2,484 (2.12×)	2,481 (2.17×)
32	5,230 (1.68×)	4,957 (1.64×)	4,011 (1.49×)	3,948 (1.25×)	5,199 (1.52×)	4,332 (1.52×)	5,122 (1.46×)	4,686 (1.51×)

Citation

Coming soon.