Qwen3-4B-NeuralRouter-ddt-e3-b32

A WhiFlash-Neural token-level router for Qwen/Qwen3-4B (paper · code).

This is not a language model. At each decoding step it maps the target's final hidden state to the predicted acceptance-length gain (ΔAL) of switching drafters; a threshold turns that scalar into the routing decision, so the operating point is tunable per hardware without retraining.

Model Details

Target model	Qwen/Qwen3-4B
Router	2-layer MLP (hidden state → GELU → scalar ΔAL)
Drafter pair (--whiflash-drafters)	ddt-e3 — DDTree (diffusion, tree-verified) vs EAGLE-3 (autoregressive)
DDTree budget (--ddtree-tree-budget)	32
Training data	Dolci Instruct SFT — task-agnostic, since the router conditions only on the hidden state
Objective	MSE on ΔAL = AL_pos − AL_neg

Installation

git clone https://github.com/SamsungLabs/WhiFlash.git
cd WhiFlash
uv sync

Quick Usage

uv run benchmark.py \
    --speculative-algorithm WHIFLASH \
    --model-name-or-path Qwen/Qwen3-4B \
    --draft-name-or-path z-lab/Qwen3-4B-DFlash-b16 \
    --whiflash-eagle-draft WhiFlash/Qwen3-4B-EAGLE3 \
    --whiflash-drafters ddt-e3 \
    --ddtree-tree-budget 32 \
    --whiflash-router neural \
    --whiflash-neural-router-path WhiFlash/Qwen3-4B-NeuralRouter-ddt-e3-b32 \
    --whiflash-neural-router-threshold 0.5 \
    --dataset mt-bench --max-samples 10 --max-new-tokens 2048

--whiflash-neural-router-path also accepts a local directory or an exported .safetensors file, so this repo id can be swapped for a router you trained yourself.

This router was trained for one drafter pair and budget, both recorded in config.json. In the repo id, df = DFlash, ddt = DDTree, e3 = EAGLE-3 and bN = DDTree tree budget N.

Evaluation

Per-category throughput gain of WhiFlash-Neural over the standalone drafters it chooses between, on an NVIDIA H100 with greedy decoding and 2048 max new tokens:

Category	vs. DDTree (b32)
Chat	+9.8%
Math	+4.8%
Coding	+2.3%
Agentic	+1.4%

Categories aggregate Chat (MT-Bench, Alpaca, UltraChat, WildChat), Math (GSM8K, MATH-500, AIME'24/'25), Coding (HumanEval, MBPP, LiveCodeBench) and Agentic (SWE-bench, τ-bench, AgentBench, AgentInstruct). DDTree postdates the paper, so these come from a separate run and are not comparable cell-to-cell with the paper's tables; each row compares this router against standalone DDTree at the same tree budget (--speculative-algorithm DDTREE --ddtree-tree-budget 32).

Citation

@misc{kwon2026whiflash,
    title={{WhiFlash: Accelerating Speculative Decoding with Token-Level Cross-Paradigm Routing}},
    author={Young D. Kwon and Miles Williams and Rui Li and Alexandros Kouris and Stylianos I. Venieris},
    year={2026}, eprint={2606.07710}, archivePrefix={arXiv}, primaryClass={cs.LG},
    url={https://arxiv.org/abs/2606.07710},
}

The Qwen3 EAGLE-3 drafters used with this router were trained with SpecVocab:

@inproceedings{williams-etal-2026-speculative,
    title = "Speculative Decoding with a Speculative Vocabulary",
    author = "Williams, Miles and Kwon, Young D. and Li, Rui and Kouris, Alexandros and Venieris, Stylianos I.",
    editor = "Liakata, Maria and Moreira, Viviane P. and Zhang, Jiajun and Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.findings-acl.2000/",
    doi = "10.18653/v1/2026.findings-acl.2000",
    pages = "40240--40254",
    ISBN = "979-8-89176-395-1"
}