wilsonzheng0327/Fun-ASR-Nano-dflash

Fun-ASR-Nano DFlash draft

A DFlash draft for greedy speculative decoding with Fun-ASR-Nano (FunAudioLLM/Fun-ASR-Nano-2512-hf), trained on English, Chinese, Japanese and Cantonese. The checkpoint layout and config fields are speculators DFlash, as in mattymchen/Qwen3-ASR-1.7B-speculator.dflash: fc, hidden_norm, 5 Qwen3 layers, norm, a reduced-vocabulary lm_head (frozen rows of the target's tied embedding) with d2t / t2d. No other draft for this model was published when this was trained, so there is no reference row below.

setting	value
draft layers	5 (hidden 1024, intermediate 3072, 16 heads / 8 KV heads, head_dim 128)
target aux layers	2, 14, 25 of the 28 decoder layers (hidden states entering those layers)
block size	8 (7 proposals per round, anchor = last committed token), causal within the block
mask token	151705
draft vocabulary	48,000 of the target's 151,936 (99.5% of the training label tokens); a 32,000 variant is compared below
sliding window	2,048: a request is at most 30 s of audio and 200 output tokens
precision	bf16

Audio as sglang-omni serves it

The draft was trained on the decoder input that sglang-omni builds, which differs from the transformers processor. sglang-omni keeps the first eighth of the audio adaptor's frames, about 63 audio tokens for 30 s, while the transformers processor expands all of them, 500 for 30 s. Both give complete transcripts, but not the same ones: on 48 clips, sglang-omni's output agreed with the first-eighth path on 0.92-0.995 of tokens per language and with the all-frames path on 0.84-0.96. A runtime that feeds all frames will likely see lower accept than the numbers below; that has not been measured. The prompt is also sglang-omni's: 语音转写： for Chinese or no language, 语音转写成英文： for English, and the raw code for ja and yue.

Training

• Audio: 599,123 clips, about 1,700 h (English 810, Chinese 459, Japanese 370, Cantonese 77). YODAS2 en000, zh000 and ja000 (CC BY 3.0, cut to 3-30 s spans), LibriSpeech train-clean-360 (CC BY 4.0), People's Speech clean (CC BY / CC BY-SA 4.0), VoxPopuli en (CC0), AISHELL-1 (Apache-2.0), Common Voice 17 ja and a Common Voice 21 yue mirror (CC0), words.hk Cantonese speech (CC BY 4.0), and FLEURS en, zh, ja and yue train (CC BY 4.0).
• Labels: the target's own greedy transcripts through sglang-omni with each clip's language hint, up to 190 new tokens. 98.7% of the served tokens are the transformers decoder's argmax on the same input, so the labels are what the target predicts.
• Speculators' DFlashDraftModel and loss (cross-entropy with D-Pace position weights), AdamW lr 6e-4, weight decay 0.01, 4% warmup, cosine, 15 epochs (30,165 steps), packed 16k-token batches on each of two H100s, up to 3,072 anchors per batch. Audio embeddings were cached from the target's encoder and adaptor, so training ran only the decoder.
• Shipped checkpoint: step 27,978, the best on a held-out subset. A 6-epoch run reached 4.47 on the full held-out set; held-out accept stopped rising at about 17,000 steps of the 15-epoch run.

Accept length

Simulated greedy speculative decoding on CUDA (bf16 target): from each anchor the draft proposes 7 tokens, the longest prefix matching the target's greedy transcript is accepted and the target adds one token; accept = generated tokens / rounds (the runtime's tokens / verify_rounds). Held-out audio only: LibriSpeech test, FLEURS test, Common Voice ja and yue test, and the last two YODAS2 shards of each language, 3,068 clips. These numbers have not been measured on a serving runtime (sglang-omni, vLLM, MLX) or with a quantized target, and sglang-omni does not yet serve a draft for this model.

draft	Libri clean	Libri other	FLEURS en	YODAS en	FLEURS zh	YODAS zh	CV ja	FLEURS ja	YODAS ja	CV yue	FLEURS yue	en	zh	ja	yue	all
this draft (48k vocab)	5.85	5.08	4.70	3.84	5.09	4.61	4.16	4.95	4.05	5.15	4.67	4.66	4.76	4.32	4.84	4.59
this draft, 32k vocab variant	5.52	4.95	4.34	3.75	4.68	4.57	4.16	4.92	4.02	5.08	4.42	4.46	4.61	4.30	4.66	4.46

Decoding is a smaller share of request time for Fun-ASR-Nano than for most ASR models, because its audio encoder has 70 layers: in sglang-omni on an H100, decode was 53% of request time at concurrency 1 and 25% at concurrency 16. End-to-end gains will be well below what the accept length suggests.