lightseekorg/kimi-k3-dflash2

Kimi-K3 DFlash2 draft

Draft model for speculative decoding on moonshotai/Kimi-K3, served with TokenSpeed (--speculative-algorithm DFLASH).

Architecture

5-layer, lookahead-8, MLA + SWA

• DFlash2DraftModel, 5 layers, hidden size 7168, bfloat16
• block_size: 8 (lookahead-8), attention_mode: mla
• target_layer_ids: [19, 37, 66, 78, 90] - five auxiliary hidden-state taps
• layer_types: 4x sliding_attention (window 4096) + 1x full_attention
• yarn RoPE, rope_theta: 50000.0, factor: 32.0
• 3.69 B parameters, 7.39 GB of weights

Benchmarks

Acceptance length acc_len = 1 + accepted/steps at 7 speculative proposals, served with TokenSpeed DFLASH on two GB300 nodes (8 GPUs, TP8), MLA target and drafter. Target is moonshotai/Kimi-K3 with the serving flags shown at the bottom of this card. Production sampling only: temperature=1.0, top_p=0.95, reasoning effort max, no top_k.

Ten-benchmark matrix

benchmark	acc_len	proposal acceptance	prompts	requests	truncated	mean out tok
GSM8K	5.901	70.0%	1319	1319	0.0%	254
HumanEval	5.080	58.3%	164	164	0.6%	989
MATH-500	4.636	51.9%	500	500	0.8%	803
SPEED-Bench coding	4.431	49.0%	80	89	1.1%	1341
SPEED-Bench RAG	3.958	42.3%	80	91	1.1%	691
SPEED-Bench multilingual	3.849	40.7%	80	80	1.2%	655
MT-Bench	3.585	36.9%	80	160	9.4%	1759
SPEED-Bench QA	3.213	31.6%	80	80	3.8%	1018
AIME 2026	2.785	25.5%	30	30	13.3%	6091
SPEED-Bench writing	2.780	25.4%	80	84	64.3%	3588

Long context

RULER multi-key needle retrieval over natural-text paragraphs, with only the input length varying. retrieval is exact match on the needle value, recorded so that acceptance cannot be credited to a model that has stopped reading its context.

bucket	input tokens	acc_len	proposal acceptance	retrieval
8k	8203	5.666	66.7%	1.000
32k	32596	4.973	56.8%	1.000
64k	65113	4.805	54.4%	1.000
128k	130151	4.854	55.1%	1.000

Serving

python3 -m tokenspeed.cli serve <Kimi-K3> \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path <this repo> \
  --speculative-num-steps 7 --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 8 \
  --speculative-draft-model-quantization unquant \
  --attention-backend mla --drafter-attention-backend mla \
  --sampling-backend flashinfer --trust-remote-code \
  --tensor-parallel-size 8 --moe-backend flashinfer_trtllm \
  --max-model-len 32768 --gpu-memory-utilization 0.90 \
  --max-num-seqs 4 --max-cudagraph-capture-size 4 \
  --cudagraph-capture-sizes 1 2 3 4 \
  --disable-prefill-graph --disable-kvstore

At startup the engine reports the auxiliary taps it captured; confirm they match (19, 37, 66, 78, 90).