GLM-5.3-Flash DFlash2 Drafter β€” run dflash2F

Self-trained DFlash2 block-diffusion speculative-decoding drafter for GLM-5.3-Flash, trained against our INT4 quant canada-quant/GLM-5.3-Flash-W4A16-MTP. Trained on self-generated data only β€” no third-party drafter weights or traces anywhere in the training path. Supersedes GLM-5.3-Flash-DFlash2-E (same architecture, same serve contract, drop-in): +0.065 mean acceptance at K=7 on the same hardware, and parity with the reference drafter incoai/GLM-5.3-Flash-DFlash2 measured on the same GPUs the same day (see below).

Architecture (unchanged from -E)

Field Value
Type DFlash2 block-diffusion drafter (DFlash2DraftModel, Qwen3-style backbone)
Layers 8 decoder layers, full attention (no sliding window)
Hidden / heads 4096 (intermediate 12288) Β· 32 attention heads / 8 KV heads Β· head_dim 128
Target taps 9 hidden-state taps at target layers [5, 9, 14, 19, 24, 28, 33, 38, 42]
Block size 8 β†’ K = 7 speculative tokens (num_speculative_tokens: 7)
Selector rank 256, top_k 16, grouped dynamic conv (kernel 2, group 16) β€” trained, not vestigial
Mask embedding learnable, shipped as mask_embedding.pt (mask_token_id 154856)
Size ~1.84B drafter parameters Β· 6.2 GB bf16 checkpoint (ships untied embed_tokens + lm_head)
Max positions 1,048,576

Training

  • Warm-started from dflash2E; one pass over 449,600 self-generated samples (the 350,260 of -E + 99,340 new completions of never-before-generated prompts), 41,279 steps, lr 1e-4 β†’ 1e-5 (cosine), gamma-6.5 tail weighting, CE + top-20 KL, 8Γ— NVIDIA B300, 16.5 h.
  • Prompts sampled from public instruction sets (ultrachat_200k Β· MIT, OpenR1-Math-220k Β· Apache-2.0, OpenMathReasoning Β· CC-BY-4.0, OpenScienceReasoning-2 Β· CC-BY-4.0, OpenCodeReasoning / OpenCodeInstruct Β· CC-BY-4.0, evol-codealpaca-v1 Β· Apache-2.0); every completion was regenerated by the target model itself (thinking ON, T=1.0 / top_p 0.95, ≀ 4,096 tokens) β€” no third-party model outputs.
  • The reference drafter incoai/GLM-5.3-Flash-DFlash2 was never a training input; it appears below only as a measured comparison.
  • PROVENANCE.txt is the verbatim training record. Recipe, code, patches and every raw measurement: canada-quant/vllm-glm53-flash-sm121/drafter (eval kit, every raw measurement, cards).

Holdout evaluation β€” same hardware, same day

500 never-trained-on prompts, thinking ON, T=1.0 / top_p 0.95, max_tokens 1024, greedy drafts, TP=4 (c16; the c1 row is 100 prompts at concurrency 1). All rows below were measured on the same 8Γ— B300 within six hours of each other, with the same vLLM build and the same target backend (FLASHINFER_MLA_SPARSE) β€” acceptance length shifts by a few hundredths between GPU generations (incoai reads 3.602–3.615 on H200 vs 3.632 on B300), so only same-hardware rows are compared.

Mean acceptance length (output tok/s) K=7, c16 K=4, c16 K=7, c1
dflash2F (this) 3.626 (1,406) 3.085 (1,339) 3.677 (313)
incoai/GLM-5.3-Flash-DFlash2 (CC-BY-NC-ND-4.0) 3.632 (1,425) 3.136 (1,369) 3.667 (328)
dflash2E (our previous release) 3.561 (1,397) β€” β€”

Per-position acceptance (K=7, c16): 0.750 Β· 0.555 Β· 0.415 Β· 0.316 Β· 0.244 Β· 0.193 Β· 0.153 (incoai: 0.75 Β· 0.55 Β· 0.41 Β· 0.32 Β· 0.25 Β· 0.20 Β· 0.16).

Honest read: parity with the reference at K=7 (βˆ’0.006) and single-stream (+0.010), both inside the protocol's Β±0.015 run-to-run noise; 1.6% short at K=4 (βˆ’0.051, the first draft positions). Against -E it is +0.065 at K=7 β€” the largest single-leg gain in the lineage, from 100K fresh self-generated samples. We ship it because it is fully ours (Apache-2.0, no NC/ND terms), trained on data we control, reproducible end-to-end, and now level with the reference. On the H200 (previous campaign) -E measured 3.568 / 3.067 / 3.585.

DGX Spark (SM121): -E reproduced its H200 acceptance on 2Γ— Spark within 0.3% (3.5788 c16 / 3.5970 c1 vs 3.568 / 3.585). -F is a drop-in for the same launcher; its Spark same-rig A/B against -E and the reference is the next measurement β€” re-measure both drafters on your rig with the eval kit in the repo before quoting a Spark number.

Serving

Pairs with canada-quant/GLM-5.3-Flash-W4A16-MTP (the quant it was trained against) or BF16 GLM-5.3-Flash. vLLM speculative config:

{"method": "dflash", "model": "/models/GLM-5.3-Flash-DFlash2-F", "num_speculative_tokens": 7}

x86 (H100 / H200 / B300), upstream vLLM nightly β‰₯ 2026-09-08 (+ the two GLM-5.3-Flash DFlash2 bind-mount patches in the repo above):

vllm serve /models/glm53-flash-w4a16-mtp --served-model-name glm53 --tensor-parallel-size 4 --enable-expert-parallel \
  --block-size 64 --no-enable-prefix-caching \
  --speculative-config '{"method":"dflash","model":"/models/GLM-5.3-Flash-DFlash2-F","num_speculative_tokens":7}'

2Γ— DGX Spark (SM121): prebuilt image + one-command launcher in canada-quant/vllm-glm53-flash-sm121 β€” the drafter is bind-mounted at runtime (DRAFTER_HOST_PATH=/models/GLM-5.3-Flash-DFlash2-F), no rebuild. launch_dflash2_tp2.sh in this repo is the same TP=2 launcher with this drafter as its default.

Hard constraints (all measured, not stylistic):

  • num_speculative_tokens: 7 (= block_size βˆ’ 1) is the trained and measured value. k=5 also boots on the SM121 image (2026-09-26, not benchmarked); an earlier version of this card said other counts boot-wedge the stack, which was wrong.
  • mask_embedding.pt must sit next to the weights. Verify the boot log carries Loaded DFlash mask embedding for mask_token_id 154856 from mask_embedding.pt β€” absence means the mask was silently ignored; do not serve.
  • The 9-tap config requires the serving stack to honor dflash_config.target_layer_ids of length 9 (upstream vLLM DFlash2 does β€” vllm-project/vllm#52816).
  • Full-attention drafter layers: the target stack must accept FullAttentionSpec drafter KV in the GLM-5 KV fast path (the SM121 image above has it; upstream nightly needs the kv_cache_utils patch from the repo).

Memory cost β€” and when to use the built-in MTP head instead (added 2026-10-04)

This drafter's 8 layers use full attention, so each keeps a KV cache for the whole context (same architecture as -E / -G); the incoai reference drafter keeps a 2,048-token sliding window. KV pools measured by the authors on 2Γ— DGX Spark (fp8 KV):

drafter KV pin max model len KV pool date
incoai reference 9 GiB 1,048,576 1,360,420 tokens (β‰ˆ7.1 KB/token) 2026-08-31
-E / -G 8 GiB 262,144 366,749 tokens (β‰ˆ22 KB/token) 2026-09-16
-G 16 GiB 800,000 888,729 tokens (β‰ˆ19 KB/token) 2026-09-26

That is about 3Γ— less context per GiB of KV. A 9 GiB pin holds β‰ˆ450K tokens with -E / -F / -G β€” less than one 1M request β€” so 1M context on 2Γ— Spark is validated only with the incoai drafter; with our drafters the validated contexts are 262K (8 GiB) and 800K (16 GiB).

  • Use this drafter for short-context English and code, where its acceptance is highest.
  • Use GLM-5.3-Flash's built-in BF16 MTP head (shipped in the W4A16 checkpoint; {"method":"mtp","num_speculative_tokens":3}) for long context, non-English text, or KV headroom.
  • Measured by the authors (W4A16 + -G, 2Γ— Spark, our SM121 image, single stream, llama-benchy pp2048/tg128): 31.4 / 30.7 / 15.3 / 10.0 tok/s at context depth 0 / 4K / 65K / 100K (2026-09-28; the 100K cell from 2026-09-26).
  • Community-reported (NVIDIA developer forum, 2026-10-04; not measured by us): on eugr's b12x vLLM build with the GPU clock capped at 1700 MHz, the W4A16 weights with the MTP head held 31.6 / 29.8 / 25.4 / 31.0 tok/s at the same depths; with -G usable context fell from β‰ˆ930K to β‰ˆ330K tokens and decode was 20–30% slower on non-English text (faster on English code).
  • Our own same-pair comparison on 2Γ— Spark (drafter vs MTP, plus eugr's b12x build) has been running since 2026-10-04 on a bilingual generation workload; results will be published in canada-quant/vllm-glm53-flash-sm121.

Provenance

  • Files: model.safetensors (sha256 305efe2aa50447cbac7d7d3ab566985103973a79c5d21eb11e1dbf870139dd8c), config.json (699bdf29…c51778, identical to -E), mask_embedding.pt (0a5eecd9…304f3a), PROVENANCE.txt (546442a6…f75721).
  • Every number on this card comes from sha256-pinned raw eval JSONs banked in the repo (angelspec-glm53/runs/all/dflash2F-*.json, dflash2E-k7-b300.json, incoai-k7-b300.json, incoai-k4-b300.json, incoai-k7-c1-b300.json).

References

Downloads last month
175
Safetensors
Model size
3B params
Tensor type
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for canada-quant/GLM-5.3-Flash-DFlash2-F

Finetuned
(43)
this model

Paper for canada-quant/GLM-5.3-Flash-DFlash2-F