DFlash2 drafter for openbmb/MiniCPM5-2B (overall 1.086x, coding 1.087x, agentic 1.109x, 4600 steps)
c93e263 verified | license: apache-2.0 | |
| library_name: transformers | |
| pipeline_tag: text-generation | |
| base_model: | |
| - openbmb/MiniCPM5-2B | |
| inference: false | |
| tags: | |
| - dflash2 | |
| - block-diffusion | |
| - speculative-decoding | |
| - draft-model | |
| - sglang | |
| - vllm | |
| - minicpm | |
| # MiniCPM5-2B-DFlash2 | |
| A community **DFlash 2** block-diffusion drafter for [`openbmb/MiniCPM5-2B`](https://huggingface.co/openbmb/MiniCPM5-2B), | |
| trained by self-distillation with the recipe of [DFlash 2](https://inco.ai/blog/dflash2/) | |
| ([`z-lab/dflash`](https://github.com/z-lab/dflash)). It is **not** a standalone language model: it drafts blocks of | |
| tokens for the frozen target to verify inside a speculative-decoding server. Decoding is lossless — greedy output | |
| matches the target exactly, and sampling preserves its distribution. | |
| ## Architecture | |
| | | | | |
| |---|---| | |
| | draft backbone | 5 qwen3-style decoder layers, hidden 2048, GQA | | |
| | block size | 8 (7 drafted tokens per verify step) | | |
| | target taps | layers [1, 10, 20, 30, 39] of 42, fused by a linear + RMSNorm | | |
| | block diffusion | bidirectional attention inside the block (`is_causal: false`), two-tap dynamic grouped convs (k=2, groups=16) | | |
| | path selector | low-rank Markov selector, rank 256, top-k 16 | | |
| | mask token | `75982` (the same unused id OpenBMB's official `MiniCPM5-2B-DSpark` uses) | | |
| | embeddings / LM head | **not stored** — reused from the target at runtime | | |
| | parameters | ~335M (0.67 GB bf16) | | |
| ## Training (this repo's `dflash2_training_meta.json` has the raw settings) | |
| - **Method:** target frozen; forward-KL distillation of the draft's block logits against the target's teacher-forced | |
| distribution (+ CE anchor weight 0.1), and the candidate-path selector trained toward the greedy-acceptance | |
| answer with teacher-forced predecessor chains. **4 blocks trained per row** with cache-streamed | |
| full-prefix contexts — the exact cache discipline of the DFlash2 inference loop, so training and serving match. | |
| - **Data:** two-phase mixture — phase 1 (general): HuggingFaceH4/ultrachat_200k, open-r1/OpenR1-Math-220k, ise-uiuc/Magicoder-Evol-Instruct-110K, HuggingFaceTB/smoltalk, HuggingFaceFW/fineweb-edu; phase 2 (code/agentic | |
| anneal): ise-uiuc/Magicoder-Evol-Instruct-110K, HuggingFaceTB/smoltalk, bigcode/self-oss-instruct-sc2-exec-filter-50k, HuggingFaceH4/ultrachat_200k, open-r1/OpenR1-Math-220k. Chat formatted with the target's template, packed into | |
| 1536-token rows, 4600 steps at an effective batch of 8 rows. | |
| - **Hardware:** Tesla T4, fp16/bf16 autocast + GradScaler. | |
| - **Validation mode:** no — full release recipe. | |
| ## Measured speed (greedy, batch 1, 6 prompts/domain, this notebook's run) | |
| | domain | plain tok/s | spec tok/s | speedup | proj | acceptance length | greedy-identical | | |
| |---|---:|---:|---:|---:|---:|---:| | |
| | coding | 17.81 | 19.36 | **1.087x** | 1.005x | 1.234 | 6/6 | | |
| | agentic | 17.98 | 19.94 | **1.109x** | 1.032x | 1.267 | 5/6 | | |
| | math | 17.91 | 19.28 | **1.076x** | 0.998x | 1.226 | 5/6 | | |
| | general | 17.9 | 19.26 | **1.076x** | 1.0x | 1.228 | 5/6 | | |
| | **overall** | 17.89 | 19.42 | **1.086x** | 1.006x | 1.236 | 21/24 | | |
| Measured per-cycle costs: target decode 57.714 ms/tok, target verify(8) 55.433 ms, draft block 15.442 ms. The **proj** column is AL * t_decode / (t_verify8 + t_draft) — what an engine-grade serving stack (SGLang/vLLM) should approach; the measured speedup additionally carries the python-side loop overhead of this eval harness. | |
| Speed targets: coding >= 2.0x, agentic >= 2.0x, overall >= 1.5x — not met within caps (measured overall 1.086x). | |
| Decoding is lossless by construction: the target verifies every drafted token, so greedy output is token-identical to | |
| plain decoding (the identity rate above is limited only by bf16 near-tie argmax flips) and sampling preserves the | |
| target distribution. The published DFlash 2 drafters report ~4-5.5 acceptance length on Qwen3.8-27B with full-scale | |
| training. | |
| ## Serving | |
| **SGLang** (needs the DFLASH support build, see [`z-lab/dflash`](https://github.com/z-lab/dflash)): | |
| ```bash | |
| python -m sglang.launch_server --model-path openbmb/MiniCPM5-2B --speculative-algorithm DFLASH --speculative-draft-model-path applepie69/MiniCPM5-2B-DFlash2 --speculative-num-draft-tokens 8 | |
| ``` | |
| **vLLM:** | |
| ```bash | |
| vllm serve openbmb/MiniCPM5-2B --speculative-config '{"method": "dflash", "model": "applepie69/MiniCPM5-2B-DFlash2", "num_speculative_tokens": 7}' | |
| ``` | |
| **Transformers / offline eval** with the `dflash` package: | |
| ```python | |
| from dflash.model import DFlash2DraftModel, dflash_generate | |
| draft = DFlash2DraftModel.from_pretrained("applepie69/MiniCPM5-2B-DFlash2") | |
| out = dflash_generate(draft, target_model, input_ids, max_new_tokens=256, | |
| stop_token_ids=[1, 130073], temperature=0.0) | |
| ``` | |
| ## Files | |
| | file | what | | |
| |---|---| | |
| | `config.json` | DFlash2 draft config (`dflash_config` block) | | |
| | `model.safetensors` | the trained drafter (bf16) | | |
| | `dflash2_training_meta.json` | training settings + measured results | | |
| | `measured_results.json` | per-domain speed/acceptance measurements + gate status | | |
| ## License | |
| Apache-2.0, matching the base model. Check the base model's terms before use. | |