AngelSlim/Hy3-DFly-Block8

A more accessible, comprehensive, and efficient toolkit for large model compression.

✒️ TechnicalReport   |    📖 Documentation   |   🤗 Hugging Face   |   🤖 ModelScope

💬 WeChat |   🫨 Discord

Hy3-A21B No-Thinking Drafters: MTP and DFly

Model Summary

This repository contains no-thinking speculative-decoding drafter checkpoints for Hy3-A21B, including an MTP drafter and a DFly drafter. These draft models are trained with AngelSpec and are intended to accelerate Hy3-A21B inference by proposing candidate tokens that are verified by the target model.

They should be deployed together with the corresponding Hy3-A21B target model under the no-thinking inference setting.

• Target model: Hy3-A21B
• Inference mode: No-thinking
• Drafter types: MTP and DFly
• Training / inference framework: AngelSpec
• Recommended serving backend: vLLM speculative decoding
• Recommended MTP speculative tokens: 3
• Recommended DFly speculative tokens: 8

Training Data

The no-thinking Hy3 drafter training starts from Open-PerfectBlend-style instruction prompts. Target responses are regenerated with Hy3-A21B under the no-thinking setting so that the drafter learns the target-side token distribution used during deployment.

Domain-Specific Data Expansion for DFly

To further strengthen code generation and mathematical reasoning, the DFly training corpus augments Open-PerfectBlend with 700K domain-specific prompts:

• 500K code prompts from OpenCodeInstruct and OpenCodeReasoning.
  • OpenCodeInstruct provides broad code-instruction coverage.
  • OpenCodeReasoning emphasizes distilled reasoning traces for competitive programming.
  • Together, they improve both general code completion and multi-step program synthesis.
• 200K math prompts from Big-Math, adding coverage of diverse and challenging reasoning problems.

Before response generation, prompts that share a contiguous sequence of at least 16 tokens with an evaluation example are removed. Responses for the filtered prompts are then regenerated with Hy3-A21B, mixed with Open-PerfectBlend, and used to train the combined corpus for six epochs with the same objective.

This expansion mainly improves math/code acceptance length. The gain is consistent across structured benchmarks, while chat-oriented performance may not improve uniformly because the final checkpoint is specialized toward code and math.

Evaluation: Accepted Length

The following table reports the mean accepted length on Hy3-A21B at temperature 1 under the no-thinking setting. The average is the arithmetic mean over the six benchmarks.

Target Model	Drafter	Math500	GSM8K	HumanEval	MBPP	LiveCodeBench	MT-Bench	Avg.
Hy3-A21B	MTP	3.30	3.30	3.13	3.04	2.84	2.40	3.00
Hy3-A21B	DFlash	4.01	4.23	4.36	4.05	3.10	2.38	3.69
Hy3-A21B	DFly	5.23	5.53	5.52	5.41	4.07	2.96	4.79

Evaluation: Throughput Benefit

The following results are measured on HY3 295B-A21B with TP=8 at temperature 1 across concurrency levels. Tok/s denotes output-token throughput. Spd. denotes throughput relative to autoregressive decoding (AR). Each cell uses 3 x 120s measurement windows, and Avg. is the arithmetic mean across the six datasets.

Conc.	Method	GSM8K Tok/s	GSM8K Spd.	Math500 Tok/s	Math500 Spd.	HumanEval Tok/s	HumanEval Spd.	MBPP Tok/s	MBPP Spd.	LiveCodeBench Tok/s	LiveCodeBench Spd.	MT-Bench Tok/s	MT-Bench Spd.	Avg. Tok/s	Avg. Spd.
c4	AR	287.9	1.00x	293.9	1.00x	279.7	1.00x	294.4	1.00x	284.5	1.00x	290.0	1.00x	288.4	1.00x
c4	MTP-3	495.5	1.72x	517.0	1.76x	473.6	1.69x	476.7	1.62x	414.0	1.46x	384.5	1.33x	460.2	1.60x
c4	DFlash-8	569.5	1.98x	587.9	2.00x	588.1	2.10x	575.7	1.96x	408.4	1.44x	371.6	1.28x	516.9	1.79x
c4	DFly-8	635.4	2.21x	643.9	2.19x	647.2	2.31x	661.5	2.25x	455.2	1.60x	384.0	1.32x	571.2	1.98x
c8	AR	426.3	1.00x	435.7	1.00x	400.3	1.00x	433.7	1.00x	413.4	1.00x	421.8	1.00x	421.8	1.00x
c8	MTP-3	756.9	1.78x	791.5	1.82x	717.4	1.79x	729.1	1.68x	620.8	1.50x	590.1	1.40x	701.0	1.66x
c8	DFlash-8	857.0	2.01x	860.1	1.97x	866.2	2.16x	866.2	2.00x	609.1	1.47x	545.5	1.29x	767.4	1.82x
c8	DFly-8	964.8	2.26x	959.5	2.20x	974.1	2.43x	1001.5	2.31x	641.3	1.55x	565.8	1.34x	851.2	2.02x
c16	AR	650.7	1.00x	670.6	1.00x	594.8	1.00x	667.9	1.00x	617.2	1.00x	628.1	1.00x	638.2	1.00x
c16	MTP-3	1146.6	1.76x	1174.0	1.75x	1076.1	1.81x	1103.3	1.65x	893.5	1.45x	876.6	1.40x	1045.0	1.64x
c16	DFlash-8	1446.1	2.22x	1430.4	2.13x	1433.3	2.41x	1440.4	2.16x	970.5	1.57x	901.0	1.43x	1270.3	1.99x
c16	DFly-8	1623.8	2.50x	1607.5	2.40x	1595.4	2.68x	1677.8	2.51x	1046.7	1.70x	961.5	1.53x	1418.8	2.22x
c32	AR	918.6	1.00x	1000.8	1.00x	850.8	1.00x	1018.7	1.00x	893.2	1.00x	921.1	1.00x	933.9	1.00x
c32	MTP-3	1923.8	2.09x	1989.8	1.99x	1780.7	2.09x	1863.6	1.83x	1394.2	1.56x	1469.4	1.60x	1736.9	1.86x
c32	DFlash-8	2261.8	2.46x	2334.3	2.33x	2170.6	2.55x	2339.3	2.30x	1489.9	1.67x	1453.4	1.58x	2008.2	2.15x
c32	DFly-8	2527.4	2.75x	2608.9	2.61x	2429.1	2.86x	2741.3	2.69x	1602.6	1.79x	1549.4	1.68x	2243.1	2.40x
c64	AR	1156.9	1.00x	1381.3	1.00x	1197.6	1.00x	1513.9	1.00x	1229.6	1.00x	1306.9	1.00x	1297.7	1.00x
c64	MTP-3	2932.0	2.53x	3170.0	2.29x	2639.6	2.20x	3011.3	1.99x	2050.4	1.67x	2350.0	1.80x	2692.2	2.08x
c64	DFlash-8	2523.4	2.18x	2947.1	2.13x	2655.2	2.22x	2671.8	1.76x	2015.8	1.64x	1815.8	1.39x	2438.2	1.89x
c64	DFly-8	2827.8	2.44x	3301.8	2.39x	2965.0	2.48x	3130.9	2.07x	2195.0	1.79x	1936.6	1.48x	2726.2	2.11x

Deployment

Set TARGET to the Hy3-A21B target model path or repository and DRAFT to the corresponding drafter checkpoint. The relevant inference code can be found in PR https://github.com/vllm-project/vllm/pull/50246.

MTP-3

export TARGET="<path-or-hf-repo-to-hy3-a21b-target>"
export DRAFT="<path-or-hf-repo-to-hy3-mtp-drafter>"

vllm serve "$TARGET" \
  --tensor-parallel-size 8 \
  --speculative-config "{"method":"mtp", "model":"$DRAFT", "num_speculative_tokens":3}"

DFly-8

export TARGET="<path-or-hf-repo-to-hy3-a21b-target>"
export DRAFT="<path-or-hf-repo-to-hy3-dfly-no-thinking-drafter>"

vllm serve "$TARGET" \
  --tensor-parallel-size 8 \
  --speculative-config "{"method":"dspark", "model":"$DRAFT","num_speculative_tokens":7}"

Technical Report

For more details, see the technical report:

AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding
arXiv: https://arxiv.org/abs/2607.25852