Up to 4.1x speedup with a DSpark draft model for K2-Horizon-3.7B (109 --> 450 tokens/sec)
K2-Horizon-3.7B-DSpark
Model Link: https://huggingface.co/rasyosef/K2-Horizon-3.7B-DSpark
A DSpark draft model for speculative decoding with IFM/K2-Horizon-3.7B as the verifier, trained with speculators. The drafter proposes 8 tokens at a time and the verifier checks them in one forward pass, so greedy output is identical to running the verifier alone — a lossless speedup. At temperature 0, mean acceptance length is 3.06 tokens committed per verification step, up to 4.86 on math_reasoning. Compared with the verifier alone, throughput is 2.72× higher on average, up to 4.12× on math_reasoning and 3.03× on HumanEval, and it is faster than the verifier alone on all nine benchmark subsets, by at least 2.16×.
Training code: rasyosef/train-dspark-draft-models.
Trained on 50,000 samples.
Usage
vLLM
vLLM loads the verifier automatically from the config — don't pass it separately. The verifier uses custom code, so --trust-remote-code is required, and K2-Horizon is a reasoning model, so enable its reasoning parser to get the thinking back in reasoning_content.
vllm serve rasyosef/K2-Horizon-3.7B-DSpark --port 8000 \
--trust-remote-code \
--reasoning-parser k2_horizon \
--max-model-len 32768 \
--gpu-memory-utilization 0.8 \
--override-generation-config '{"temperature": 0}'
Then query the OpenAI-compatible endpoint at http://localhost:8000/v1.
For agent use, add --enable-auto-tool-choice --tool-call-parser k2_horizon. To serve with the sampling settings used in the evaluation below, which are also the verifier's recommended settings, swap the last flag:
--override-generation-config '{"temperature":1.0,"top_p":0.95}'
Details
Architecture
- Qwen3-style decoder with 4 draft layers, ~0.5B params, bfloat16
- Hidden size 2560, MLP size 8192
- 24 query heads over 8 KV heads (GQA, head dim 128)
- First three layers use sliding-window attention (window 512); the last uses full attention
- Block size 8 (8 draft tokens per step)
- Draft vocabulary of 32,000 tokens, reduced from the verifier's 250,624 by token frequency over the training data
- Aux hidden states from verifier layers 2/10/18/26/34
Training
- 4 epochs at lr 4e-4 (AdamW, cosine schedule with 4% warmup)
- Loss:
{"ce": 0.1, "tv": 0.9} - Data: 50,000 filtered prompts from Open PerfectBlend, with every response generated by the verifier itself
- 96/4 train/validation split
- Samples prepared at 8192 tokens
- Training sequence length 8192, up to 1536 anchors per sample
- Verifier hidden states pulled on demand from a running vLLM server and deleted after use, rather than staged to disk up front
Evaluation
evaluate.py throughput across the nine RedHatAI/speculator_benchmarks subsets.
Setup
- Acceptance measured at two settings: temperature 0 (greedy decoding), and sampling with temperature 1.0, top_p 0.95 (the verifier's recommended settings)
- Throughput reported at temperature 0 only
- Hardware: single A100 40GB, max concurrency 1
- Same settings for the baseline and DSpark runs
acceptance_length is mean tokens committed per verification step, including the bonus token. Its floor is 1.0 and its ceiling is 9.0 at block size 8. Rows are in the same order in every table, sorted by acceptance length at temperature 0.
Acceptance at temperature 0
| subset | acceptance_length | pos_0 | pos_1 | pos_2 | pos_3 | pos_4 | pos_5 | pos_6 | pos_7 |
|---|---|---|---|---|---|---|---|---|---|
| math_reasoning | 4.857 | 85.5% | 71.4% | 59.5% | 48.8% | 40.0% | 32.6% | 26.7% | 21.2% |
| HumanEval | 3.420 | 76.1% | 54.1% | 38.0% | 26.4% | 18.6% | 13.0% | 9.2% | 6.6% |
| rag | 3.014 | 72.2% | 49.0% | 31.9% | 20.3% | 12.8% | 7.8% | 4.7% | 2.7% |
| writing | 2.925 | 67.4% | 43.8% | 28.8% | 19.6% | 13.5% | 9.1% | 6.2% | 4.1% |
| question | 2.905 | 66.8% | 43.2% | 28.5% | 19.4% | 13.3% | 9.0% | 6.1% | 4.2% |
| qa | 2.828 | 66.7% | 42.7% | 27.6% | 18.3% | 11.8% | 7.5% | 4.9% | 3.2% |
| tool_call | 2.803 | 69.5% | 44.7% | 28.0% | 16.6% | 10.1% | 5.8% | 3.5% | 2.0% |
| translation | 2.797 | 64.9% | 43.4% | 28.4% | 18.3% | 11.3% | 6.9% | 4.0% | 2.4% |
| summarization | 2.572 | 64.8% | 40.0% | 23.8% | 13.7% | 7.8% | 4.2% | 1.9% | 0.9% |
Weighted across all subsets: 3.063 over 276,769 verification steps. math_reasoning is well ahead of every other subset at 4.857, and it and HumanEval are the only two that still accept more than 5% of drafts at pos_7 (21.2% and 6.6%). The other seven are tightly grouped between 2.57 and 3.01. summarization is lowest and falls below 1% by pos_7.
Acceptance with sampling
Same benchmark with {"temperature":1.0,"top_p":0.95}, the verifier's recommended sampling settings.
| subset | acceptance_length | pos_0 | pos_1 | pos_2 | pos_3 | pos_4 | pos_5 | pos_6 | pos_7 |
|---|---|---|---|---|---|---|---|---|---|
| math_reasoning | 3.946 | 75.0% | 58.0% | 44.9% | 35.4% | 28.2% | 22.3% | 17.3% | 13.5% |
| HumanEval | 2.621 | 60.5% | 36.8% | 23.4% | 15.3% | 10.3% | 7.2% | 5.1% | 3.5% |
| rag | 2.269 | 56.0% | 30.8% | 17.6% | 10.3% | 5.8% | 3.3% | 1.9% | 1.1% |
| writing | 2.280 | 53.1% | 29.1% | 17.5% | 11.1% | 7.2% | 4.8% | 3.2% | 2.1% |
| question | 2.256 | 52.7% | 28.8% | 17.0% | 10.6% | 6.9% | 4.6% | 3.0% | 2.1% |
| qa | 2.068 | 50.7% | 25.8% | 13.8% | 7.6% | 4.3% | 2.4% | 1.3% | 0.7% |
| tool_call | 2.286 | 56.4% | 31.1% | 17.7% | 10.4% | 6.1% | 3.5% | 2.2% | 1.3% |
| translation | 2.322 | 56.5% | 32.2% | 18.6% | 11.0% | 6.6% | 3.8% | 2.2% | 1.3% |
| summarization | 2.146 | 53.7% | 28.7% | 15.6% | 8.3% | 4.4% | 2.3% | 1.1% | 0.5% |
Weighted across all subsets: 2.371 over 478,155 verification steps, against 3.063 at temperature 0. Acceptance length is lower on every subset, by 17–27%. The drop is largest in absolute terms on math_reasoning (4.857 to 3.946) and HumanEval (3.420 to 2.621), and largest in relative terms on qa (2.828 to 2.068), which becomes the lowest subset. It is smallest on summarization (2.572 to 2.146) and translation (2.797 to 2.322). math_reasoning stays highest at 3.946. Acceptance at pos_7 is under 5% on every subset except math_reasoning (13.5%), and under 1% on qa and summarization.
Throughput at temperature 0 (tokens/s)
Mean output throughput in tokens/s, using the setup above. Baseline is K2-Horizon-3.7B with no speculative decoding. The last column gives this model's throughput divided by baseline throughput for each subset.
| subset | baseline (no drafter) | DSpark (this model) | speedup |
|---|---|---|---|
| math_reasoning | 109.4 | 450.2 | 4.12× |
| HumanEval | 109.1 | 330.7 | 3.03× |
| rag | 108.1 | 265.1 | 2.45× |
| writing | 109.1 | 293.3 | 2.69× |
| question | 109.1 | 293.3 | 2.69× |
| qa | 108.3 | 275.8 | 2.55× |
| tool_call | 107.5 | 257.1 | 2.39× |
| translation | 109.5 | 268.2 | 2.45× |
| summarization | 108.0 | 233.2 | 2.16× |
| average speedup | 1.00× | 2.72× | 2.72× |
Average speedup is 2.72× (unweighted mean of per-subset speedups), from about 108 tokens/s for the verifier alone to about 296 tokens/s with the drafter. The gains are largest on math_reasoning (4.12×) and HumanEval (3.03×), and smallest on tool_call (2.39×) and summarization (2.16×).
Speedup roughly tracks acceptance length, at about 0.8–0.9× of it on every subset. The ratio is highest on writing and question (2.69× from about 2.9 accepted tokens per step) and lowest on rag (2.45× from 3.01), so rag ranks third on acceptance but only sixth on speedup.
Baseline throughput is nearly flat across subsets (107.5–109.5 tokens/s), so the differences in speedup come from the drafter rather than the baseline.
Limitations
Works only with K2-Horizon-3.7B and is not usable as a standalone model. With sampling at the verifier's recommended settings (temperature=1.0, top_p=0.95), weighted acceptance length falls from 3.063 to 2.371; throughput with sampling is not reported. All results are for a single concurrent request; throughput under load is not reported. The drafter was trained on Open PerfectBlend only, with samples up to 8,192 tokens, and three of its four layers see only a 512-token window, while the verifier supports a 512K context and long reasoning traces at high reasoning effort; acceptance on very long contexts is untested. Acceptance at the last draft position at temperature 0 is under 5% on seven of the nine subsets and under 1% on summarization, so a block size of 8 is partly wasted there; with sampling it is under 5% on eight subsets. Real-world speedup depends on your traffic mix, and because verification is lossless, the verifier's own behavior and biases carry through unchanged.
License
Apache 2.0, inherited from the verifier. The speculators training code is Apache-2.0.