A DSpark draft model for Gemma-4-E4B-it (up to 4.7x speedup; higher acceptance than MTP on 4/9 benchmarks)
Gemma-4-E4B-it-DSpark
Model Link: https://huggingface.co/rasyosef/gemma-4-E4B-it-dspark
A DSpark draft model for speculative decoding with google/gemma-4-E4B-it as the verifier, trained with speculators. The drafter proposes 8 tokens at a time and the verifier checks them in one forward pass, so output is identical to running the verifier alone. This makes it a lossless speedup. Mean acceptance length is 3.24 tokens committed per verification step, up to 6.05 on math_reasoning and 4.91 on HumanEval. On an A100 at concurrency 1, that gives a throughput speedup over the verifier alone of 4.66× on math_reasoning and 3.85× on HumanEval.
Training code: rasyosef/train-dspark-draft-models.
Trained on 216,000 samples.
Usage
vLLM
vLLM loads the verifier automatically from the config, so don't pass it separately.
vllm serve rasyosef/gemma-4-E4B-it-dspark --port 8000 --gpu-memory-utilization 0.8
Then query the OpenAI-compatible endpoint at http://localhost:8000/v1.
Details
Architecture. A Qwen3-style decoder with 4 draft layers, ~0.55B params, bfloat16. It has hidden size 2560, MLP size 8192, and 32 query heads over 8 KV heads (GQA, head dim 128). The first three layers use sliding-window attention (window 2048) and the last uses full attention. Block size is 8. The draft vocabulary is reduced to 32,000 tokens from the verifier's full 262,144, selected by token frequency over the training data. Aux hidden states come from verifier layers 2/10/20/30/40.
Training. 3 epochs at lr 5e-4 (Muon optimizer, cosine schedule with 4% warmup) with a {"ce": 0.1, "tv": 0.9} loss. The data is 216,000 filtered Open PerfectBlend prompts regenerated by the verifier itself, split 96/4 into train and validation. Prompts were prepared at 2560 tokens. Training sequence length was 4096, with up to 512 anchors per sample. Verifier hidden states were pulled on demand from a running vLLM server during training and deleted after use, rather than staged to disk up front.
Evaluation
evaluate.py across the nine RedHatAI/speculator_benchmarks subsets. acceptance_length is mean tokens committed per verification step, including the bonus token. Its floor is 1.0 and its ceiling is 9.0 at block size 8.
| subset | acceptance_length | pos_0 | pos_1 | pos_2 | pos_3 | pos_4 | pos_5 | pos_6 | pos_7 |
|---|---|---|---|---|---|---|---|---|---|
| math_reasoning | 6.048 | 88.2% | 79.1% | 72.0% | 65.2% | 58.0% | 52.5% | 47.3% | 42.5% |
| HumanEval | 4.905 | 81.8% | 67.9% | 57.2% | 48.9% | 42.5% | 36.2% | 30.6% | 25.5% |
| question | 2.904 | 63.0% | 39.9% | 26.5% | 19.4% | 14.7% | 11.4% | 8.9% | 6.6% |
| writing | 2.882 | 62.5% | 39.4% | 26.3% | 19.1% | 14.6% | 11.2% | 8.5% | 6.6% |
| translation | 2.757 | 59.6% | 38.5% | 27.6% | 18.9% | 13.2% | 8.9% | 5.7% | 3.2% |
| rag | 2.753 | 64.3% | 41.9% | 27.1% | 17.2% | 11.2% | 7.0% | 4.0% | 2.6% |
| tool_call | 2.697 | 60.9% | 38.6% | 24.7% | 16.8% | 11.6% | 8.1% | 5.4% | 3.6% |
| qa | 2.281 | 55.7% | 31.1% | 17.5% | 10.2% | 6.1% | 3.8% | 2.3% | 1.4% |
| summarization | 2.189 | 55.9% | 29.0% | 15.5% | 8.8% | 5.1% | 2.7% | 1.2% | 0.7% |
Weighted across all subsets: 3.244 over 96,410 verification steps.
Acceptance is highest where the verifier's next token is most predictable. math_reasoning leads at 6.05, more than 1.1 tokens ahead of HumanEval, and still accepts more than half of pos_5 drafts (52.5%). Its last position (pos_7, 42.5%) matches HumanEval's pos_4 (42.5%). HumanEval is clearly second at 4.91 and also makes real use of the full 8-token block, with a quarter of pos_7 drafts accepted. The other seven subsets sit between 2.19 and 2.90.
Throughput (tokens/s)
Mean output throughput in tokens/s on a single A100 40GB at max concurrency 1. The last column is DSpark's speedup over the verifier alone, and the last row is each method's mean per-subset speedup over the baseline. Gemma's MTP drafter (google/gemma-4-E4B-it-assistant, 8 draft tokens) is included for comparison.
| subset | baseline (no drafter) | MTP | DSpark (this model) | DSpark speedup |
|---|---|---|---|---|
| math_reasoning | 98.0 | 394.5 | 456.4 | 4.66× |
| HumanEval | 97.7 | 362.2 | 376.0 | 3.85× |
| question | 97.9 | 231.1 | 254.0 | 2.59× |
| writing | 97.8 | 225.8 | 257.1 | 2.63× |
| translation | 98.2 | 280.3 | 212.6 | 2.16× |
| rag | 93.7 | 208.6 | 176.5 | 1.88× |
| tool_call | 94.0 | 219.0 | 180.2 | 1.92× |
| qa | 97.7 | 174.6 | 182.5 | 1.87× |
| summarization | 94.5 | 168.5 | 148.4 | 1.57× |
| average speedup | 1.00× | 2.60× | 2.57× | 2.57× |
The throughput ranking follows acceptance length: math_reasoning and HumanEval see the largest gains, summarization the smallest. DSpark is the fastest on 5 of 9 subsets and MTP on 4, and the two are nearly level on average speedup (2.57× vs 2.60×). DSpark's best result is math_reasoning at 456.4 tokens/s, 4.66× the baseline and 16% ahead of MTP. It also leads MTP by 10–14% on question and writing. MTP is ahead on translation, rag, tool_call and summarization.
Comparison with MTP
Acceptance length on the same nine subsets for Gemma's MTP drafter (google/gemma-4-E4B-it-assistant). Both drafters propose 8 tokens per step, so they share the same 9.0 ceiling. Per-step draft cost differs between the two, so acceptance length alone doesn't determine speed. See the throughput section above for that.
| subset | MTP (8 draft tokens) | DSpark (8 draft tokens) | best |
|---|---|---|---|
| math_reasoning | 5.370 | 6.048 | DSpark |
| HumanEval | 4.851 | 4.905 | DSpark |
| question | 2.820 | 2.904 | DSpark |
| writing | 2.851 | 2.882 | DSpark |
| translation | 3.810 | 2.757 | MTP |
| rag | 3.293 | 2.753 | MTP |
| tool_call | 3.154 | 2.697 | MTP |
| qa | 2.303 | 2.281 | MTP |
| summarization | 2.500 | 2.189 | MTP |
| mean (unweighted) | 3.439 | 3.269 | MTP |
DSpark has the higher acceptance length on 4 of 9 subsets and MTP on 5, with MTP ahead on average. DSpark's clearest win is math_reasoning, where it leads by 0.68 tokens. HumanEval, question, writing and qa are close, all within 0.1 tokens either way. MTP's lead comes from translation (by 1.05 tokens), rag, tool_call and summarization, the same four subsets where it is faster. On qa, DSpark is slightly faster despite a slightly lower acceptance length. The mean here is unweighted across subsets, so DSpark's 3.269 differs slightly from its weighted 3.244 above.
Limitations
Works only with gemma-4-E4B-it and is not usable as a standalone model. Acceptance falls off steeply past the first two positions on prose-like traffic, most of all on summarization and qa, so the gains concentrate in math and code. On prose-heavy workloads the tail of the 8-token block rarely gets accepted. Real-world speedup depends on your traffic mix. Because verification is lossless, the verifier's own behavior and biases carry through unchanged.
License
Apache 2.0, matching the verifier's license. The speculators training code is Apache-2.0.