Masked MTP drafts

With LLMs it seems that progress is not intuitive: acceptance dropped, and the tok/s got higher. Full draft head: 66.7% acceptance, 117.1 tok/s. Trimmed head: 63.6%, 121.7 tok/s. +3.9% end to end, on an early K=3 held-out run.

I knew it would release some compute before I ran it, it simply wasn't intuitive. The sequence was ordinary: I measured, saw I'm compute bound, checked, and saw the draft head is pushing the limit. I can't do both, the full head read and the step. So I looked for ways to shrink the compute. And masking worked. The surprising part was: shrinking by almost 7x keeps the acceptance pretty close, even though it's not even close to the full vocab options. 66.7% to 63.6%. That's a small drop for cutting the head from 248,320 rows to 32,768. Acceptable.

Not my idea. It comes from FR-Spec (ACL 2025). In short: when speculative decoding is used, the draft model's vocabulary can be trimmed to a frequency-ranked subset of the full vocab. The verification is the same, but it simply can't offer tokens outside this subset. It simply dies immediately if it tries to, and the next token is generated from the full model. There's less options to consider, so the acceptance drops a little, but the compute is much cheaper. The only difference is that the paper's authors measured it on a separate draft model, smaller, which I also tried, but it requires training and the head is already there, so why not just use it. So I wired it for native MTP.

That read is especially expensive on this model, and its family. Qwen3.8-27B has a 248,320-token vocabulary, and the MTP head produces a logit for all of it, every draft step, but on average the model will never say most of them. Keep the top 32,768 by what the model actually emits and the head read shrinks 248,320 / 32,768 = 7.6x. That's the trick. When each draft step gets that much cheaper you can afford a few percent acceptance drop.

So the credit really is: FR-Spec is the paper's idea. My part is wiring it for native MTP, the multi-token-prediction head that ships inside the checkpoint and not a separate draft model, and publishing a card you can actually download and run. And this write-up.

Important: it's not what people mostly use, it's specifically what the model probably emits. The ranks come from the model's own generations, 163k tokens of agentic output and 154k of prose, with external text used as prompts only.

It kept its truth on re-measure too. memra v0.86.2, interleaved x5 against the full embedded head: +5.1% on an RTX PRO 6000, 137.9 vs 131.2 decode p50. +6.4% on an RTX 5090 Laptop, 73.6 vs 69.2. And the full head still accepts more, 0.76 vs 0.74. Loses anyway. But - measured on three Qwen 3.6 models and two Gemmas, plus the 3.8 here. The per-model drafts, rank files and board rows are public on memra-bench. Two things that sweep taught me: ranks don't transfer between models, I measured -12 acceptance points when I tried; and requantizing the trimmed head to NVFP4 costs zero measured acceptance. I can't claim more without measuring.

Two ways to run it in memra, because checkpoints come in two shapes.

GGUF wants a separate pre-trimmed draft file. You attach it:

MEMRA_MTP_DRAFT=mtp-Qwen3.8-27B-NVFP4-frspec-sxc32768.gguf \
memra-server --model Qwen3.8-27B-NVFP4-Q5K-mtp.gguf

Safetensors doesn't need a separate file. Drop a .txt ranks card next to the model and memra trims the head weight at load:

MEMRA_FRSPEC_TRIM=<ranks.txt>

Why two mechanisms for one trick? GGUF is what people download, so the trim is baked into the file and nobody needs conversion tooling. The safetensors path leaves the official checkpoint alone and masks at load, so trying a different flavor means swapping one text file. The .txt is literally the list that produced the trimmed GGUF.

If memra isn't your engine, steal the idea anyway. I opened a llama.cpp issue for it: Research: FR-Spec-style draft-vocab trimming for native MTP speculative decoding. Research plus a branch, to be clear, not something I shipped there. Most of the plumbing already exists in their EAGLE-3 support: a d2t index mapping, scatter-after-matmul. That's exactly what a frequency-ranked subset needs. Native trimmed MTP just has to reuse it.

Everything downloadable sits on the Hugging Face card: pre-trimmed draft GGUFs in agentic, prose and mixed flavors, plus the ranks lists for the safetensors path. 32,768 ranked token ids each, and the agentic one is my serving default. Based on my usage. And it makes a difference.

• Draft GGUF: mtp-Qwen3.8-27B-NVFP4-frspec-sxc32768.gguf, plus prose and mixed flavors.
• Ranks lists: q38-ranks-sxc32768.gguf.txt (agentic), q38-ranks-prose-32768.txt, q38-ranks-mixed-32768.txt.

For short generic stuff it doesn't really matter. My quick checks put all three inside noise of each other, acceptance around 0.40 on prose text and 0.59 to 0.62 on code, whichever ranking is loaded. One good read is that the top of the ranking is almost the same across domains and the differences only live in the tail. Agentic workloads take the agentic list, prose takes prose, and if you have no idea, mixed exists so you can stop thinking about it.

I'm probably already measuring the next head by the time you read this.

Update (November 2026)

It held. Three months later I ran the same trim on more models and more hardware. Same rule as before: with and without, same machine, same build, and the run count next to every number.

The 27B kept its gain with a different drafter. With the DFlash2 drafter instead of the native MTP head: +4.8% on chat and +1.7% on agentic work on an RTX PRO 6000 (3 interleaved runs), and +3.9% on the RTX 5090 Laptop (ABBA). Qwen3.8-Flash-Next took it best: +6.6% with thinking on, +7.8% with thinking off, 5 interleaved runs on a PRO 6000. Ornith-1.5-35B-A3B: +10% on short single-user work and +2.9% at 8 users, 6 runs per shape.

The Mac took it too. I ported the llama.cpp patch onto current master and ran the 27B on a Mac mini M5 Pro, Q8_0 on Metal: 23.22 to 24.38 tok/s at one user, +5.0%. At four users, +1.3%. One run each. On a DGX Spark, Flash-Next FP4 under vLLM with a 65,536-row trim: +4.5% to +9.7% at one user, +2.2% to +3.0% at eight. Also one run.

Where it stops paying. Long context first. On Ornith 35B a long document and a long session came out flat. Flint 40B-A2.5B at ~245K context on MLX lost on 11 of 12 cells. When most of a round is reading the KV cache, the head is a small part of it, and the acceptance cost wins. Pierpaolo saw the same on a 70K document in the llama.cpp issue. Many users second: the round turns into verify, and the head's share shrinks with it.

Ranks matter more than I thought. Flint lost on every cell on the Mac, but that rank list had only 11,114 distinct ids padded to 32,768. That is a bad list, not a verdict. Ranks from another model still cost 12 acceptance points. And a static trim can miss. On Gemma 4 31B it lost 17% on chat until the server learned the escaped ids into spare head rows. After that it matched or beat the untrimmed head on every cell (2 interleaved runs).

DFlash2 is the same idea from the other side. It never carries a head of its own. It scores candidates through the target's own lm_head, and the trim stacks on top. Tencent's DSpine drafter ships its own 1.27 GB FP8 head, so I moved its candidates onto the target's NVFP4 head. Its numbers come in a later update, once the repeated runs finish.

Lossless still holds where I can check it. In memra, greedy spec on and spec off are byte-identical with the trim on. On the Mac the trimmed and full heads gave the same text on 16 of 16 prompts at one user.

The full table is in the llama.cpp issue: https://github.com/ggml-org/llama.cpp/issues/25187#issuecomment-6728848783

Already measuring the next one.

The DSpine numbers are in. On one DGX Spark, taking DSpine's coarse candidates from the target's NVFP4 head took 8-user throughput from 122.6 to 151.5 tok/s, one run each. It still wasn't enough to beat DFlash2 at load. Four interleaved runs each on another Spark: 153.9 vs 164.3 at 8 users, 259.1 vs 287.4 at 32. At one user DSpine wins, 38.3 vs 33.9. It accepts more per round and still loses, because it verifies 16 tokens a round against DFlash2's 8. The head trick pays where the head is a big share of the step, same as the trim. Full table: https://github.com/ggml-org/llama.cpp/issues/25187#issuecomment-6731776833