Lead with one results table across the Qwen3.5 sizes
Browse files
README.md
CHANGED
|
@@ -19,6 +19,22 @@ The code is on GitHub at **[Cyronius/memdraft](https://github.com/Cyronius/memdr
|
|
| 19 |
|
| 20 |
MemDraft needs a drafter made for your model: a lookup table (`.mdt`) and, optionally, a reranker (`.mdr`). MemDraft does not build them while serving. They are built once, offline: the target model first answers many prompts, then scripts count those answers into a table and train the reranker. The drafter behind the results below took about 40M generated tokens, most of them from 14 hours on one rented A40.
|
| 21 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
## Drafters
|
| 23 |
|
| 24 |
Ready-made drafters are on Hugging Face at **[Cyronius/memdraft](https://huggingface.co/Cyronius/memdraft)**, one folder per target model. For any other model, [build one](#building-a-table).
|
|
@@ -42,7 +58,7 @@ The table counts 77.5M tokens of text, keeping contexts seen at least 3 times (1
|
|
| 42 |
|
| 43 |
The reranker was fine-tuned on positions from the 9B's code. It includes 64-wide projections of the 9B's token embeddings.
|
| 44 |
|
| 45 |
-
- **Compatibility.** The table works with any model that uses the same tokenizer: the Qwen3.5, 3.6 and 3.8 family, 248,070 tokens. `vocab_hash` makes the plugin refuse any other tokenizer.
|
| 46 |
|
| 47 |
## How it works
|
| 48 |
|
|
@@ -70,70 +86,57 @@ The 2B draft model gets more tokens per step than MemDraft, but it is the slowes
|
|
| 70 |
- **Smaller targets.** A smaller target takes less time per step, so a drafter that runs a model takes a larger share of each step. MemDraft's cost stays in microseconds at any target size. On Qwen3.5-0.8B, at 244 tok/s plain, MemDraft still gives 1.85× on code. Its step cost rises from 1.19× on the 9B to 1.38×; a draft model's step cost (2.36× on the 9B) would leave nothing.
|
| 71 |
- **No VRAM.** The table and reranker load into system RAM (380 MiB for the Qwen3.5-9B drafter) and run on the CPU. A draft model needs VRAM for its weights and its own KV cache, and in the results the MTP head also ran on the GPU. The target still checks MemDraft's guesses on the GPU.
|
| 72 |
|
| 73 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
| 74 |
|
| 75 |
-
|
| 76 |
|
| 77 |
-
|
| 78 |
-
-
|
| 79 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 80 |
|
| 81 |
| Setup | Coding prompts | HumanEval |
|
| 82 |
|---|---:|---:|
|
| 83 |
| Plain decoding | 38.0 tok/s (1.00×) | 38.2 tok/s (1.00×) |
|
| 84 |
-
|
|
| 85 |
| `ngram-simple` alone¹ | 43.3 tok/s (1.13×) | 73.7 tok/s (1.93×) |
|
| 86 |
-
|
|
| 87 |
| The MTP head alone (`draft-mtp`, 3 tokens) | 73.9 tok/s (1.94×) | 88.0 tok/s (2.30×) |
|
| 88 |
| **`ngram-simple`, then the MTP head, then MemDraft** | **81.6 tok/s (2.14×)** | **136.3 tok/s (3.57×)** |
|
|
|
|
|
|
|
| 89 |
|
| 90 |
-
Rows without a mark are from 2026-11-18. ¹ From 2026-11-21,
|
| 91 |
-
|
| 92 |
-
Two other drafters were also tested and did worse than every setup above:
|
| 93 |
-
|
| 94 |
-
- **`ngram-mod`**, the lookup drafter that llama.cpp's `--spec-default` turns on: 41.0 tok/s (1.08×) on coding prompts, and 67.8 tok/s (1.78×) on HumanEval in a run on 2026-11-19.
|
| 95 |
-
- **Qwen3.5-2B as a separate draft model**, 3 tokens: 44.6 tok/s (1.17×) on coding prompts. On HumanEval only 2 problems completed, at 51.7 tok/s (1.36×).
|
| 96 |
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
- **On the coding prompts, nearly all the gain is MemDraft's**: 69.4 tok/s (1.82×) alone, 70.5 tok/s (1.85×) behind `ngram-simple`. On HumanEval, `ngram-simple` adds the long copies, taking MemDraft from 88.2 tok/s (2.31×) to 112.8 tok/s (2.96×) in the same run. HumanEval answers often restate code from the prompt, so the coding prompts are the more typical case.
|
| 100 |
-
- **The MTP head alone beats MemDraft alone on the coding prompts** (73.9 tok/s (1.94×) against 69.4 tok/s (1.82×)). It accepts more tokens per step, but each step costs more because the head is a real model. Putting the head first and MemDraft after it is fastest.
|
| 101 |
- With both chained MemDraft setups, every prompt in every round ran faster than plain decoding.
|
| 102 |
-
- **Verification is lossless, but llama.cpp's batched decoding is not bit-exact.** The output text matched plain decoding on 11 of 12 HumanEval problems and on 8 of 24 coding-prompt runs. Where it differs, it splits at a near-tie between two tokens; plain decoding also changes this way when only the batch size changes.
|
| 103 |
-
- **Answer quality held up on HumanEval.** Over all 164 problems, pass@1 was 83.5% against 84.8% for plain decoding: two problems flipped. That run was MemDraft alone (87.4 tok/s (2.29×) over the 164 problems, against 38.1 tok/s plain), with the same table and reranker, in the earlier patched build of llama.cpp.
|
| 104 |
-
- **The setup is narrow so far.** It has been measured on Qwen3.5 0.8B to 9B, on code, on one GPU. On a mix of chat, math and editing prompts, an earlier table built from the 9B's chat and math text gave 40.2 tok/s (1.24×) overall against 32.4 tok/s plain, but only 35.3 tok/s (1.09×) on chat.
|
| 105 |
-
- Raw results are in [`bench/results/`](https://github.com/Cyronius/memdraft/tree/main/bench/results); the rows marked ¹ are in `plugin-9b-vk-20261121-2038` (coding) and `plugin-9b-vk-20261121-1835` (HumanEval). How MemDraft got here, including what failed, is in [`docs/history.md`](https://github.com/Cyronius/memdraft/blob/main/docs/history.md).
|
| 106 |
-
|
| 107 |
-
### Other Qwen3.5 sizes
|
| 108 |
-
|
| 109 |
-
The same drafter, unchanged, with the smaller Qwen3.5 models. Same GPU, prompts and settings as above, 3-token drafts, 2026-11-21. Each turn compares every setup with plain decoding on the same prompts.
|
| 110 |
-
|
| 111 |
-
**Coding prompts:**
|
| 112 |
-
|
| 113 |
-
| Model | Plain decoding | MemDraft alone | `ngram-simple`, then MemDraft | The same, no reranker | Acceptance¹ | Step cost¹ |
|
| 114 |
-
|---|---:|---:|---:|---:|---:|---:|
|
| 115 |
-
| 0.8B | 244.1 tok/s | 435.5 tok/s (1.78×) | **451.3 tok/s (1.85×)** | 412.6 tok/s (1.69×) | 41.7% | 1.38× |
|
| 116 |
-
| 2B | 135.6 tok/s | 246.4 tok/s (1.82×) | **247.8 tok/s (1.83×)** | 226.6 tok/s (1.67×) | 37.2% | 1.27× |
|
| 117 |
-
| 4B | 65.1 tok/s | **110.4 tok/s (1.70×)** | 110.1 tok/s (1.69×) | 104.5 tok/s (1.61×) | 33.4% | 1.27× |
|
| 118 |
-
| 9B | 38.2 tok/s | 69.4 tok/s (1.82×) | **70.3 tok/s (1.84×)** | 66.2 tok/s (1.74×) | 35.2% | 1.19× |
|
| 119 |
-
|
| 120 |
-
**HumanEval:**
|
| 121 |
|
| 122 |
-
|
| 123 |
-
|---|---:|---:|---:|---:|
|
| 124 |
-
| 0.8B | 246.8 tok/s | 505.7 tok/s (2.05×) | 625.7 tok/s (2.54×) | **640.0 tok/s (2.59×)** |
|
| 125 |
-
| 2B | 136.5 tok/s | 303.2 tok/s (2.22×) | **352.2 tok/s (2.58×)** | 335.5 tok/s (2.46×) |
|
| 126 |
-
| 4B | 65.5 tok/s | 146.6 tok/s (2.24×) | **199.0 tok/s (3.04×)** | 189.1 tok/s (2.89×) |
|
| 127 |
-
| 9B | 38.2 tok/s | 88.2 tok/s (2.31×) | 112.8 tok/s (2.96×) | **118.1 tok/s (3.09×)** |
|
| 128 |
|
| 129 |
-
|
|
|
|
|
|
|
| 130 |
|
| 131 |
-
|
| 132 |
-
- **On ordinary code, `ngram-simple` adds nothing.** MemDraft alone ties the chain on the 2B, 4B and 9B. On HumanEval, where answers copy from the prompt, the chain is well ahead at every size.
|
| 133 |
-
- **The reranker pays on code at every size** (5% to 10%), though it was trained against the 9B. On HumanEval it helps the 2B and 4B and loses a little on the 0.8B and 9B, all within the 95% intervals.
|
| 134 |
-
- **Step cost rises as the target shrinks**, from 1.19× on the 9B to 1.38× on the 0.8B, as [the cost model](#why-a-lookup-drafter) predicts. MemDraft's own microseconds do not change; the target's check of 4 tokens weighs more on a fast model.
|
| 135 |
-
- **CPU load matters.** MemDraft runs on the CPU and plain decoding on the GPU, so a busy CPU slows only MemDraft. A first 4B run at a load average of 8 gave 1.56× on code; the rerun on a quiet machine, with the same accepted tokens, gave 1.69×. At the start of each turn the load average was 0.1 to 3.7, except for the 9B coding turns (5.3 to 7.0, yet plain decoding and the chain both matched 2026-11-18), the 9B HumanEval turn (4.4, which may explain its lower 2.96×) and the 4B HumanEval turn (5.3).
|
| 136 |
-
- Raw results: `bench/results/plugin-{0_8b,2b,4b,9b}-vk-20261121-*`. `bench/run_qwen_family.sh` reruns them.
|
| 137 |
|
| 138 |
## Quick start
|
| 139 |
|
|
|
|
| 19 |
|
| 20 |
MemDraft needs a drafter made for your model: a lookup table (`.mdt`) and, optionally, a reranker (`.mdr`). MemDraft does not build them while serving. They are built once, offline: the target model first answers many prompts, then scripts count those answers into a table and train the reranker. The drafter behind the results below took about 40M generated tokens, most of them from 14 hours on one rented A40.
|
| 21 |
|
| 22 |
+
## Results
|
| 23 |
+
|
| 24 |
+
One drafter, built from Qwen3.5-9B's own text, used unchanged on four Qwen3.5 sizes:
|
| 25 |
+
|
| 26 |
+
| Model | Plain decoding | Coding prompts | HumanEval |
|
| 27 |
+
|---|---:|---:|---:|
|
| 28 |
+
| Qwen3.5-0.8B | 244 tok/s | 451 tok/s (1.85×) | 626 tok/s (2.54×) |
|
| 29 |
+
| Qwen3.5-2B | 136 tok/s | 248 tok/s (1.83×) | 352 tok/s (2.58×) |
|
| 30 |
+
| Qwen3.5-4B | 65 tok/s | 110 tok/s (1.69×) | 199 tok/s (3.04×) |
|
| 31 |
+
| Qwen3.5-9B | 38 tok/s | 70 tok/s (1.84×) | 113 tok/s (2.96×) |
|
| 32 |
+
|
| 33 |
+
- **Setup:** `ngram-simple`, then MemDraft with the reranker, drafting 3 tokens. Q4_K_M models on an AMD Radeon 8060S through Vulkan, greedy decoding, 2026-11-21.
|
| 34 |
+
- **Prompts:** 12 held-out OSS-Instruct coding prompts (2 rounds) and HumanEval problems 0–11. No table holds answers to them. The speedup is over plain decoding on the same prompts; plain decoding runs within about 1% of the same speed on both sets.
|
| 35 |
+
- **Output:** the target model checks every drafted token, so the text is the target's own. See [Quality](#quality).
|
| 36 |
+
- **Limits:** Qwen3.5 only, code only, one GPU. The 27B and 35B-A3B are not measured yet. On chat, an earlier table gave only 1.09×.
|
| 37 |
+
|
| 38 |
## Drafters
|
| 39 |
|
| 40 |
Ready-made drafters are on Hugging Face at **[Cyronius/memdraft](https://huggingface.co/Cyronius/memdraft)**, one folder per target model. For any other model, [build one](#building-a-table).
|
|
|
|
| 58 |
|
| 59 |
The reranker was fine-tuned on positions from the 9B's code. It includes 64-wide projections of the 9B's token embeddings.
|
| 60 |
|
| 61 |
+
- **Compatibility.** The table works with any model that uses the same tokenizer: the Qwen3.5, 3.6 and 3.8 family, 248,070 tokens. `vocab_hash` makes the plugin refuse any other tokenizer. It has been measured with Qwen3.5 0.8B, 2B, 4B and 9B (see [Results](#results)), and the reranker helps on all four, although it was trained against the 9B. The 27B and 35B-A3B have not been measured.
|
| 62 |
|
| 63 |
## How it works
|
| 64 |
|
|
|
|
| 86 |
- **Smaller targets.** A smaller target takes less time per step, so a drafter that runs a model takes a larger share of each step. MemDraft's cost stays in microseconds at any target size. On Qwen3.5-0.8B, at 244 tok/s plain, MemDraft still gives 1.85× on code. Its step cost rises from 1.19× on the 9B to 1.38×; a draft model's step cost (2.36× on the 9B) would leave nothing.
|
| 87 |
- **No VRAM.** The table and reranker load into system RAM (380 MiB for the Qwen3.5-9B drafter) and run on the CPU. A draft model needs VRAM for its weights and its own KV cache, and in the results the MTP head also ran on the GPU. The target still checks MemDraft's guesses on the GPU.
|
| 88 |
|
| 89 |
+
## Detailed results
|
| 90 |
+
|
| 91 |
+
Settings and prompts are as in [Results](#results). Raw results are in [`bench/results/`](https://github.com/Cyronius/memdraft/tree/main/bench/results); `bench/run_qwen_family.sh` reruns the size comparison.
|
| 92 |
+
|
| 93 |
+
### Which setup to use
|
| 94 |
|
| 95 |
+
Speedup over plain decoding, all sizes, 2026-11-21:
|
| 96 |
|
| 97 |
+
| Model | MemDraft alone | `ngram-simple`, then MemDraft | The same, no reranker |
|
| 98 |
+
|---|---:|---:|---:|
|
| 99 |
+
| **Coding prompts** | | | |
|
| 100 |
+
| 0.8B | 1.78× | 1.85× | 1.69× |
|
| 101 |
+
| 2B | 1.82× | 1.83× | 1.67× |
|
| 102 |
+
| 4B | 1.70× | 1.69× | 1.61× |
|
| 103 |
+
| 9B | 1.82× | 1.84× | 1.74× |
|
| 104 |
+
| **HumanEval** | | | |
|
| 105 |
+
| 0.8B | 2.05× | 2.54× | 2.59× |
|
| 106 |
+
| 2B | 2.22× | 2.58× | 2.46× |
|
| 107 |
+
| 4B | 2.24× | 3.04× | 2.89× |
|
| 108 |
+
| 9B | 2.31× | 2.96× | 3.09× |
|
| 109 |
+
|
| 110 |
+
- **`ngram-simple` pays only when answers copy from the prompt.** On the coding prompts, MemDraft alone ties the chain on the 2B, 4B and 9B. On HumanEval, whose answers often restate code from the prompt, the chain is well ahead at every size.
|
| 111 |
+
- **The reranker pays on code at every size**, by 5% to 10%. On HumanEval it helps the 2B and 4B and loses a little on the 0.8B and 9B, all within the 95% intervals.
|
| 112 |
+
- **Smaller models accept more but pay more per step.** On the coding prompts, the chain's acceptance (draft tokens accepted over draft tokens proposed) is 41.7%, 37.2%, 33.4% and 35.2% from 0.8B to 9B. Its step cost, in plain decoding steps, is 1.38×, 1.27×, 1.27× and 1.19×: the target's check of 4 tokens weighs more on a fast model, as [the cost model](#why-a-lookup-drafter) predicts.
|
| 113 |
+
- **CPU load matters.** MemDraft runs on the CPU and plain decoding on the GPU, so a busy CPU slows only MemDraft. A first 4B run at a load average of 8 gave 1.56× on code; the rerun on a quiet machine, with the same accepted tokens, gave 1.69×. Turns started at a load average of 0.1 to 3.7, except the 9B coding turns (5.3 to 7.0, yet plain decoding and the chain both matched 2026-11-18), the 9B HumanEval turn (4.4) and the 4B HumanEval turn (5.3).
|
| 114 |
+
|
| 115 |
+
### Against other drafters (Qwen3.5-9B)
|
| 116 |
|
| 117 |
| Setup | Coding prompts | HumanEval |
|
| 118 |
|---|---:|---:|
|
| 119 |
| Plain decoding | 38.0 tok/s (1.00×) | 38.2 tok/s (1.00×) |
|
| 120 |
+
| MemDraft alone¹ | 69.4 tok/s (1.82×) | 88.2 tok/s (2.31×) |
|
| 121 |
| `ngram-simple` alone¹ | 43.3 tok/s (1.13×) | 73.7 tok/s (1.93×) |
|
| 122 |
+
| `ngram-simple`, then MemDraft | 70.5 tok/s (1.85×) | 125.5 tok/s (3.29×) |
|
| 123 |
| The MTP head alone (`draft-mtp`, 3 tokens) | 73.9 tok/s (1.94×) | 88.0 tok/s (2.30×) |
|
| 124 |
| **`ngram-simple`, then the MTP head, then MemDraft** | **81.6 tok/s (2.14×)** | **136.3 tok/s (3.57×)** |
|
| 125 |
+
| `ngram-mod`² | 41.0 tok/s (1.08×) | 67.8 tok/s (1.78×) |
|
| 126 |
+
| Qwen3.5-2B as a separate draft model, 3 tokens³ | 44.6 tok/s (1.17×) | 51.7 tok/s (1.36×) |
|
| 127 |
|
| 128 |
+
Rows without a mark are from 2026-11-18. ¹ From 2026-11-21, the run in [Results](#results), where `ngram-simple`, then MemDraft gave 70.3 tok/s (1.84×) on coding prompts and 113 tok/s (2.96×) on HumanEval. ² The lookup drafter that llama.cpp's `--spec-default` turns on; HumanEval from 2026-11-19. ³ On HumanEval only 2 problems completed.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 129 |
|
| 130 |
+
- **The MTP head alone beats MemDraft alone on the coding prompts** (1.94× against 1.82×). It accepts more tokens per step, but each step costs more because the head is a real model. Putting the head first and MemDraft after it is fastest.
|
|
|
|
|
|
|
|
|
|
| 131 |
- With both chained MemDraft setups, every prompt in every round ran faster than plain decoding.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 132 |
|
| 133 |
+
### Quality
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 134 |
|
| 135 |
+
- **Verification is lossless, but llama.cpp's batched decoding is not bit-exact.** On the 9B, the output text matched plain decoding on 11 of 12 HumanEval problems and on 8 of 24 coding-prompt runs. Where it differs, it splits at a near-tie between two tokens; plain decoding also changes this way when only the batch size changes.
|
| 136 |
+
- **Answer quality held up on HumanEval.** Over all 164 problems with the 9B, pass@1 was 83.5% against 84.8% for plain decoding: two problems flipped. That run was MemDraft alone (87.4 tok/s (2.29×) over the 164 problems, against 38.1 tok/s plain), with the same table and reranker, in an earlier patched build of llama.cpp.
|
| 137 |
+
- **Chat is weak.** On a mix of chat, math and editing prompts, an earlier table built from the 9B's chat and math text gave 40.2 tok/s (1.24×) overall against 32.4 tok/s plain, but only 35.3 tok/s (1.09×) on chat.
|
| 138 |
|
| 139 |
+
How MemDraft got here, including what failed, is in [`docs/history.md`](https://github.com/Cyronius/memdraft/blob/main/docs/history.md).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 140 |
|
| 141 |
## Quick start
|
| 142 |
|