Cyronius commited on
Commit
6130d86
·
verified ·
1 Parent(s): 3aeec22

Lead with one results table across the Qwen3.5 sizes

Browse files
Files changed (1) hide show
  1. README.md +53 -50
README.md CHANGED
@@ -19,6 +19,22 @@ The code is on GitHub at **[Cyronius/memdraft](https://github.com/Cyronius/memdr
19
 
20
  MemDraft needs a drafter made for your model: a lookup table (`.mdt`) and, optionally, a reranker (`.mdr`). MemDraft does not build them while serving. They are built once, offline: the target model first answers many prompts, then scripts count those answers into a table and train the reranker. The drafter behind the results below took about 40M generated tokens, most of them from 14 hours on one rented A40.
21
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
22
  ## Drafters
23
 
24
  Ready-made drafters are on Hugging Face at **[Cyronius/memdraft](https://huggingface.co/Cyronius/memdraft)**, one folder per target model. For any other model, [build one](#building-a-table).
@@ -42,7 +58,7 @@ The table counts 77.5M tokens of text, keeping contexts seen at least 3 times (1
42
 
43
  The reranker was fine-tuned on positions from the 9B's code. It includes 64-wide projections of the 9B's token embeddings.
44
 
45
- - **Compatibility.** The table works with any model that uses the same tokenizer: the Qwen3.5, 3.6 and 3.8 family, 248,070 tokens. `vocab_hash` makes the plugin refuse any other tokenizer. Unchanged, the drafter gives 1.69× to 1.85× on code with Qwen3.5 0.8B, 2B, 4B and 9B, and the reranker helps on all four, although it was trained against the 9B. See [Other Qwen3.5 sizes](#other-qwen35-sizes). The 27B and 35B-A3B have not been measured.
46
 
47
  ## How it works
48
 
@@ -70,70 +86,57 @@ The 2B draft model gets more tokens per step than MemDraft, but it is the slowes
70
  - **Smaller targets.** A smaller target takes less time per step, so a drafter that runs a model takes a larger share of each step. MemDraft's cost stays in microseconds at any target size. On Qwen3.5-0.8B, at 244 tok/s plain, MemDraft still gives 1.85× on code. Its step cost rises from 1.19× on the 9B to 1.38×; a draft model's step cost (2.36× on the 9B) would leave nothing.
71
  - **No VRAM.** The table and reranker load into system RAM (380 MiB for the Qwen3.5-9B drafter) and run on the CPU. A draft model needs VRAM for its weights and its own KV cache, and in the results the MTP head also ran on the GPU. The target still checks MemDraft's guesses on the GPU.
72
 
73
- ## Results
 
 
 
 
74
 
75
- These numbers are for Qwen3.5-9B Q4_K_M on an AMD Radeon 8060S through Vulkan, with greedy decoding and no other process on the GPU, using the [Qwen3.5-9B drafter](#the-qwen35-9b-drafter). The coding prompts below are held-out OSS-Instruct prompts: no table holds answers to them, and the reranker never saw them.
76
 
77
- - **Coding prompts:** 12 OSS-Instruct prompts, 2 rounds.
78
- - **HumanEval:** problems 0–11.
79
- - Each figure is generated tokens per second, with the speedup over plain decoding on the same prompts in parentheses.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
80
 
81
  | Setup | Coding prompts | HumanEval |
82
  |---|---:|---:|
83
  | Plain decoding | 38.0 tok/s (1.00×) | 38.2 tok/s (1.00×) |
84
- | **MemDraft alone**¹ | **69.4 tok/s (1.82×)** | **88.2 tok/s (2.31×)** |
85
  | `ngram-simple` alone¹ | 43.3 tok/s (1.13×) | 73.7 tok/s (1.93×) |
86
- | **`ngram-simple`, then MemDraft** | **70.5 tok/s (1.85×)** | **125.5 tok/s (3.29×)** |
87
  | The MTP head alone (`draft-mtp`, 3 tokens) | 73.9 tok/s (1.94×) | 88.0 tok/s (2.30×) |
88
  | **`ngram-simple`, then the MTP head, then MemDraft** | **81.6 tok/s (2.14×)** | **136.3 tok/s (3.57×)** |
 
 
89
 
90
- Rows without a mark are from 2026-11-18. ¹ From 2026-11-21, with the same build and prompts. In those runs, `ngram-simple`, then MemDraft gave 70.3 tok/s (1.84×) on coding prompts, matching 2026-11-18, but 112.8 tok/s (2.96×) on HumanEval, where its step cost was higher.
91
-
92
- Two other drafters were also tested and did worse than every setup above:
93
-
94
- - **`ngram-mod`**, the lookup drafter that llama.cpp's `--spec-default` turns on: 41.0 tok/s (1.08×) on coding prompts, and 67.8 tok/s (1.78×) on HumanEval in a run on 2026-11-19.
95
- - **Qwen3.5-2B as a separate draft model**, 3 tokens: 44.6 tok/s (1.17×) on coding prompts. On HumanEval only 2 problems completed, at 51.7 tok/s (1.36×).
96
 
97
- What the results show:
98
-
99
- - **On the coding prompts, nearly all the gain is MemDraft's**: 69.4 tok/s (1.82×) alone, 70.5 tok/s (1.85×) behind `ngram-simple`. On HumanEval, `ngram-simple` adds the long copies, taking MemDraft from 88.2 tok/s (2.31×) to 112.8 tok/s (2.96×) in the same run. HumanEval answers often restate code from the prompt, so the coding prompts are the more typical case.
100
- - **The MTP head alone beats MemDraft alone on the coding prompts** (73.9 tok/s (1.94×) against 69.4 tok/s (1.82×)). It accepts more tokens per step, but each step costs more because the head is a real model. Putting the head first and MemDraft after it is fastest.
101
  - With both chained MemDraft setups, every prompt in every round ran faster than plain decoding.
102
- - **Verification is lossless, but llama.cpp's batched decoding is not bit-exact.** The output text matched plain decoding on 11 of 12 HumanEval problems and on 8 of 24 coding-prompt runs. Where it differs, it splits at a near-tie between two tokens; plain decoding also changes this way when only the batch size changes.
103
- - **Answer quality held up on HumanEval.** Over all 164 problems, pass@1 was 83.5% against 84.8% for plain decoding: two problems flipped. That run was MemDraft alone (87.4 tok/s (2.29×) over the 164 problems, against 38.1 tok/s plain), with the same table and reranker, in the earlier patched build of llama.cpp.
104
- - **The setup is narrow so far.** It has been measured on Qwen3.5 0.8B to 9B, on code, on one GPU. On a mix of chat, math and editing prompts, an earlier table built from the 9B's chat and math text gave 40.2 tok/s (1.24×) overall against 32.4 tok/s plain, but only 35.3 tok/s (1.09×) on chat.
105
- - Raw results are in [`bench/results/`](https://github.com/Cyronius/memdraft/tree/main/bench/results); the rows marked ¹ are in `plugin-9b-vk-20261121-2038` (coding) and `plugin-9b-vk-20261121-1835` (HumanEval). How MemDraft got here, including what failed, is in [`docs/history.md`](https://github.com/Cyronius/memdraft/blob/main/docs/history.md).
106
-
107
- ### Other Qwen3.5 sizes
108
-
109
- The same drafter, unchanged, with the smaller Qwen3.5 models. Same GPU, prompts and settings as above, 3-token drafts, 2026-11-21. Each turn compares every setup with plain decoding on the same prompts.
110
-
111
- **Coding prompts:**
112
-
113
- | Model | Plain decoding | MemDraft alone | `ngram-simple`, then MemDraft | The same, no reranker | Acceptance¹ | Step cost¹ |
114
- |---|---:|---:|---:|---:|---:|---:|
115
- | 0.8B | 244.1 tok/s | 435.5 tok/s (1.78×) | **451.3 tok/s (1.85×)** | 412.6 tok/s (1.69×) | 41.7% | 1.38× |
116
- | 2B | 135.6 tok/s | 246.4 tok/s (1.82×) | **247.8 tok/s (1.83×)** | 226.6 tok/s (1.67×) | 37.2% | 1.27× |
117
- | 4B | 65.1 tok/s | **110.4 tok/s (1.70×)** | 110.1 tok/s (1.69×) | 104.5 tok/s (1.61×) | 33.4% | 1.27× |
118
- | 9B | 38.2 tok/s | 69.4 tok/s (1.82×) | **70.3 tok/s (1.84×)** | 66.2 tok/s (1.74×) | 35.2% | 1.19× |
119
-
120
- **HumanEval:**
121
 
122
- | Model | Plain decoding | MemDraft alone | `ngram-simple`, then MemDraft | The same, no reranker |
123
- |---|---:|---:|---:|---:|
124
- | 0.8B | 246.8 tok/s | 505.7 tok/s (2.05×) | 625.7 tok/s (2.54×) | **640.0 tok/s (2.59×)** |
125
- | 2B | 136.5 tok/s | 303.2 tok/s (2.22×) | **352.2 tok/s (2.58×)** | 335.5 tok/s (2.46×) |
126
- | 4B | 65.5 tok/s | 146.6 tok/s (2.24×) | **199.0 tok/s (3.04×)** | 189.1 tok/s (2.89×) |
127
- | 9B | 38.2 tok/s | 88.2 tok/s (2.31×) | 112.8 tok/s (2.96×) | **118.1 tok/s (3.09×)** |
128
 
129
- ¹ For `ngram-simple`, then MemDraft. Acceptance: draft tokens accepted over draft tokens proposed. Step cost: the time of one step in plain decoding steps.
 
 
130
 
131
- - **The drafter carries across sizes.** On code it gives 1.69× to 1.85× at every size, though its table holds only the 9B's text. The 0.8B accepts more of it than the 9B does (41.7% against 35.2%).
132
- - **On ordinary code, `ngram-simple` adds nothing.** MemDraft alone ties the chain on the 2B, 4B and 9B. On HumanEval, where answers copy from the prompt, the chain is well ahead at every size.
133
- - **The reranker pays on code at every size** (5% to 10%), though it was trained against the 9B. On HumanEval it helps the 2B and 4B and loses a little on the 0.8B and 9B, all within the 95% intervals.
134
- - **Step cost rises as the target shrinks**, from 1.19× on the 9B to 1.38× on the 0.8B, as [the cost model](#why-a-lookup-drafter) predicts. MemDraft's own microseconds do not change; the target's check of 4 tokens weighs more on a fast model.
135
- - **CPU load matters.** MemDraft runs on the CPU and plain decoding on the GPU, so a busy CPU slows only MemDraft. A first 4B run at a load average of 8 gave 1.56× on code; the rerun on a quiet machine, with the same accepted tokens, gave 1.69×. At the start of each turn the load average was 0.1 to 3.7, except for the 9B coding turns (5.3 to 7.0, yet plain decoding and the chain both matched 2026-11-18), the 9B HumanEval turn (4.4, which may explain its lower 2.96×) and the 4B HumanEval turn (5.3).
136
- - Raw results: `bench/results/plugin-{0_8b,2b,4b,9b}-vk-20261121-*`. `bench/run_qwen_family.sh` reruns them.
137
 
138
  ## Quick start
139
 
 
19
 
20
  MemDraft needs a drafter made for your model: a lookup table (`.mdt`) and, optionally, a reranker (`.mdr`). MemDraft does not build them while serving. They are built once, offline: the target model first answers many prompts, then scripts count those answers into a table and train the reranker. The drafter behind the results below took about 40M generated tokens, most of them from 14 hours on one rented A40.
21
 
22
+ ## Results
23
+
24
+ One drafter, built from Qwen3.5-9B's own text, used unchanged on four Qwen3.5 sizes:
25
+
26
+ | Model | Plain decoding | Coding prompts | HumanEval |
27
+ |---|---:|---:|---:|
28
+ | Qwen3.5-0.8B | 244 tok/s | 451 tok/s (1.85×) | 626 tok/s (2.54×) |
29
+ | Qwen3.5-2B | 136 tok/s | 248 tok/s (1.83×) | 352 tok/s (2.58×) |
30
+ | Qwen3.5-4B | 65 tok/s | 110 tok/s (1.69×) | 199 tok/s (3.04×) |
31
+ | Qwen3.5-9B | 38 tok/s | 70 tok/s (1.84×) | 113 tok/s (2.96×) |
32
+
33
+ - **Setup:** `ngram-simple`, then MemDraft with the reranker, drafting 3 tokens. Q4_K_M models on an AMD Radeon 8060S through Vulkan, greedy decoding, 2026-11-21.
34
+ - **Prompts:** 12 held-out OSS-Instruct coding prompts (2 rounds) and HumanEval problems 0–11. No table holds answers to them. The speedup is over plain decoding on the same prompts; plain decoding runs within about 1% of the same speed on both sets.
35
+ - **Output:** the target model checks every drafted token, so the text is the target's own. See [Quality](#quality).
36
+ - **Limits:** Qwen3.5 only, code only, one GPU. The 27B and 35B-A3B are not measured yet. On chat, an earlier table gave only 1.09×.
37
+
38
  ## Drafters
39
 
40
  Ready-made drafters are on Hugging Face at **[Cyronius/memdraft](https://huggingface.co/Cyronius/memdraft)**, one folder per target model. For any other model, [build one](#building-a-table).
 
58
 
59
  The reranker was fine-tuned on positions from the 9B's code. It includes 64-wide projections of the 9B's token embeddings.
60
 
61
+ - **Compatibility.** The table works with any model that uses the same tokenizer: the Qwen3.5, 3.6 and 3.8 family, 248,070 tokens. `vocab_hash` makes the plugin refuse any other tokenizer. It has been measured with Qwen3.5 0.8B, 2B, 4B and 9B (see [Results](#results)), and the reranker helps on all four, although it was trained against the 9B. The 27B and 35B-A3B have not been measured.
62
 
63
  ## How it works
64
 
 
86
  - **Smaller targets.** A smaller target takes less time per step, so a drafter that runs a model takes a larger share of each step. MemDraft's cost stays in microseconds at any target size. On Qwen3.5-0.8B, at 244 tok/s plain, MemDraft still gives 1.85× on code. Its step cost rises from 1.19× on the 9B to 1.38×; a draft model's step cost (2.36× on the 9B) would leave nothing.
87
  - **No VRAM.** The table and reranker load into system RAM (380 MiB for the Qwen3.5-9B drafter) and run on the CPU. A draft model needs VRAM for its weights and its own KV cache, and in the results the MTP head also ran on the GPU. The target still checks MemDraft's guesses on the GPU.
88
 
89
+ ## Detailed results
90
+
91
+ Settings and prompts are as in [Results](#results). Raw results are in [`bench/results/`](https://github.com/Cyronius/memdraft/tree/main/bench/results); `bench/run_qwen_family.sh` reruns the size comparison.
92
+
93
+ ### Which setup to use
94
 
95
+ Speedup over plain decoding, all sizes, 2026-11-21:
96
 
97
+ | Model | MemDraft alone | `ngram-simple`, then MemDraft | The same, no reranker |
98
+ |---|---:|---:|---:|
99
+ | **Coding prompts** | | | |
100
+ | 0.8B | 1.78× | 1.85× | 1.69× |
101
+ | 2B | 1.82× | 1.83× | 1.67× |
102
+ | 4B | 1.70× | 1.69× | 1.61× |
103
+ | 9B | 1.82× | 1.84× | 1.74× |
104
+ | **HumanEval** | | | |
105
+ | 0.8B | 2.05× | 2.54× | 2.59× |
106
+ | 2B | 2.22× | 2.58× | 2.46× |
107
+ | 4B | 2.24× | 3.04× | 2.89× |
108
+ | 9B | 2.31× | 2.96× | 3.09× |
109
+
110
+ - **`ngram-simple` pays only when answers copy from the prompt.** On the coding prompts, MemDraft alone ties the chain on the 2B, 4B and 9B. On HumanEval, whose answers often restate code from the prompt, the chain is well ahead at every size.
111
+ - **The reranker pays on code at every size**, by 5% to 10%. On HumanEval it helps the 2B and 4B and loses a little on the 0.8B and 9B, all within the 95% intervals.
112
+ - **Smaller models accept more but pay more per step.** On the coding prompts, the chain's acceptance (draft tokens accepted over draft tokens proposed) is 41.7%, 37.2%, 33.4% and 35.2% from 0.8B to 9B. Its step cost, in plain decoding steps, is 1.38×, 1.27×, 1.27× and 1.19×: the target's check of 4 tokens weighs more on a fast model, as [the cost model](#why-a-lookup-drafter) predicts.
113
+ - **CPU load matters.** MemDraft runs on the CPU and plain decoding on the GPU, so a busy CPU slows only MemDraft. A first 4B run at a load average of 8 gave 1.56× on code; the rerun on a quiet machine, with the same accepted tokens, gave 1.69×. Turns started at a load average of 0.1 to 3.7, except the 9B coding turns (5.3 to 7.0, yet plain decoding and the chain both matched 2026-11-18), the 9B HumanEval turn (4.4) and the 4B HumanEval turn (5.3).
114
+
115
+ ### Against other drafters (Qwen3.5-9B)
116
 
117
  | Setup | Coding prompts | HumanEval |
118
  |---|---:|---:|
119
  | Plain decoding | 38.0 tok/s (1.00×) | 38.2 tok/s (1.00×) |
120
+ | MemDraft alone¹ | 69.4 tok/s (1.82×) | 88.2 tok/s (2.31×) |
121
  | `ngram-simple` alone¹ | 43.3 tok/s (1.13×) | 73.7 tok/s (1.93×) |
122
+ | `ngram-simple`, then MemDraft | 70.5 tok/s (1.85×) | 125.5 tok/s (3.29×) |
123
  | The MTP head alone (`draft-mtp`, 3 tokens) | 73.9 tok/s (1.94×) | 88.0 tok/s (2.30×) |
124
  | **`ngram-simple`, then the MTP head, then MemDraft** | **81.6 tok/s (2.14×)** | **136.3 tok/s (3.57×)** |
125
+ | `ngram-mod`² | 41.0 tok/s (1.08×) | 67.8 tok/s (1.78×) |
126
+ | Qwen3.5-2B as a separate draft model, 3 tokens³ | 44.6 tok/s (1.17×) | 51.7 tok/s (1.36×) |
127
 
128
+ Rows without a mark are from 2026-11-18. ¹ From 2026-11-21, the run in [Results](#results), where `ngram-simple`, then MemDraft gave 70.3 tok/s (1.84×) on coding prompts and 113 tok/s (2.96×) on HumanEval. ² The lookup drafter that llama.cpp's `--spec-default` turns on; HumanEval from 2026-11-19. ³ On HumanEval only 2 problems completed.
 
 
 
 
 
129
 
130
+ - **The MTP head alone beats MemDraft alone on the coding prompts** (1.94× against 1.82×). It accepts more tokens per step, but each step costs more because the head is a real model. Putting the head first and MemDraft after it is fastest.
 
 
 
131
  - With both chained MemDraft setups, every prompt in every round ran faster than plain decoding.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
132
 
133
+ ### Quality
 
 
 
 
 
134
 
135
+ - **Verification is lossless, but llama.cpp's batched decoding is not bit-exact.** On the 9B, the output text matched plain decoding on 11 of 12 HumanEval problems and on 8 of 24 coding-prompt runs. Where it differs, it splits at a near-tie between two tokens; plain decoding also changes this way when only the batch size changes.
136
+ - **Answer quality held up on HumanEval.** Over all 164 problems with the 9B, pass@1 was 83.5% against 84.8% for plain decoding: two problems flipped. That run was MemDraft alone (87.4 tok/s (2.29×) over the 164 problems, against 38.1 tok/s plain), with the same table and reranker, in an earlier patched build of llama.cpp.
137
+ - **Chat is weak.** On a mix of chat, math and editing prompts, an earlier table built from the 9B's chat and math text gave 40.2 tok/s (1.24×) overall against 32.4 tok/s plain, but only 35.3 tok/s (1.09×) on chat.
138
 
139
+ How MemDraft got here, including what failed, is in [`docs/history.md`](https://github.com/Cyronius/memdraft/blob/main/docs/history.md).
 
 
 
 
 
140
 
141
  ## Quick start
142