Cyronius commited on
Commit
57a1dcf
·
verified ·
1 Parent(s): f98ff40

Drop predicted speeds

Browse files
Files changed (1) hide show
  1. README.md +187 -188
README.md CHANGED
@@ -11,191 +11,190 @@ tags:
11
  - code
12
  ---
13
 
14
- # MemDraft
15
-
16
- MemDraft is a speculative-decoding method for llama.cpp that doesn't need a separate draft model. It proposes the next tokens from a lookup table, and the target model checks the guesses as in any speculative decoding. A draft step costs microseconds, so even guesses that are right only some of the time make decoding faster.
17
-
18
- The code is on GitHub at **[Cyronius/memdraft](https://github.com/Cyronius/memdraft)**. It builds as a plugin, `libmemdraft.so`, loaded by llama.cpp's `--spec-type plugin`. That option comes from a small patch to llama.cpp in the GitHub repo.
19
-
20
- MemDraft needs a drafter made for your model: a lookup table (`.mdt`) and, optionally, a reranker (`.mdr`). MemDraft does not build them while serving. They are built once, offline: the target model first answers many prompts, then scripts count those answers into a table and train the reranker. The drafter behind the results below took about 40M generated tokens, most of them from 14 hours on one rented A40.
21
-
22
- ## Drafters
23
-
24
- Ready-made drafters are on Hugging Face at **[Cyronius/memdraft](https://huggingface.co/Cyronius/memdraft)**, one folder per target model. For any other model, [build one](#building-a-table).
25
-
26
- | Target model | Domain | Folder | Table | Reranker | Vocabulary hash |
27
- |---|---|---|---:|---:|---|
28
- | [Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) | Code | [`qwen3.5-9b/`](https://huggingface.co/Cyronius/memdraft/tree/main/qwen3.5-9b) | 256 MiB | 124 MiB | `c52114e74bc3a20f` |
29
-
30
- | File | SHA-256 |
31
- |---|---|
32
- | `qwen3.5-9b/code.mdt` | `5d835a951a03e86f3cda826738e14f3ea1d38b06487f567a5afe80af7eebf3db` |
33
- | `qwen3.5-9b/code.mdr` | `6458dcf9b905fb0b28338b19abb99a6ce42b73681063f83858860fd234c84fd7` |
34
-
35
- ### The Qwen3.5-9B drafter
36
-
37
- The table counts 77.5M tokens of text, keeping contexts seen at least 3 times (14.8M entries):
38
-
39
- - **Code the 9B wrote:** answers to about 74,000 prompts from [Magicoder-OSS-Instruct-75K](https://huggingface.co/datasets/ise-uiuc/Magicoder-OSS-Instruct-75K), at temperature 0 with thinking off.
40
- - **That dataset's own gpt-3.5 solutions**, except for the held-out prompts.
41
- - **Chat and math text the 9B wrote:** answers to prompts from GSM8K, MATH, OpenThoughts and UltraChat.
42
-
43
- The reranker was fine-tuned on positions from the 9B's code. It includes 64-wide projections of the 9B's token embeddings.
44
-
45
- - **Compatibility.** The table works with any model that uses the same tokenizer: the Qwen3.5, 3.6 and 3.8 family, 248,070 tokens. `vocab_hash` makes the plugin refuse any other tokenizer. The table has only been measured with the 9B. The reranker was trained against the 9B; it loads with other models in the family, but it has not been measured there.
46
- - **Offline, on held-out answers.** Drafting 3 tokens: 2.01 tokens per step with the reranker and 1.86 without it. At plain decoding's 38.0 tok/s and the 9B's measured GPU step costs, that predicts 62.7 tok/s (1.65×) and 58.2 tok/s (1.53×).
47
-
48
- ## How it works
49
-
50
- At each step MemDraft proposes tokens from two lookups:
51
-
52
- 1. **The table.** For each of the last 1 to 6 tokens of context, it stores the token the model most often wrote next and how often that guess was right on held-out text. MemDraft loads it at startup and only reads it.
53
- 2. **The current request.** The same kind of lookup over the prompt and the tokens generated so far, built up as the request runs.
54
-
55
- An optional reranker picks among the proposals. It is a small MLP over the proposals and the target's token embeddings, trained offline alongside the table and stored in a `.mdr` file. MemDraft always drafts the same number of tokens (`pad=1`), so llama.cpp's compute graph keeps its shape and is never rebuilt. A draft call takes about 60 µs with the reranker and 3 µs without it.
56
-
57
- It works best behind llama.cpp's `ngram-simple` drafter. `ngram-simple` also needs no draft model, and no table either: when the last 12 tokens appeared earlier in the same request, it drafts up to 48 tokens that followed that earlier match. It catches long copies, such as an answer that repeats code from the prompt. When it finds no match, MemDraft drafts. Optionally, the model's own multi-token prediction (MTP) head drafts the first token and MemDraft continues from it. The MTP head is a small extra layer that some models ship with (Qwen3.5 does) to predict tokens past the next one.
58
-
59
- ## Why a lookup drafter
60
-
61
- A drafter helps only when it is cheap next to the target. The speedup is about the tokens accepted per step divided by the cost of a step, so a drafter that guesses well but runs slowly gives back its gain. On the coding prompts with Qwen3.5-9B, with step cost relative to one plain decoding step:
62
-
63
- | Drafter (3 tokens) | Tokens per step | Step cost | Speed |
64
- |---|---:|---:|---:|
65
- | Qwen3.5-2B as a separate draft model | 2.77 | 2.36× | 44.6 tok/s (1.17×) |
66
- | The 9B's MTP head (`draft-mtp`) | 3.08 | 1.59× | 73.9 tok/s (1.94×) |
67
- | `ngram-simple`, then MemDraft | 2.19 | 1.18× | 70.5 tok/s (1.85×) |
68
-
69
- The 2B draft model gets more tokens per step than MemDraft, but it is the slowest setup. A MemDraft draft call takes 3 to 60 µs, against about 26 ms for one plain step of the 9B. Most of MemDraft's 1.18× step cost is the target checking 4 tokens instead of 1.
70
-
71
- - **Smaller targets.** A smaller target takes less time per step, so a drafter that runs a model takes a larger share of each step. MemDraft's cost stays in microseconds at any target size. This follows from the cost model; MemDraft has only been measured with the 9B.
72
- - **No VRAM.** The table and reranker load into system RAM (380 MiB for the Qwen3.5-9B drafter) and run on the CPU. A draft model needs VRAM for its weights and its own KV cache, and in the results the MTP head also ran on the GPU. The target still checks MemDraft's guesses on the GPU.
73
-
74
- ## Results
75
-
76
- These numbers are for Qwen3.5-9B Q4_K_M on an AMD Radeon 8060S through Vulkan, with greedy decoding and no other process on the GPU, using the [Qwen3.5-9B drafter](#the-qwen35-9b-drafter). The coding prompts below are held-out OSS-Instruct prompts: no table holds answers to them, and the reranker never saw them.
77
-
78
- - **Coding prompts:** 12 OSS-Instruct prompts, 2 rounds.
79
- - **HumanEval:** problems 0–11.
80
- - Each figure is generated tokens per second, with the speedup over plain decoding on the same prompts in parentheses.
81
-
82
- | Setup | Coding prompts | HumanEval |
83
- |---|---:|---:|
84
- | Plain decoding | 38.0 tok/s (1.00×) | 38.2 tok/s (1.00×) |
85
- | **MemDraft alone**¹ | **68.7 tok/s (1.81×)** | **91.1 tok/s (2.39×)** |
86
- | `ngram-simple` alone¹ | 43.1 tok/s (1.14×) | 74.3 tok/s (1.95×) |
87
- | **`ngram-simple`, then MemDraft** | **70.5 tok/s (1.85×)** | **125.5 tok/s (3.29×)** |
88
- | The MTP head alone (`draft-mtp`, 3 tokens) | 73.9 tok/s (1.94×) | 88.0 tok/s (2.30×) |
89
- | **`ngram-simple`, then the MTP head, then MemDraft** | **81.6 tok/s (2.14×)** | **136.3 tok/s (3.57×)** |
90
-
91
- Rows without a mark are from 2026-11-18. ¹ From 2026-11-17, in the earlier patched build of llama.cpp, with one round of the coding prompts and HumanEval problems 0–35. The patched build and the plugin ran at the same speed for `ngram-simple`, then MemDraft: 70.1 tok/s (1.84×) and 70.5 tok/s (1.85×).
92
-
93
- Two other drafters were also tested and did worse than every setup above:
94
-
95
- - **`ngram-mod`**, the lookup drafter that llama.cpp's `--spec-default` turns on: 41.0 tok/s (1.08×) on coding prompts, and 67.8 tok/s (1.78×) on HumanEval in a run on 2026-11-19.
96
- - **Qwen3.5-2B as a separate draft model**, 3 tokens: 44.6 tok/s (1.17×) on coding prompts. On HumanEval only 2 problems completed, at 51.7 tok/s (1.36×).
97
-
98
- What the results show:
99
-
100
- - **On the coding prompts, nearly all the gain is MemDraft's**: 68.7 tok/s (1.81×) alone, 70.5 tok/s (1.85×) behind `ngram-simple`. On HumanEval, `ngram-simple` adds the long copies, taking MemDraft from 91.1 tok/s (2.39×) to 125.5 tok/s (3.29×). HumanEval answers often restate code from the prompt, so the coding prompts are the more typical case.
101
- - **The MTP head alone beats MemDraft alone on the coding prompts** (73.9 tok/s (1.94×) against 68.7 tok/s (1.81×)). It accepts more tokens per step, but each step costs more because the head is a real model. Putting the head first and MemDraft after it is fastest.
102
- - With both chained MemDraft setups, every prompt in every round ran faster than plain decoding.
103
- - **Verification is lossless, but llama.cpp's batched decoding is not bit-exact.** The output text matched plain decoding on 11 of 12 HumanEval problems and on 8 of 24 coding-prompt runs. Where it differs, it splits at a near-tie between two tokens; plain decoding also changes this way when only the batch size changes.
104
- - **Answer quality held up on HumanEval.** Over all 164 problems, pass@1 was 83.5% against 84.8% for plain decoding: two problems flipped. That run was MemDraft alone (87.4 tok/s (2.29×) over the 164 problems, against 38.1 tok/s plain), with the same table and reranker, in the earlier patched build of llama.cpp.
105
- - **The setup is narrow so far.** It has been measured on one model, on code, on one GPU. On a mix of chat, math and editing prompts, an earlier table built from the 9B's chat and math text gave 40.2 tok/s (1.24×) overall against 32.4 tok/s plain, but only 35.3 tok/s (1.09×) on chat.
106
- - Raw results are in [`bench/results/`](https://github.com/Cyronius/memdraft/tree/main/bench/results); the rows marked ¹ are in [`research/results/prior-9b-vk-20261117-1614/`](https://github.com/Cyronius/memdraft/tree/main/research/results/prior-9b-vk-20261117-1614). How MemDraft got here, including what failed, is in [`docs/history.md`](https://github.com/Cyronius/memdraft/blob/main/docs/history.md).
107
-
108
- ## Quick start
109
-
110
- You need:
111
-
112
- - llama.cpp at commit `edec462`
113
- - the [MemDraft code](https://github.com/Cyronius/memdraft)
114
- - a drafter for your model: a table (`.mdt`) and, optionally, a reranker (`.mdr`). See [Drafters](#drafters).
115
-
116
- The commands below use the Qwen3.5-9B drafter from the results.
117
-
118
- ```bash
119
- # 0. Get MemDraft and download the drafter
120
- git clone https://github.com/Cyronius/memdraft
121
- hf download Cyronius/memdraft --include 'qwen3.5-9b/*' --local-dir drafters
122
-
123
- # 1. Patch and build llama.cpp
124
- git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp && git checkout edec462
125
- git apply /path/to/memdraft/patches/llama-plugin-hook.patch
126
- cmake -B build -DGGML_VULKAN=ON && cmake --build build -j # or the backend you use
127
-
128
- # 2. Build the plugin
129
- /path/to/memdraft/plugin/build.sh /path/to/llama.cpp # writes plugin/libmemdraft.so
130
-
131
- # 3. Serve: ngram-simple first, then MemDraft drafting 3 tokens
132
- llama-server -m Qwen3.5-9B-Q4_K_M.gguf -ngl 99 \
133
- --spec-type ngram-simple,plugin --spec-plugin-n-max 3 \
134
- --spec-plugin /path/to/memdraft/plugin/libmemdraft.so \
135
- --spec-plugin-args table=drafters/qwen3.5-9b/code.mdt,reranker=drafters/qwen3.5-9b/code.mdr,pad=1,vocab_hash=c52114e74bc3a20f
136
- ```
137
-
138
- To put the MTP head in front as well, add it as a draft model and let it draft the first token:
139
-
140
- ```bash
141
- --spec-type ngram-simple,draft-mtp,plugin --spec-draft-model HEAD.gguf --spec-draft-p-min 0 \
142
- --spec-plugin-prefix 1 --spec-plugin-n-max 4
143
- ```
144
-
145
- To make the head, export it from the Hugging Face weights with `convert_hf_to_gguf.py --mtp --outtype q8_0`. Then re-quantize it to Q4_K_M with a Q4_K output tensor (for example `llama-quantize --allow-requantize --output-tensor-type q4_k HEAD-q8_0.gguf HEAD.gguf Q4_K_M`). The head's output projection is most of its cost, so the smaller head is faster overall.
146
-
147
- ### Plugin options (`--spec-plugin-args`)
148
-
149
- | Key | Meaning |
150
- |---|---|
151
- | `table=FILE` | The lookup table (required) |
152
- | `reranker=FILE` | Pick among the proposals with this reranker |
153
- | `pad=0\|1` | Always draft `n_max` tokens, so the batch shape never changes (default 0; use 1) |
154
- | `tau=P` | Without `pad`: keep drafting while the chain's chance of full acceptance is at least P (default 0.3) |
155
- | `n_max=N` | Longest draft (default: whatever the host allows) |
156
- | `source=both\|table\|context` | Which lookups to use (default both) |
157
- | `vocab_hash=HEX` | Refuse to load unless the model's vocabulary matches. `python plugin/vocab_hash.py MODEL.gguf` prints the hash. |
158
-
159
- ## Building a table
160
-
161
- A table is counted from text the target model wrote itself. Its own output is worth far more per token than anyone else's. For code, the steps are:
162
-
163
- 1. **Generate.** `tools/gen_code.py` has a llama-server answer OSS-Instruct coding prompts (`ise-uiuc/Magicoder-OSS-Instruct-75K`) and stores the answers as token IDs. The table in the results used about 74,000 answers (39.5M tokens): 5,000 answered on a CPU and 69,000 on one rented A40 in 14 hours.
164
- 2. **Count and calibrate.** `tools/code_study.py --export` counts the table, calibrates its confidences on held-out answers and writes a `.mdt` file. `tools/build_table.py` does the same for general text. `--check` on either replays the C++ drafter against the Python one.
165
- 3. **Reranker (optional, about +5% tokens per step).** The reranker is trained with `research/reranker/rerank7.py`. `tools/export_reranker.py` then writes the final pair of files: the table in its version 2 format and the reranker. It also checks that the C++ reranker picks the same token as the Python one (it did at 99.975% of positions). Training is still research code; see [`research/`](https://github.com/Cyronius/memdraft/tree/main/research).
166
-
167
- The published Qwen3.5-9B table also contains 9B chat and math text from another project; that text itself is not published. Without it, `code_study.py --skip-shards` builds a code-only table.
168
-
169
- This README is also the Hugging Face model card. To add a drafter for another model:
170
-
171
- 1. Upload its `.mdt` and `.mdr` to a new folder in the Hugging Face repo.
172
- 2. Add a row to [Drafters](#drafters), with the SHA-256 of both files.
173
- 3. Add the model under `base_model` in `hf/card-metadata.yml`.
174
- 4. Publish the card: `bash hf/publish-card.sh`.
175
-
176
- ## Benchmarking
177
-
178
- `bench/run_plugin_live.sh` produced the results table. It runs plain decoding and each drafter setup on the same prompts, in turns of at most 9 minutes, only while no other process is on the GPU. `tools/show_wide.py` then summarizes the turns with confidence intervals.
179
-
180
- The script wraps each turn in `arbiter run …`, a local scheduler for shared GPUs; remove that prefix on another machine. The prompt sets are in [`bench/prompts/`](https://github.com/Cyronius/memdraft/tree/main/bench/prompts).
181
-
182
- ## Repository layout
183
-
184
- | Path | Contents |
185
- |---|---|
186
- | `src/memdraft.h` | The drafter: table, request index, reranker, chain rule. Header-only C++17, with no llama.cpp types. |
187
- | `plugin/` | The llama.cpp plugin, its build script, `vocab_hash.py`, and example plugins (`ngram-mod`, a draft model, an MTP head) with equivalence tests |
188
- | `patches/llama-plugin-hook.patch` | Adds `--spec-type plugin` to llama.cpp `edec462` |
189
- | `tools/` | Generating training text, building and checking tables, the benchmark client |
190
- | `bench/` | The benchmark script, prompt sets and results |
191
- | `hf/` | The Hugging Face card metadata and the script that publishes this README as the card |
192
- | `tests/` | `python -m pytest tests` |
193
- | `docs/history.md` | How the project got here, including the failed first attempt |
194
- | `research/` | Experiments that did not ship, or that led to the shipped setup, with their raw results |
195
-
196
- Large files live outside git. `data/` holds tables and generated rows. `$MEMDRAFT_CACHE` (default `~/.cache/model-experiments/memdraft`) holds llama.cpp builds, GGUFs, and the tokenizer and prompt dataset. Python dependencies are listed in `requirements.txt`.
197
-
198
- ## License
199
-
200
- - **Code:** MIT or Apache 2.0, at your option; see [LICENSE-MIT](https://github.com/Cyronius/memdraft/blob/main/LICENSE-MIT) and [LICENSE-APACHE](https://github.com/Cyronius/memdraft/blob/main/LICENSE-APACHE).
201
- - **Drafter files on Hugging Face:** Apache 2.0, the license of the Qwen3.5-9B weights they derive from.
 
11
  - code
12
  ---
13
 
14
+ # MemDraft
15
+
16
+ MemDraft is a speculative-decoding method for llama.cpp that doesn't need a separate draft model. It proposes the next tokens from a lookup table, and the target model checks the guesses as in any speculative decoding. A draft step costs microseconds, so even guesses that are right only some of the time make decoding faster.
17
+
18
+ The code is on GitHub at **[Cyronius/memdraft](https://github.com/Cyronius/memdraft)**. It builds as a plugin, `libmemdraft.so`, loaded by llama.cpp's `--spec-type plugin`. That option comes from a small patch to llama.cpp in the GitHub repo.
19
+
20
+ MemDraft needs a drafter made for your model: a lookup table (`.mdt`) and, optionally, a reranker (`.mdr`). MemDraft does not build them while serving. They are built once, offline: the target model first answers many prompts, then scripts count those answers into a table and train the reranker. The drafter behind the results below took about 40M generated tokens, most of them from 14 hours on one rented A40.
21
+
22
+ ## Drafters
23
+
24
+ Ready-made drafters are on Hugging Face at **[Cyronius/memdraft](https://huggingface.co/Cyronius/memdraft)**, one folder per target model. For any other model, [build one](#building-a-table).
25
+
26
+ | Target model | Domain | Folder | Table | Reranker | Vocabulary hash |
27
+ |---|---|---|---:|---:|---|
28
+ | [Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) | Code | [`qwen3.5-9b/`](https://huggingface.co/Cyronius/memdraft/tree/main/qwen3.5-9b) | 256 MiB | 124 MiB | `c52114e74bc3a20f` |
29
+
30
+ | File | SHA-256 |
31
+ |---|---|
32
+ | `qwen3.5-9b/code.mdt` | `5d835a951a03e86f3cda826738e14f3ea1d38b06487f567a5afe80af7eebf3db` |
33
+ | `qwen3.5-9b/code.mdr` | `6458dcf9b905fb0b28338b19abb99a6ce42b73681063f83858860fd234c84fd7` |
34
+
35
+ ### The Qwen3.5-9B drafter
36
+
37
+ The table counts 77.5M tokens of text, keeping contexts seen at least 3 times (14.8M entries):
38
+
39
+ - **Code the 9B wrote:** answers to about 74,000 prompts from [Magicoder-OSS-Instruct-75K](https://huggingface.co/datasets/ise-uiuc/Magicoder-OSS-Instruct-75K), at temperature 0 with thinking off.
40
+ - **That dataset's own gpt-3.5 solutions**, except for the held-out prompts.
41
+ - **Chat and math text the 9B wrote:** answers to prompts from GSM8K, MATH, OpenThoughts and UltraChat.
42
+
43
+ The reranker was fine-tuned on positions from the 9B's code. It includes 64-wide projections of the 9B's token embeddings.
44
+
45
+ - **Compatibility.** The table works with any model that uses the same tokenizer: the Qwen3.5, 3.6 and 3.8 family, 248,070 tokens. `vocab_hash` makes the plugin refuse any other tokenizer. The table has only been measured with the 9B. The reranker was trained against the 9B; it loads with other models in the family, but it has not been measured there.
46
+
47
+ ## How it works
48
+
49
+ At each step MemDraft proposes tokens from two lookups:
50
+
51
+ 1. **The table.** For each of the last 1 to 6 tokens of context, it stores the token the model most often wrote next and how often that guess was right on held-out text. MemDraft loads it at startup and only reads it.
52
+ 2. **The current request.** The same kind of lookup over the prompt and the tokens generated so far, built up as the request runs.
53
+
54
+ An optional reranker picks among the proposals. It is a small MLP over the proposals and the target's token embeddings, trained offline alongside the table and stored in a `.mdr` file. MemDraft always drafts the same number of tokens (`pad=1`), so llama.cpp's compute graph keeps its shape and is never rebuilt. A draft call takes about 60 µs with the reranker and 3 µs without it.
55
+
56
+ It works best behind llama.cpp's `ngram-simple` drafter. `ngram-simple` also needs no draft model, and no table either: when the last 12 tokens appeared earlier in the same request, it drafts up to 48 tokens that followed that earlier match. It catches long copies, such as an answer that repeats code from the prompt. When it finds no match, MemDraft drafts. Optionally, the model's own multi-token prediction (MTP) head drafts the first token and MemDraft continues from it. The MTP head is a small extra layer that some models ship with (Qwen3.5 does) to predict tokens past the next one.
57
+
58
+ ## Why a lookup drafter
59
+
60
+ A drafter helps only when it is cheap next to the target. The speedup is about the tokens accepted per step divided by the cost of a step, so a drafter that guesses well but runs slowly gives back its gain. On the coding prompts with Qwen3.5-9B, with step cost relative to one plain decoding step:
61
+
62
+ | Drafter (3 tokens) | Tokens per step | Step cost | Speed |
63
+ |---|---:|---:|---:|
64
+ | Qwen3.5-2B as a separate draft model | 2.77 | 2.36× | 44.6 tok/s (1.17×) |
65
+ | The 9B's MTP head (`draft-mtp`) | 3.08 | 1.59× | 73.9 tok/s (1.94×) |
66
+ | `ngram-simple`, then MemDraft | 2.19 | 1.18× | 70.5 tok/s (1.85×) |
67
+
68
+ The 2B draft model gets more tokens per step than MemDraft, but it is the slowest setup. A MemDraft draft call takes 3 to 60 µs, against about 26 ms for one plain step of the 9B. Most of MemDraft's 1.18× step cost is the target checking 4 tokens instead of 1.
69
+
70
+ - **Smaller targets.** A smaller target takes less time per step, so a drafter that runs a model takes a larger share of each step. MemDraft's cost stays in microseconds at any target size. This follows from the cost model; MemDraft has only been measured with the 9B.
71
+ - **No VRAM.** The table and reranker load into system RAM (380 MiB for the Qwen3.5-9B drafter) and run on the CPU. A draft model needs VRAM for its weights and its own KV cache, and in the results the MTP head also ran on the GPU. The target still checks MemDraft's guesses on the GPU.
72
+
73
+ ## Results
74
+
75
+ These numbers are for Qwen3.5-9B Q4_K_M on an AMD Radeon 8060S through Vulkan, with greedy decoding and no other process on the GPU, using the [Qwen3.5-9B drafter](#the-qwen35-9b-drafter). The coding prompts below are held-out OSS-Instruct prompts: no table holds answers to them, and the reranker never saw them.
76
+
77
+ - **Coding prompts:** 12 OSS-Instruct prompts, 2 rounds.
78
+ - **HumanEval:** problems 0–11.
79
+ - Each figure is generated tokens per second, with the speedup over plain decoding on the same prompts in parentheses.
80
+
81
+ | Setup | Coding prompts | HumanEval |
82
+ |---|---:|---:|
83
+ | Plain decoding | 38.0 tok/s (1.00×) | 38.2 tok/s (1.00×) |
84
+ | **MemDraft alone**¹ | **68.7 tok/s (1.81×)** | **91.1 tok/s (2.39×)** |
85
+ | `ngram-simple` alone¹ | 43.1 tok/s (1.14×) | 74.3 tok/s (1.95×) |
86
+ | **`ngram-simple`, then MemDraft** | **70.5 tok/s (1.85×)** | **125.5 tok/s (3.29×)** |
87
+ | The MTP head alone (`draft-mtp`, 3 tokens) | 73.9 tok/s (1.94×) | 88.0 tok/s (2.30×) |
88
+ | **`ngram-simple`, then the MTP head, then MemDraft** | **81.6 tok/s (2.14×)** | **136.3 tok/s (3.57×)** |
89
+
90
+ Rows without a mark are from 2026-11-18. ¹ From 2026-11-17, in the earlier patched build of llama.cpp, with one round of the coding prompts and HumanEval problems 0–35. The patched build and the plugin ran at the same speed for `ngram-simple`, then MemDraft: 70.1 tok/s (1.84×) and 70.5 tok/s (1.85×).
91
+
92
+ Two other drafters were also tested and did worse than every setup above:
93
+
94
+ - **`ngram-mod`**, the lookup drafter that llama.cpp's `--spec-default` turns on: 41.0 tok/s (1.08×) on coding prompts, and 67.8 tok/s (1.78×) on HumanEval in a run on 2026-11-19.
95
+ - **Qwen3.5-2B as a separate draft model**, 3 tokens: 44.6 tok/s (1.17×) on coding prompts. On HumanEval only 2 problems completed, at 51.7 tok/s (1.36×).
96
+
97
+ What the results show:
98
+
99
+ - **On the coding prompts, nearly all the gain is MemDraft's**: 68.7 tok/s (1.81×) alone, 70.5 tok/s (1.85×) behind `ngram-simple`. On HumanEval, `ngram-simple` adds the long copies, taking MemDraft from 91.1 tok/s (2.39×) to 125.5 tok/s (3.29×). HumanEval answers often restate code from the prompt, so the coding prompts are the more typical case.
100
+ - **The MTP head alone beats MemDraft alone on the coding prompts** (73.9 tok/s (1.94×) against 68.7 tok/s (1.81×)). It accepts more tokens per step, but each step costs more because the head is a real model. Putting the head first and MemDraft after it is fastest.
101
+ - With both chained MemDraft setups, every prompt in every round ran faster than plain decoding.
102
+ - **Verification is lossless, but llama.cpp's batched decoding is not bit-exact.** The output text matched plain decoding on 11 of 12 HumanEval problems and on 8 of 24 coding-prompt runs. Where it differs, it splits at a near-tie between two tokens; plain decoding also changes this way when only the batch size changes.
103
+ - **Answer quality held up on HumanEval.** Over all 164 problems, pass@1 was 83.5% against 84.8% for plain decoding: two problems flipped. That run was MemDraft alone (87.4 tok/s (2.29×) over the 164 problems, against 38.1 tok/s plain), with the same table and reranker, in the earlier patched build of llama.cpp.
104
+ - **The setup is narrow so far.** It has been measured on one model, on code, on one GPU. On a mix of chat, math and editing prompts, an earlier table built from the 9B's chat and math text gave 40.2 tok/s (1.24×) overall against 32.4 tok/s plain, but only 35.3 tok/s (1.09×) on chat.
105
+ - Raw results are in [`bench/results/`](https://github.com/Cyronius/memdraft/tree/main/bench/results); the rows marked ¹ are in [`research/results/prior-9b-vk-20261117-1614/`](https://github.com/Cyronius/memdraft/tree/main/research/results/prior-9b-vk-20261117-1614). How MemDraft got here, including what failed, is in [`docs/history.md`](https://github.com/Cyronius/memdraft/blob/main/docs/history.md).
106
+
107
+ ## Quick start
108
+
109
+ You need:
110
+
111
+ - llama.cpp at commit `edec462`
112
+ - the [MemDraft code](https://github.com/Cyronius/memdraft)
113
+ - a drafter for your model: a table (`.mdt`) and, optionally, a reranker (`.mdr`). See [Drafters](#drafters).
114
+
115
+ The commands below use the Qwen3.5-9B drafter from the results.
116
+
117
+ ```bash
118
+ # 0. Get MemDraft and download the drafter
119
+ git clone https://github.com/Cyronius/memdraft
120
+ hf download Cyronius/memdraft --include 'qwen3.5-9b/*' --local-dir drafters
121
+
122
+ # 1. Patch and build llama.cpp
123
+ git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp && git checkout edec462
124
+ git apply /path/to/memdraft/patches/llama-plugin-hook.patch
125
+ cmake -B build -DGGML_VULKAN=ON && cmake --build build -j # or the backend you use
126
+
127
+ # 2. Build the plugin
128
+ /path/to/memdraft/plugin/build.sh /path/to/llama.cpp # writes plugin/libmemdraft.so
129
+
130
+ # 3. Serve: ngram-simple first, then MemDraft drafting 3 tokens
131
+ llama-server -m Qwen3.5-9B-Q4_K_M.gguf -ngl 99 \
132
+ --spec-type ngram-simple,plugin --spec-plugin-n-max 3 \
133
+ --spec-plugin /path/to/memdraft/plugin/libmemdraft.so \
134
+ --spec-plugin-args table=drafters/qwen3.5-9b/code.mdt,reranker=drafters/qwen3.5-9b/code.mdr,pad=1,vocab_hash=c52114e74bc3a20f
135
+ ```
136
+
137
+ To put the MTP head in front as well, add it as a draft model and let it draft the first token:
138
+
139
+ ```bash
140
+ --spec-type ngram-simple,draft-mtp,plugin --spec-draft-model HEAD.gguf --spec-draft-p-min 0 \
141
+ --spec-plugin-prefix 1 --spec-plugin-n-max 4
142
+ ```
143
+
144
+ To make the head, export it from the Hugging Face weights with `convert_hf_to_gguf.py --mtp --outtype q8_0`. Then re-quantize it to Q4_K_M with a Q4_K output tensor (for example `llama-quantize --allow-requantize --output-tensor-type q4_k HEAD-q8_0.gguf HEAD.gguf Q4_K_M`). The head's output projection is most of its cost, so the smaller head is faster overall.
145
+
146
+ ### Plugin options (`--spec-plugin-args`)
147
+
148
+ | Key | Meaning |
149
+ |---|---|
150
+ | `table=FILE` | The lookup table (required) |
151
+ | `reranker=FILE` | Pick among the proposals with this reranker |
152
+ | `pad=0\|1` | Always draft `n_max` tokens, so the batch shape never changes (default 0; use 1) |
153
+ | `tau=P` | Without `pad`: keep drafting while the chain's chance of full acceptance is at least P (default 0.3) |
154
+ | `n_max=N` | Longest draft (default: whatever the host allows) |
155
+ | `source=both\|table\|context` | Which lookups to use (default both) |
156
+ | `vocab_hash=HEX` | Refuse to load unless the model's vocabulary matches. `python plugin/vocab_hash.py MODEL.gguf` prints the hash. |
157
+
158
+ ## Building a table
159
+
160
+ A table is counted from text the target model wrote itself. Its own output is worth far more per token than anyone else's. For code, the steps are:
161
+
162
+ 1. **Generate.** `tools/gen_code.py` has a llama-server answer OSS-Instruct coding prompts (`ise-uiuc/Magicoder-OSS-Instruct-75K`) and stores the answers as token IDs. The table in the results used about 74,000 answers (39.5M tokens): 5,000 answered on a CPU and 69,000 on one rented A40 in 14 hours.
163
+ 2. **Count and calibrate.** `tools/code_study.py --export` counts the table, calibrates its confidences on held-out answers and writes a `.mdt` file. `tools/build_table.py` does the same for general text. `--check` on either replays the C++ drafter against the Python one.
164
+ 3. **Reranker (optional, about +5% tokens per step).** The reranker is trained with `research/reranker/rerank7.py`. `tools/export_reranker.py` then writes the final pair of files: the table in its version 2 format and the reranker. It also checks that the C++ reranker picks the same token as the Python one (it did at 99.975% of positions). Training is still research code; see [`research/`](https://github.com/Cyronius/memdraft/tree/main/research).
165
+
166
+ The published Qwen3.5-9B table also contains 9B chat and math text from another project; that text itself is not published. Without it, `code_study.py --skip-shards` builds a code-only table.
167
+
168
+ This README is also the Hugging Face model card. To add a drafter for another model:
169
+
170
+ 1. Upload its `.mdt` and `.mdr` to a new folder in the Hugging Face repo.
171
+ 2. Add a row to [Drafters](#drafters), with the SHA-256 of both files.
172
+ 3. Add the model under `base_model` in `hf/card-metadata.yml`.
173
+ 4. Publish the card: `bash hf/publish-card.sh`.
174
+
175
+ ## Benchmarking
176
+
177
+ `bench/run_plugin_live.sh` produced the results table. It runs plain decoding and each drafter setup on the same prompts, in turns of at most 9 minutes, only while no other process is on the GPU. `tools/show_wide.py` then summarizes the turns with confidence intervals.
178
+
179
+ The script wraps each turn in `arbiter run …`, a local scheduler for shared GPUs; remove that prefix on another machine. The prompt sets are in [`bench/prompts/`](https://github.com/Cyronius/memdraft/tree/main/bench/prompts).
180
+
181
+ ## Repository layout
182
+
183
+ | Path | Contents |
184
+ |---|---|
185
+ | `src/memdraft.h` | The drafter: table, request index, reranker, chain rule. Header-only C++17, with no llama.cpp types. |
186
+ | `plugin/` | The llama.cpp plugin, its build script, `vocab_hash.py`, and example plugins (`ngram-mod`, a draft model, an MTP head) with equivalence tests |
187
+ | `patches/llama-plugin-hook.patch` | Adds `--spec-type plugin` to llama.cpp `edec462` |
188
+ | `tools/` | Generating training text, building and checking tables, the benchmark client |
189
+ | `bench/` | The benchmark script, prompt sets and results |
190
+ | `hf/` | The Hugging Face card metadata and the script that publishes this README as the card |
191
+ | `tests/` | `python -m pytest tests` |
192
+ | `docs/history.md` | How the project got here, including the failed first attempt |
193
+ | `research/` | Experiments that did not ship, or that led to the shipped setup, with their raw results |
194
+
195
+ Large files live outside git. `data/` holds tables and generated rows. `$MEMDRAFT_CACHE` (default `~/.cache/model-experiments/memdraft`) holds llama.cpp builds, GGUFs, and the tokenizer and prompt dataset. Python dependencies are listed in `requirements.txt`.
196
+
197
+ ## License
198
+
199
+ - **Code:** MIT or Apache 2.0, at your option; see [LICENSE-MIT](https://github.com/Cyronius/memdraft/blob/main/LICENSE-MIT) and [LICENSE-APACHE](https://github.com/Cyronius/memdraft/blob/main/LICENSE-APACHE).
200
+ - **Drafter files on Hugging Face:** Apache 2.0, the license of the Qwen3.5-9B weights they derive from.