Cyronius commited on
Commit
792c8dc
·
verified ·
1 Parent(s): 52da732

Quote speeds as tok/s with the speedup in parentheses

Browse files
Files changed (1) hide show
  1. README.md +142 -34
README.md CHANGED
@@ -11,68 +11,176 @@ tags:
11
  - code
12
  ---
13
 
14
- # MemDraft drafters
15
 
16
- These are drafters for [MemDraft](https://github.com/Cyronius/memdraft), a speculative-decoding method for llama.cpp. MemDraft proposes tokens from a lookup table, so it doesn't need a separate draft model.
17
 
18
- Each drafter is built for one target model and has two files:
19
 
20
- - **`.mdt`**: the lookup table, built from text generated by the target model.
21
- - **`.mdr`**: a reranker, a small MLP that chooses among the table's proposals.
22
 
23
- Load them with MemDraft's llama.cpp plugin. The [GitHub repo](https://github.com/Cyronius/memdraft) explains how to build the plugin and patch llama.cpp.
24
 
25
- ## Available drafters
26
 
27
- | Target model | Folder | Domain | Table | Reranker | Vocabulary hash |
28
  |---|---|---|---:|---:|---|
29
- | [Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) | [`qwen3.5-9b/`](qwen3.5-9b/) | Code | 256 MiB | 124 MiB | `c52114e74bc3a20f` |
30
 
31
- More target models will be added here.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
32
 
33
- ## Qwen3.5-9B, code
34
 
35
- **Speed** with Qwen3.5-9B Q4_K_M on an AMD Radeon 8060S through Vulkan, greedy decoding, against plain decoding (38 tok/s):
36
 
37
- | Setup | Coding prompts (12 OSS-Instruct prompts, 2 rounds) | HumanEval (12 problems) |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
38
  |---|---:|---:|
39
- | `ngram-simple`, then MemDraft (3 tokens) | 1.85× | 3.29× |
40
- | `ngram-simple`, then the 9B's MTP head (1 token), then MemDraft (3 tokens) | 2.14× | 3.57× |
41
- | llama.cpp `draft-mtp` alone, for comparison | 1.94× | 2.30× |
 
 
 
42
 
43
- Verification is lossless. Over all 164 HumanEval problems, pass@1 was 83.5% with MemDraft and 84.8% with plain decoding. llama.cpp's batched decoding is not bit-exact, so a few outputs split at near-ties.
44
 
45
- **What went into it.** The table counts 77.5M tokens of text, keeping contexts seen at least 3 times (14.8M entries):
46
 
47
- - **Code the 9B wrote:** answers to about 74,000 prompts from [Magicoder-OSS-Instruct-75K](https://huggingface.co/datasets/ise-uiuc/Magicoder-OSS-Instruct-75K), at temperature 0 with thinking off.
48
- - **That dataset's own gpt-3.5 solutions.**
49
- - **Chat and math text the 9B wrote:** answers to prompts from GSM8K, MATH, OpenThoughts and UltraChat.
50
 
51
- The reranker was fine-tuned on positions from the 9B's code. It includes 64-wide projections of the 9B's token embeddings.
52
 
53
- **Compatibility.**
 
 
 
 
 
 
54
 
55
- - The table works with any model that uses this tokenizer: the Qwen3.5, 3.6 and 3.8 family, 248,070 tokens. `vocab_hash` makes the plugin refuse any other tokenizer.
56
- - The reranker was trained against the 9B. It loads with other models in the family, but it has not been measured there.
57
 
58
- **Held-out answers, offline.** Using the 9B's measured GPU step costs, drafting 3 tokens: 2.01 tokens per step and a predicted 1.65× with the reranker; 1.86 tokens per step and 1.53× without it.
59
 
60
- ### Usage
 
 
 
 
61
 
62
  ```bash
63
- hf download Cyronius/memdraft --include 'qwen3.5-9b/*' --local-dir memdraft-drafters
 
 
 
 
 
 
 
64
 
 
 
 
 
65
  llama-server -m Qwen3.5-9B-Q4_K_M.gguf -ngl 99 \
66
  --spec-type ngram-simple,plugin --spec-plugin-n-max 3 \
67
- --spec-plugin /path/to/libmemdraft.so \
68
- --spec-plugin-args table=memdraft-drafters/qwen3.5-9b/code.mdt,reranker=memdraft-drafters/qwen3.5-9b/code.mdr,pad=1,vocab_hash=c52114e74bc3a20f
69
  ```
70
 
71
- | File | SHA-256 |
 
 
 
 
 
 
 
 
 
 
 
72
  |---|---|
73
- | `qwen3.5-9b/code.mdt` | `5d835a951a03e86f3cda826738e14f3ea1d38b06487f567a5afe80af7eebf3db` |
74
- | `qwen3.5-9b/code.mdr` | `6458dcf9b905fb0b28338b19abb99a6ce42b73681063f83858860fd234c84fd7` |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
75
 
76
  ## License
77
 
78
- These files are released under Apache 2.0, the license of the Qwen3.5-9B weights they derive from. The MemDraft code is under the MIT license.
 
 
11
  - code
12
  ---
13
 
14
+ # MemDraft
15
 
16
+ MemDraft is a speculative-decoding method for llama.cpp that doesn't need a separate draft model. It proposes the next tokens from a lookup table, and the target model checks the guesses as in any speculative decoding. A draft step costs microseconds, so even guesses that are right only some of the time make decoding faster.
17
 
18
+ The code is on GitHub at **[Cyronius/memdraft](https://github.com/Cyronius/memdraft)**. It builds as a plugin, `libmemdraft.so`, loaded by llama.cpp's `--spec-type plugin`. That option comes from a small patch to llama.cpp in the GitHub repo.
19
 
20
+ MemDraft needs a drafter made for your model: a lookup table (`.mdt`) and, optionally, a reranker (`.mdr`). MemDraft does not build them while serving. They are built once, offline: the target model first answers many prompts, then scripts count those answers into a table and train the reranker. The drafter behind the results below took about 40M generated tokens, most of them from 14 hours on one rented A40.
 
21
 
22
+ ## Drafters
23
 
24
+ Ready-made drafters are on Hugging Face at **[Cyronius/memdraft](https://huggingface.co/Cyronius/memdraft)**, one folder per target model. For any other model, [build one](#building-a-table).
25
 
26
+ | Target model | Domain | Folder | Table | Reranker | Vocabulary hash |
27
  |---|---|---|---:|---:|---|
28
+ | [Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) | Code | [`qwen3.5-9b/`](https://huggingface.co/Cyronius/memdraft/tree/main/qwen3.5-9b) | 256 MiB | 124 MiB | `c52114e74bc3a20f` |
29
 
30
+ | File | SHA-256 |
31
+ |---|---|
32
+ | `qwen3.5-9b/code.mdt` | `5d835a951a03e86f3cda826738e14f3ea1d38b06487f567a5afe80af7eebf3db` |
33
+ | `qwen3.5-9b/code.mdr` | `6458dcf9b905fb0b28338b19abb99a6ce42b73681063f83858860fd234c84fd7` |
34
+
35
+ ### The Qwen3.5-9B drafter
36
+
37
+ The table counts 77.5M tokens of text, keeping contexts seen at least 3 times (14.8M entries):
38
+
39
+ - **Code the 9B wrote:** answers to about 74,000 prompts from [Magicoder-OSS-Instruct-75K](https://huggingface.co/datasets/ise-uiuc/Magicoder-OSS-Instruct-75K), at temperature 0 with thinking off.
40
+ - **That dataset's own gpt-3.5 solutions**, except for the held-out prompts.
41
+ - **Chat and math text the 9B wrote:** answers to prompts from GSM8K, MATH, OpenThoughts and UltraChat.
42
+
43
+ The reranker was fine-tuned on positions from the 9B's code. It includes 64-wide projections of the 9B's token embeddings.
44
+
45
+ - **Compatibility.** The table works with any model that uses the same tokenizer: the Qwen3.5, 3.6 and 3.8 family, 248,070 tokens. `vocab_hash` makes the plugin refuse any other tokenizer. The table has only been measured with the 9B. The reranker was trained against the 9B; it loads with other models in the family, but it has not been measured there.
46
+ - **Offline, on held-out answers.** Drafting 3 tokens: 2.01 tokens per step with the reranker and 1.86 without it. At plain decoding's 38.0 tok/s and the 9B's measured GPU step costs, that predicts 62.7 tok/s (1.65×) and 58.2 tok/s (1.53×).
47
 
48
+ ## How it works
49
 
50
+ At each step MemDraft proposes tokens from two lookups:
51
 
52
+ 1. **The table.** For each of the last 1 to 6 tokens of context, it stores the token the model most often wrote next and how often that guess was right on held-out text. MemDraft loads it at startup and only reads it.
53
+ 2. **The current request.** The same kind of lookup over the prompt and the tokens generated so far, built up as the request runs.
54
+
55
+ An optional reranker picks among the proposals. It is a small MLP over the proposals and the target's token embeddings, trained offline alongside the table and stored in a `.mdr` file. MemDraft always drafts the same number of tokens (`pad=1`), so llama.cpp's compute graph keeps its shape and is never rebuilt. A draft call takes about 60 µs with the reranker and 3 µs without it.
56
+
57
+ It works best behind llama.cpp's `ngram-simple` drafter. `ngram-simple` also needs no draft model, and no table either: when the last 12 tokens appeared earlier in the same request, it drafts up to 48 tokens that followed that earlier match. It catches long copies, such as an answer that repeats code from the prompt. When it finds no match, MemDraft drafts. Optionally, the model's own multi-token prediction (MTP) head drafts the first token and MemDraft continues from it. The MTP head is a small extra layer that some models ship with (Qwen3.5 does) to predict tokens past the next one.
58
+
59
+ ## Results
60
+
61
+ These numbers are for Qwen3.5-9B Q4_K_M on an AMD Radeon 8060S through Vulkan, with greedy decoding and no other process on the GPU, using the [Qwen3.5-9B drafter](#the-qwen35-9b-drafter). The coding prompts below are held-out OSS-Instruct prompts: no table holds answers to them, and the reranker never saw them.
62
+
63
+ - **Coding prompts:** 12 OSS-Instruct prompts, 2 rounds.
64
+ - **HumanEval:** problems 0–11.
65
+ - Each figure is generated tokens per second, with the speedup over plain decoding on the same prompts in parentheses.
66
+
67
+ | Setup | Coding prompts | HumanEval |
68
  |---|---:|---:|
69
+ | Plain decoding | 38.0 tok/s (1.00×) | 38.2 tok/s (1.00×) |
70
+ | **MemDraft alone**¹ | **68.7 tok/s (1.81×)** | **91.1 tok/s (2.39×)** |
71
+ | `ngram-simple` alone¹ | 43.1 tok/s (1.14×) | 74.3 tok/s (1.95×) |
72
+ | **`ngram-simple`, then MemDraft** | **70.5 tok/s (1.85×)** | **125.5 tok/s (3.29×)** |
73
+ | The MTP head alone (`draft-mtp`, 3 tokens) | 73.9 tok/s (1.94×) | 88.0 tok/s (2.30×) |
74
+ | **`ngram-simple`, then the MTP head, then MemDraft** | **81.6 tok/s (2.14×)** | **136.3 tok/s (3.57×)** |
75
 
76
+ Rows without a mark are from 2026-11-18. ¹ From 2026-11-17, in the earlier patched build of llama.cpp, with one round of the coding prompts and HumanEval problems 0–35. The patched build and the plugin ran at the same speed for `ngram-simple`, then MemDraft: 70.1 tok/s (1.84×) and 70.5 tok/s (1.85×).
77
 
78
+ Two other drafters were also tested and did worse than every setup above:
79
 
80
+ - **`ngram-mod`**, the lookup drafter that llama.cpp's `--spec-default` turns on: 41.0 tok/s (1.08×) on coding prompts, and 67.8 tok/s (1.78×) on HumanEval in a run on 2026-11-19.
81
+ - **Qwen3.5-2B as a separate draft model**, 3 tokens: 44.6 tok/s (1.17×) on coding prompts. On HumanEval only 2 problems completed, at 51.7 tok/s (1.36×).
 
82
 
83
+ What the results show:
84
 
85
+ - **On the coding prompts, nearly all the gain is MemDraft's**: 68.7 tok/s (1.81×) alone, 70.5 tok/s (1.85×) behind `ngram-simple`. On HumanEval, `ngram-simple` adds the long copies, taking MemDraft from 91.1 tok/s (2.39×) to 125.5 tok/s (3.29×). HumanEval answers often restate code from the prompt, so the coding prompts are the more typical case.
86
+ - **The MTP head alone beats MemDraft alone on the coding prompts** (73.9 tok/s (1.94×) against 68.7 tok/s (1.81×)). It accepts more tokens per step, but each step costs more because the head is a real model. Putting the head first and MemDraft after it is fastest.
87
+ - With both chained MemDraft setups, every prompt in every round ran faster than plain decoding.
88
+ - **Verification is lossless, but llama.cpp's batched decoding is not bit-exact.** The output text matched plain decoding on 11 of 12 HumanEval problems and on 8 of 24 coding-prompt runs. Where it differs, it splits at a near-tie between two tokens; plain decoding also changes this way when only the batch size changes.
89
+ - **Answer quality held up on HumanEval.** Over all 164 problems, pass@1 was 83.5% against 84.8% for plain decoding: two problems flipped. That run was MemDraft alone (87.4 tok/s (2.29×) over the 164 problems, against 38.1 tok/s plain), with the same table and reranker, in the earlier patched build of llama.cpp.
90
+ - **The setup is narrow so far.** It has been measured on one model, on code, on one GPU. On a mix of chat, math and editing prompts, an earlier table built from the 9B's chat and math text gave 40.2 tok/s (1.24×) overall against 32.4 tok/s plain, but only 35.3 tok/s (1.09×) on chat.
91
+ - Raw results are in [`bench/results/`](https://github.com/Cyronius/memdraft/tree/main/bench/results); the rows marked ¹ are in [`research/results/prior-9b-vk-20261117-1614/`](https://github.com/Cyronius/memdraft/tree/main/research/results/prior-9b-vk-20261117-1614). How MemDraft got here, including what failed, is in [`docs/history.md`](https://github.com/Cyronius/memdraft/blob/main/docs/history.md).
92
 
93
+ ## Quick start
 
94
 
95
+ You need:
96
 
97
+ - llama.cpp at commit `edec462`
98
+ - the [MemDraft code](https://github.com/Cyronius/memdraft)
99
+ - a drafter for your model: a table (`.mdt`) and, optionally, a reranker (`.mdr`). See [Drafters](#drafters).
100
+
101
+ The commands below use the Qwen3.5-9B drafter from the results.
102
 
103
  ```bash
104
+ # 0. Get MemDraft and download the drafter
105
+ git clone https://github.com/Cyronius/memdraft
106
+ hf download Cyronius/memdraft --include 'qwen3.5-9b/*' --local-dir drafters
107
+
108
+ # 1. Patch and build llama.cpp
109
+ git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp && git checkout edec462
110
+ git apply /path/to/memdraft/patches/llama-plugin-hook.patch
111
+ cmake -B build -DGGML_VULKAN=ON && cmake --build build -j # or the backend you use
112
 
113
+ # 2. Build the plugin
114
+ /path/to/memdraft/plugin/build.sh /path/to/llama.cpp # writes plugin/libmemdraft.so
115
+
116
+ # 3. Serve: ngram-simple first, then MemDraft drafting 3 tokens
117
  llama-server -m Qwen3.5-9B-Q4_K_M.gguf -ngl 99 \
118
  --spec-type ngram-simple,plugin --spec-plugin-n-max 3 \
119
+ --spec-plugin /path/to/memdraft/plugin/libmemdraft.so \
120
+ --spec-plugin-args table=drafters/qwen3.5-9b/code.mdt,reranker=drafters/qwen3.5-9b/code.mdr,pad=1,vocab_hash=c52114e74bc3a20f
121
  ```
122
 
123
+ To put the MTP head in front as well, add it as a draft model and let it draft the first token:
124
+
125
+ ```bash
126
+ --spec-type ngram-simple,draft-mtp,plugin --spec-draft-model HEAD.gguf --spec-draft-p-min 0 \
127
+ --spec-plugin-prefix 1 --spec-plugin-n-max 4
128
+ ```
129
+
130
+ To make the head, export it from the Hugging Face weights with `convert_hf_to_gguf.py --mtp --outtype q8_0`. Then re-quantize it to Q4_K_M with a Q4_K output tensor (for example `llama-quantize --allow-requantize --output-tensor-type q4_k HEAD-q8_0.gguf HEAD.gguf Q4_K_M`). The head's output projection is most of its cost, so the smaller head is faster overall.
131
+
132
+ ### Plugin options (`--spec-plugin-args`)
133
+
134
+ | Key | Meaning |
135
  |---|---|
136
+ | `table=FILE` | The lookup table (required) |
137
+ | `reranker=FILE` | Pick among the proposals with this reranker |
138
+ | `pad=0\|1` | Always draft `n_max` tokens, so the batch shape never changes (default 0; use 1) |
139
+ | `tau=P` | Without `pad`: keep drafting while the chain's chance of full acceptance is at least P (default 0.3) |
140
+ | `n_max=N` | Longest draft (default: whatever the host allows) |
141
+ | `source=both\|table\|context` | Which lookups to use (default both) |
142
+ | `vocab_hash=HEX` | Refuse to load unless the model's vocabulary matches. `python plugin/vocab_hash.py MODEL.gguf` prints the hash. |
143
+
144
+ ## Building a table
145
+
146
+ A table is counted from text the target model wrote itself. Its own output is worth far more per token than anyone else's. For code, the steps are:
147
+
148
+ 1. **Generate.** `tools/gen_code.py` has a llama-server answer OSS-Instruct coding prompts (`ise-uiuc/Magicoder-OSS-Instruct-75K`) and stores the answers as token IDs. The table in the results used about 74,000 answers (39.5M tokens): 5,000 answered on a CPU and 69,000 on one rented A40 in 14 hours.
149
+ 2. **Count and calibrate.** `tools/code_study.py --export` counts the table, calibrates its confidences on held-out answers and writes a `.mdt` file. `tools/build_table.py` does the same for general text. `--check` on either replays the C++ drafter against the Python one.
150
+ 3. **Reranker (optional, about +5% tokens per step).** The reranker is trained with `research/reranker/rerank7.py`. `tools/export_reranker.py` then writes the final pair of files: the table in its version 2 format and the reranker. It also checks that the C++ reranker picks the same token as the Python one (it did at 99.975% of positions). Training is still research code; see [`research/`](https://github.com/Cyronius/memdraft/tree/main/research).
151
+
152
+ The published Qwen3.5-9B table also contains 9B chat and math text from another project; that text itself is not published. Without it, `code_study.py --skip-shards` builds a code-only table.
153
+
154
+ This README is also the Hugging Face model card. To add a drafter for another model:
155
+
156
+ 1. Upload its `.mdt` and `.mdr` to a new folder in the Hugging Face repo.
157
+ 2. Add a row to [Drafters](#drafters), with the SHA-256 of both files.
158
+ 3. Add the model under `base_model` in `hf/card-metadata.yml`.
159
+ 4. Publish the card: `bash hf/publish-card.sh`.
160
+
161
+ ## Benchmarking
162
+
163
+ `bench/run_plugin_live.sh` produced the results table. It runs plain decoding and each drafter setup on the same prompts, in turns of at most 9 minutes, only while no other process is on the GPU. `tools/show_wide.py` then summarizes the turns with confidence intervals.
164
+
165
+ The script wraps each turn in `arbiter run …`, a local scheduler for shared GPUs; remove that prefix on another machine. The prompt sets are in [`bench/prompts/`](https://github.com/Cyronius/memdraft/tree/main/bench/prompts).
166
+
167
+ ## Repository layout
168
+
169
+ | Path | Contents |
170
+ |---|---|
171
+ | `src/memdraft.h` | The drafter: table, request index, reranker, chain rule. Header-only C++17, with no llama.cpp types. |
172
+ | `plugin/` | The llama.cpp plugin, its build script, `vocab_hash.py`, and example plugins (`ngram-mod`, a draft model, an MTP head) with equivalence tests |
173
+ | `patches/llama-plugin-hook.patch` | Adds `--spec-type plugin` to llama.cpp `edec462` |
174
+ | `tools/` | Generating training text, building and checking tables, the benchmark client |
175
+ | `bench/` | The benchmark script, prompt sets and results |
176
+ | `hf/` | The Hugging Face card metadata and the script that publishes this README as the card |
177
+ | `tests/` | `python -m pytest tests` |
178
+ | `docs/history.md` | How the project got here, including the failed first attempt |
179
+ | `research/` | Experiments that did not ship, or that led to the shipped setup, with their raw results |
180
+
181
+ Large files live outside git. `data/` holds tables and generated rows. `$MEMDRAFT_CACHE` (default `~/.cache/model-experiments/memdraft`) holds llama.cpp builds, GGUFs, and the tokenizer and prompt dataset. Python dependencies are listed in `requirements.txt`.
182
 
183
  ## License
184
 
185
+ - **Code:** MIT; see [LICENSE](https://github.com/Cyronius/memdraft/blob/main/LICENSE).
186
+ - **Drafter files on Hugging Face:** Apache 2.0, the license of the Qwen3.5-9B weights they derive from.