Cyronius commited on
Commit
e0eb645
·
verified ·
1 Parent(s): 792c8dc

Sync the model card with the GitHub README

Browse files
Files changed (1) hide show
  1. README.md +33 -18
README.md CHANGED
@@ -43,7 +43,7 @@ The table counts 77.5M tokens of text, keeping contexts seen at least 3 times (1
43
  The reranker was fine-tuned on positions from the 9B's code. It includes 64-wide projections of the 9B's token embeddings.
44
 
45
  - **Compatibility.** The table works with any model that uses the same tokenizer: the Qwen3.5, 3.6 and 3.8 family, 248,070 tokens. `vocab_hash` makes the plugin refuse any other tokenizer. The table has only been measured with the 9B. The reranker was trained against the 9B; it loads with other models in the family, but it has not been measured there.
46
- - **Offline, on held-out answers.** Drafting 3 tokens: 2.01 tokens per step with the reranker and 1.86 without it. At plain decoding's 38.0 tok/s and the 9B's measured GPU step costs, that predicts 62.7 tok/s (1.65×) and 58.2 tok/s (1.53×).
47
 
48
  ## How it works
49
 
@@ -56,38 +56,53 @@ An optional reranker picks among the proposals. It is a small MLP over the propo
56
 
57
  It works best behind llama.cpp's `ngram-simple` drafter. `ngram-simple` also needs no draft model, and no table either: when the last 12 tokens appeared earlier in the same request, it drafts up to 48 tokens that followed that earlier match. It catches long copies, such as an answer that repeats code from the prompt. When it finds no match, MemDraft drafts. Optionally, the model's own multi-token prediction (MTP) head drafts the first token and MemDraft continues from it. The MTP head is a small extra layer that some models ship with (Qwen3.5 does) to predict tokens past the next one.
58
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
59
  ## Results
60
 
61
  These numbers are for Qwen3.5-9B Q4_K_M on an AMD Radeon 8060S through Vulkan, with greedy decoding and no other process on the GPU, using the [Qwen3.5-9B drafter](#the-qwen35-9b-drafter). The coding prompts below are held-out OSS-Instruct prompts: no table holds answers to them, and the reranker never saw them.
62
 
63
  - **Coding prompts:** 12 OSS-Instruct prompts, 2 rounds.
64
  - **HumanEval:** problems 0–11.
65
- - Each figure is generated tokens per second, with the speedup over plain decoding on the same prompts in parentheses.
66
 
67
- | Setup | Coding prompts | HumanEval |
68
- |---|---:|---:|
69
- | Plain decoding | 38.0 tok/s (1.00×) | 38.2 tok/s (1.00×) |
70
- | **MemDraft alone**¹ | **68.7 tok/s (1.81×)** | **91.1 tok/s (2.39×)** |
71
- | `ngram-simple` alone¹ | 43.1 tok/s (1.14×) | 74.3 tok/s (1.95×) |
72
- | **`ngram-simple`, then MemDraft** | **70.5 tok/s (1.85×)** | **125.5 tok/s (3.29×)** |
73
- | The MTP head alone (`draft-mtp`, 3 tokens) | 73.9 tok/s (1.94×) | 88.0 tok/s (2.30×) |
74
- | **`ngram-simple`, then the MTP head, then MemDraft** | **81.6 tok/s (2.14×)** | **136.3 tok/s (3.57×)** |
75
 
76
- Rows without a mark are from 2026-11-18. ¹ From 2026-11-17, in the earlier patched build of llama.cpp, with one round of the coding prompts and HumanEval problems 0–35. The patched build and the plugin ran at the same speed for `ngram-simple`, then MemDraft: 70.1 tok/s (1.84×) and 70.5 tok/s (1.85×).
77
 
78
  Two other drafters were also tested and did worse than every setup above:
79
 
80
- - **`ngram-mod`**, the lookup drafter that llama.cpp's `--spec-default` turns on: 41.0 tok/s (1.08×) on coding prompts, and 67.8 tok/s (1.78×) on HumanEval in a run on 2026-11-19.
81
- - **Qwen3.5-2B as a separate draft model**, 3 tokens: 44.6 tok/s (1.17×) on coding prompts. On HumanEval only 2 problems completed, at 51.7 tok/s (1.36×).
82
 
83
  What the results show:
84
 
85
- - **On the coding prompts, nearly all the gain is MemDraft's**: 68.7 tok/s (1.81×) alone, 70.5 tok/s (1.85×) behind `ngram-simple`. On HumanEval, `ngram-simple` adds the long copies, taking MemDraft from 91.1 tok/s (2.39×) to 125.5 tok/s (3.29×). HumanEval answers often restate code from the prompt, so the coding prompts are the more typical case.
86
- - **The MTP head alone beats MemDraft alone on the coding prompts** (73.9 tok/s (1.94×) against 68.7 tok/s (1.81×)). It accepts more tokens per step, but each step costs more because the head is a real model. Putting the head first and MemDraft after it is fastest.
87
  - With both chained MemDraft setups, every prompt in every round ran faster than plain decoding.
88
  - **Verification is lossless, but llama.cpp's batched decoding is not bit-exact.** The output text matched plain decoding on 11 of 12 HumanEval problems and on 8 of 24 coding-prompt runs. Where it differs, it splits at a near-tie between two tokens; plain decoding also changes this way when only the batch size changes.
89
- - **Answer quality held up on HumanEval.** Over all 164 problems, pass@1 was 83.5% against 84.8% for plain decoding: two problems flipped. That run was MemDraft alone (87.4 tok/s (2.29×) over the 164 problems, against 38.1 tok/s plain), with the same table and reranker, in the earlier patched build of llama.cpp.
90
- - **The setup is narrow so far.** It has been measured on one model, on code, on one GPU. On a mix of chat, math and editing prompts, an earlier table built from the 9B's chat and math text gave 40.2 tok/s (1.24×) overall against 32.4 tok/s plain, but only 35.3 tok/s (1.09×) on chat.
91
  - Raw results are in [`bench/results/`](https://github.com/Cyronius/memdraft/tree/main/bench/results); the rows marked ¹ are in [`research/results/prior-9b-vk-20261117-1614/`](https://github.com/Cyronius/memdraft/tree/main/research/results/prior-9b-vk-20261117-1614). How MemDraft got here, including what failed, is in [`docs/history.md`](https://github.com/Cyronius/memdraft/blob/main/docs/history.md).
92
 
93
  ## Quick start
@@ -182,5 +197,5 @@ Large files live outside git. `data/` holds tables and generated rows. `$MEMDRAF
182
 
183
  ## License
184
 
185
- - **Code:** MIT; see [LICENSE](https://github.com/Cyronius/memdraft/blob/main/LICENSE).
186
  - **Drafter files on Hugging Face:** Apache 2.0, the license of the Qwen3.5-9B weights they derive from.
 
43
  The reranker was fine-tuned on positions from the 9B's code. It includes 64-wide projections of the 9B's token embeddings.
44
 
45
  - **Compatibility.** The table works with any model that uses the same tokenizer: the Qwen3.5, 3.6 and 3.8 family, 248,070 tokens. `vocab_hash` makes the plugin refuse any other tokenizer. The table has only been measured with the 9B. The reranker was trained against the 9B; it loads with other models in the family, but it has not been measured there.
46
+ - **Offline, on held-out answers.** With the 9B's measured GPU step costs and 3-token drafts: 2.01 tokens per step and a predicted 1.65× with the reranker, 1.86 tokens per step and 1.53× without it.
47
 
48
  ## How it works
49
 
 
56
 
57
  It works best behind llama.cpp's `ngram-simple` drafter. `ngram-simple` also needs no draft model, and no table either: when the last 12 tokens appeared earlier in the same request, it drafts up to 48 tokens that followed that earlier match. It catches long copies, such as an answer that repeats code from the prompt. When it finds no match, MemDraft drafts. Optionally, the model's own multi-token prediction (MTP) head drafts the first token and MemDraft continues from it. The MTP head is a small extra layer that some models ship with (Qwen3.5 does) to predict tokens past the next one.
58
 
59
+ ## Why a lookup drafter
60
+
61
+ A drafter helps only when it is cheap next to the target. The speedup is about the tokens accepted per step divided by the cost of a step, so a drafter that guesses well but runs slowly gives back its gain. On the coding prompts with Qwen3.5-9B, with step cost relative to one plain decoding step:
62
+
63
+ | Drafter (3 tokens) | Tokens per step | Step cost | Speedup |
64
+ |---|---:|---:|---:|
65
+ | Qwen3.5-2B as a separate draft model | 2.77 | 2.36× | 1.17× |
66
+ | The 9B's MTP head (`draft-mtp`) | 3.08 | 1.59× | 1.94× |
67
+ | `ngram-simple`, then MemDraft | 2.19 | 1.18× | 1.85× |
68
+
69
+ The 2B draft model gets more tokens per step than MemDraft, but it is the slowest setup. A MemDraft draft call takes 3 to 60 µs, against about 26 ms for one plain step of the 9B. Most of MemDraft's 1.18× step cost is the target checking 4 tokens instead of 1.
70
+
71
+ - **Smaller targets.** A smaller target takes less time per step, so a drafter that runs a model takes a larger share of each step. MemDraft's cost stays in microseconds at any target size. This follows from the cost model; MemDraft has only been measured with the 9B.
72
+ - **No VRAM.** The table and reranker load into system RAM (380 MiB for the Qwen3.5-9B drafter) and run on the CPU. A draft model needs VRAM for its weights and its own KV cache, and in the results the MTP head also ran on the GPU. The target still checks MemDraft's guesses on the GPU.
73
+
74
  ## Results
75
 
76
  These numbers are for Qwen3.5-9B Q4_K_M on an AMD Radeon 8060S through Vulkan, with greedy decoding and no other process on the GPU, using the [Qwen3.5-9B drafter](#the-qwen35-9b-drafter). The coding prompts below are held-out OSS-Instruct prompts: no table holds answers to them, and the reranker never saw them.
77
 
78
  - **Coding prompts:** 12 OSS-Instruct prompts, 2 rounds.
79
  - **HumanEval:** problems 0–11.
80
+ - **Tok/s:** generated tokens per second. **Speedup:** tok/s divided by plain decoding's tok/s on the same prompts.
81
 
82
+ | Setup | Coding tok/s | Coding speedup | HumanEval tok/s | HumanEval speedup |
83
+ |---|---:|---:|---:|---:|
84
+ | Plain decoding | 38.0 | 1.00× | 38.2 | 1.00× |
85
+ | **MemDraft alone**¹ | 68.7 | **1.81×** | 91.1 | **2.39×** |
86
+ | `ngram-simple` alone¹ | 43.1 | 1.14× | 74.3 | 1.95× |
87
+ | **`ngram-simple`, then MemDraft** | 70.5 | **1.85×** | 125.5 | **3.29×** |
88
+ | The MTP head alone (`draft-mtp`, 3 tokens) | 73.9 | 1.94× | 88.0 | 2.30× |
89
+ | **`ngram-simple`, then the MTP head, then MemDraft** | 81.6 | **2.14×** | 136.3 | **3.57×** |
90
 
91
+ Rows without a mark are from 2026-11-18. ¹ From 2026-11-17, in the earlier patched build of llama.cpp, with one round of the coding prompts and HumanEval problems 0–35. The patched build and the plugin ran at the same speed (1.84× and 1.85× for `ngram-simple`, then MemDraft).
92
 
93
  Two other drafters were also tested and did worse than every setup above:
94
 
95
+ - **`ngram-mod`**, the lookup drafter that llama.cpp's `--spec-default` turns on: 1.08× on coding prompts (41.0 tok/s), 1.78× on HumanEval (67.8 tok/s, from a run on 2026-11-19).
96
+ - **Qwen3.5-2B as a separate draft model**, 3 tokens: 1.17× on coding prompts (44.6 tok/s). On HumanEval only 2 problems completed, at 1.36× (51.7 tok/s).
97
 
98
  What the results show:
99
 
100
+ - **On the coding prompts, nearly all the gain is MemDraft's**: 1.81× alone, 1.85× behind `ngram-simple`. On HumanEval, `ngram-simple` adds the long copies, taking MemDraft from 2.39× to 3.29×. HumanEval answers often restate code from the prompt, so the coding prompts are the more typical case.
101
+ - **The MTP head alone beats MemDraft alone on the coding prompts** (1.94× against 1.81×). It accepts more tokens per step, but each step costs more because the head is a real model. Putting the head first and MemDraft after it is fastest.
102
  - With both chained MemDraft setups, every prompt in every round ran faster than plain decoding.
103
  - **Verification is lossless, but llama.cpp's batched decoding is not bit-exact.** The output text matched plain decoding on 11 of 12 HumanEval problems and on 8 of 24 coding-prompt runs. Where it differs, it splits at a near-tie between two tokens; plain decoding also changes this way when only the batch size changes.
104
+ - **Answer quality held up on HumanEval.** Over all 164 problems, pass@1 was 83.5% against 84.8% for plain decoding: two problems flipped. That run was MemDraft alone (2.29× over the 164 problems), with the same table and reranker, in the earlier patched build of llama.cpp.
105
+ - **The setup is narrow so far.** It has been measured on one model, on code, on one GPU. On a mix of chat, math and editing prompts, an earlier table built from the 9B's chat and math text gave 1.24× overall, but only 1.09× on chat.
106
  - Raw results are in [`bench/results/`](https://github.com/Cyronius/memdraft/tree/main/bench/results); the rows marked ¹ are in [`research/results/prior-9b-vk-20261117-1614/`](https://github.com/Cyronius/memdraft/tree/main/research/results/prior-9b-vk-20261117-1614). How MemDraft got here, including what failed, is in [`docs/history.md`](https://github.com/Cyronius/memdraft/blob/main/docs/history.md).
107
 
108
  ## Quick start
 
197
 
198
  ## License
199
 
200
+ - **Code:** MIT or Apache 2.0, at your option; see [LICENSE-MIT](https://github.com/Cyronius/memdraft/blob/main/LICENSE-MIT) and [LICENSE-APACHE](https://github.com/Cyronius/memdraft/blob/main/LICENSE-APACHE).
201
  - **Drafter files on Hugging Face:** Apache 2.0, the license of the Qwen3.5-9B weights they derive from.