Sync the model card with the GitHub README
Browse files
README.md
CHANGED
|
@@ -43,7 +43,7 @@ The table counts 77.5M tokens of text, keeping contexts seen at least 3 times (1
|
|
| 43 |
The reranker was fine-tuned on positions from the 9B's code. It includes 64-wide projections of the 9B's token embeddings.
|
| 44 |
|
| 45 |
- **Compatibility.** The table works with any model that uses the same tokenizer: the Qwen3.5, 3.6 and 3.8 family, 248,070 tokens. `vocab_hash` makes the plugin refuse any other tokenizer. The table has only been measured with the 9B. The reranker was trained against the 9B; it loads with other models in the family, but it has not been measured there.
|
| 46 |
-
- **Offline, on held-out answers.**
|
| 47 |
|
| 48 |
## How it works
|
| 49 |
|
|
@@ -56,38 +56,53 @@ An optional reranker picks among the proposals. It is a small MLP over the propo
|
|
| 56 |
|
| 57 |
It works best behind llama.cpp's `ngram-simple` drafter. `ngram-simple` also needs no draft model, and no table either: when the last 12 tokens appeared earlier in the same request, it drafts up to 48 tokens that followed that earlier match. It catches long copies, such as an answer that repeats code from the prompt. When it finds no match, MemDraft drafts. Optionally, the model's own multi-token prediction (MTP) head drafts the first token and MemDraft continues from it. The MTP head is a small extra layer that some models ship with (Qwen3.5 does) to predict tokens past the next one.
|
| 58 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 59 |
## Results
|
| 60 |
|
| 61 |
These numbers are for Qwen3.5-9B Q4_K_M on an AMD Radeon 8060S through Vulkan, with greedy decoding and no other process on the GPU, using the [Qwen3.5-9B drafter](#the-qwen35-9b-drafter). The coding prompts below are held-out OSS-Instruct prompts: no table holds answers to them, and the reranker never saw them.
|
| 62 |
|
| 63 |
- **Coding prompts:** 12 OSS-Instruct prompts, 2 rounds.
|
| 64 |
- **HumanEval:** problems 0–11.
|
| 65 |
-
-
|
| 66 |
|
| 67 |
-
| Setup | Coding
|
| 68 |
-
|---|---:|---:|
|
| 69 |
-
| Plain decoding | 38.0
|
| 70 |
-
| **MemDraft alone**¹ |
|
| 71 |
-
| `ngram-simple` alone¹ | 43.1
|
| 72 |
-
| **`ngram-simple`, then MemDraft** |
|
| 73 |
-
| The MTP head alone (`draft-mtp`, 3 tokens) | 73.9
|
| 74 |
-
| **`ngram-simple`, then the MTP head, then MemDraft** |
|
| 75 |
|
| 76 |
-
Rows without a mark are from 2026-11-18. ¹ From 2026-11-17, in the earlier patched build of llama.cpp, with one round of the coding prompts and HumanEval problems 0–35. The patched build and the plugin ran at the same speed for `ngram-simple`, then MemDraft
|
| 77 |
|
| 78 |
Two other drafters were also tested and did worse than every setup above:
|
| 79 |
|
| 80 |
-
- **`ngram-mod`**, the lookup drafter that llama.cpp's `--spec-default` turns on:
|
| 81 |
-
- **Qwen3.5-2B as a separate draft model**, 3 tokens:
|
| 82 |
|
| 83 |
What the results show:
|
| 84 |
|
| 85 |
-
- **On the coding prompts, nearly all the gain is MemDraft's**:
|
| 86 |
-
- **The MTP head alone beats MemDraft alone on the coding prompts** (
|
| 87 |
- With both chained MemDraft setups, every prompt in every round ran faster than plain decoding.
|
| 88 |
- **Verification is lossless, but llama.cpp's batched decoding is not bit-exact.** The output text matched plain decoding on 11 of 12 HumanEval problems and on 8 of 24 coding-prompt runs. Where it differs, it splits at a near-tie between two tokens; plain decoding also changes this way when only the batch size changes.
|
| 89 |
-
- **Answer quality held up on HumanEval.** Over all 164 problems, pass@1 was 83.5% against 84.8% for plain decoding: two problems flipped. That run was MemDraft alone (
|
| 90 |
-
- **The setup is narrow so far.** It has been measured on one model, on code, on one GPU. On a mix of chat, math and editing prompts, an earlier table built from the 9B's chat and math text gave
|
| 91 |
- Raw results are in [`bench/results/`](https://github.com/Cyronius/memdraft/tree/main/bench/results); the rows marked ¹ are in [`research/results/prior-9b-vk-20261117-1614/`](https://github.com/Cyronius/memdraft/tree/main/research/results/prior-9b-vk-20261117-1614). How MemDraft got here, including what failed, is in [`docs/history.md`](https://github.com/Cyronius/memdraft/blob/main/docs/history.md).
|
| 92 |
|
| 93 |
## Quick start
|
|
@@ -182,5 +197,5 @@ Large files live outside git. `data/` holds tables and generated rows. `$MEMDRAF
|
|
| 182 |
|
| 183 |
## License
|
| 184 |
|
| 185 |
-
- **Code:** MIT; see [LICENSE](https://github.com/Cyronius/memdraft/blob/main/LICENSE).
|
| 186 |
- **Drafter files on Hugging Face:** Apache 2.0, the license of the Qwen3.5-9B weights they derive from.
|
|
|
|
| 43 |
The reranker was fine-tuned on positions from the 9B's code. It includes 64-wide projections of the 9B's token embeddings.
|
| 44 |
|
| 45 |
- **Compatibility.** The table works with any model that uses the same tokenizer: the Qwen3.5, 3.6 and 3.8 family, 248,070 tokens. `vocab_hash` makes the plugin refuse any other tokenizer. The table has only been measured with the 9B. The reranker was trained against the 9B; it loads with other models in the family, but it has not been measured there.
|
| 46 |
+
- **Offline, on held-out answers.** With the 9B's measured GPU step costs and 3-token drafts: 2.01 tokens per step and a predicted 1.65× with the reranker, 1.86 tokens per step and 1.53× without it.
|
| 47 |
|
| 48 |
## How it works
|
| 49 |
|
|
|
|
| 56 |
|
| 57 |
It works best behind llama.cpp's `ngram-simple` drafter. `ngram-simple` also needs no draft model, and no table either: when the last 12 tokens appeared earlier in the same request, it drafts up to 48 tokens that followed that earlier match. It catches long copies, such as an answer that repeats code from the prompt. When it finds no match, MemDraft drafts. Optionally, the model's own multi-token prediction (MTP) head drafts the first token and MemDraft continues from it. The MTP head is a small extra layer that some models ship with (Qwen3.5 does) to predict tokens past the next one.
|
| 58 |
|
| 59 |
+
## Why a lookup drafter
|
| 60 |
+
|
| 61 |
+
A drafter helps only when it is cheap next to the target. The speedup is about the tokens accepted per step divided by the cost of a step, so a drafter that guesses well but runs slowly gives back its gain. On the coding prompts with Qwen3.5-9B, with step cost relative to one plain decoding step:
|
| 62 |
+
|
| 63 |
+
| Drafter (3 tokens) | Tokens per step | Step cost | Speedup |
|
| 64 |
+
|---|---:|---:|---:|
|
| 65 |
+
| Qwen3.5-2B as a separate draft model | 2.77 | 2.36× | 1.17× |
|
| 66 |
+
| The 9B's MTP head (`draft-mtp`) | 3.08 | 1.59× | 1.94× |
|
| 67 |
+
| `ngram-simple`, then MemDraft | 2.19 | 1.18× | 1.85× |
|
| 68 |
+
|
| 69 |
+
The 2B draft model gets more tokens per step than MemDraft, but it is the slowest setup. A MemDraft draft call takes 3 to 60 µs, against about 26 ms for one plain step of the 9B. Most of MemDraft's 1.18× step cost is the target checking 4 tokens instead of 1.
|
| 70 |
+
|
| 71 |
+
- **Smaller targets.** A smaller target takes less time per step, so a drafter that runs a model takes a larger share of each step. MemDraft's cost stays in microseconds at any target size. This follows from the cost model; MemDraft has only been measured with the 9B.
|
| 72 |
+
- **No VRAM.** The table and reranker load into system RAM (380 MiB for the Qwen3.5-9B drafter) and run on the CPU. A draft model needs VRAM for its weights and its own KV cache, and in the results the MTP head also ran on the GPU. The target still checks MemDraft's guesses on the GPU.
|
| 73 |
+
|
| 74 |
## Results
|
| 75 |
|
| 76 |
These numbers are for Qwen3.5-9B Q4_K_M on an AMD Radeon 8060S through Vulkan, with greedy decoding and no other process on the GPU, using the [Qwen3.5-9B drafter](#the-qwen35-9b-drafter). The coding prompts below are held-out OSS-Instruct prompts: no table holds answers to them, and the reranker never saw them.
|
| 77 |
|
| 78 |
- **Coding prompts:** 12 OSS-Instruct prompts, 2 rounds.
|
| 79 |
- **HumanEval:** problems 0–11.
|
| 80 |
+
- **Tok/s:** generated tokens per second. **Speedup:** tok/s divided by plain decoding's tok/s on the same prompts.
|
| 81 |
|
| 82 |
+
| Setup | Coding tok/s | Coding speedup | HumanEval tok/s | HumanEval speedup |
|
| 83 |
+
|---|---:|---:|---:|---:|
|
| 84 |
+
| Plain decoding | 38.0 | 1.00× | 38.2 | 1.00× |
|
| 85 |
+
| **MemDraft alone**¹ | 68.7 | **1.81×** | 91.1 | **2.39×** |
|
| 86 |
+
| `ngram-simple` alone¹ | 43.1 | 1.14× | 74.3 | 1.95× |
|
| 87 |
+
| **`ngram-simple`, then MemDraft** | 70.5 | **1.85×** | 125.5 | **3.29×** |
|
| 88 |
+
| The MTP head alone (`draft-mtp`, 3 tokens) | 73.9 | 1.94× | 88.0 | 2.30× |
|
| 89 |
+
| **`ngram-simple`, then the MTP head, then MemDraft** | 81.6 | **2.14×** | 136.3 | **3.57×** |
|
| 90 |
|
| 91 |
+
Rows without a mark are from 2026-11-18. ¹ From 2026-11-17, in the earlier patched build of llama.cpp, with one round of the coding prompts and HumanEval problems 0–35. The patched build and the plugin ran at the same speed (1.84× and 1.85× for `ngram-simple`, then MemDraft).
|
| 92 |
|
| 93 |
Two other drafters were also tested and did worse than every setup above:
|
| 94 |
|
| 95 |
+
- **`ngram-mod`**, the lookup drafter that llama.cpp's `--spec-default` turns on: 1.08× on coding prompts (41.0 tok/s), 1.78× on HumanEval (67.8 tok/s, from a run on 2026-11-19).
|
| 96 |
+
- **Qwen3.5-2B as a separate draft model**, 3 tokens: 1.17× on coding prompts (44.6 tok/s). On HumanEval only 2 problems completed, at 1.36× (51.7 tok/s).
|
| 97 |
|
| 98 |
What the results show:
|
| 99 |
|
| 100 |
+
- **On the coding prompts, nearly all the gain is MemDraft's**: 1.81× alone, 1.85× behind `ngram-simple`. On HumanEval, `ngram-simple` adds the long copies, taking MemDraft from 2.39× to 3.29×. HumanEval answers often restate code from the prompt, so the coding prompts are the more typical case.
|
| 101 |
+
- **The MTP head alone beats MemDraft alone on the coding prompts** (1.94× against 1.81×). It accepts more tokens per step, but each step costs more because the head is a real model. Putting the head first and MemDraft after it is fastest.
|
| 102 |
- With both chained MemDraft setups, every prompt in every round ran faster than plain decoding.
|
| 103 |
- **Verification is lossless, but llama.cpp's batched decoding is not bit-exact.** The output text matched plain decoding on 11 of 12 HumanEval problems and on 8 of 24 coding-prompt runs. Where it differs, it splits at a near-tie between two tokens; plain decoding also changes this way when only the batch size changes.
|
| 104 |
+
- **Answer quality held up on HumanEval.** Over all 164 problems, pass@1 was 83.5% against 84.8% for plain decoding: two problems flipped. That run was MemDraft alone (2.29× over the 164 problems), with the same table and reranker, in the earlier patched build of llama.cpp.
|
| 105 |
+
- **The setup is narrow so far.** It has been measured on one model, on code, on one GPU. On a mix of chat, math and editing prompts, an earlier table built from the 9B's chat and math text gave 1.24× overall, but only 1.09× on chat.
|
| 106 |
- Raw results are in [`bench/results/`](https://github.com/Cyronius/memdraft/tree/main/bench/results); the rows marked ¹ are in [`research/results/prior-9b-vk-20261117-1614/`](https://github.com/Cyronius/memdraft/tree/main/research/results/prior-9b-vk-20261117-1614). How MemDraft got here, including what failed, is in [`docs/history.md`](https://github.com/Cyronius/memdraft/blob/main/docs/history.md).
|
| 107 |
|
| 108 |
## Quick start
|
|
|
|
| 197 |
|
| 198 |
## License
|
| 199 |
|
| 200 |
+
- **Code:** MIT or Apache 2.0, at your option; see [LICENSE-MIT](https://github.com/Cyronius/memdraft/blob/main/LICENSE-MIT) and [LICENSE-APACHE](https://github.com/Cyronius/memdraft/blob/main/LICENSE-APACHE).
|
| 201 |
- **Drafter files on Hugging Face:** Apache 2.0, the license of the Qwen3.5-9B weights they derive from.
|