Add the Qwen3.5-9B code drafter
Browse files- .gitattributes +2 -0
- README.md +76 -0
- qwen3.5-9b/code.mdr +3 -0
- qwen3.5-9b/code.mdt +3 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
qwen3.5-9b/code.mdr filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
qwen3.5-9b/code.mdt filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
library_name: llama.cpp
|
| 4 |
+
pipeline_tag: text-generation
|
| 5 |
+
base_model:
|
| 6 |
+
- Qwen/Qwen3.5-9B
|
| 7 |
+
tags:
|
| 8 |
+
- speculative-decoding
|
| 9 |
+
- llama.cpp
|
| 10 |
+
- drafter
|
| 11 |
+
- code
|
| 12 |
+
---
|
| 13 |
+
|
| 14 |
+
# MemDraft drafters
|
| 15 |
+
|
| 16 |
+
These are lookup-table drafters for [MemDraft](https://github.com/Cyronius/memdraft), a speculative-decoding drafter for llama.cpp that runs no model. Each drafter has two files and is built for one target model:
|
| 17 |
+
|
| 18 |
+
- **a table (`.mdt`)**, counted from text that the target model wrote;
|
| 19 |
+
- **a reranker (`.mdr`)**, a small MLP that picks among the table's proposals.
|
| 20 |
+
|
| 21 |
+
Load them with MemDraft's llama.cpp plugin. The [GitHub repo](https://github.com/Cyronius/memdraft) explains how to build the plugin and patch llama.cpp.
|
| 22 |
+
|
| 23 |
+
## Available drafters
|
| 24 |
+
|
| 25 |
+
| Target model | Folder | Domain | Table | Reranker | Vocabulary hash |
|
| 26 |
+
|---|---|---|---:|---:|---|
|
| 27 |
+
| [Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) | [`qwen3.5-9b/`](qwen3.5-9b/) | Code | 256 MiB | 124 MiB | `c52114e74bc3a20f` |
|
| 28 |
+
|
| 29 |
+
More target models will be added here.
|
| 30 |
+
|
| 31 |
+
## Qwen3.5-9B, code
|
| 32 |
+
|
| 33 |
+
**Speed** with Qwen3.5-9B Q4_K_M on an AMD Radeon 8060S through Vulkan, greedy decoding, against plain decoding (38 tok/s):
|
| 34 |
+
|
| 35 |
+
| Setup | Coding prompts (12 OSS-Instruct prompts, 2 rounds) | HumanEval (12 problems) |
|
| 36 |
+
|---|---:|---:|
|
| 37 |
+
| `ngram-simple`, then MemDraft (3 tokens) | 1.85× | 3.29× |
|
| 38 |
+
| `ngram-simple`, then the 9B's MTP head (1 token), then MemDraft (3 tokens) | 2.14× | 3.57× |
|
| 39 |
+
| llama.cpp `draft-mtp` alone, for comparison | 1.94× | 2.30× |
|
| 40 |
+
|
| 41 |
+
Verification is lossless. Over all 164 HumanEval problems, pass@1 was 83.5% with MemDraft and 84.8% with plain decoding. llama.cpp's batched decoding is not bit-exact, so a few outputs split at near-ties.
|
| 42 |
+
|
| 43 |
+
**What went into it.** The table counts 77.5M tokens of text, keeping contexts seen at least 3 times (14.8M entries):
|
| 44 |
+
|
| 45 |
+
- **Code the 9B wrote:** answers to about 74,000 prompts from [Magicoder-OSS-Instruct-75K](https://huggingface.co/datasets/ise-uiuc/Magicoder-OSS-Instruct-75K), at temperature 0 with thinking off.
|
| 46 |
+
- **That dataset's own gpt-3.5 solutions.**
|
| 47 |
+
- **Chat and math text the 9B wrote:** answers to prompts from GSM8K, MATH, OpenThoughts and UltraChat.
|
| 48 |
+
|
| 49 |
+
The reranker was fine-tuned on positions from the 9B's code. It includes 64-wide projections of the 9B's token embeddings.
|
| 50 |
+
|
| 51 |
+
**Compatibility.**
|
| 52 |
+
|
| 53 |
+
- The table works with any model that uses this tokenizer: the Qwen3.5, 3.6 and 3.8 family, 248,070 tokens. `vocab_hash` makes the plugin refuse any other tokenizer.
|
| 54 |
+
- The reranker was trained against the 9B. It loads with other models in the family, but it has not been measured there.
|
| 55 |
+
|
| 56 |
+
**Held-out answers, offline.** Using the 9B's measured GPU step costs, drafting 3 tokens: 2.01 tokens per step and a predicted 1.65× with the reranker; 1.86 tokens per step and 1.53× without it.
|
| 57 |
+
|
| 58 |
+
### Usage
|
| 59 |
+
|
| 60 |
+
```bash
|
| 61 |
+
hf download Cyronius/memdraft --include 'qwen3.5-9b/*' --local-dir memdraft-drafters
|
| 62 |
+
|
| 63 |
+
llama-server -m Qwen3.5-9B-Q4_K_M.gguf -ngl 99 \
|
| 64 |
+
--spec-type ngram-simple,plugin --spec-plugin-n-max 3 \
|
| 65 |
+
--spec-plugin /path/to/libmemdraft.so \
|
| 66 |
+
--spec-plugin-args table=memdraft-drafters/qwen3.5-9b/code.mdt,reranker=memdraft-drafters/qwen3.5-9b/code.mdr,pad=1,vocab_hash=c52114e74bc3a20f
|
| 67 |
+
```
|
| 68 |
+
|
| 69 |
+
| File | SHA-256 |
|
| 70 |
+
|---|---|
|
| 71 |
+
| `qwen3.5-9b/code.mdt` | `5d835a951a03e86f3cda826738e14f3ea1d38b06487f567a5afe80af7eebf3db` |
|
| 72 |
+
| `qwen3.5-9b/code.mdr` | `6458dcf9b905fb0b28338b19abb99a6ce42b73681063f83858860fd234c84fd7` |
|
| 73 |
+
|
| 74 |
+
## License
|
| 75 |
+
|
| 76 |
+
These files are released under Apache 2.0, the license of the Qwen3.5-9B weights they derive from. The MemDraft code is under the MIT license.
|
qwen3.5-9b/code.mdr
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6458dcf9b905fb0b28338b19abb99a6ce42b73681063f83858860fd234c84fd7
|
| 3 |
+
size 130222660
|
qwen3.5-9b/code.mdt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5d835a951a03e86f3cda826738e14f3ea1d38b06487f567a5afe80af7eebf3db
|
| 3 |
+
size 268438568
|