Cyronius commited on
Commit
f78547f
·
verified ·
1 Parent(s): bc643dc

Add the Qwen3.5-9B code drafter

Browse files
Files changed (4) hide show
  1. .gitattributes +2 -0
  2. README.md +76 -0
  3. qwen3.5-9b/code.mdr +3 -0
  4. qwen3.5-9b/code.mdt +3 -0
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ qwen3.5-9b/code.mdr filter=lfs diff=lfs merge=lfs -text
37
+ qwen3.5-9b/code.mdt filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,76 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: llama.cpp
4
+ pipeline_tag: text-generation
5
+ base_model:
6
+ - Qwen/Qwen3.5-9B
7
+ tags:
8
+ - speculative-decoding
9
+ - llama.cpp
10
+ - drafter
11
+ - code
12
+ ---
13
+
14
+ # MemDraft drafters
15
+
16
+ These are lookup-table drafters for [MemDraft](https://github.com/Cyronius/memdraft), a speculative-decoding drafter for llama.cpp that runs no model. Each drafter has two files and is built for one target model:
17
+
18
+ - **a table (`.mdt`)**, counted from text that the target model wrote;
19
+ - **a reranker (`.mdr`)**, a small MLP that picks among the table's proposals.
20
+
21
+ Load them with MemDraft's llama.cpp plugin. The [GitHub repo](https://github.com/Cyronius/memdraft) explains how to build the plugin and patch llama.cpp.
22
+
23
+ ## Available drafters
24
+
25
+ | Target model | Folder | Domain | Table | Reranker | Vocabulary hash |
26
+ |---|---|---|---:|---:|---|
27
+ | [Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) | [`qwen3.5-9b/`](qwen3.5-9b/) | Code | 256 MiB | 124 MiB | `c52114e74bc3a20f` |
28
+
29
+ More target models will be added here.
30
+
31
+ ## Qwen3.5-9B, code
32
+
33
+ **Speed** with Qwen3.5-9B Q4_K_M on an AMD Radeon 8060S through Vulkan, greedy decoding, against plain decoding (38 tok/s):
34
+
35
+ | Setup | Coding prompts (12 OSS-Instruct prompts, 2 rounds) | HumanEval (12 problems) |
36
+ |---|---:|---:|
37
+ | `ngram-simple`, then MemDraft (3 tokens) | 1.85× | 3.29× |
38
+ | `ngram-simple`, then the 9B's MTP head (1 token), then MemDraft (3 tokens) | 2.14× | 3.57× |
39
+ | llama.cpp `draft-mtp` alone, for comparison | 1.94× | 2.30× |
40
+
41
+ Verification is lossless. Over all 164 HumanEval problems, pass@1 was 83.5% with MemDraft and 84.8% with plain decoding. llama.cpp's batched decoding is not bit-exact, so a few outputs split at near-ties.
42
+
43
+ **What went into it.** The table counts 77.5M tokens of text, keeping contexts seen at least 3 times (14.8M entries):
44
+
45
+ - **Code the 9B wrote:** answers to about 74,000 prompts from [Magicoder-OSS-Instruct-75K](https://huggingface.co/datasets/ise-uiuc/Magicoder-OSS-Instruct-75K), at temperature 0 with thinking off.
46
+ - **That dataset's own gpt-3.5 solutions.**
47
+ - **Chat and math text the 9B wrote:** answers to prompts from GSM8K, MATH, OpenThoughts and UltraChat.
48
+
49
+ The reranker was fine-tuned on positions from the 9B's code. It includes 64-wide projections of the 9B's token embeddings.
50
+
51
+ **Compatibility.**
52
+
53
+ - The table works with any model that uses this tokenizer: the Qwen3.5, 3.6 and 3.8 family, 248,070 tokens. `vocab_hash` makes the plugin refuse any other tokenizer.
54
+ - The reranker was trained against the 9B. It loads with other models in the family, but it has not been measured there.
55
+
56
+ **Held-out answers, offline.** Using the 9B's measured GPU step costs, drafting 3 tokens: 2.01 tokens per step and a predicted 1.65× with the reranker; 1.86 tokens per step and 1.53× without it.
57
+
58
+ ### Usage
59
+
60
+ ```bash
61
+ hf download Cyronius/memdraft --include 'qwen3.5-9b/*' --local-dir memdraft-drafters
62
+
63
+ llama-server -m Qwen3.5-9B-Q4_K_M.gguf -ngl 99 \
64
+ --spec-type ngram-simple,plugin --spec-plugin-n-max 3 \
65
+ --spec-plugin /path/to/libmemdraft.so \
66
+ --spec-plugin-args table=memdraft-drafters/qwen3.5-9b/code.mdt,reranker=memdraft-drafters/qwen3.5-9b/code.mdr,pad=1,vocab_hash=c52114e74bc3a20f
67
+ ```
68
+
69
+ | File | SHA-256 |
70
+ |---|---|
71
+ | `qwen3.5-9b/code.mdt` | `5d835a951a03e86f3cda826738e14f3ea1d38b06487f567a5afe80af7eebf3db` |
72
+ | `qwen3.5-9b/code.mdr` | `6458dcf9b905fb0b28338b19abb99a6ce42b73681063f83858860fd234c84fd7` |
73
+
74
+ ## License
75
+
76
+ These files are released under Apache 2.0, the license of the Qwen3.5-9B weights they derive from. The MemDraft code is under the MIT license.
qwen3.5-9b/code.mdr ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6458dcf9b905fb0b28338b19abb99a6ce42b73681063f83858860fd234c84fd7
3
+ size 130222660
qwen3.5-9b/code.mdt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5d835a951a03e86f3cda826738e14f3ea1d38b06487f567a5afe80af7eebf3db
3
+ size 268438568