yanghaojin commited on
Commit
7693666
·
verified ·
1 Parent(s): d665bad

Update draft-head speed tables (v0.8.0)

Browse files
Files changed (1) hide show
  1. README.md +21 -16
README.md CHANGED
@@ -181,19 +181,24 @@ alone. It is off unless asked for:
181
  GBX_GLM53_MTP=on gbx_lm --model GreenBitAI/GLM-5.3-Flash-4bit-paged
182
  ```
183
 
184
- **It grows with the context.** Measured 2026-09-18 on a 512 GB Mac Studio
185
- (M3 Ultra), greedy, decode timed from the first token, both columns from the same
186
- build:
187
-
188
- | context | head off | head on | speedup | accepted |
189
- |---|---|---|---|---|
190
- | 11,317 | 27.1 tok/s | **38.3** | **1.41x** | 78% |
191
- | 2,854 | 27.5 | **35.7** | 1.30x | 76% |
192
- | 715 | 31.6 | 31.8 | 1.01x | 65% |
193
-
194
- A short context has little to go on, so the head proposes less well and the win
195
- is small; by 11k it accepts 78% of what it proposes. Through the server on
196
- chat-shaped questions it accepts 63-75%.
197
-
198
- Every token the head proposes is verified by the model itself, so the reply is
199
- the model's own either way; the head only saves passes over the weights.
 
 
 
 
 
 
181
  GBX_GLM53_MTP=on gbx_lm --model GreenBitAI/GLM-5.3-Flash-4bit-paged
182
  ```
183
 
184
+ **Up to 1.4x, more as the context grows.**
185
+ Measured 2026-10-13 with `gbx_lm` v0.8.0 on a 512 GB Mac Studio (M3 Ultra), weights
186
+ resident, greedy, thinking off, 256 tokens a reply, decode timed from the first token.
187
+ Short prompts: three each of code, prose and chat, three runs each, the median per kind.
188
+ Long contexts: a slice of source code and a question about it, the median of three runs.
189
+
190
+ | prompt | head off | head on | speedup |
191
+ |---|---|---|---|
192
+ | short, code | 39.5 tok/s | **55.5** | 1.41x |
193
+ | short, prose | 39.5 | **45.8** | 1.16x |
194
+ | short, chat | 39.5 | **50.8** | 1.29x |
195
+ | 3,400-token context | 32.6 | **40.0** | 1.23x |
196
+ | 14,000-token context | 32.2 | **43.8** | 1.36x |
197
+ | 29,000-token context | 31.4 | **42.5** | 1.35x |
198
+
199
+ Code gains most and prose least. With more context behind it the head proposes
200
+ better -- it accepts about 63% of what it proposes on short prompts and 75-77%
201
+ from 14k tokens on.
202
+
203
+ Every token the head proposes is checked by the model itself, so the reply is the
204
+ model's own either way; the head only saves passes over the weights.