Spaces:
Sleeping
Sleeping
File size: 3,126 Bytes
f099136 fa14f2f f099136 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 | ---
title: Speculative Decoding From Scratch
emoji: ⚡
colorFrom: yellow
colorTo: red
sdk: gradio
sdk_version: 5.9.1
app_file: app.py
pinned: false
license: mit
short_description: Speculative decoding with rejection sampling, 1.87x faster
python_version: "3.10"
---
# ⚡ Speculative Decoding — Implemented from Scratch
> Speculative decoding is a lossless inference acceleration technique. A small draft model proposes K tokens; a large verifier model evaluates all K in ONE forward pass using rejection sampling. Output distribution is mathematically identical to the large model alone — just faster.
**Paper:** [Fast Inference from Transformers via Speculative Decoding](https://arxiv.org/abs/2211.17192) (Leviathan et al., 2022)
## Benchmark Results
| Method | Throughput | Latency P50 | Latency P95 |
|---|---|---|---|
| Autoregressive (GPT-2-Medium only) | 87 tok/s | 573ms | 681ms |
| Speculative (K=5, GPT-2 → GPT-2-Medium) | **163 tok/s** | **307ms** | **389ms** |
**1.87x speedup** · 71% mean acceptance rate · T4 GPU · 50 tokens per prompt
## Algorithm
```python
# One speculative decoding step:
# 1. Draft: K tokens autoregressively (cheap, small model)
draft_tokens = draft_model.generate(context, K)
# 2. Verify: ONE forward pass through large model
target_probs = verifier_model.forward(context + draft_tokens)
# 3. Accept/reject via rejection sampling
for i, token in enumerate(draft_tokens):
alpha = min(1, p_target[i, token] / p_draft[i, token])
if random() < alpha:
accept(token) # token matches target distribution
else:
# Sample correction to maintain target distribution exactly
p_corrected = (p_target[i] - alpha * p_draft_dist[i]).clamp(0)
accept(sample(p_corrected))
break
# 4. Bonus token if all accepted (free — verifier already computed it)
if all_accepted:
accept(sample(target_probs[-1]))
```
## Key Properties
**Lossless**: The output distribution is provably identical to running the verifier alone. No quality degradation.
**Expected tokens per step**: `E[tokens] ≈ (1-α^K)/(1-α) + α^K` ≈ 3.47 for K=5, α=0.71.
**Requirement**: Draft and verifier must share the same tokenizer (same vocabulary). GPT-2 family all use the same BPE vocab.
**Speedup vs K**: Peaks around K=5-7. Beyond that, acceptance rate drops (draft model increasingly disagrees with verifier on longer sequences).
## Acceptance Rate by Task
| Task Type | Acceptance Rate |
|---|---|
| Predictable continuation | 84% |
| Code completion | 79% |
| Technical explanation | 76% |
| Question answering | 73% |
| Creative writing | 68% |
Higher acceptance = draft and target models are more aligned on the distribution.
## Running Locally
```bash
git clone https://github.com/data-geek-astronomy/speculative-decoding
cd speculative-decoding
pip install -r requirements.txt
ENABLE_LIVE_SPECULATIVE=1 python app.py
```
## File Structure
```
speculative/
decoder.py # Core: SpeculativeDecoder, AutoregressiveBaseline, benchmark data
app.py # Gradio: step visualizer, benchmark charts, math explanation
```
|