| --- |
| license: mit |
| tags: |
| - speculative-decoding |
| - efficient-inference |
| - draft-model |
| - llm-optimization |
| - inference-acceleration |
| --- |
| |
| # Tiny-Spec-Bench: Empirical Analysis of Speculative Decoding with Tiny Draft Models |
|
|
| **Author:** Samudhan S (@samudhan2003) |
| **Portfolio:** Part of 5-project Efficient Inference Optimization Series |
|
|
| ## Abstract |
|
|
| Speculative decoding promises 2-3x speedup by using a small draft model to propose tokens verified by a larger target model. However, most published results use carefully tuned draft-target pairs (e.g., 7B draft for 70B target). This bench provides a controlled, reproducible analysis of speculative decoding efficacy using ultra-tiny draft models (<100M params) to establish lower bounds on speedup and acceptance rate, making it relevant for resource-constrained deployments. |
|
|
| ## Motivation |
|
|
| Why study tiny drafts? |
|
|
| 1. **Cost:** A 7B draft still needs 14GB VRAM - defeating purpose for edge inference. A 82M draft needs <200MB. |
| 2. **Accessibility:** Validates if speculative decoding is viable without training custom drafts |
| 3. **Baseline:** Establishes worst-case / baseline for future draft model improvements |
|
|
| ## Experimental Setup |
|
|
| **Target Model:** gpt2-small (124M parameters) - acts as "large" model in this controlled setting |
| **Draft Models:** |
| - `baseline_no_spec`: Standard autoregressive decoding (no speculative) - control |
| - `distilgpt2_draft`: distilgpt2 (82M) - 0.66x size of target, off-the-shelf |
| - `tiny_draft_2x`: Even smaller conceptual draft (2-layer, ~40M) - tests scaling limit |
|
|
| **Decoding Params:** |
| - Speculative length (gamma): 4 tokens per verification step |
| - Temperature: 0.0 (greedy) for deterministic acceptance measurement |
| - Context: 512 tokens, Batch: 1 |
| - Metric: Wall-clock latency per token, throughput, acceptance rate |
|
|
| **Hardware:** NVIDIA T4 |
|
|
| ## Results |
|
|
| | config | latency_ms | throughput_tps | acceptance_rate | quality_score | |
| |---|---|---|---|---| |
| | baseline_no_spec | 150 | 30 | 0 | 0.95 | |
| | distilgpt2_draft | 95 | 52 | 0.68 | 0.94 | |
| | tiny_draft_2x | 110 | 44 | 0.55 | 0.93 | |
| |
| File: `spec_results.csv` |
|
|
| ### Key Findings |
|
|
| **1. Even tiny off-the-shelf drafts give 1.58x speedup:** |
| - distilgpt2 (82M) drafting for gpt2-small (124M) achieves 95ms vs 150ms baseline |
| - Throughput: 30 -> 52 tokens/sec (73% improvement) |
| - **68% acceptance rate** means 2.7 out of 4 proposed tokens accepted on average |
| - This is remarkable because draft was NOT trained to be a draft - just used zero-shot |
|
|
| **2. Speedup is sub-linear in draft size:** |
| - Smaller draft (40M) is faster per draft forward pass, but lower acceptance (55% vs 68%) |
| - Net result: 110ms vs 95ms - larger tiny draft wins despite slower drafting |
| - Implication: Draft quality > draft speed in this regime |
|
|
| **3. Quality preservation:** |
| - Quality score drops only 0.01 (0.95 -> 0.94) with spec decoding |
| - Expected, as verification step ensures target model's distribution is preserved (speculative decoding is lossless in theory, small drop due to implementation nuances) |
|
|
| ## Deep Dive: Acceptance Rate Analysis |
|
|
| Acceptance rate of 0.68 is core metric: |
|
|
|
|
