File size: 3,388 Bytes
f82a0e3 a1ad09c f82a0e3 a1ad09c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 | ---
library_name: transformers
tags:
- speculative-decoding
- medusa
- qwen2.5
- edge-ai
- inference-acceleration
- pytorch
base_model: Qwen/Qwen2.5-1.5B-Instruct
license: apache-2.0
pipeline_tag: text-generation
---
# SpecDecode Edge: Medusa-3Heads for Qwen2.5-1.5B-Instruct
This repository provides trained **Medusa multi-head speculative decoding** weights for accelerating **`Qwen/Qwen2.5-1.5B-Instruct`** on single GPU and edge devices.
- **Target Model:** `Qwen/Qwen2.5-1.5B-Instruct` (Frozen)
- **Speculative Method:** Medusa Multi-Head Speculative Decoding (Cai et al., 2024)
- **Heads Configuration:** 3 speculative lookahead heads (offsets $t+2, t+3, t+4$)
- **Architecture per Head:** 1 ResBlock (SiLU) + Linear projection ($1536 \rightarrow 151936$)
- **Precision:** `bfloat16`
- **Trained Parameters:** ~707M parameters
- **Hardware Verified:** Single NVIDIA GeForce RTX 5060 Ti (15.93 GB VRAM)
- **Source Code & GitHub Repository:** [https://github.com/adityapande403/SpecDecode-edge](https://github.com/adityapande403/SpecDecode-edge)
---
## Performance & Acceleration
Empirical benchmarks on an NVIDIA RTX 5060 Ti:
| Workload Category | Baseline Decode TPS | Speculative Decode TPS | Speedup Ratio | Mean Acc / Step |
| :--- | :--- | :--- | :--- | :--- |
| **Code Generation** | 39.99 tok/s | **43.17 tok/s** | **1.08x** | **1.14** |
| **Mathematical Reasoning** | 40.41 tok/s | **42.25 tok/s** | **1.05x** | **1.12** |
| **Structured Output** | 39.23 tok/s | **39.44 tok/s** | **1.01x** | **1.05** |
| **Technical Explanations** | 38.64 tok/s | **39.40 tok/s** | **1.02x** | **1.05** |
- **Peak VRAM Consumption:** **`2.96` GB** (18.5% of 16GB GPU memory).
- **Authoritative Semantic Preservation:** Target model remains 100% authoritative. In 6 out of 8 categories, speculative decoding produced an exact bitwise token-by-token match with standard autoregressive generation.
---
## Usage with SpecDecode Edge
Clone the repository:
```bash
git clone https://github.com/adityapande403/SpecDecode-edge.git
cd SpecDecode-edge
pip install -r requirements.txt
```
Run speculative generation:
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from huggingface_hub import hf_hub_download
from specdecode.models.medusa_model import MedusaModel
from specdecode.inference.speculative_generator import SpeculativeGenerator
base_model_id = "Qwen/Qwen2.5-1.5B-Instruct"
medusa_repo_id = "Adityapande403/SpecDecode-edge"
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = AutoModelForCausalLM.from_pretrained(base_model_id, dtype=torch.bfloat16, device_map="cuda:0")
medusa_model = MedusaModel(base_model, num_heads=3, num_layers=1, freeze_base=True)
# Download and load Medusa heads
weights_path = hf_hub_download(repo_id=medusa_repo_id, filename="medusa_heads.safetensors")
medusa_model.load_medusa_heads(weights_path)
medusa_model.eval()
generator = SpeculativeGenerator(medusa_model, tokenizer)
out = generator.generate("Write a quicksort implementation in Python:", max_new_tokens=80)
print(out.generated_text)
```
---
## Citation & Acknowledgements
```bibtex
@software{specdecode_edge_2026,
author = {Aditya Pande},
title = {SpecDecode Edge: Training and Deploying Speculative Decoding Models for Local LLM Inference},
year = {2026},
url = {https://github.com/adityapande403/SpecDecode-edge}
}
```
|