| --- |
| library_name: transformers |
| tags: |
| - speculative-decoding |
| - medusa |
| - qwen2.5 |
| - edge-ai |
| - inference-acceleration |
| - pytorch |
| base_model: Qwen/Qwen2.5-1.5B-Instruct |
| license: apache-2.0 |
| pipeline_tag: text-generation |
| --- |
| |
| # SpecDecode Edge: Medusa-3Heads for Qwen2.5-1.5B-Instruct |
|
|
| This repository provides trained **Medusa multi-head speculative decoding** weights for accelerating **`Qwen/Qwen2.5-1.5B-Instruct`** on single GPU and edge devices. |
|
|
| - **Target Model:** `Qwen/Qwen2.5-1.5B-Instruct` (Frozen) |
| - **Speculative Method:** Medusa Multi-Head Speculative Decoding (Cai et al., 2024) |
| - **Heads Configuration:** 3 speculative lookahead heads (offsets $t+2, t+3, t+4$) |
| - **Architecture per Head:** 1 ResBlock (SiLU) + Linear projection ($1536 \rightarrow 151936$) |
| - **Precision:** `bfloat16` |
| - **Trained Parameters:** ~707M parameters |
| - **Hardware Verified:** Single NVIDIA GeForce RTX 5060 Ti (15.93 GB VRAM) |
| - **Source Code & GitHub Repository:** [https://github.com/adityapande403/SpecDecode-edge](https://github.com/adityapande403/SpecDecode-edge) |
|
|
| --- |
|
|
| ## Performance & Acceleration |
|
|
| Empirical benchmarks on an NVIDIA RTX 5060 Ti: |
|
|
| | Workload Category | Baseline Decode TPS | Speculative Decode TPS | Speedup Ratio | Mean Acc / Step | |
| | :--- | :--- | :--- | :--- | :--- | |
| | **Code Generation** | 39.99 tok/s | **43.17 tok/s** | **1.08x** | **1.14** | |
| | **Mathematical Reasoning** | 40.41 tok/s | **42.25 tok/s** | **1.05x** | **1.12** | |
| | **Structured Output** | 39.23 tok/s | **39.44 tok/s** | **1.01x** | **1.05** | |
| | **Technical Explanations** | 38.64 tok/s | **39.40 tok/s** | **1.02x** | **1.05** | |
|
|
| - **Peak VRAM Consumption:** **`2.96` GB** (18.5% of 16GB GPU memory). |
| - **Authoritative Semantic Preservation:** Target model remains 100% authoritative. In 6 out of 8 categories, speculative decoding produced an exact bitwise token-by-token match with standard autoregressive generation. |
|
|
| --- |
|
|
| ## Usage with SpecDecode Edge |
|
|
| Clone the repository: |
| ```bash |
| git clone https://github.com/adityapande403/SpecDecode-edge.git |
| cd SpecDecode-edge |
| pip install -r requirements.txt |
| ``` |
|
|
| Run speculative generation: |
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| from huggingface_hub import hf_hub_download |
| from specdecode.models.medusa_model import MedusaModel |
| from specdecode.inference.speculative_generator import SpeculativeGenerator |
| |
| base_model_id = "Qwen/Qwen2.5-1.5B-Instruct" |
| medusa_repo_id = "Adityapande403/SpecDecode-edge" |
| |
| tokenizer = AutoTokenizer.from_pretrained(base_model_id) |
| base_model = AutoModelForCausalLM.from_pretrained(base_model_id, dtype=torch.bfloat16, device_map="cuda:0") |
| |
| medusa_model = MedusaModel(base_model, num_heads=3, num_layers=1, freeze_base=True) |
| |
| # Download and load Medusa heads |
| weights_path = hf_hub_download(repo_id=medusa_repo_id, filename="medusa_heads.safetensors") |
| medusa_model.load_medusa_heads(weights_path) |
| medusa_model.eval() |
| |
| generator = SpeculativeGenerator(medusa_model, tokenizer) |
| out = generator.generate("Write a quicksort implementation in Python:", max_new_tokens=80) |
| print(out.generated_text) |
| ``` |
|
|
| --- |
|
|
| ## Citation & Acknowledgements |
|
|
| ```bibtex |
| @software{specdecode_edge_2026, |
| author = {Aditya Pande}, |
| title = {SpecDecode Edge: Training and Deploying Speculative Decoding Models for Local LLM Inference}, |
| year = {2026}, |
| url = {https://github.com/adityapande403/SpecDecode-edge} |
| } |
| ``` |
|
|
