File size: 1,198 Bytes
a181ec9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 | # Serve in vLLM
This tutorial shows you how to deploy your trained speculator model for production inference using vLLM.
After training a speculator, deploying it with vLLM is straightforward. vLLM automatically recognizes the `speculators_config` in your model and enables speculative decoding.
## Basic Serving
```bash
vllm serve ./checkpoints/checkpoint_best
```
That's it! vLLM will:
1. Load your speculator model
2. Read the `speculators_config` from config.json
3. Load the target/verifier model
4. Enable speculative decoding automatically
## Long form command
```bash
vllm serve Qwen/Qwen3-8B \
-tp 1 \
--speculative-config '{
"model": "RedHatAI/Qwen3-8B-speculator.eagle3",
"num_speculative_tokens": 3,
"method": "eagle3"
}'
```
For the long form command, pass in the target model first (e.g. `Qwen/Qwen3-8B`) and then specify the draft model in the speculative config (e.g. `RedHatAI/Qwen3-8B-speculator.eagle3`).
The long form command can be used to change the `num_speculative_tokens` or to use a different target model with the speculator. This can be used to combine a quantized target model, with a speculative decoding model for even better performance.
|