| # Quantized Meta-LLaMA-3.1-8B-Instruct vLLM Inference Project |
|
|
| ## 🚀 Project Overview |
| This project benchmarks the inference performance of: |
| - **vLLM** server with **Quantized (W8A8)** Meta-LLaMA-3.1-8B-Instruct model using **LLM Compressor** |
| - **Hugging Face Transformers** running full-precision **Meta-LLaMA-3.1-8B-Instruct** in float16. |
|
|
| We evaluate: |
| - Average latency (seconds) |
| - Throughput (tokens/sec) |
| - Output quality for identical prompts |
|
|
| --- |
|
|
| ## 🖥️ System Setup |
|
|
| - **GPU**: NVIDIA A100-SXM4-40GB |
| - **CUDA Version**: 12.4 |
| - **Driver Version**: 550.54.15 |
| - **Environment**: Google Colab (Virtual Environment: `llm_comp`) |
|
|
| --- |
|
|
| ## ⚙️ Tools and Libraries |
|
|
| - [vLLM](https://github.com/vllm-project/vllm) (Optimized LLM serving engine) |
| - [llmcompressor](https://github.com/vllm-project/llm-compressor) (One-shot quantization compression) |
|
|
|
|
| --- |
|
|
| ## 🛠 Workflow Summary |
|
|
| 1. **Quantized** Meta-LLaMA-3.1-8B-Instruct model using [LLM Compressor](https://github.com/vllm-project/llm-compressor), applying **SmoothQuant + GPTQ** techniques with **W8A8** quantization scheme. |
| 2. **Uploaded** the quantized model to Hugging Face Hub: `Roy2144/Meta-LLama`. |
| 3. **Deployed** the quantized model using vLLM OpenAI-compatible API server. |
| 4. **Ran Benchmarks**: |
| - vLLM server performance (Quantized) |
| - Hugging Face Transformers local inference (float16) |
| 5. **Measured**: |
| - Latency per request |
| - Throughput (tokens/sec) |
| - Response quality comparison |
| 6. **Analyzed** differences between quantized and float16 models. |
|
|
| --- |
|
|
| ## 📈 Benchmark Results |
|
|
| | Metric | vLLM + Quantized Meta-LLaMA W8A8 | HF Transformers + Meta-LLaMA float16 | |
| |:-------|:---------------------------------|:-------------------------------------| |
| | Avg Latency | 0.62 seconds | 2.66 seconds | |
| | Avg Throughput | 163.99 tokens/sec | 30.46 tokens/sec | |
|
|
| --- |
|
|
| ## 🧠 Key Findings |
|
|
| - **vLLM + Quantization** massively boosts inference speed (~4–5x faster than float16). |
| - **Quantized Meta-LLaMA-3.1-8B-Instruct** maintains excellent response quality despite INT8 compression. |
| - **HF Transformers** serve as a baseline for unoptimized float16 inference. |
|
|
| --- |
|
|
| ## ✨ Sample Prompt for Comparison |
|
|
| > **Prompt:** "Explain quantization in simple words for a beginner." |
|
|
| Both vLLM Quantized and HF float16 models generated detailed, coherent responses. |
| Quantized model responses were slightly more concise. |
|
|
| --- |
|
|
| ## 📚 Future Enhancements |
|
|
| - Stress testing with GuideLLM: simulate 100+ concurrent users. |
| - Deploy a chatbot frontend with Gradio connected to vLLM server. |
| - Experiment with even lighter quantization (e.g., W4A8, W6A8). |
| - Multi-GPU tensor parallelism scaling. |
|
|
| --- |
|
|
| ## 🏷️ Repo Details |
|
|
| - **Model Repo:** [Roy2144/Meta-LLama](https://huggingface.co/Roy2144/Meta-LLama) |
| - **Quantization Tool:** [LLM Compressor](https://github.com/vllm-project/llm-compressor) |
| - **Quantization Scheme:** SmoothQuant + GPTQ (W8A8) |
| - **Inference Engines:** vLLM, Hugging Face Transformers |
|
|
| --- |
|
|
