# Quantized Meta-LLaMA-3.1-8B-Instruct vLLM Inference Project

## 🚀 Project Overview
This project benchmarks the inference performance of:
- **vLLM** server with **Quantized (W8A8)** Meta-LLaMA-3.1-8B-Instruct model using **LLM Compressor**
- **Hugging Face Transformers** running full-precision **Meta-LLaMA-3.1-8B-Instruct** in float16.

We evaluate:
- Average latency (seconds)
- Throughput (tokens/sec)
- Output quality for identical prompts

---

## 🖥️ System Setup

- **GPU**: NVIDIA A100-SXM4-40GB
- **CUDA Version**: 12.4
- **Driver Version**: 550.54.15
- **Environment**: Google Colab (Virtual Environment: `llm_comp`)

---

## ⚙️ Tools and Libraries

- [vLLM](https://github.com/vllm-project/vllm) (Optimized LLM serving engine)
- [llmcompressor](https://github.com/vllm-project/llm-compressor) (One-shot quantization compression)


---

## 🛠 Workflow Summary

1. **Quantized** Meta-LLaMA-3.1-8B-Instruct model using [LLM Compressor](https://github.com/vllm-project/llm-compressor), applying **SmoothQuant + GPTQ** techniques with **W8A8** quantization scheme.
2. **Uploaded** the quantized model to Hugging Face Hub: `Roy2144/Meta-LLama`.
3. **Deployed** the quantized model using vLLM OpenAI-compatible API server.
4. **Ran Benchmarks**:
   - vLLM server performance (Quantized)
   - Hugging Face Transformers local inference (float16)
5. **Measured**:
   - Latency per request
   - Throughput (tokens/sec)
   - Response quality comparison
6. **Analyzed** differences between quantized and float16 models.

---

## 📈 Benchmark Results

| Metric | vLLM + Quantized Meta-LLaMA W8A8 | HF Transformers + Meta-LLaMA float16 |
|:-------|:---------------------------------|:-------------------------------------|
| Avg Latency | 0.62 seconds | 2.66 seconds |
| Avg Throughput | 163.99 tokens/sec | 30.46 tokens/sec |

---

## 🧠 Key Findings

- **vLLM + Quantization** massively boosts inference speed (~4–5x faster than float16).
- **Quantized Meta-LLaMA-3.1-8B-Instruct** maintains excellent response quality despite INT8 compression.
- **HF Transformers** serve as a baseline for unoptimized float16 inference.

---

## ✨ Sample Prompt for Comparison

> **Prompt:** "Explain quantization in simple words for a beginner."

Both vLLM Quantized and HF float16 models generated detailed, coherent responses.  
Quantized model responses were slightly more concise.

---

## 📚 Future Enhancements

- Stress testing with GuideLLM: simulate 100+ concurrent users.
- Deploy a chatbot frontend with Gradio connected to vLLM server.
- Experiment with even lighter quantization (e.g., W4A8, W6A8).
- Multi-GPU tensor parallelism scaling.

---

## 🏷️ Repo Details

- **Model Repo:** [Roy2144/Meta-LLama](https://huggingface.co/Roy2144/Meta-LLama)
- **Quantization Tool:** [LLM Compressor](https://github.com/vllm-project/llm-compressor)
- **Quantization Scheme:** SmoothQuant + GPTQ (W8A8)
- **Inference Engines:** vLLM, Hugging Face Transformers

---