Added README.md with benchmark details

README.md

@@ -0,0 +1,75 @@
+# Quantized LLaMA vLLM Inference Project
+
+## 🚀 Project Overview
+This project benchmarks the inference performance of:
+- **vLLM** server with **Quantized (W8A8)** Mistral-7B model.
+- **Hugging Face Transformers** with **Meta-LLaMA-3.1-8B-Instruct** float16 model.
+
+We compare:
+- Average latency (seconds)
+- Throughput (tokens/sec)
+- Output quality for same prompts
+
+---
+
+## 🖥️ System Setup
+
+- **GPU**: A100-SXM4-40GB
+- **CUDA**: 12.4
+- **Driver**: 550.54.15
+- **Environment**: Google Colab Virtual Environment (`llm_comp`)
+
+---
+
+## ⚙️ Tools Used
+
+- [vLLM](https://github.com/vllm-project/vllm) (OpenAI API server for optimized LLM inference)
+- [llmcompressor](https://github.com/vllm-project/llm-compressor) (Quantization compression toolkit)
+- Hugging Face Transformers
+- PyTorch
+
+---
+
+## 🛠 Workflow
+
+1. **Quantized** `mistralai/Mistral-7B-Instruct-v0.2` model using `SmoothQuant + GPTQModifier (W8A8)`.
+2. **Uploaded** quantized model to Hugging Face Hub: `Roy2144/Meta-LLama`.
+3. **Started** vLLM server to serve quantized model.
+4. **Benchmark 1**: 
+    - vLLM + Quantized Mistral
+    - 5 prompts
+    - Measured average latency and tokens/sec.
+5. **Benchmark 2**: 
+    - HF Transformers + Meta-LLaMA-3.1-8B-Instruct
+    - 5 prompts
+    - Measured average latency and tokens/sec.
+6. **Compared** speed, memory usage, and quality.
+
+---
+
+## 📈 Benchmark Results
+
+| Metric | vLLM + Quantized Mistral-7B W8A8 | HF Transformers + Meta-LLaMA-3.1-8B fp16 |
+|:-------|:---------------------------------|:---------------------------------------|
+| Avg Latency | 0.62 s | 2.66 s |
+| Avg Throughput | 163.99 tokens/sec | 30.46 tokens/sec |
+
+---
+
+## 🧠 Conclusion
+
+- **vLLM + Quantization** massively improves inference speed (~4-5x faster).
+- **Output quality** remains acceptable even after W8A8 quantization.
+- **HF Transformers float16** slower but preserves maximum precision.
+
+**vLLM is strongly recommended** for scalable production deployment of quantized LLMs!
+
+---
+
+## ✍️ Future Work
+
+- Multi-user load testing with GuideLLM.
+- Full chatbot webapp via Gradio frontend.
+- Stress test up to 50 concurrent users.
+
+---