senthilsdglakhsg/phi3-Q4.gguf

Phi-3 Mini (GGUF Quantized - Q4_K_M)

Model Description

This repository contains a quantized GGUF version of the Phi-3 Mini 128K Instruct model.

• Base Model: microsoft/Phi-3-mini-128k-instruct
• Format: GGUF
• Quantization: Q4_K_M
• Framework: llama.cpp

This model is optimized for efficient local inference with reduced memory usage.

---

Intended Use

• Local LLM inference
• Chatbots
• Lightweight deployments
• CPU/GPU inference using llama.cpp

---

Model Details

Base Model

• Microsoft Phi-3 Mini (128K context)

Conversion

• Converted from Hugging Face format → GGUF

Quantization

• Method: Q4_K_M
• Tool: llama.cpp quantize

This reduces model size while maintaining reasonable performance.

---

How to Use

Using llama.cpp

./llama-cli -m model-q4.gguf -p "Explain AI simply"