| # Lightweight LLM with MCP Integration |
|
|
| This directory contains a lightweight LLM inference service that can connect to external MCP (Model Context Protocol) servers for enhanced capabilities. |
|
|
| ## 🏗️ Quick Setup |
|
|
| ### Local Development |
| ```bash |
| # Install dependencies |
| pip install -r requirements.txt |
| |
| # Run the service |
| python app.py |
| |
| # Or use the start script |
| ./start.sh |
| ``` |
|
|
| ### Docker Deployment |
| ```bash |
| # Build the image |
| docker build -t lightweight-llm . |
| |
| # Run the container |
| docker run -p 8000:8000 \ |
| -e MCP_SERVER_URL=https://your-mcp-server.hf.space \ |
| lightweight-llm |
| ``` |
|
|
| ### Hugging Face Spaces |
| 1. Create a new Space with Docker SDK |
| 2. Upload all files from this directory |
| 3. Set `app_port: 8000` in README.md header |
| 4. Configure environment variables in Space settings |
|
|
| ## 📡 API Usage |
|
|
| ### Basic Inference |
| ```bash |
| curl -X POST http://localhost:8000/ \ |
| -H "Content-Type: application/json" \ |
| -d '{ |
| "inputs": "Hello, how are you?", |
| "parameters": {"max_new_tokens": 100} |
| }' |
| ``` |
|
|
| ### MCP-Enhanced Inference |
| ```bash |
| curl -X POST http://localhost:8000/ \ |
| -H "Content-Type: application/json" \ |
| -d '{ |
| "inputs": "Tell me about product 123", |
| "parameters": { |
| "max_new_tokens": 200, |
| "mcp_server_url": "https://your-mcp-server.hf.space" |
| } |
| }' |
| ``` |
|
|
| ## 🔧 Configuration |
|
|
| Set these environment variables: |
| - `MCP_SERVER_URL`: Default MCP server URL |
| - `MODEL_NAME`: Hugging Face model name |
| - `MAX_NEW_TOKENS`: Default max tokens |
| - `PORT`: Service port (default: 8000) |
|
|
| ## 🧪 Testing |
|
|
| Run the test suite: |
| ```bash |
| python test_service.py |
| ``` |
|
|
| ## 🚀 Production Notes |
|
|
| - The current implementation uses a simple text generation placeholder |
| - For production, integrate with: |
| - Transformers library with a small model |
| - vLLM for faster inference |
| - TensorRT for optimized inference |
| - Any other inference engine |
|
|
| ## 🔗 MCP Integration |
|
|
| This service automatically detects when to use MCP tools based on keywords in the input. It can connect to any MCP server that implements the standard protocol. |
|
|
| **Keywords that trigger MCP usage:** |
| - product, price, stock, inventory |
| - order, customer, database |
| - search, find, get, fetch, check |
| - prestashop, shop, cart, purchase |
|
|
| ## 📊 Endpoints |
|
|
| - `POST /` - Main inference endpoint (HF compatible) |
| - `GET /health` - Health check |
| - `GET /info` - Service information |
| - `GET /docs` - API documentation |
|
|
