Copy main.py to Docker image to pick up syntax fixes
Andrew McCracken+1: Claude
7c501a5
- Ensures latest main.py with f-string fixes is included
- Fixes SyntaxError in streaming endpoint
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add concurrent request handling with model pool
Andrew McCracken+1: Claude
efd4459
Implemented ModelPool for true concurrent processing:
- Created ModelPool class with thread-safe queue
- Initializes 10 model instances (configurable via MODEL_POOL_SIZE)
- Each instance can handle one request simultaneously
- Automatic model checkout/return from pool
- Added pool statistics to /health endpoint
Configuration:
- MODEL_POOL_SIZE=10 (supports 10 concurrent users)
- 60s timeout if all instances busy
- Each model instance ~2.4GB VRAM
- Total VRAM: ~24GB for 10 instances (fits in 48GB GPU)
Sessions are handled via session_id parameter (already present)
Pool automatically balances load across instances
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add GPU support
Andrew McCracken+1: Claude
bfa102d
Added GPU-enabled Docker configuration:
- Dockerfile.base.gpu: CUDA 12.1 base with llama-cpp-python GPU support
- Dockerfile.gpu: HF Spaces GPU deployment dockerfile
- build-and-push-gpu.sh: Script to build and push GPU image
- Updated llm_handler.py to use N_GPU_LAYERS env variable
To use GPU:
1. Build: ./build-and-push-gpu.sh
2. Switch HF Space to GPU hardware
3. Use Dockerfile.gpu for deployment
Expected speedup: ~15s → 1-3s per response
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
