Added GPU-enabled Docker configuration:
- Dockerfile.base.gpu: CUDA 12.1 base with llama-cpp-python GPU support
- Dockerfile.gpu: HF Spaces GPU deployment dockerfile
- build-and-push-gpu.sh: Script to build and push GPU image
- Updated llm_handler.py to use N_GPU_LAYERS env variable
To use GPU:
1. Build: ./build-and-push-gpu.sh
2. Switch HF Space to GPU hardware
3. Use Dockerfile.gpu for deployment
Expected speedup: ~15s → 1-3s per response
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Revert to simpler configuration - optimizations caused slowdown
Andrew McCracken+1: Claude
457c9e1
- Reverted n_ctx back to 4096 (2048 was too small)
- Reverted streaming to simple token-by-token (buffering caused issues)
- Reduced threads to 6 (8 may have been overloading)
- Kept max_tokens at 256 (only change that should help)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Optimize for faster inference
Andrew McCracken+1: Claude
8cfe5b7
Performance optimizations:
1. Reduced max_tokens from 512 to 256 (faster responses)
2. Reduced n_ctx from 4096 to 2048 (faster prompt processing)
3. Added token buffering in streaming (better perceived speed)
- Buffers 3 tokens or until whitespace
- Reduces network overhead
Expected speedup: 15s → 8-10s per response
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Increase threads to 8 for faster inference
Andrew McCracken+1: Claude
3f2ee19
- Use all 8 vCPUs for maximum inference speed
- Should reduce response time from ~15s to ~10-12s
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Optimize for 8vCPU/32GB instance
Andrew McCracken+1: Claude
b7fb901
- Use 6 threads (leave 2 for system/API)
- Restore n_ctx=4096 and n_batch=512 for better quality
- Restore max_tokens=512 for complete responses
- Enable verbose logging for debugging
- f16_kv=True for memory efficiency with sufficient RAM
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Optimize model parameters for faster CPU inference
Andrew McCracken+1: Claude
6e83384
- Reduced n_ctx from 4096 to 2048 for faster inference
- Reduced n_batch from 512 to 256 for lower memory usage
- Reduced n_threads from 8 to 4 for HF Spaces CPU limits
- Changed f16_kv to False for better CPU performance
- Reduced default max_tokens from 512 to 256 for faster responses
These changes need to be rebuilt into the base Docker image.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
