Spaces:
Sleeping
Sleeping
Commit History
-
Remove outlines library and all related code
ming- Remove outlines dependency from requirements.txt (NumPy compatibility issue) - Remove outlines import validation from Dockerfile - Remove /stream-json endpoint from V4 API (outlines-based) - Remove all outlines-related code from structured_summarizer.py - Remove outlines import tests - Delete test_v4_live.py (outlines integration tests) - Remove all stream-json endpoint tests from test_v4_api.py This resolves the Hugging Face build failure caused by outlines==0.0.44 incompatibility with newer NumPy versions (ModuleNotFoundError: numpy.lib.function_base). V4 API now only supports /stream and /stream-ndjson endpoints using NDJSON patch-based structured summarization (no external dependencies).d25a17f
-
Add Outlines installation verification to Dockerfile
ming- Verify Outlines installs correctly during Docker build - Print version if successful, fail build if installation fails2ff86e9 -
Add OMP_NUM_THREADS env var and improve Outlines requirement comment
ming- Set OMP_NUM_THREADS=1 to fix libgomp warning - Add comment about Python 3.10+ requirement for Outlines55d10ea -
Fix Python 3.10 requirement and torch_dtype deprecation
ming- Update Dockerfile to Python 3.10 (Outlines requires 3.10+) - Replace deprecated torch_dtype with dtype parameter - Fixes TypeError with Outlines library on Python 3.96b859f2
-
Implement Option 3: Use FP16 for 2-3x faster inference
mingSpeed optimization: - Added v4_use_fp16_for_speed config option - When enabled, uses FP16 instead of 4-bit quantization - FP16 is 2-3x faster than 4-bit NF4 quantization - Enabled by default in Dockerfile for maximum speed Memory trade-off: - FP16 uses ~2-3GB GPU memory (vs ~1GB for 4-bit) - Still fits comfortably on T4 GPU (16GB total) Expected results: - Generation time: 24.9s → 8-12s (2-3x speedup) - Same output quality - Faster token generation (~20-30 tokens/sec vs ~6 tokens/sec) This completes the speed optimization plan (Option 1 + 2 + 3)7fff563 -
Fix bitsandbytes UID error with getpass patch
ming- Added getpass.getuser() mock in structured_summarizer.py - Mock catches KeyError and returns USER env var or 'user' - Added HOME=/tmp and USER=user to Dockerfile - Fixes 'getpwuid(): uid not found: 1000' error in HF Spaces - Allows bitsandbytes 4-bit NF4 quantization to initializedd29a6d -
Switch V4 to GPU INT4 quantization with Qwen-1.5B
ming- Upgraded transformers to 4.44+ and accelerate to 0.33+ - Added bitsandbytes for 4-bit NF4 quantization on GPU - Removed torchao dependency (causes HF Spaces errors) - Added TRANSFORMERS_NO_TORCHAO=1 to prevent import errors - Updated to use Qwen/Qwen2.5-1.5B-Instruct model - GPU: 4-bit NF4 quantization via bitsandbytes - CPU fallback: FP32 + dynamic INT8 quantizationa36f560
-
Add support for sshleifer/distilbart-cnn-6-6 model for V2 API
ming- Added BART model detection and specific input handling - Updated default model from t5-small to sshleifer/distilbart-cnn-6-6 - BART models now receive direct text input without prefixes - Updated warmup and health check methods for BART compatibility - Updated Dockerfile and README documentation6e48ad3 -
Optimize V2 API for HuggingFace Spaces with t5-small
ming- Change default model from Phi-3-mini to t5-small (~250MB vs 7GB) - Update Dockerfile for V2-only deployment (remove Ollama) - Add t5-small specific handling in HF streaming service - Add HuggingFace cache directory configuration - Update README with V2-primary deployment info - Optimize for HuggingFace Spaces free tier limits - Fast startup: ~30-60 seconds vs 2-3 minutes - Memory efficient: ~500MB RAM vs 8GB+ for Phi-3 - V2 model downloads during warmup for instant first requestd174be4
-
Switch to llama3.2:1b base model for better availability
ming- Use llama3.2:1b instead of llama3.2:1b-instruct - Base model should be more readily available in Ollama - Still small enough to fit within storage limits - Should resolve the model not found error64e4f7a -
Fix storage limit issue by switching to llama3.2:1b-instruct
ming- Switch to 1B parameter model to reduce storage usage - Add model cleanup step to remove existing models and free space - 1B model should fit within Hugging Face Spaces 50GB limit - Still instruction-fine-tuned for good summarization quality - Much faster inference than larger models43c3d8e -
Switch to llama3.2:3b-instruct model for faster performance
ming- Update default model to llama3.2:3b-instruct (3B parameters) - Smaller model should provide faster response times - Still instruction-fine-tuned for better summarization quality - Reduced memory and resource usage4cc543a -
Switch to mistral:7b-instruct model for better instruction following
ming- Update default model in config.py to mistral:7b-instruct - Update Dockerfile to pull the new model during startup - The instruct variant is better optimized for summarization tasksaf820b0 -
Use /tmp directory for Ollama to bypass file system permission restrictions
mingc3ba37c -
Run as root to bypass Hugging Face Spaces permission restrictions
ming6660cd8 -
Fix Ollama directory permissions by setting proper ownership and permissions during build
ming26568af -
Remove chmod command that's not permitted in Hugging Face Spaces
mingd72587d -
Fix Ollama permissions by using /app/.ollama directory instead of /root/.ollama
ming7716cd4 -
Improve Ollama startup with better error handling and health checks
mingb75970a -
Fix Ollama permissions and model configuration for Hugging Face deployment
mingbd3417a -
Add Hugging Face Spaces configuration and deployment files
ming3a7a125
-
feat: add Docker containerization with Ollama integration
ming- Add multi-stage Dockerfile with security best practices - Add docker-compose.yml for full stack deployment - Add docker-compose.dev.yml for development with hot reload - Add nginx.conf for production reverse proxy with rate limiting - Add .dockerignore for optimized builds - Add setup-ollama.sh script for model management - Add run-tests.sh script for Docker-based testing - Update README with comprehensive Docker deployment guide2043365