Spaces:
Sleeping
Sleeping
Commit History
-
Fix vLLM device detection for ZeroGPU
- Set CUDA_VISIBLE_DEVICES environment variable for vLLM - Add CUDA detection logging at module level - Ensure CUDA is available before attempting vLLM load - Improve error messages for device detection issues2ddfeca -
Fix vLLM token parameter and improve streaming error handling
- Remove 'token' parameter from vLLM LLM() call (uses HF_TOKEN env var) - Add better error handling for generation thread - Add debug logging for streamer token consumption - Add timeout handling for generation threadb4fd5e9
-
Add debug logging for model loading and generation issues
54880b1 -
Fix streaming loop break condition - only break when finished is True
d6f9002 -
Add deployment status document after re-authentication
1fb66ec
-
Add permission fix guide for spherical-gate-477614-q7 project
162c75a -
Add Cloud Build deployment script and permission setup helper
- deploy-cloud-build.sh: Uses Cloud Build service (no local Docker needed) - setup-gcp-permissions.sh: Helper to create project and enable APIs - Updated cloudbuild.yaml with better substitutions - QUICK_DEPLOY.md: Troubleshooting guide for permission issuesfd26b3d -
Add Cloud Run PORT environment variable support
1b04006 -
Add Google Cloud Platform deployment configurations
- Dockerfile for containerization - Cloud Run deployment script (serverless, CPU) - Compute Engine deployment script (GPU support) - Cloud Build configuration - Comprehensive deployment documentation - Support for PORT and GRADIO_SERVER_PORT env vars for Cloud Run compatibilityaa65d00 -
Fix Gradio UI structure and add comprehensive fallback logging
- Fix UI structure: move buttons to separate Row, fix Column nesting - Add detailed fallback chain logging with emoji indicators - Show clear progression: vLLM → Transformers AWQ → BitsAndBytes → FP16/FP32 - Improve error messages to show which fallback path is being used - All fallback paths now properly logged for debugging03689e3 -
Fix all indentation errors in Gradio UI components
06aef1b -
Fix syntax error: correct indentation in BitsAndBytes fallback block
f43bdac -
Suppress AutoAWQ deprecation warnings and improve vLLM logging
- Filter AutoAWQ deprecation warnings (vLLM handles AWQ natively via llm-compressor) - Improve vLLM loading logs with detailed status information - Add better error handling with traceback for debugging - Clarify that vLLM is primary path (AutoAWQ only for Transformers fallback)83a232d -
Implement vLLM with LLM Compressor and performance optimizations
Major optimizations for faster deployment and inference: 1. vLLM Integration (Primary Engine): - Native AWQ quantization support (auto-detects AWQ weights) - Continuous batching (max_num_seqs=256) for concurrent requests - PagedAttention for efficient KV cache management - Prefix caching enabled for faster TTFT on repeated prompts - Chunked prefill for long context handling - Optimized for ZeroGPU H200 slice (gpu_memory_utilization=0.90) 2. Performance Optimizations: - torch.compile with mode='reduce-overhead' (~10-20% speedup after first call) - FlashAttention-2 enabled when available - Prefer bfloat16 over int8 for speed (bf16 ~15-23% faster than int8) - Non-blocking streaming with TextIteratorStreamer - CUDA kernel warmup on startup 3. Fallback Chain: - vLLM AWQ (primary) → vLLM FP16 → Transformers AWQ → BitsAndBytes 8-bit → FP16/FP32 - Graceful degradation ensures compatibility 4. Deployment Speed: - Model caching in VLLM_MODELS dict …a79facb -
Migrate to AWQ quantization with FlashAttention-2
- Replace BitsAndBytes 8-bit with AWQ 4-bit quantization (primary path) - Add FlashAttention-2 support for optimized attention - Enable TF32 math for Ampere+ GPUs - Add CUDA kernel warmup on startup to reduce first-token latency - Update model names to reflect AWQ optimization - Add graceful fallback chain: AWQ -> BitsAndBytes -> bf16/fp16/fp32 - Update requirements.txt with flash-attn>=2.5.0 - Update README with performance optimizations documentation06b4cf5 -
Fix: Pre-create GPU wrappers at module load time for startup detection
- Create GPU-decorated functions at module load time (not dynamically) - This ensures spaces.GPU decorators are detected during startup - Pre-create wrappers for all durations from 60-1800 seconds (every 60s) - Fix closure variable capture issue - Slider value now properly controls GPU duration allocationcdac920 -
Make GPU duration slider functional with dynamic wrapper creation
- Create GPU wrappers dynamically based on user's gpu_duration slider value - Cache wrappers to avoid recreating them for the same duration - Round duration to nearest 60 seconds for efficient caching - Now the slider value actually controls GPU allocation durationfc0ab14 -
Fix indentation errors in _generate_router_plan_streaming_internal
c454e43 -
Fix: Remove context manager usage for spaces.GPU decorator
- spaces.GPU is a decorator, not a context manager - Use decorator with maximum duration (1800s) to allow user flexibility - Keep gpu_duration parameter for user awareness/UI - Fix AttributeError: __enter__ errora217627 -
Add user-configurable GPU duration slider (60-1800 seconds)
- Add GPU Duration slider in UI (default: 600 seconds) - Refactor to use spaces.GPU context manager with dynamic duration - Allow users to set GPU time allocation per request - Maintain backward compatibility with default 600s wrapper9a4d6d3 -
Fix: Move trim_at_stop_sequences function before it's used
597f1a9 -
Add Gradio client API test script
de18e95 -
Fix API launch configuration
9773e4b -
Enable API in Gradio launch configuration
1b16b00
-
Update README and clean up old files
9592189 -
Improve streaming with incremental JSON parsing and plan end token
- Add PLAN_END_TOKEN for explicit plan termination - Enhance system prompt with detailed JSON structure rules - Implement incremental JSON parsing during streaming - Update default max_new_tokens to 16000 - Add API names for endpoints - Improve validation feedback during streamingf5a609d -
Add streaming support and increase max tokens to 20000
- Implement token streaming using TextIteratorStreamer - Increase max_new_tokens slider from 1024 to 20000 - Convert generation function to generator for real-time token output - Add progress indicator during generation4f65341 -
Fix deprecation warnings and improve error handling
- Replace deprecated load_in_8bit with BitsAndBytesConfig - Fix dropdown value to dynamically use first model - Increase GPU duration to 600s for model loading - Add better error handling for GPU task aborted errors - Add model_choice validationbf2fdae -
Update app.py and requirements.txt for CourseGPT-Pro router models
4c3d05b -
Update README: Focus on CourseGPT-Pro router checkpoints
4706b45 -
Update README with correct space URL
9af4b77 -
Add .gitignore and remove cache files
7bc8a45 -
Initial commit: ZeroGPU LLM Inference Space
f91e906