Spaces:
Runtime error
Download documentation/performance-optimization.md from Agents-MCP-Hackathon/TubeGenius: direct link, hf CLI and curl.
- Browser
- Download file 9.39 kB
-
https://huggingface.co/spaces/Agents-MCP-Hackathon/TubeGenius/resolve/main/documentation/performance-optimization.md
- Command line
-
hf download hf://spaces/Agents-MCP-Hackathon/TubeGenius/documentation/performance-optimization.md
-
curl -L -o performance-optimization.md https://huggingface.co/spaces/Agents-MCP-Hackathon/TubeGenius/resolve/main/documentation/performance-optimization.md
A newer version of the Gradio SDK is available: 6.32.0
Performance Optimization Guide
SDXL Model Caching
Problem
The SmartThumbnailer uses Stable Diffusion XL (SDXL) to generate AI thumbnails. By default, the base SDXL model (~9GB) was being downloaded from HuggingFace every time a thumbnail generation was requested, causing:
- Long wait times: 30-40 seconds just to download the model
- Bandwidth waste: Re-downloading the same 9GB model repeatedly
- Poor user experience: Most time spent waiting, not generating
Solution
We implemented SDXL model caching using Modal volumes:
- One-time setup: Download and cache the base SDXL model to Modal storage
- Fast loading: Load cached model from Modal volume (local storage) instead of downloading
- Automatic fallback: If cache fails, automatically falls back to HuggingFace download
Performance Impact
| Scenario | Before Optimization | After Optimization | Time Saved |
|---|---|---|---|
| First run (no cache) | 45 seconds | 45 seconds | 0 seconds |
| Subsequent runs | 45 seconds | 5-8 seconds | 30-40 seconds |
| Typical usage | 45 seconds | 5-8 seconds | π 85% faster |
How to Enable
Step 1: One-time Setup (5-10 minutes)
python modal_deployments/cache_model.py
This downloads the SDXL model once and stores it in Modal's persistent storage.
Step 2: Automatic Fast Loading
After setup, every thumbnail generation automatically uses the cached model for much faster processing.
Technical Details
Architecture
Before:
HuggingFace β Download (30-40s) β Generate Thumbnails
After:
Modal Cache β Load (2-3s) β Generate Thumbnails
β (if cache fails)
HuggingFace β Download β Cache β Generate Thumbnails
Storage
- Volume:
sdxl-base-cacheModal volume - Model path:
/sdxl-base-cache/stable-diffusion-xl-base-1.0/ - Model format: SafeTensors (optimized loading)
- Size: ~9GB
Functions Added
cache_base_sdxl_model(): Downloads and caches the modelload_cached_or_download_sdxl_model(): Smart loading with fallbackinitialize_model_cache(): User-friendly initialization endpoint
Monitoring Cache Status
The system provides clear logging:
β
Successfully loaded cached SDXL model! # Cache hit (fast)
π₯ No cached model found, downloading... # Cache miss (slow)
β οΈ Failed to load from cache: [error] # Cache error (fallback)
Advanced Options
Manual Cache Check
# Check if model is cached
import os
cached = os.path.exists("/sdxl-base-cache/stable-diffusion-xl-base-1.0")
print(f"Model cached: {cached}")
Force Recache
If you need to refresh the cached model:
# This will re-download and cache the model
python modal_deployments/cache_model.py # Will detect existing cache and skip
To force a fresh download, clear the cache first through Modal dashboard.
Troubleshooting
| Issue | Solution |
|---|---|
| Cache script fails | Check Modal authentication: modal token new |
| Still slow after caching | Verify cache success in logs |
| Out of storage space | Modal volumes have generous limits |
| Cache corruption | Re-run python modal_deployments/cache_model.py |
Cost Impact
- Initial cache setup: ~$0.50-1.00 (one-time)
- Storage cost: ~$0.10/month for cached model
- Compute savings: Significant reduction in GPU billing from faster loading
The caching pays for itself after just a few thumbnail generations!
Future Improvements
Potential optimizations being considered:
- Pre-warm containers with cached models
- Compress cached models further
- Cache fine-tuned models similarly
- Implement cache versioning for model updates
Whisper Model Caching
Problem
The SmartThumbnailer uses OpenAI Whisper (turbo model) for audio transcription. By default, the Whisper model was being downloaded every time a transcription was requested, causing:
- Repeated downloads: Model downloaded fresh for each video upload
- Transcription delays: Extra time spent downloading before transcription starts
- Resource waste: Re-downloading the same model files repeatedly
- Poor user experience: Inconsistent transcription start times
Solution
We implemented Whisper model caching using Modal volumes:
- One-time setup: Download and cache the Whisper turbo model to Modal storage
- Fast loading: Load cached model from Modal volume instead of downloading
- Automatic fallback: If cache fails, automatically falls back to OpenAI download
- Smart caching: Automatically cache models on first download
Performance Impact
| Scenario | Before Optimization | After Optimization | Time Saved |
|---|---|---|---|
| First transcription (no cache) | ~60 seconds total | ~60 seconds total | 0 seconds |
| Subsequent transcriptions | ~60 seconds total | ~30 seconds total | ~30 seconds |
| Model loading only | 15-25 seconds | 2-3 seconds | π 85% faster |
How to Enable
Step 1: One-time Setup (5-10 minutes)
python modal_deployments/cache_whisper.py
This downloads the Whisper turbo model once and stores it in Modal's persistent storage.
Step 2: Automatic Fast Loading
After setup, every transcription automatically uses the cached model for much faster loading.
Technical Details
Architecture
Before:
OpenAI/HuggingFace β Download Whisper (15-25s) β Transcribe Audio
After:
Modal Cache β Load Whisper (2-3s) β Transcribe Audio
β (if cache fails)
OpenAI/HuggingFace β Download β Cache β Transcribe Audio
Storage
- Volume:
whisper-cacheModal volume - Model path:
/whisper-cache/whisper-turbo/ - Model format: PyTorch (
.ptfiles) - Size: ~1.5GB
Functions Added
cache_whisper_turbo_model(): Downloads and caches the Whisper modelload_cached_or_download_whisper_model(): Smart loading with fallbackinitialize_whisper_cache(): User-friendly initialization endpoint
Monitoring Cache Status
The system provides clear logging:
β
Successfully loaded cached Whisper turbo model! # Cache hit (fast)
π₯ No cached Whisper model found, downloading... # Cache miss (slow)
β οΈ Failed to load from cache: [error] # Cache error (fallback)
Advanced Options
Manual Cache Check
# Check if models are cached
import os
sdxl_cached = os.path.exists("/sdxl-base-cache/stable-diffusion-xl-base-1.0")
whisper_cached = os.path.exists("/whisper-cache/whisper-turbo")
print(f"SDXL model cached: {sdxl_cached}")
print(f"Whisper model cached: {whisper_cached}")
Force Recache
If you need to refresh the cached models:
# Re-run caching scripts (will detect existing cache and skip unless forced)
python modal_deployments/cache_model.py
python modal_deployments/cache_whisper.py
To force a fresh download, clear the respective cache through Modal dashboard.
Troubleshooting
| Issue | Solution |
|---|---|
| Cache scripts fail | Check Modal authentication: modal token new |
| Still slow after caching | Verify cache success in function logs |
| Out of storage space | Modal volumes have generous limits |
| Cache corruption | Re-run respective caching script |
| Models out of date | Clear cache and re-run caching scripts |
Cost Impact
- SDXL cache setup: ~$0.50-1.00 (one-time)
- Whisper cache setup: ~$0.25-0.50 (one-time)
- Monthly storage: ~$0.15/month for both cached models
- Compute savings: Significant reduction in GPU billing from faster loading
The caching investment pays for itself after just a few operations!
Future Improvements
Potential optimizations being considered:
- Pre-warm containers with cached models
- Compress cached models further
- Cache additional model variants
- Implement cache versioning for model updates
- Smart cache invalidation based on model updates
Combined Performance Benefits
Total Time Savings
When both SDXL and Whisper models are cached:
| Operation | Before Caching | After Caching | Time Saved |
|---|---|---|---|
| Video Upload + Transcription | 60 seconds | 30 seconds | 50% faster |
| Thumbnail Generation | 45 seconds | 5-8 seconds | 85% faster |
| Complete workflow | 105 seconds | 35-38 seconds | 65% faster |
Setup Commands
Run both caching commands for maximum performance:
# Cache SDXL model for thumbnails (5-10 minutes)
python modal_deployments/cache_model.py
# Cache Whisper model for transcriptions (5-10 minutes)
python modal_deployments/cache_whisper.py
Cost Impact
- SDXL cache setup: ~$0.50-1.00 (one-time)
- Whisper cache setup: ~$0.25-0.50 (one-time)
- Monthly storage: ~$0.15/month for both cached models
- Compute savings: Significant reduction in GPU billing from faster loading
The caching investment pays for itself after just a few operations!
Future Improvements
Potential optimizations being considered:
- Pre-warm containers with cached models
- Compress cached models further
- Cache additional model variants
- Implement cache versioning for model updates
- Smart cache invalidation based on model updates