TubeGenius / documentation /performance-optimization.md
Shreyashgupta5's picture
Enhance performance by implementing caching for the Whisper turbo model to optimize audio transcription. Update README.md with instructions for one-time model caching setup and document performance benefits. Add caching functions in modal_functions.py to streamline model loading and improve user experience during transcriptions.
0e6ddab
|
Raw History Blame Contribute Delete
9.39 kB
# Performance Optimization Guide
## SDXL Model Caching
### Problem
The SmartThumbnailer uses Stable Diffusion XL (SDXL) to generate AI thumbnails. By default, the base SDXL model (~9GB) was being downloaded from HuggingFace every time a thumbnail generation was requested, causing:
- **Long wait times**: 30-40 seconds just to download the model
- **Bandwidth waste**: Re-downloading the same 9GB model repeatedly
- **Poor user experience**: Most time spent waiting, not generating
### Solution
We implemented **SDXL model caching** using Modal volumes:
1. **One-time setup**: Download and cache the base SDXL model to Modal storage
2. **Fast loading**: Load cached model from Modal volume (local storage) instead of downloading
3. **Automatic fallback**: If cache fails, automatically falls back to HuggingFace download
### Performance Impact
| Scenario | Before Optimization | After Optimization | Time Saved |
|----------|-------------------|-------------------|------------|
| First run (no cache) | 45 seconds | 45 seconds | 0 seconds |
| Subsequent runs | 45 seconds | 5-8 seconds | **30-40 seconds** |
| **Typical usage** | **45 seconds** | **5-8 seconds** | **πŸš€ 85% faster** |
### How to Enable
#### Step 1: One-time Setup (5-10 minutes)
```bash
python modal_deployments/cache_model.py
```
This downloads the SDXL model once and stores it in Modal's persistent storage.
#### Step 2: Automatic Fast Loading
After setup, every thumbnail generation automatically uses the cached model for much faster processing.
### Technical Details
#### Architecture
```
Before:
HuggingFace β†’ Download (30-40s) β†’ Generate Thumbnails
After:
Modal Cache β†’ Load (2-3s) β†’ Generate Thumbnails
↓ (if cache fails)
HuggingFace β†’ Download β†’ Cache β†’ Generate Thumbnails
```
#### Storage
- **Volume**: `sdxl-base-cache` Modal volume
- **Model path**: `/sdxl-base-cache/stable-diffusion-xl-base-1.0/`
- **Model format**: SafeTensors (optimized loading)
- **Size**: ~9GB
#### Functions Added
- `cache_base_sdxl_model()`: Downloads and caches the model
- `load_cached_or_download_sdxl_model()`: Smart loading with fallback
- `initialize_model_cache()`: User-friendly initialization endpoint
### Monitoring Cache Status
The system provides clear logging:
```
βœ… Successfully loaded cached SDXL model! # Cache hit (fast)
πŸ“₯ No cached model found, downloading... # Cache miss (slow)
⚠️ Failed to load from cache: [error] # Cache error (fallback)
```
### Advanced Options
#### Manual Cache Check
```python
# Check if model is cached
import os
cached = os.path.exists("/sdxl-base-cache/stable-diffusion-xl-base-1.0")
print(f"Model cached: {cached}")
```
#### Force Recache
If you need to refresh the cached model:
```bash
# This will re-download and cache the model
python modal_deployments/cache_model.py # Will detect existing cache and skip
```
To force a fresh download, clear the cache first through Modal dashboard.
### Troubleshooting
| Issue | Solution |
|-------|----------|
| Cache script fails | Check Modal authentication: `modal token new` |
| Still slow after caching | Verify cache success in logs |
| Out of storage space | Modal volumes have generous limits |
| Cache corruption | Re-run `python modal_deployments/cache_model.py` |
### Cost Impact
- **Initial cache setup**: ~$0.50-1.00 (one-time)
- **Storage cost**: ~$0.10/month for cached model
- **Compute savings**: Significant reduction in GPU billing from faster loading
The caching pays for itself after just a few thumbnail generations!
### Future Improvements
Potential optimizations being considered:
- Pre-warm containers with cached models
- Compress cached models further
- Cache fine-tuned models similarly
- Implement cache versioning for model updates
## Whisper Model Caching
### Problem
The SmartThumbnailer uses OpenAI Whisper (turbo model) for audio transcription. By default, the Whisper model was being downloaded every time a transcription was requested, causing:
- **Repeated downloads**: Model downloaded fresh for each video upload
- **Transcription delays**: Extra time spent downloading before transcription starts
- **Resource waste**: Re-downloading the same model files repeatedly
- **Poor user experience**: Inconsistent transcription start times
### Solution
We implemented **Whisper model caching** using Modal volumes:
1. **One-time setup**: Download and cache the Whisper turbo model to Modal storage
2. **Fast loading**: Load cached model from Modal volume instead of downloading
3. **Automatic fallback**: If cache fails, automatically falls back to OpenAI download
4. **Smart caching**: Automatically cache models on first download
### Performance Impact
| Scenario | Before Optimization | After Optimization | Time Saved |
|----------|-------------------|-------------------|------------|
| First transcription (no cache) | ~60 seconds total | ~60 seconds total | 0 seconds |
| Subsequent transcriptions | ~60 seconds total | ~30 seconds total | **~30 seconds** |
| **Model loading only** | **15-25 seconds** | **2-3 seconds** | **πŸš€ 85% faster** |
### How to Enable
#### Step 1: One-time Setup (5-10 minutes)
```bash
python modal_deployments/cache_whisper.py
```
This downloads the Whisper turbo model once and stores it in Modal's persistent storage.
#### Step 2: Automatic Fast Loading
After setup, every transcription automatically uses the cached model for much faster loading.
### Technical Details
#### Architecture
```
Before:
OpenAI/HuggingFace β†’ Download Whisper (15-25s) β†’ Transcribe Audio
After:
Modal Cache β†’ Load Whisper (2-3s) β†’ Transcribe Audio
↓ (if cache fails)
OpenAI/HuggingFace β†’ Download β†’ Cache β†’ Transcribe Audio
```
#### Storage
- **Volume**: `whisper-cache` Modal volume
- **Model path**: `/whisper-cache/whisper-turbo/`
- **Model format**: PyTorch (`.pt` files)
- **Size**: ~1.5GB
#### Functions Added
- `cache_whisper_turbo_model()`: Downloads and caches the Whisper model
- `load_cached_or_download_whisper_model()`: Smart loading with fallback
- `initialize_whisper_cache()`: User-friendly initialization endpoint
### Monitoring Cache Status
The system provides clear logging:
```
βœ… Successfully loaded cached Whisper turbo model! # Cache hit (fast)
πŸ“₯ No cached Whisper model found, downloading... # Cache miss (slow)
⚠️ Failed to load from cache: [error] # Cache error (fallback)
```
### Advanced Options
#### Manual Cache Check
```python
# Check if models are cached
import os
sdxl_cached = os.path.exists("/sdxl-base-cache/stable-diffusion-xl-base-1.0")
whisper_cached = os.path.exists("/whisper-cache/whisper-turbo")
print(f"SDXL model cached: {sdxl_cached}")
print(f"Whisper model cached: {whisper_cached}")
```
#### Force Recache
If you need to refresh the cached models:
```bash
# Re-run caching scripts (will detect existing cache and skip unless forced)
python modal_deployments/cache_model.py
python modal_deployments/cache_whisper.py
```
To force a fresh download, clear the respective cache through Modal dashboard.
### Troubleshooting
| Issue | Solution |
|-------|----------|
| Cache scripts fail | Check Modal authentication: `modal token new` |
| Still slow after caching | Verify cache success in function logs |
| Out of storage space | Modal volumes have generous limits |
| Cache corruption | Re-run respective caching script |
| Models out of date | Clear cache and re-run caching scripts |
### Cost Impact
- **SDXL cache setup**: ~$0.50-1.00 (one-time)
- **Whisper cache setup**: ~$0.25-0.50 (one-time)
- **Monthly storage**: ~$0.15/month for both cached models
- **Compute savings**: Significant reduction in GPU billing from faster loading
The caching investment pays for itself after just a few operations!
### Future Improvements
Potential optimizations being considered:
- Pre-warm containers with cached models
- Compress cached models further
- Cache additional model variants
- Implement cache versioning for model updates
- Smart cache invalidation based on model updates
## Combined Performance Benefits
### Total Time Savings
When **both** SDXL and Whisper models are cached:
| Operation | Before Caching | After Caching | Time Saved |
|-----------|----------------|---------------|------------|
| **Video Upload + Transcription** | 60 seconds | 30 seconds | 50% faster |
| **Thumbnail Generation** | 45 seconds | 5-8 seconds | 85% faster |
| **Complete workflow** | 105 seconds | 35-38 seconds | **65% faster** |
### Setup Commands
Run both caching commands for maximum performance:
```bash
# Cache SDXL model for thumbnails (5-10 minutes)
python modal_deployments/cache_model.py
# Cache Whisper model for transcriptions (5-10 minutes)
python modal_deployments/cache_whisper.py
```
### Cost Impact
- **SDXL cache setup**: ~$0.50-1.00 (one-time)
- **Whisper cache setup**: ~$0.25-0.50 (one-time)
- **Monthly storage**: ~$0.15/month for both cached models
- **Compute savings**: Significant reduction in GPU billing from faster loading
The caching investment pays for itself after just a few operations!
### Future Improvements
Potential optimizations being considered:
- Pre-warm containers with cached models
- Compress cached models further
- Cache additional model variants
- Implement cache versioning for model updates
- Smart cache invalidation based on model updates