Spaces:
Runtime error
Runtime error
File size: 9,393 Bytes
abf90a0 1a92033 abf90a0 1a92033 abf90a0 1a92033 abf90a0 0e6ddab | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 | # Performance Optimization Guide
## SDXL Model Caching
### Problem
The SmartThumbnailer uses Stable Diffusion XL (SDXL) to generate AI thumbnails. By default, the base SDXL model (~9GB) was being downloaded from HuggingFace every time a thumbnail generation was requested, causing:
- **Long wait times**: 30-40 seconds just to download the model
- **Bandwidth waste**: Re-downloading the same 9GB model repeatedly
- **Poor user experience**: Most time spent waiting, not generating
### Solution
We implemented **SDXL model caching** using Modal volumes:
1. **One-time setup**: Download and cache the base SDXL model to Modal storage
2. **Fast loading**: Load cached model from Modal volume (local storage) instead of downloading
3. **Automatic fallback**: If cache fails, automatically falls back to HuggingFace download
### Performance Impact
| Scenario | Before Optimization | After Optimization | Time Saved |
|----------|-------------------|-------------------|------------|
| First run (no cache) | 45 seconds | 45 seconds | 0 seconds |
| Subsequent runs | 45 seconds | 5-8 seconds | **30-40 seconds** |
| **Typical usage** | **45 seconds** | **5-8 seconds** | **π 85% faster** |
### How to Enable
#### Step 1: One-time Setup (5-10 minutes)
```bash
python modal_deployments/cache_model.py
```
This downloads the SDXL model once and stores it in Modal's persistent storage.
#### Step 2: Automatic Fast Loading
After setup, every thumbnail generation automatically uses the cached model for much faster processing.
### Technical Details
#### Architecture
```
Before:
HuggingFace β Download (30-40s) β Generate Thumbnails
After:
Modal Cache β Load (2-3s) β Generate Thumbnails
β (if cache fails)
HuggingFace β Download β Cache β Generate Thumbnails
```
#### Storage
- **Volume**: `sdxl-base-cache` Modal volume
- **Model path**: `/sdxl-base-cache/stable-diffusion-xl-base-1.0/`
- **Model format**: SafeTensors (optimized loading)
- **Size**: ~9GB
#### Functions Added
- `cache_base_sdxl_model()`: Downloads and caches the model
- `load_cached_or_download_sdxl_model()`: Smart loading with fallback
- `initialize_model_cache()`: User-friendly initialization endpoint
### Monitoring Cache Status
The system provides clear logging:
```
β
Successfully loaded cached SDXL model! # Cache hit (fast)
π₯ No cached model found, downloading... # Cache miss (slow)
β οΈ Failed to load from cache: [error] # Cache error (fallback)
```
### Advanced Options
#### Manual Cache Check
```python
# Check if model is cached
import os
cached = os.path.exists("/sdxl-base-cache/stable-diffusion-xl-base-1.0")
print(f"Model cached: {cached}")
```
#### Force Recache
If you need to refresh the cached model:
```bash
# This will re-download and cache the model
python modal_deployments/cache_model.py # Will detect existing cache and skip
```
To force a fresh download, clear the cache first through Modal dashboard.
### Troubleshooting
| Issue | Solution |
|-------|----------|
| Cache script fails | Check Modal authentication: `modal token new` |
| Still slow after caching | Verify cache success in logs |
| Out of storage space | Modal volumes have generous limits |
| Cache corruption | Re-run `python modal_deployments/cache_model.py` |
### Cost Impact
- **Initial cache setup**: ~$0.50-1.00 (one-time)
- **Storage cost**: ~$0.10/month for cached model
- **Compute savings**: Significant reduction in GPU billing from faster loading
The caching pays for itself after just a few thumbnail generations!
### Future Improvements
Potential optimizations being considered:
- Pre-warm containers with cached models
- Compress cached models further
- Cache fine-tuned models similarly
- Implement cache versioning for model updates
## Whisper Model Caching
### Problem
The SmartThumbnailer uses OpenAI Whisper (turbo model) for audio transcription. By default, the Whisper model was being downloaded every time a transcription was requested, causing:
- **Repeated downloads**: Model downloaded fresh for each video upload
- **Transcription delays**: Extra time spent downloading before transcription starts
- **Resource waste**: Re-downloading the same model files repeatedly
- **Poor user experience**: Inconsistent transcription start times
### Solution
We implemented **Whisper model caching** using Modal volumes:
1. **One-time setup**: Download and cache the Whisper turbo model to Modal storage
2. **Fast loading**: Load cached model from Modal volume instead of downloading
3. **Automatic fallback**: If cache fails, automatically falls back to OpenAI download
4. **Smart caching**: Automatically cache models on first download
### Performance Impact
| Scenario | Before Optimization | After Optimization | Time Saved |
|----------|-------------------|-------------------|------------|
| First transcription (no cache) | ~60 seconds total | ~60 seconds total | 0 seconds |
| Subsequent transcriptions | ~60 seconds total | ~30 seconds total | **~30 seconds** |
| **Model loading only** | **15-25 seconds** | **2-3 seconds** | **π 85% faster** |
### How to Enable
#### Step 1: One-time Setup (5-10 minutes)
```bash
python modal_deployments/cache_whisper.py
```
This downloads the Whisper turbo model once and stores it in Modal's persistent storage.
#### Step 2: Automatic Fast Loading
After setup, every transcription automatically uses the cached model for much faster loading.
### Technical Details
#### Architecture
```
Before:
OpenAI/HuggingFace β Download Whisper (15-25s) β Transcribe Audio
After:
Modal Cache β Load Whisper (2-3s) β Transcribe Audio
β (if cache fails)
OpenAI/HuggingFace β Download β Cache β Transcribe Audio
```
#### Storage
- **Volume**: `whisper-cache` Modal volume
- **Model path**: `/whisper-cache/whisper-turbo/`
- **Model format**: PyTorch (`.pt` files)
- **Size**: ~1.5GB
#### Functions Added
- `cache_whisper_turbo_model()`: Downloads and caches the Whisper model
- `load_cached_or_download_whisper_model()`: Smart loading with fallback
- `initialize_whisper_cache()`: User-friendly initialization endpoint
### Monitoring Cache Status
The system provides clear logging:
```
β
Successfully loaded cached Whisper turbo model! # Cache hit (fast)
π₯ No cached Whisper model found, downloading... # Cache miss (slow)
β οΈ Failed to load from cache: [error] # Cache error (fallback)
```
### Advanced Options
#### Manual Cache Check
```python
# Check if models are cached
import os
sdxl_cached = os.path.exists("/sdxl-base-cache/stable-diffusion-xl-base-1.0")
whisper_cached = os.path.exists("/whisper-cache/whisper-turbo")
print(f"SDXL model cached: {sdxl_cached}")
print(f"Whisper model cached: {whisper_cached}")
```
#### Force Recache
If you need to refresh the cached models:
```bash
# Re-run caching scripts (will detect existing cache and skip unless forced)
python modal_deployments/cache_model.py
python modal_deployments/cache_whisper.py
```
To force a fresh download, clear the respective cache through Modal dashboard.
### Troubleshooting
| Issue | Solution |
|-------|----------|
| Cache scripts fail | Check Modal authentication: `modal token new` |
| Still slow after caching | Verify cache success in function logs |
| Out of storage space | Modal volumes have generous limits |
| Cache corruption | Re-run respective caching script |
| Models out of date | Clear cache and re-run caching scripts |
### Cost Impact
- **SDXL cache setup**: ~$0.50-1.00 (one-time)
- **Whisper cache setup**: ~$0.25-0.50 (one-time)
- **Monthly storage**: ~$0.15/month for both cached models
- **Compute savings**: Significant reduction in GPU billing from faster loading
The caching investment pays for itself after just a few operations!
### Future Improvements
Potential optimizations being considered:
- Pre-warm containers with cached models
- Compress cached models further
- Cache additional model variants
- Implement cache versioning for model updates
- Smart cache invalidation based on model updates
## Combined Performance Benefits
### Total Time Savings
When **both** SDXL and Whisper models are cached:
| Operation | Before Caching | After Caching | Time Saved |
|-----------|----------------|---------------|------------|
| **Video Upload + Transcription** | 60 seconds | 30 seconds | 50% faster |
| **Thumbnail Generation** | 45 seconds | 5-8 seconds | 85% faster |
| **Complete workflow** | 105 seconds | 35-38 seconds | **65% faster** |
### Setup Commands
Run both caching commands for maximum performance:
```bash
# Cache SDXL model for thumbnails (5-10 minutes)
python modal_deployments/cache_model.py
# Cache Whisper model for transcriptions (5-10 minutes)
python modal_deployments/cache_whisper.py
```
### Cost Impact
- **SDXL cache setup**: ~$0.50-1.00 (one-time)
- **Whisper cache setup**: ~$0.25-0.50 (one-time)
- **Monthly storage**: ~$0.15/month for both cached models
- **Compute savings**: Significant reduction in GPU billing from faster loading
The caching investment pays for itself after just a few operations!
### Future Improvements
Potential optimizations being considered:
- Pre-warm containers with cached models
- Compress cached models further
- Cache additional model variants
- Implement cache versioning for model updates
- Smart cache invalidation based on model updates |