Spaces:
Sleeping
Sleeping
| title: BLT Patch View | |
| emoji: π§ | |
| colorFrom: blue | |
| colorTo: purple | |
| sdk: gradio | |
| app_file: app.py | |
| python_version: 3.12 | |
| pinned: false | |
| # BLT Patching Visualization | |
| Interactive Gradio app for exploring entropy-based patching strategies in Byte Latent Transformers (BLT). | |
| ## Features | |
| - **Entropy & Perplexity Metrics**: Switch between entropy and perplexity as the underlying metric | |
| - **Multi-Constraint Patching**: Combine global, additive, multiplicative, additive-from-min, multiplicative-from-min, cumulative, and cumulative-max thresholds with an optional minimum-metric guard (OR logic) | |
| - **4-Panel Visualization**: Waveform, spectrogram, entropy/perplexity plot, patch size histogram | |
| - **Audio Playback**: | |
| - Compare original vs WavTokenizer reconstructed audio | |
| - Play audio from token position until next patch boundary | |
| - **Statistics**: Patch counts, sizes, inference speedup calculations | |
| ## Quick Start | |
| ### 1. Precompute Data (One-time setup) | |
| This processes the 10 audio samples in `data/test-10` and computes entropy/perplexity using the trained model: | |
| ```bash | |
| uv run python apps/patch_view/scripts/precompute_patch_view.py | |
| ``` | |
| This will: | |
| - Load the entropy model from `archive/checkpoints/audio_peoplespeech_50k_scratch/checkpoints/0000050000` | |
| - Process each audio file (encode with WavTokenizer, compute entropies) | |
| - Save JSON files to `data/test-10/precomputed/` | |
| - Save reconstructed audio files for comparison | |
| Expected output: | |
| - 10 JSON files (~20KB each) with tokens, entropies, perplexities | |
| - 10 reconstructed WAV files for audio comparison | |
| - Total processing time: ~5-10 minutes (depends on GPU) | |
| ### 2. Launch Gradio App | |
| ```bash | |
| uv run python -m patch_view.app | |
| ``` | |
| Open in browser: http://localhost:7860 | |
| ## Usage | |
| 1. **Select Audio**: Choose from 10 precomputed audio samples | |
| 2. **Configure Patching**: | |
| - Metric: Entropy or Perplexity | |
| - Constraints: Select one or more patching rules (global, additive, multiplicative, additive-from-min, multiplicative-from-min, cumulative, cumulative-max) | |
| - Thresholds: Individual sliders per constraint plus an optional minimum-metric guard | |
| 3. **Apply Patching**: Click "Apply Patching" to see results | |
| 4. **Explore**: | |
| - View 4-panel visualization with patch boundaries | |
| - Compare original vs reconstructed audio | |
| - Play individual patches by token position | |
| ## File Structure | |
| ``` | |
| apps/patch_view/ | |
| βββ README.md # This file | |
| βββ requirements.txt # Python dependencies | |
| βββ SIMPLIFIED_DESIGN.md # Design document | |
| βββ app.py # Shim that delegates to `patch_view.app:main` | |
| βββ data/ # Bundled test-10 snapshot for the demo | |
| βββ scripts/ | |
| β βββ precompute_patch_view.py # Data preprocessing script | |
| βββ patch_view/ | |
| β βββ __init__.py | |
| β βββ app.py # Gradio wiring (UI only) | |
| β βββ audio.py # Cached audio utilities | |
| β βββ constraints.py # Slider metadata & helpers | |
| β βββ data_loader.py # Precomputed dataset discovery | |
| β βββ legacy_patching.py # Legacy single-constraint helpers | |
| β βββ patching.py # Multi-constraint evaluator | |
| β βββ services.py # UI-facing service layer | |
| β βββ visualization.py # Plotly visualization generation | |
| βββ tests/ | |
| βββ patch_view/ | |
| βββ conftest.py | |
| βββ test_audio_cache.py | |
| βββ test_constraints.py | |
| βββ test_patching.py | |
| ``` | |
| ## Data Format | |
| Precomputed JSON files contain: | |
| ```json | |
| { | |
| "file_id": "121-121726-0000", | |
| "audio_path": "data/test-10/121-121726-0000.flac", | |
| "reconstructed_audio_path": "data/test-10/precomputed/121-121726-0000_reconstructed.wav", | |
| "transcript": "...", | |
| "audio_duration_sec": 4.35, | |
| "tokens": [1234, 5678, ...], | |
| "entropies": [2.5, 3.1, ...], | |
| "perplexities": [5.66, 8.57, ...], | |
| "avg_entropy": 3.2, | |
| "avg_perplexity": 9.2 | |
| } | |
| ``` | |
| ## Technical Details | |
| - **Entropy Model**: `archive/checkpoints/audio_peoplespeech_50k_scratch` (step 50000) | |
| - **WavTokenizer**: 4096 vocab, 40 tokens/sec, 24kHz | |
| - **Entropy Calculation**: `H = -Ξ£ p(x) logβ(p(x))` (bits) | |
| - **Perplexity**: `PPL = 2^H` | |
| - **Spectrogram**: Computed at runtime using librosa (not stored in JSON) | |
| ## Patching Strategies | |
| ### Global Threshold | |
| Creates a new patch when metric exceeds threshold. | |
| ### Additive Monotonic | |
| Creates a new patch when metric increase exceeds threshold: | |
| - `delta = H(x_i) - H(x_{i-1})` | |
| - If `delta > threshold`: new patch | |
| ### Multiplicative Monotonic | |
| Creates a new patch when metric ratio exceeds threshold: | |
| - `ratio = H(x_i) / H(x_{i-1})` | |
| - If `ratio > threshold`: new patch | |
| ### Additive from Minimum | |
| Creates a patch when the metric rises above the minimum observed in the current patch by more than the threshold. | |
| ### Multiplicative from Minimum | |
| Creates a patch when the metric grows proportionally compared to the current patch's minimum. | |
| ### Cumulative Sum (Min) | |
| Accumulates metric values and inserts a boundary immediately after the running sum exceeds the threshold. | |
| ### Cumulative Sum (Max) | |
| Attempts to keep patch sums below the threshold by deferring boundaries until the next token would overflow. | |
| ### Minimum Metric Guard | |
| Optional AND guard that delays any boundary creation until the metric itself passes a minimum value (applies alongside the OR-combined constraints). | |
| ## Statistics | |
| The app displays: | |
| - Total number of patches | |
| - Average, min, max, median patch sizes (in tokens and seconds) | |
| - Standard deviation | |
| - **Inference speedup**: `num_tokens / num_patches` | |
| - Shows how many fewer transformer steps are needed vs. token-level processing | |
| ## Requirements | |
| - Python 3.10+ | |
| - CUDA-capable GPU (for precomputation) | |
| - ~2GB GPU memory for entropy model inference | |
| - See `requirements.txt` for Python dependencies | |