README.md

metadata

title: BLT Patch View
emoji: 🎧
colorFrom: blue
colorTo: purple
sdk: gradio
app_file: app.py
python_version: '3.12'
pinned: false

BLT Patching Visualization

Interactive Gradio app for exploring entropy-based patching strategies in Byte Latent Transformers (BLT).

Features

• Entropy & Perplexity Metrics: Switch between entropy and perplexity as the underlying metric
• Multi-Constraint Patching: Combine global, additive, multiplicative, additive-from-min, multiplicative-from-min, cumulative, and cumulative-max thresholds with an optional minimum-metric guard (OR logic)
• 4-Panel Visualization: Waveform, spectrogram, entropy/perplexity plot, patch size histogram
• Audio Playback:
  • Compare original vs WavTokenizer reconstructed audio
  • Play audio from token position until next patch boundary
• Statistics: Patch counts, sizes, inference speedup calculations

Quick Start

1. Precompute Data (One-time setup)

This processes the 10 audio samples in data/test-10 and computes entropy/perplexity using the trained model:

uv run python apps/patch_view/scripts/precompute_patch_view.py

This will:

• Load the entropy model from archive/checkpoints/audio_peoplespeech_50k_scratch/checkpoints/0000050000
• Process each audio file (encode with WavTokenizer, compute entropies)
• Save JSON files to data/test-10/precomputed/
• Save reconstructed audio files for comparison

Expected output:

• 10 JSON files (~20KB each) with tokens, entropies, perplexities
• 10 reconstructed WAV files for audio comparison
• Total processing time: ~5-10 minutes (depends on GPU)

2. Launch Gradio App

uv run python -m patch_view.app

Open in browser: http://localhost:7860

Usage

1. Select Audio: Choose from 10 precomputed audio samples
2. Configure Patching:
   • Metric: Entropy or Perplexity
   • Constraints: Select one or more patching rules (global, additive, multiplicative, additive-from-min, multiplicative-from-min, cumulative, cumulative-max)
   • Thresholds: Individual sliders per constraint plus an optional minimum-metric guard
3. Apply Patching: Click "Apply Patching" to see results
4. Explore:
   • View 4-panel visualization with patch boundaries
   • Compare original vs reconstructed audio
   • Play individual patches by token position

File Structure

apps/patch_view/
├── README.md              # This file
├── requirements.txt       # Python dependencies
├── SIMPLIFIED_DESIGN.md   # Design document
├── app.py                 # Shim that delegates to `patch_view.app:main`
├── data/                  # Bundled test-10 snapshot for the demo
├── scripts/
│   └── precompute_patch_view.py  # Data preprocessing script
├── patch_view/
│   ├── __init__.py
│   ├── app.py             # Gradio wiring (UI only)
│   ├── audio.py           # Cached audio utilities
│   ├── constraints.py     # Slider metadata & helpers
│   ├── data_loader.py     # Precomputed dataset discovery
│   ├── legacy_patching.py # Legacy single-constraint helpers
│   ├── patching.py        # Multi-constraint evaluator
│   ├── services.py        # UI-facing service layer
│   └── visualization.py   # Plotly visualization generation
└── tests/
    └── patch_view/
        ├── conftest.py
        ├── test_audio_cache.py
        ├── test_constraints.py
        └── test_patching.py

Data Format

Precomputed JSON files contain:

{
  "file_id": "121-121726-0000",
  "audio_path": "data/test-10/121-121726-0000.flac",
  "reconstructed_audio_path": "data/test-10/precomputed/121-121726-0000_reconstructed.wav",
  "transcript": "...",
  "audio_duration_sec": 4.35,
  "tokens": [1234, 5678, ...],
  "entropies": [2.5, 3.1, ...],
  "perplexities": [5.66, 8.57, ...],
  "avg_entropy": 3.2,
  "avg_perplexity": 9.2
}

Technical Details

• Entropy Model: archive/checkpoints/audio_peoplespeech_50k_scratch (step 50000)
• WavTokenizer: 4096 vocab, 40 tokens/sec, 24kHz
• Entropy Calculation: H = -Σ p(x) log₂(p(x)) (bits)
• Perplexity: PPL = 2^H
• Spectrogram: Computed at runtime using librosa (not stored in JSON)

Patching Strategies

Global Threshold

Creates a new patch when metric exceeds threshold.

Additive Monotonic

Creates a new patch when metric increase exceeds threshold:

• delta = H(x_i) - H(x_{i-1})
• If delta > threshold: new patch

Multiplicative Monotonic

Creates a new patch when metric ratio exceeds threshold:

• ratio = H(x_i) / H(x_{i-1})
• If ratio > threshold: new patch

Additive from Minimum

Creates a patch when the metric rises above the minimum observed in the current patch by more than the threshold.

Multiplicative from Minimum

Creates a patch when the metric grows proportionally compared to the current patch's minimum.

Cumulative Sum (Min)

Accumulates metric values and inserts a boundary immediately after the running sum exceeds the threshold.

Cumulative Sum (Max)

Attempts to keep patch sums below the threshold by deferring boundaries until the next token would overflow.

Minimum Metric Guard

Optional AND guard that delays any boundary creation until the metric itself passes a minimum value (applies alongside the OR-combined constraints).

Statistics

The app displays:

• Total number of patches
• Average, min, max, median patch sizes (in tokens and seconds)
• Standard deviation
• Inference speedup: num_tokens / num_patches
  • Shows how many fewer transformer steps are needed vs. token-level processing

Requirements

• Python 3.10+
• CUDA-capable GPU (for precomputation)
• ~2GB GPU memory for entropy model inference
• See requirements.txt for Python dependencies