Hugging Face's logo

Spaces:
benhong
/
blt-patch-view
Sleeping

App Files Community
blt-patch-view / SIMPLIFIED_DESIGN.md
3v324v23's picture
3v324v23
Add app
98c0f96
preview code
|
Raw History Blame Contribute Delete
28.3 kB
# Patch Visualization App - Simplified Design
**Purpose**: Interactive exploration of entropy-based patching strategies on pre-encoded audio
---
## Scope (Minimal)
### What We're Building
- Gradio app to explore **entropy-based patching only**
- Uses **10 pre-selected audio files** from `data/test-10/`
- Uses **single entropy model**: `archive/checkpoints/audio_peoplespeech_50k_scratch/best`
- **Precomputed data**: All entropy/perplexity computed offline
- **Focus**: Parameter exploration, not model inference
### What We're NOT Building
- ❌ No audio upload
- ❌ No model selection/switching
- ❌ No other patching strategies (strided, space, BPE)
- ❌ No export functionality
- ❌ No batch processing
- ❌ No comparison mode
---
## Data Pipeline
### Step 1: Precomputation Script
**Input**: 10 FLAC files from `data/test-10/`
**Process**:
1. Load audio with WavTokenizer
2. Encode to tokens (40 tok/sec, 4096 vocab)
3. Load entropy model from `archive/checkpoints/audio_peoplespeech_50k_scratch/best`
4. Compute entropy for each token position
5. Compute perplexity: `PPL(x_i) = 2^{H(x_i)}`
6. Generate spectrogram (for visualization)
7. Save to JSON
**Output**: `data/test-10/precomputed/<file_id>.json`
**JSON Schema**:
```json
{
"file_id": "1089-134691-0016",
"audio_path": "data/test-10/1089-134691-0016.flac",
"transcript": "...",
"audio_duration_sec": 7.23,
"sample_rate": 24000,
"tokens": [234, 1023, 892, ...], // WavTokenizer tokens
"entropies": [2.34, 1.89, 2.56, ...], // bits
"perplexities": [5.07, 3.71, 5.91, ...] // 2^entropy
}
```
**Note**: Spectrogram is computed at runtime from audio file (keeps JSON small)
**Storage Format**: JSON (simple, human-readable, easy to debug)
- Each audio file → 1 JSON file
- Typical size: ~20KB per file (without spectrogram)
- Total: ~200KB for 10 files
---
### Step 2: Runtime Patching
**Input**:
- Precomputed JSON
- User parameters (threshold, metric, constraint type)
**Process**:
1. Load metric array (entropy or perplexity)
2. Apply patching strategy
3. Return patch boundaries
**Patching Strategies**:
#### A. Global Threshold
```python
def patch_global_threshold(metric, threshold):
patches = []
current_patch = []
for i, value in enumerate(metric):
current_patch.append(i)
if value > threshold:
patches.append(current_patch)
current_patch = []
if current_patch:
patches.append(current_patch)
return patches
```
#### B. Additive Monotonicity
```python
def patch_additive_monotonic(metric, threshold):
patches = []
current_patch = [0]
for i in range(1, len(metric)):
delta = metric[i] - metric[i-1]
if delta > threshold:
patches.append(current_patch)
current_patch = [i]
else:
current_patch.append(i)
if current_patch:
patches.append(current_patch)
return patches
```
#### C. Multiplicative Monotonicity
```python
def patch_multiplicative_monotonic(metric, threshold):
patches = []
current_patch = [0]
for i in range(1, len(metric)):
if metric[i-1] > 0: # Avoid division by zero
ratio = metric[i] / metric[i-1]
else:
ratio = 1.0
if ratio > threshold:
patches.append(current_patch)
current_patch = [i]
else:
current_patch.append(i)
if current_patch:
patches.append(current_patch)
return patches
```
---
## Gradio UI
### Layout
```
┌─────────────────────────────────────────────────┐
│ BLT Patching Visualization Explorer │
└─────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────┐
│ Controls │
├──────────────────┬──────────────────────────────┤
│ Audio File: │ Patching Parameters: │
│ [Dropdown ▼] │ Metric: │
│ │ ● Entropy ○ Perplexity │
│ 1089-134691-0016 │ │
│ Duration: 7.2s │ Constraint: │
│ Tokens: 289 │ ● Global Threshold │
│ │ ○ Additive Monotonic │
│ Transcript: │ ○ Multiplicative Monotonic│
│ "..." (truncated)│ │
│ │ Threshold: [2.0 ━━━━━━━] │
│ │ (range: 0.0 - 10.0) │
│ │ │
│ │ [Apply Patching] │
└──────────────────┴──────────────────────────────┘
┌─────────────────────────────────────────────────┐
│ Waveform + Patch Boundaries │
│ [Plotly Interactive Plot] │
│ - X-axis: Time (sec) │
│ - Y-axis: Amplitude │
│ - Red vertical lines: Patch boundaries │
└─────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────┐
│ Spectrogram + Patch Boundaries │
│ [Plotly Interactive Plot] │
│ - X-axis: Time (sec) │
│ - Y-axis: Frequency (Hz) │
│ - Color: Magnitude (dB) │
│ - Red vertical lines: Patch boundaries │
└─────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────┐
│ Entropy/Perplexity + Patch Boundaries │
│ [Plotly Interactive Plot] │
│ - X-axis: Token position │
│ - Y-axis: Metric value │
│ - Blue line: Metric values │
│ - Red horizontal line: Threshold │
│ - Gray vertical lines: Patch boundaries │
└─────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────┐
│ Patch Statistics │
│ Total Patches: 42 │
│ Avg Size: 6.9 tokens │
│ Min Size: 2 tokens │
│ Max Size: 18 tokens │
│ │
│ [Bar Chart: Patch Size Distribution] │
└─────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────┐
│ Audio Playback │
│ [▶ Play Full Audio] │
│ [▶ Play Until Next Boundary] (from current pos)│
│ │
│ Current Position: 2.34s (Token 94, Patch 15) │
│ Next Boundary: 2.89s (Token 116) │
└─────────────────────────────────────────────────┘
```
### Interactions
1. **Select Audio**: Dropdown changes which precomputed data is loaded
2. **Adjust Parameters**: Sliders/radio buttons update patching in real-time
3. **Click Plots**: Click on waveform/spectrogram to set playback position
4. **Play Until Boundary**: Plays from current position until next patch boundary
- Useful for hearing what's in a single patch
- Automatically stops at boundary
---
## Implementation
### File Structure
```
apps/patch_view/
├── precompute.py # Offline: Compute entropy/perplexity for all samples
├── app.py # Gradio app
├── patching.py # Patching strategies
├── visualization.py # Plotly plot generation
├── audio.py # Audio playback utilities
└── README.md # Usage instructions
```
### Dependencies
```python
# requirements.txt
gradio>=4.0
plotly>=5.0
numpy>=1.24
torch>=2.0
torchaudio>=2.0
librosa>=0.10
soundfile>=0.12
```
### Core Functions
#### precompute.py
```python
import torch
import torchaudio
import librosa
import numpy as np
import json
from pathlib import Path
def load_entropy_model(checkpoint_path):
"""Load entropy model from checkpoint"""
# Load from archive/checkpoints/audio_peoplespeech_50k_scratch/best
pass
def compute_entropies(audio_path, entropy_model):
"""
Load audio, encode with WavTokenizer, compute entropies
Returns:
tokens: np.array of token IDs
entropies: np.array of entropy values (bits)
"""
pass
def compute_spectrogram(audio, sr=24000):
"""
Generate mel-spectrogram for visualization
Returns:
times: np.array of time points
frequencies: np.array of frequency bins
magnitudes: 2D np.array (freq x time)
"""
S = librosa.feature.melspectrogram(
y=audio, sr=sr, n_mels=128, fmax=8000
)
S_db = librosa.power_to_db(S, ref=np.max)
times = librosa.frames_to_time(
np.arange(S.shape[1]), sr=sr, hop_length=512
)
frequencies = librosa.mel_frequencies(n_mels=128, fmax=8000)
return times, frequencies, S_db
def precompute_all():
"""Precompute entropy/perplexity for all test-10 samples"""
entropy_model = load_entropy_model(
"archive/checkpoints/audio_peoplespeech_50k_scratch/best"
)
test_dir = Path("data/test-10")
output_dir = Path("data/test-10/precomputed")
output_dir.mkdir(exist_ok=True)
for audio_file in sorted(test_dir.glob("*.flac")):
print(f"Processing {audio_file.name}...")
# Load audio
audio, sr = librosa.load(audio_file, sr=24000, mono=True)
# Compute tokens and entropies
tokens, entropies = compute_entropies(audio_file, entropy_model)
# Compute perplexities
perplexities = 2 ** entropies
# Compute spectrogram
times, freqs, mags = compute_spectrogram(audio, sr)
# Load transcript
txt_file = audio_file.with_suffix(".txt")
transcript = txt_file.read_text().strip() if txt_file.exists() else ""
# Save
data = {
"file_id": audio_file.stem,
"audio_path": str(audio_file),
"transcript": transcript,
"audio_duration_sec": float(len(audio) / sr),
"sample_rate": int(sr),
"tokens": tokens.tolist(),
"entropies": entropies.tolist(),
"perplexities": perplexities.tolist(),
"spectrogram": {
"times": times.tolist(),
"frequencies": freqs.tolist(),
"magnitudes": mags.tolist()
}
}
output_file = output_dir / f"{audio_file.stem}.json"
with open(output_file, 'w') as f:
json.dump(data, f, indent=2)
print(f" Saved to {output_file}")
if __name__ == "__main__":
precompute_all()
```
#### patching.py
```python
import numpy as np
from typing import List
def patch_global_threshold(metric: np.ndarray, threshold: float) -> List[List[int]]:
"""Patch when metric exceeds threshold"""
patches = []
current_patch = []
for i, value in enumerate(metric):
current_patch.append(i)
if value > threshold:
patches.append(current_patch)
current_patch = []
if current_patch:
patches.append(current_patch)
return patches
def patch_additive_monotonic(metric: np.ndarray, threshold: float) -> List[List[int]]:
"""Patch when metric increase exceeds threshold"""
patches = []
current_patch = [0]
for i in range(1, len(metric)):
delta = metric[i] - metric[i-1]
if delta > threshold:
patches.append(current_patch)
current_patch = [i]
else:
current_patch.append(i)
if current_patch:
patches.append(current_patch)
return patches
def patch_multiplicative_monotonic(metric: np.ndarray, threshold: float) -> List[List[int]]:
"""Patch when metric ratio exceeds threshold"""
patches = []
current_patch = [0]
for i in range(1, len(metric)):
if metric[i-1] > 1e-6: # Avoid division by zero
ratio = metric[i] / metric[i-1]
else:
ratio = 1.0
if ratio > threshold:
patches.append(current_patch)
current_patch = [i]
else:
current_patch.append(i)
if current_patch:
patches.append(current_patch)
return patches
def apply_patching(metric: np.ndarray,
constraint: str,
threshold: float) -> List[List[int]]:
"""
Apply patching strategy
Args:
metric: Entropy or perplexity values
constraint: "global", "additive", or "multiplicative"
threshold: Threshold value
Returns:
List of patches (each patch is list of token indices)
"""
if constraint == "global":
return patch_global_threshold(metric, threshold)
elif constraint == "additive":
return patch_additive_monotonic(metric, threshold)
elif constraint == "multiplicative":
return patch_multiplicative_monotonic(metric, threshold)
else:
raise ValueError(f"Unknown constraint: {constraint}")
def compute_patch_statistics(patches: List[List[int]]) -> dict:
"""Compute statistics about patches"""
sizes = [len(p) for p in patches]
return {
"num_patches": len(patches),
"avg_size": np.mean(sizes),
"min_size": np.min(sizes),
"max_size": np.max(sizes),
"median_size": np.median(sizes),
"std_size": np.std(sizes),
"sizes": sizes
}
```
#### visualization.py
```python
import plotly.graph_objects as go
from plotly.subplots import make_subplots
import numpy as np
def create_visualization(data: dict, patches: List[List[int]],
metric_name: str, threshold: float):
"""
Create 4-panel visualization
Args:
data: Precomputed data dict
patches: Patch boundaries
metric_name: "entropy" or "perplexity"
threshold: Threshold value
Returns:
Plotly figure
"""
fig = make_subplots(
rows=4, cols=1,
subplot_titles=(
'Waveform',
'Spectrogram',
f'{metric_name.capitalize()} + Patch Boundaries',
'Patch Size Distribution'
),
specs=[
[{"type": "scatter"}],
[{"type": "heatmap"}],
[{"type": "scatter"}],
[{"type": "bar"}]
],
vertical_spacing=0.08,
row_heights=[0.2, 0.3, 0.3, 0.2]
)
# Get metric data
metric = np.array(data['entropies'] if metric_name == 'entropy'
else data['perplexities'])
# Calculate patch boundary times
tokens_per_sec = 40 # WavTokenizer rate
boundary_times = []
for patch in patches:
if patch:
boundary_times.append(patch[0] / tokens_per_sec)
# 1. Waveform (simplified - just show patch boundaries)
duration = data['audio_duration_sec']
fig.add_trace(
go.Scatter(
x=[0, duration],
y=[0, 0],
mode='lines',
name='Audio',
line=dict(color='lightgray', width=1)
),
row=1, col=1
)
# Add patch boundaries to waveform
for t in boundary_times:
fig.add_vline(
x=t, line_width=1, line_dash="solid",
line_color="red", opacity=0.5,
row=1, col=1
)
# 2. Spectrogram
spec = data['spectrogram']
fig.add_trace(
go.Heatmap(
x=spec['times'],
y=spec['frequencies'],
z=spec['magnitudes'],
colorscale='Viridis',
showscale=False
),
row=2, col=1
)
# Add patch boundaries to spectrogram
for t in boundary_times:
fig.add_vline(
x=t, line_width=1, line_dash="solid",
line_color="red", opacity=0.7,
row=2, col=1
)
# 3. Entropy/Perplexity plot
token_positions = np.arange(len(metric))
fig.add_trace(
go.Scatter(
x=token_positions,
y=metric,
mode='lines',
name=metric_name.capitalize(),
line=dict(color='blue', width=2)
),
row=3, col=1
)
# Add threshold line
fig.add_hline(
y=threshold, line_width=2, line_dash="dash",
line_color="red", opacity=0.7,
annotation_text=f"Threshold: {threshold:.2f}",
row=3, col=1
)
# Add patch boundaries
for patch in patches:
if patch:
fig.add_vline(
x=patch[0], line_width=1, line_dash="solid",
line_color="gray", opacity=0.5,
row=3, col=1
)
# 4. Patch size distribution
sizes = [len(p) for p in patches]
size_counts = {}
for size in sizes:
size_counts[size] = size_counts.get(size, 0) + 1
fig.add_trace(
go.Bar(
x=list(size_counts.keys()),
y=list(size_counts.values()),
name='Patch Sizes',
marker_color='steelblue'
),
row=4, col=1
)
# Update layout
fig.update_xaxes(title_text="Time (s)", row=1, col=1)
fig.update_xaxes(title_text="Time (s)", row=2, col=1)
fig.update_yaxes(title_text="Frequency (Hz)", row=2, col=1)
fig.update_xaxes(title_text="Token Position", row=3, col=1)
fig.update_yaxes(title_text=metric_name.capitalize(), row=3, col=1)
fig.update_xaxes(title_text="Patch Size (tokens)", row=4, col=1)
fig.update_yaxes(title_text="Count", row=4, col=1)
fig.update_layout(
height=1200,
showlegend=False,
hovermode='x unified'
)
return fig
```
#### audio.py
```python
import soundfile as sf
import numpy as np
from pathlib import Path
class AudioPlayer:
"""Handle audio playback for patches"""
def __init__(self, audio_path: str, sample_rate: int = 24000):
self.audio, self.sr = sf.read(audio_path)
self.sample_rate = sample_rate
self.current_position = 0 # In samples
def get_full_audio(self):
"""Return full audio for playback"""
return self.sr, self.audio
def play_until_boundary(self, patches: List[List[int]],
current_token: int) -> tuple:
"""
Get audio from current position until next patch boundary
Args:
patches: List of patches
current_token: Current token position
Returns:
(sample_rate, audio_segment)
"""
# Find which patch contains current token
current_patch_idx = None
for i, patch in enumerate(patches):
if current_token in patch:
current_patch_idx = i
break
if current_patch_idx is None:
return self.sr, np.array([])
# Get tokens in current patch from current position onward
current_patch = patches[current_patch_idx]
start_token = current_token
end_token = current_patch[-1]
# Convert to audio samples (40 tokens/sec)
tokens_per_sec = 40
start_sample = int(start_token / tokens_per_sec * self.sr)
end_sample = int((end_token + 1) / tokens_per_sec * self.sr)
# Extract segment
segment = self.audio[start_sample:end_sample]
return self.sr, segment
```
#### app.py
```python
import gradio as gr
import json
import numpy as np
from pathlib import Path
from patching import apply_patching, compute_patch_statistics
from visualization import create_visualization
from audio import AudioPlayer
# Load all precomputed data
PRECOMPUTED_DIR = Path("data/test-10/precomputed")
precomputed_data = {}
def load_precomputed_data():
"""Load all precomputed JSON files"""
for json_file in PRECOMPUTED_DIR.glob("*.json"):
with open(json_file) as f:
data = json.load(f)
precomputed_data[data['file_id']] = data
def get_audio_choices():
"""Get list of available audio files"""
return sorted(precomputed_data.keys())
def process_patching(file_id, metric_name, constraint, threshold):
"""
Apply patching and generate visualization
Returns:
figure, stats_text, audio
"""
# Get data
data = precomputed_data[file_id]
# Select metric
metric = np.array(data['entropies'] if metric_name == 'Entropy'
else data['perplexities'])
# Apply patching
patches = apply_patching(metric, constraint.lower(), threshold)
# Compute statistics
stats = compute_patch_statistics(patches)
# Generate visualization
fig = create_visualization(data, patches, metric_name.lower(), threshold)
# Format statistics
stats_text = f"""
**Patch Statistics:**
- Total Patches: {stats['num_patches']}
- Average Size: {stats['avg_size']:.2f} tokens
- Min Size: {stats['min_size']} tokens
- Max Size: {stats['max_size']} tokens
- Median Size: {stats['median_size']:.1f} tokens
- Std Dev: {stats['std_size']:.2f} tokens
"""
# Load audio
audio_path = data['audio_path']
player = AudioPlayer(audio_path)
sr, audio = player.get_full_audio()
return fig, stats_text, (sr, audio), patches
def play_until_boundary(file_id, current_token, patches):
"""Play audio from current token until next boundary"""
data = precomputed_data[file_id]
audio_path = data['audio_path']
player = AudioPlayer(audio_path)
sr, segment = player.play_until_boundary(patches, current_token)
return (sr, segment)
# Gradio Interface
def create_app():
load_precomputed_data()
with gr.Blocks(title="BLT Patching Visualization") as app:
gr.Markdown("# BLT Patching Visualization Explorer")
gr.Markdown("Explore entropy-based patching strategies on audio")
with gr.Row():
# Left column: Controls
with gr.Column(scale=1):
gr.Markdown("### Audio Selection")
audio_dropdown = gr.Dropdown(
choices=get_audio_choices(),
value=get_audio_choices()[0],
label="Audio File"
)
file_info = gr.Markdown()
gr.Markdown("### Patching Parameters")
metric_radio = gr.Radio(
choices=["Entropy", "Perplexity"],
value="Entropy",
label="Metric"
)
constraint_radio = gr.Radio(
choices=["Global", "Additive", "Multiplicative"],
value="Global",
label="Constraint Type"
)
threshold_slider = gr.Slider(
minimum=0.0,
maximum=10.0,
value=2.0,
step=0.1,
label="Threshold"
)
apply_btn = gr.Button("Apply Patching", variant="primary")
# Visualization area
with gr.Column(scale=2):
plot_output = gr.Plot(label="Visualization")
stats_output = gr.Markdown()
# Audio playback
gr.Markdown("### Audio Playback")
with gr.Row():
audio_output = gr.Audio(label="Full Audio", type="numpy")
with gr.Row():
current_token_input = gr.Number(
value=0,
label="Current Token Position",
precision=0
)
play_boundary_btn = gr.Button("▶ Play Until Next Boundary")
boundary_audio = gr.Audio(label="Patch Audio", type="numpy")
# State to store patches
patches_state = gr.State([])
# Update file info when audio changes
def update_file_info(file_id):
data = precomputed_data[file_id]
info = f"""
**Duration:** {data['audio_duration_sec']:.2f}s
**Tokens:** {len(data['tokens'])}
**Transcript:** {data['transcript'][:100]}...
"""
return info
audio_dropdown.change(
update_file_info,
inputs=[audio_dropdown],
outputs=[file_info]
)
# Apply patching
apply_btn.click(
process_patching,
inputs=[audio_dropdown, metric_radio, constraint_radio, threshold_slider],
outputs=[plot_output, stats_output, audio_output, patches_state]
)
# Play until boundary
play_boundary_btn.click(
play_until_boundary,
inputs=[audio_dropdown, current_token_input, patches_state],
outputs=[boundary_audio]
)
# Initialize
app.load(
update_file_info,
inputs=[audio_dropdown],
outputs=[file_info]
)
return app
if __name__ == "__main__":
app = create_app()
app.launch(server_name="0.0.0.0", server_port=7860)
```
---
## Development Steps
### Step 1: Precomputation (1 day)
1. ✅ Adapt existing entropy analysis script
2. ✅ Load entropy model checkpoint
3. ✅ Process all 10 audio files
4. ✅ Save JSON files
**Command**:
```bash
uv run python apps/patch_view/precompute.py
```
### Step 2: Core Patching (0.5 days)
1. ✅ Implement 3 patching strategies
2. ✅ Test on sample data
3. ✅ Validate outputs
### Step 3: Visualization (1 day)
1. ✅ Create 4-panel Plotly figure
2. ✅ Add patch boundaries
3. ✅ Test interactivity
### Step 4: Gradio UI (1 day)
1. ✅ Build basic interface
2. ✅ Wire up controls
3. ✅ Add audio playback
### Step 5: Audio Playback (0.5 days)
1. ✅ Implement "play until boundary"
2. ✅ Test timing accuracy
### Step 6: Polish (0.5 days)
1. ✅ Fix bugs
2. ✅ Improve layout
3. ✅ Add documentation
**Total Time**: ~4 days
---
## Success Criteria
### Functional
- ✅ All 10 audio files precomputed
- ✅ 3 patching strategies working
- ✅ Real-time threshold adjustment
- ✅ Interactive visualizations
- ✅ "Play until boundary" feature
### Performance
- ✅ <1s latency for parameter changes
- ✅ Smooth plot interactions
- ✅ Accurate audio playback timing
### Quality
- ✅ Clear visualizations
- ✅ Intuitive controls
- ✅ No crashes on edge cases (threshold=0, etc.)
---
## Future Enhancements (Optional)
1. **Click-to-play**: Click on waveform → play from that position
2. **Patch highlighting**: Click patch → highlight in all plots
3. **Comparison view**: Show 2 strategies side-by-side
4. **Statistics table**: Per-patch details in table
5. **Download data**: Export patches as JSON
These are nice-to-have but not essential for v1.