SIMPLIFIED_DESIGN.md

Patch Visualization App - Simplified Design

Purpose: Interactive exploration of entropy-based patching strategies on pre-encoded audio

---

Scope (Minimal)

What We're Building

• Gradio app to explore entropy-based patching only
• Uses 10 pre-selected audio files from data/test-10/
• Uses single entropy model: archive/checkpoints/audio_peoplespeech_50k_scratch/best
• Precomputed data: All entropy/perplexity computed offline
• Focus: Parameter exploration, not model inference

What We're NOT Building

• ❌ No audio upload
• ❌ No model selection/switching
• ❌ No other patching strategies (strided, space, BPE)
• ❌ No export functionality
• ❌ No batch processing
• ❌ No comparison mode

---

Data Pipeline

Step 1: Precomputation Script

Input: 10 FLAC files from data/test-10/

Process:

1. Load audio with WavTokenizer
2. Encode to tokens (40 tok/sec, 4096 vocab)
3. Load entropy model from archive/checkpoints/audio_peoplespeech_50k_scratch/best
4. Compute entropy for each token position
5. Compute perplexity: PPL(x_i) = 2^{H(x_i)}
6. Generate spectrogram (for visualization)
7. Save to JSON

Output: data/test-10/precomputed/<file_id>.json

JSON Schema:

{
  "file_id": "1089-134691-0016",
  "audio_path": "data/test-10/1089-134691-0016.flac",
  "transcript": "...",
  "audio_duration_sec": 7.23,
  "sample_rate": 24000,
  "tokens": [234, 1023, 892, ...],  // WavTokenizer tokens
  "entropies": [2.34, 1.89, 2.56, ...],  // bits
  "perplexities": [5.07, 3.71, 5.91, ...]  // 2^entropy
}

Note: Spectrogram is computed at runtime from audio file (keeps JSON small)

Storage Format: JSON (simple, human-readable, easy to debug)

• Each audio file → 1 JSON file
• Typical size: ~20KB per file (without spectrogram)
• Total: ~200KB for 10 files

---

Step 2: Runtime Patching

Input:

• Precomputed JSON
• User parameters (threshold, metric, constraint type)

Process:

1. Load metric array (entropy or perplexity)
2. Apply patching strategy
3. Return patch boundaries

Patching Strategies:

A. Global Threshold

def patch_global_threshold(metric, threshold):
    patches = []
    current_patch = []

    for i, value in enumerate(metric):
        current_patch.append(i)
        if value > threshold:
            patches.append(current_patch)
            current_patch = []

    if current_patch:
        patches.append(current_patch)

    return patches

B. Additive Monotonicity

def patch_additive_monotonic(metric, threshold):
    patches = []
    current_patch = [0]

    for i in range(1, len(metric)):
        delta = metric[i] - metric[i-1]

        if delta > threshold:
            patches.append(current_patch)
            current_patch = [i]
        else:
            current_patch.append(i)

    if current_patch:
        patches.append(current_patch)

    return patches

C. Multiplicative Monotonicity

def patch_multiplicative_monotonic(metric, threshold):
    patches = []
    current_patch = [0]

    for i in range(1, len(metric)):
        if metric[i-1] > 0:  # Avoid division by zero
            ratio = metric[i] / metric[i-1]
        else:
            ratio = 1.0

        if ratio > threshold:
            patches.append(current_patch)
            current_patch = [i]
        else:
            current_patch.append(i)

    if current_patch:
        patches.append(current_patch)

    return patches

---

Gradio UI

Layout

┌─────────────────────────────────────────────────┐
│  BLT Patching Visualization Explorer            │
└─────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────┐
│  Controls                                        │
├──────────────────┬──────────────────────────────┤
│ Audio File:      │  Patching Parameters:        │
│ [Dropdown ▼]     │  Metric:                     │
│                  │    ● Entropy  ○ Perplexity   │
│ 1089-134691-0016 │                              │
│ Duration: 7.2s   │  Constraint:                 │
│ Tokens: 289      │    ● Global Threshold        │
│                  │    ○ Additive Monotonic      │
│ Transcript:      │    ○ Multiplicative Monotonic│
│ "..." (truncated)│                              │
│                  │  Threshold: [2.0  ━━━━━━━]  │
│                  │  (range: 0.0 - 10.0)         │
│                  │                              │
│                  │  [Apply Patching]            │
└──────────────────┴──────────────────────────────┘

┌─────────────────────────────────────────────────┐
│  Waveform + Patch Boundaries                    │
│  [Plotly Interactive Plot]                      │
│  - X-axis: Time (sec)                           │
│  - Y-axis: Amplitude                            │
│  - Red vertical lines: Patch boundaries         │
└─────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────┐
│  Spectrogram + Patch Boundaries                 │
│  [Plotly Interactive Plot]                      │
│  - X-axis: Time (sec)                           │
│  - Y-axis: Frequency (Hz)                       │
│  - Color: Magnitude (dB)                        │
│  - Red vertical lines: Patch boundaries         │
└─────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────┐
│  Entropy/Perplexity + Patch Boundaries          │
│  [Plotly Interactive Plot]                      │
│  - X-axis: Token position                       │
│  - Y-axis: Metric value                         │
│  - Blue line: Metric values                     │
│  - Red horizontal line: Threshold               │
│  - Gray vertical lines: Patch boundaries        │
└─────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────┐
│  Patch Statistics                                │
│  Total Patches: 42                              │
│  Avg Size: 6.9 tokens                           │
│  Min Size: 2 tokens                             │
│  Max Size: 18 tokens                            │
│                                                  │
│  [Bar Chart: Patch Size Distribution]           │
└─────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────┐
│  Audio Playback                                  │
│  [▶ Play Full Audio]                            │
│  [▶ Play Until Next Boundary] (from current pos)│
│                                                  │
│  Current Position: 2.34s (Token 94, Patch 15)  │
│  Next Boundary: 2.89s (Token 116)               │
└─────────────────────────────────────────────────┘

Interactions

1. Select Audio: Dropdown changes which precomputed data is loaded
2. Adjust Parameters: Sliders/radio buttons update patching in real-time
3. Click Plots: Click on waveform/spectrogram to set playback position
4. Play Until Boundary: Plays from current position until next patch boundary
   • Useful for hearing what's in a single patch
   • Automatically stops at boundary

---

Implementation

File Structure

apps/patch_view/
├── precompute.py          # Offline: Compute entropy/perplexity for all samples
├── app.py                 # Gradio app
├── patching.py            # Patching strategies
├── visualization.py       # Plotly plot generation
├── audio.py               # Audio playback utilities
└── README.md              # Usage instructions

Dependencies

# requirements.txt
gradio>=4.0
plotly>=5.0
numpy>=1.24
torch>=2.0
torchaudio>=2.0
librosa>=0.10
soundfile>=0.12

Core Functions

precompute.py

import torch
import torchaudio
import librosa
import numpy as np
import json
from pathlib import Path

def load_entropy_model(checkpoint_path):
    """Load entropy model from checkpoint"""
    # Load from archive/checkpoints/audio_peoplespeech_50k_scratch/best
    pass

def compute_entropies(audio_path, entropy_model):
    """
    Load audio, encode with WavTokenizer, compute entropies

    Returns:
        tokens: np.array of token IDs
        entropies: np.array of entropy values (bits)
    """
    pass

def compute_spectrogram(audio, sr=24000):
    """
    Generate mel-spectrogram for visualization

    Returns:
        times: np.array of time points
        frequencies: np.array of frequency bins
        magnitudes: 2D np.array (freq x time)
    """
    S = librosa.feature.melspectrogram(
        y=audio, sr=sr, n_mels=128, fmax=8000
    )
    S_db = librosa.power_to_db(S, ref=np.max)

    times = librosa.frames_to_time(
        np.arange(S.shape[1]), sr=sr, hop_length=512
    )
    frequencies = librosa.mel_frequencies(n_mels=128, fmax=8000)

    return times, frequencies, S_db

def precompute_all():
    """Precompute entropy/perplexity for all test-10 samples"""
    entropy_model = load_entropy_model(
        "archive/checkpoints/audio_peoplespeech_50k_scratch/best"
    )

    test_dir = Path("data/test-10")
    output_dir = Path("data/test-10/precomputed")
    output_dir.mkdir(exist_ok=True)

    for audio_file in sorted(test_dir.glob("*.flac")):
        print(f"Processing {audio_file.name}...")

        # Load audio
        audio, sr = librosa.load(audio_file, sr=24000, mono=True)

        # Compute tokens and entropies
        tokens, entropies = compute_entropies(audio_file, entropy_model)

        # Compute perplexities
        perplexities = 2 ** entropies

        # Compute spectrogram
        times, freqs, mags = compute_spectrogram(audio, sr)

        # Load transcript
        txt_file = audio_file.with_suffix(".txt")
        transcript = txt_file.read_text().strip() if txt_file.exists() else ""

        # Save
        data = {
            "file_id": audio_file.stem,
            "audio_path": str(audio_file),
            "transcript": transcript,
            "audio_duration_sec": float(len(audio) / sr),
            "sample_rate": int(sr),
            "tokens": tokens.tolist(),
            "entropies": entropies.tolist(),
            "perplexities": perplexities.tolist(),
            "spectrogram": {
                "times": times.tolist(),
                "frequencies": freqs.tolist(),
                "magnitudes": mags.tolist()
            }
        }

        output_file = output_dir / f"{audio_file.stem}.json"
        with open(output_file, 'w') as f:
            json.dump(data, f, indent=2)

        print(f"  Saved to {output_file}")

if __name__ == "__main__":
    precompute_all()

patching.py

import numpy as np
from typing import List

def patch_global_threshold(metric: np.ndarray, threshold: float) -> List[List[int]]:
    """Patch when metric exceeds threshold"""
    patches = []
    current_patch = []

    for i, value in enumerate(metric):
        current_patch.append(i)
        if value > threshold:
            patches.append(current_patch)
            current_patch = []

    if current_patch:
        patches.append(current_patch)

    return patches

def patch_additive_monotonic(metric: np.ndarray, threshold: float) -> List[List[int]]:
    """Patch when metric increase exceeds threshold"""
    patches = []
    current_patch = [0]

    for i in range(1, len(metric)):
        delta = metric[i] - metric[i-1]

        if delta > threshold:
            patches.append(current_patch)
            current_patch = [i]
        else:
            current_patch.append(i)

    if current_patch:
        patches.append(current_patch)

    return patches

def patch_multiplicative_monotonic(metric: np.ndarray, threshold: float) -> List[List[int]]:
    """Patch when metric ratio exceeds threshold"""
    patches = []
    current_patch = [0]

    for i in range(1, len(metric)):
        if metric[i-1] > 1e-6:  # Avoid division by zero
            ratio = metric[i] / metric[i-1]
        else:
            ratio = 1.0

        if ratio > threshold:
            patches.append(current_patch)
            current_patch = [i]
        else:
            current_patch.append(i)

    if current_patch:
        patches.append(current_patch)

    return patches

def apply_patching(metric: np.ndarray,
                  constraint: str,
                  threshold: float) -> List[List[int]]:
    """
    Apply patching strategy

    Args:
        metric: Entropy or perplexity values
        constraint: "global", "additive", or "multiplicative"
        threshold: Threshold value

    Returns:
        List of patches (each patch is list of token indices)
    """
    if constraint == "global":
        return patch_global_threshold(metric, threshold)
    elif constraint == "additive":
        return patch_additive_monotonic(metric, threshold)
    elif constraint == "multiplicative":
        return patch_multiplicative_monotonic(metric, threshold)
    else:
        raise ValueError(f"Unknown constraint: {constraint}")

def compute_patch_statistics(patches: List[List[int]]) -> dict:
    """Compute statistics about patches"""
    sizes = [len(p) for p in patches]

    return {
        "num_patches": len(patches),
        "avg_size": np.mean(sizes),
        "min_size": np.min(sizes),
        "max_size": np.max(sizes),
        "median_size": np.median(sizes),
        "std_size": np.std(sizes),
        "sizes": sizes
    }

visualization.py

import plotly.graph_objects as go
from plotly.subplots import make_subplots
import numpy as np

def create_visualization(data: dict, patches: List[List[int]],
                        metric_name: str, threshold: float):
    """
    Create 4-panel visualization

    Args:
        data: Precomputed data dict
        patches: Patch boundaries
        metric_name: "entropy" or "perplexity"
        threshold: Threshold value

    Returns:
        Plotly figure
    """
    fig = make_subplots(
        rows=4, cols=1,
        subplot_titles=(
            'Waveform',
            'Spectrogram',
            f'{metric_name.capitalize()} + Patch Boundaries',
            'Patch Size Distribution'
        ),
        specs=[
            [{"type": "scatter"}],
            [{"type": "heatmap"}],
            [{"type": "scatter"}],
            [{"type": "bar"}]
        ],
        vertical_spacing=0.08,
        row_heights=[0.2, 0.3, 0.3, 0.2]
    )

    # Get metric data
    metric = np.array(data['entropies'] if metric_name == 'entropy'
                     else data['perplexities'])

    # Calculate patch boundary times
    tokens_per_sec = 40  # WavTokenizer rate
    boundary_times = []
    for patch in patches:
        if patch:
            boundary_times.append(patch[0] / tokens_per_sec)

    # 1. Waveform (simplified - just show patch boundaries)
    duration = data['audio_duration_sec']
    fig.add_trace(
        go.Scatter(
            x=[0, duration],
            y=[0, 0],
            mode='lines',
            name='Audio',
            line=dict(color='lightgray', width=1)
        ),
        row=1, col=1
    )

    # Add patch boundaries to waveform
    for t in boundary_times:
        fig.add_vline(
            x=t, line_width=1, line_dash="solid",
            line_color="red", opacity=0.5,
            row=1, col=1
        )

    # 2. Spectrogram
    spec = data['spectrogram']
    fig.add_trace(
        go.Heatmap(
            x=spec['times'],
            y=spec['frequencies'],
            z=spec['magnitudes'],
            colorscale='Viridis',
            showscale=False
        ),
        row=2, col=1
    )

    # Add patch boundaries to spectrogram
    for t in boundary_times:
        fig.add_vline(
            x=t, line_width=1, line_dash="solid",
            line_color="red", opacity=0.7,
            row=2, col=1
        )

    # 3. Entropy/Perplexity plot
    token_positions = np.arange(len(metric))

    fig.add_trace(
        go.Scatter(
            x=token_positions,
            y=metric,
            mode='lines',
            name=metric_name.capitalize(),
            line=dict(color='blue', width=2)
        ),
        row=3, col=1
    )

    # Add threshold line
    fig.add_hline(
        y=threshold, line_width=2, line_dash="dash",
        line_color="red", opacity=0.7,
        annotation_text=f"Threshold: {threshold:.2f}",
        row=3, col=1
    )

    # Add patch boundaries
    for patch in patches:
        if patch:
            fig.add_vline(
                x=patch[0], line_width=1, line_dash="solid",
                line_color="gray", opacity=0.5,
                row=3, col=1
            )

    # 4. Patch size distribution
    sizes = [len(p) for p in patches]
    size_counts = {}
    for size in sizes:
        size_counts[size] = size_counts.get(size, 0) + 1

    fig.add_trace(
        go.Bar(
            x=list(size_counts.keys()),
            y=list(size_counts.values()),
            name='Patch Sizes',
            marker_color='steelblue'
        ),
        row=4, col=1
    )

    # Update layout
    fig.update_xaxes(title_text="Time (s)", row=1, col=1)
    fig.update_xaxes(title_text="Time (s)", row=2, col=1)
    fig.update_yaxes(title_text="Frequency (Hz)", row=2, col=1)
    fig.update_xaxes(title_text="Token Position", row=3, col=1)
    fig.update_yaxes(title_text=metric_name.capitalize(), row=3, col=1)
    fig.update_xaxes(title_text="Patch Size (tokens)", row=4, col=1)
    fig.update_yaxes(title_text="Count", row=4, col=1)

    fig.update_layout(
        height=1200,
        showlegend=False,
        hovermode='x unified'
    )

    return fig

audio.py

import soundfile as sf
import numpy as np
from pathlib import Path

class AudioPlayer:
    """Handle audio playback for patches"""

    def __init__(self, audio_path: str, sample_rate: int = 24000):
        self.audio, self.sr = sf.read(audio_path)
        self.sample_rate = sample_rate
        self.current_position = 0  # In samples

    def get_full_audio(self):
        """Return full audio for playback"""
        return self.sr, self.audio

    def play_until_boundary(self, patches: List[List[int]],
                           current_token: int) -> tuple:
        """
        Get audio from current position until next patch boundary

        Args:
            patches: List of patches
            current_token: Current token position

        Returns:
            (sample_rate, audio_segment)
        """
        # Find which patch contains current token
        current_patch_idx = None
        for i, patch in enumerate(patches):
            if current_token in patch:
                current_patch_idx = i
                break

        if current_patch_idx is None:
            return self.sr, np.array([])

        # Get tokens in current patch from current position onward
        current_patch = patches[current_patch_idx]
        start_token = current_token
        end_token = current_patch[-1]

        # Convert to audio samples (40 tokens/sec)
        tokens_per_sec = 40
        start_sample = int(start_token / tokens_per_sec * self.sr)
        end_sample = int((end_token + 1) / tokens_per_sec * self.sr)

        # Extract segment
        segment = self.audio[start_sample:end_sample]

        return self.sr, segment

app.py

import gradio as gr
import json
import numpy as np
from pathlib import Path
from patching import apply_patching, compute_patch_statistics
from visualization import create_visualization
from audio import AudioPlayer

# Load all precomputed data
PRECOMPUTED_DIR = Path("data/test-10/precomputed")
precomputed_data = {}

def load_precomputed_data():
    """Load all precomputed JSON files"""
    for json_file in PRECOMPUTED_DIR.glob("*.json"):
        with open(json_file) as f:
            data = json.load(f)
            precomputed_data[data['file_id']] = data

def get_audio_choices():
    """Get list of available audio files"""
    return sorted(precomputed_data.keys())

def process_patching(file_id, metric_name, constraint, threshold):
    """
    Apply patching and generate visualization

    Returns:
        figure, stats_text, audio
    """
    # Get data
    data = precomputed_data[file_id]

    # Select metric
    metric = np.array(data['entropies'] if metric_name == 'Entropy'
                     else data['perplexities'])

    # Apply patching
    patches = apply_patching(metric, constraint.lower(), threshold)

    # Compute statistics
    stats = compute_patch_statistics(patches)

    # Generate visualization
    fig = create_visualization(data, patches, metric_name.lower(), threshold)

    # Format statistics
    stats_text = f"""
    **Patch Statistics:**
    - Total Patches: {stats['num_patches']}
    - Average Size: {stats['avg_size']:.2f} tokens
    - Min Size: {stats['min_size']} tokens
    - Max Size: {stats['max_size']} tokens
    - Median Size: {stats['median_size']:.1f} tokens
    - Std Dev: {stats['std_size']:.2f} tokens
    """

    # Load audio
    audio_path = data['audio_path']
    player = AudioPlayer(audio_path)
    sr, audio = player.get_full_audio()

    return fig, stats_text, (sr, audio), patches

def play_until_boundary(file_id, current_token, patches):
    """Play audio from current token until next boundary"""
    data = precomputed_data[file_id]
    audio_path = data['audio_path']

    player = AudioPlayer(audio_path)
    sr, segment = player.play_until_boundary(patches, current_token)

    return (sr, segment)

# Gradio Interface
def create_app():
    load_precomputed_data()

    with gr.Blocks(title="BLT Patching Visualization") as app:
        gr.Markdown("# BLT Patching Visualization Explorer")
        gr.Markdown("Explore entropy-based patching strategies on audio")

        with gr.Row():
            # Left column: Controls
            with gr.Column(scale=1):
                gr.Markdown("### Audio Selection")
                audio_dropdown = gr.Dropdown(
                    choices=get_audio_choices(),
                    value=get_audio_choices()[0],
                    label="Audio File"
                )

                file_info = gr.Markdown()

                gr.Markdown("### Patching Parameters")

                metric_radio = gr.Radio(
                    choices=["Entropy", "Perplexity"],
                    value="Entropy",
                    label="Metric"
                )

                constraint_radio = gr.Radio(
                    choices=["Global", "Additive", "Multiplicative"],
                    value="Global",
                    label="Constraint Type"
                )

                threshold_slider = gr.Slider(
                    minimum=0.0,
                    maximum=10.0,
                    value=2.0,
                    step=0.1,
                    label="Threshold"
                )

                apply_btn = gr.Button("Apply Patching", variant="primary")

        # Visualization area
        with gr.Column(scale=2):
            plot_output = gr.Plot(label="Visualization")
            stats_output = gr.Markdown()

        # Audio playback
        gr.Markdown("### Audio Playback")
        with gr.Row():
            audio_output = gr.Audio(label="Full Audio", type="numpy")

        with gr.Row():
            current_token_input = gr.Number(
                value=0,
                label="Current Token Position",
                precision=0
            )
            play_boundary_btn = gr.Button("▶ Play Until Next Boundary")
            boundary_audio = gr.Audio(label="Patch Audio", type="numpy")

        # State to store patches
        patches_state = gr.State([])

        # Update file info when audio changes
        def update_file_info(file_id):
            data = precomputed_data[file_id]
            info = f"""
            **Duration:** {data['audio_duration_sec']:.2f}s
            **Tokens:** {len(data['tokens'])}
            **Transcript:** {data['transcript'][:100]}...
            """
            return info

        audio_dropdown.change(
            update_file_info,
            inputs=[audio_dropdown],
            outputs=[file_info]
        )

        # Apply patching
        apply_btn.click(
            process_patching,
            inputs=[audio_dropdown, metric_radio, constraint_radio, threshold_slider],
            outputs=[plot_output, stats_output, audio_output, patches_state]
        )

        # Play until boundary
        play_boundary_btn.click(
            play_until_boundary,
            inputs=[audio_dropdown, current_token_input, patches_state],
            outputs=[boundary_audio]
        )

        # Initialize
        app.load(
            update_file_info,
            inputs=[audio_dropdown],
            outputs=[file_info]
        )

    return app

if __name__ == "__main__":
    app = create_app()
    app.launch(server_name="0.0.0.0", server_port=7860)

---

Development Steps

Step 1: Precomputation (1 day)

1. ✅ Adapt existing entropy analysis script
2. ✅ Load entropy model checkpoint
3. ✅ Process all 10 audio files
4. ✅ Save JSON files

Command:

uv run python apps/patch_view/precompute.py

Step 2: Core Patching (0.5 days)

1. ✅ Implement 3 patching strategies
2. ✅ Test on sample data
3. ✅ Validate outputs

Step 3: Visualization (1 day)

1. ✅ Create 4-panel Plotly figure
2. ✅ Add patch boundaries
3. ✅ Test interactivity

Step 4: Gradio UI (1 day)

1. ✅ Build basic interface
2. ✅ Wire up controls
3. ✅ Add audio playback

Step 5: Audio Playback (0.5 days)

1. ✅ Implement "play until boundary"
2. ✅ Test timing accuracy

Step 6: Polish (0.5 days)

1. ✅ Fix bugs
2. ✅ Improve layout
3. ✅ Add documentation

Total Time: ~4 days

---

Success Criteria

Functional

• ✅ All 10 audio files precomputed
• ✅ 3 patching strategies working
• ✅ Real-time threshold adjustment
• ✅ Interactive visualizations
• ✅ "Play until boundary" feature

Performance

• ✅ <1s latency for parameter changes
• ✅ Smooth plot interactions
• ✅ Accurate audio playback timing

Quality

• ✅ Clear visualizations
• ✅ Intuitive controls
• ✅ No crashes on edge cases (threshold=0, etc.)

---

Future Enhancements (Optional)

1. Click-to-play: Click on waveform → play from that position
2. Patch highlighting: Click patch → highlight in all plots
3. Comparison view: Show 2 strategies side-by-side
4. Statistics table: Per-patch details in table
5. Download data: Export patches as JSON

These are nice-to-have but not essential for v1.