plan.md

Interactive Kernel Convolution Visualizer - Project Plan

Project Overview

Build an interactive web application that demonstrates how image convolution works in real time. The application serves both as an image-processing tool and an educational visualization platform for understanding convolution kernels, their mathematical operations, and their effects on images.

Target Audience

• Students learning computer vision and digital image processing
• Developers learning about CNNs and kernel operations
• Educators teaching image processing concepts
• Anyone curious about how image filters work

Core Features

1. Image Management

• Upload any image (drag & drop, file picker)
• Support common formats: PNG, JPEG, WebP, BMP
• Display original image with zoom/pan capabilities
• Image metadata display (dimensions, file size, color channels)

2. Kernel Library

Predefined kernels:

• Blur Filters: Box Blur (3×3, 5×5, 7×7), Gaussian Blur (3×3, 5×5)
• Sharpen Filters: Sharpen, Unsharp Mask
• Edge Detection: Sobel X, Sobel Y, Sobel Combined, Prewitt, Roberts, Laplacian
• Emboss: Emboss (multiple directions)
• Special Effects: Motion Blur, Outline, Ridge Detection
• Custom Kernel Builder: User-defined matrix with size selector (3×3, 5×5, 7×7, 9×9)

3. Convolution Visualization (Core Feature)

• Step-by-step Animation: Kernel matrix moves across image pixel by pixel
• Current Patch Highlight: Visual indication of image region under kernel
• Element-wise Multiplication: Show kernel values × pixel values
• Summation Display: Real-time calculation of output pixel value
• Continuous Animation: Smooth traversal across entire image
• Progressive Output Rendering: Output image builds up as computation completes

4. Interactive Controls

• Play / Pause / Restart / Step Forward / Step Backward
• Animation speed slider (0.5x - 10x)
• Zoom controls for pixel-level inspection
• Toggle overlays:
  • Kernel boundary visualization
  • Pixel value display
  • Multiplication calculation overlay
  • Summation formula display

5. Comparison & Analysis

• Side-by-side original vs processed view
• Split-view slider (before/after)
• Synchronized zoom/pan between views
• Multiple kernel comparison mode (tabbed results)
• Processing statistics: execution time, operations count, memory usage

6. Padding Strategies

• Visual demonstration of padding effects
• Options: Zero Padding, Reflect Padding, Replicate Padding, Wrap Padding
• Border handling visualization at image edges

7. Educational Insights

• Real-time mathematical equation display
• Kernel purpose/effect description
• Convolution formula with current values
• Channel-wise processing explanation (RGB)
• Links to learning resources

8. Export & Sharing

• Save processed image (PNG, JPEG)
• Export visualization as GIF/MP4
• Shareable URL with current state
• Download convolution computation log (CSV/JSON)

Technical Architecture

Frontend Stack

• Framework: Next.js 14+ (App Router) with TypeScript
• State Management: Zustand or React Context
• Animation: Framer Motion for UI, custom Canvas/WebGL for convolution viz
• Styling: Tailwind CSS
• Canvas: HTML5 Canvas API (primary), WebGL fallback for performance
• Image Processing: Custom JavaScript implementation (no heavy deps)

Component Structure

src/
├── app/                    # Next.js App Router pages
├── components/
│   ├── canvas/            # Canvas-based visualization components
│   ├── controls/          # Playback, speed, zoom controls
│   ├── kernel/            # Kernel selector, editor, library
│   ├── comparison/        # Split view, side-by-side comparison
│   ├── education/         # Math display, kernel info panels
│   └── ui/                # Reusable UI components (buttons, sliders, etc.)
├── hooks/                 # Custom React hooks
├── lib/
│   ├── convolution/       # Core convolution algorithms
│   ├── kernels/           # Predefined kernel definitions
│   ├── image/             # Image loading, processing utilities
│   └── export/            # Export functionality
├── store/                 # State management
├── types/                 # TypeScript definitions
└── utils/                 # Helper functions

Core Algorithms

Convolution Engine

interface ConvolutionOptions {
  kernel: number[][];
  padding: 'zero' | 'reflect' | 'replicate' | 'wrap';
  normalize: boolean;
  stride: number;
}

function convolve(
  imageData: ImageData,
  kernel: number[][],
  options: ConvolutionOptions,
  onProgress?: (progress: ConvolutionProgress) => void
): Promise<ImageData>;

Step-by-step Generator

interface ConvolutionStep {
  x: number;
  y: number;
  inputPatch: number[][][];
  kernel: number[][];
  multiplications: number[][][];
  sum: number;
  outputValue: number;
  channel: number;
}

function* convolutionSteps(
  imageData: ImageData,
  kernel: number[][],
  options: ConvolutionOptions
): Generator<ConvolutionStep>;

Animation System

• RequestAnimationFrame-based loop
• Configurable frame rate (target 60fps)
• Step batching for performance (process N pixels per frame)
• Pause/resume without losing state
• Step-by-step mode with manual advancement

Performance Considerations

• Web Workers for heavy computation
• Canvas pooling to reduce GC pressure
• TypedArrays for numerical computation
• Progressive rendering (don't block UI)
• Lazy evaluation for comparison views

Development Phases

Phase 1: Foundation (Week 1-2)

• Project setup (Next.js, TypeScript, Tailwind, ESLint, Prettier)
• Basic project structure and component architecture
• Image upload and display component
• Basic Canvas rendering setup
• Kernel data structure and predefined kernels library

Phase 2: Core Convolution (Week 2-3)

• Implement convolution algorithm (CPU)
• Implement padding strategies
• Create step-by-step generator
• Basic convolution visualization (static)
• Output image generation

Phase 3: Animation & Visualization (Week 3-4)

• Animation loop with play/pause/step controls
• Speed control implementation
• Current patch highlighting
• Element-wise multiplication visualization
• Summation process display
• Progressive output rendering

Phase 4: Interactive Controls (Week 4-5)

• Zoom/pan functionality
• Overlay toggles (kernel boundary, values, math)
• Kernel selector UI with categories
• Custom kernel editor
• Padding strategy selector with visualization

Phase 5: Comparison & Analysis (Week 5-6)

• Side-by-side comparison view
• Split slider component
• Synchronized zoom/pan
• Multi-kernel comparison tabs
• Statistics panel (time, ops, memory)

Phase 6: Educational Features (Week 6-7)

• Real-time equation display
• Kernel information panels
• Channel processing visualization
• Tooltip/explanation system
• Learning resource links

Phase 7: Export & Polish (Week 7-8)

• Save processed image
• Export animation (GIF/Video)
• Shareable state URLs
• Performance optimization
• Mobile responsiveness
• Accessibility improvements
• Documentation & help system

Phase 8: Advanced Features (Week 8+)

• WebGL acceleration option
• Web Worker for computation
• Batch processing multiple images
• Kernel composition (sequential filters)
• Frequency domain visualization (FFT)
• 3D kernel visualization

Data Structures

Kernel Definition

interface Kernel {
  id: string;
  name: string;
  category: 'blur' | 'sharpen' | 'edge' | 'emboss' | 'custom' | 'other';
  matrix: number[][];
  size: number;
  normalize: boolean;
  description: string;
  formula?: string;
  reference?: string;
}

Convolution State

interface ConvolutionState {
  status: 'idle' | 'running' | 'paused' | 'completed' | 'error';
  currentStep: ConvolutionStep | null;
  progress: number; // 0-1
  startTime: number;
  elapsedTime: number;
  processedPixels: number;
  totalPixels: number;
  outputImageData: ImageData | null;
}

App Settings

interface AppSettings {
  animationSpeed: number; // 0.5 - 10
  showOverlay: {
    kernelBoundary: boolean;
    pixelValues: boolean;
    multiplications: boolean;
    summation: boolean;
  };
  padding: 'zero' | 'reflect' | 'replicate' | 'wrap';
  zoomLevel: number;
  panOffset: { x: number; y: number };
  comparisonMode: 'side-by-side' | 'split' | 'overlay';
}

Testing Strategy

Unit Tests

• Convolution algorithm correctness
• Padding strategy implementations
• Kernel normalization
• Step generator accuracy
• Export functionality

Integration Tests

• Image upload → convolution → output pipeline
• Animation playback controls
• Comparison view synchronization
• State persistence/restoration

Visual Regression Tests

• Canvas rendering at different zoom levels
• Overlay accuracy
• Split slider positions
• Kernel matrix display

Performance Benchmarks

• Convolution speed (pixels/second)
• Animation frame rate
• Memory usage during large image processing
• Web Worker vs main thread performance

Accessibility

• Keyboard navigation for all controls
• Screen reader announcements for animation state
• High contrast mode support
• Reduced motion preference respect
• Focus indicators
• Alt text for all visual elements

Browser Support

• Chrome 90+
• Firefox 88+
• Safari 14+
• Edge 90+
• Mobile browsers (iOS Safari, Chrome Android)

Deployment

• Static export to Vercel/Netlify
• CDN for kernel definitions and assets
• PWA support for offline use
• Analytics (privacy-friendly)

Success Metrics

• Frame rate ≥ 30fps during animation
• Convolution of 1MP image < 2 seconds
• Time to interactive < 3 seconds
• Lighthouse score > 90
• Zero critical accessibility violations

Risk Mitigation

Risk	Mitigation
Performance on large images	Web Workers, progressive rendering, WebGL fallback
Animation smoothness	RequestAnimationFrame, frame budgeting, step batching
Memory leaks	Canvas pooling, proper cleanup, WeakRef for caches
Cross-browser canvas differences	Feature detection, polyfills, fallback rendering
Complex state management	Zustand with devtools, immutable updates

Future Extensions

• Video convolution visualization
• 3D convolution (volumetric data)
• CNN layer visualization (multiple kernels, feature maps)
• Interactive kernel design with real-time preview
• Frequency domain analysis
• Integration with ML frameworks (TensorFlow.js, ONNX.js)
• Collaborative annotation/sharing
• Plugin system for custom kernels/effects