# Experiment Plan: Confluent Distillation for Vision-Language Models

## Design Principle
Maximize visual richness and experimental credibility while minimizing implementation cost.
Prioritize: existing logs → existing checkpoints (eval only) → lightweight post-hoc analysis.
Avoid: retraining, re-running full pipelines, expensive new computation.

---

## 1. Main Results Table (Table 1)

Full benchmark suite across model scales. Columns: OCR, InfoVQA, DocVQA, ChartQA, TextVQA, V*Bench, HR4K, HR8K, RWQA, MME-RW.

### 1.1 Qwen3-VL-4B

| Method | Checkpoint | Eval Status |
|--------|------------|-------------|
| Vanilla | Qwen3-VL-4B-Instruct | HAVE (baseline-mar30) |
| SD-RPN | ddp-verify | HAVE (sdrpn-mar30) |
| + Confluent (α=2/3) | selfdistill-a0.5 | HAVE partial (selfdistill-v1-a0.5-mar30), running OCR+Info now |
| + Confluent + IWA | run7b-sdpa-iwa | HAVE (selfdistill-run7b-sdpa-iwa) |

### 1.2 Qwen3-VL-8B

| Method | Checkpoint | Status |
|--------|------------|--------|
| Vanilla | Qwen3-VL-8B-Instruct | Running OCR+Info eval now (GPU 3) |
| SD-RPN | 8b-roi-K24T3-185k-sdrpn | Just finished training → NEED eval |
| + Confluent | TODO | NEED train+eval |
| + Confluent + IWA | TODO | NEED train+eval |

### 1.3 Qwen3-VL-30B-A3B (MoE)

On hold per user — model on NAS (58GB, 48 layers). User will set up prerequisites.

---

## 2. Ablation Studies — Low-Cost, High-Impact

### Ablation A: Training Dynamics Comparison (FREE — from existing logs)

**Goal**: Show that Confluent Distillation changes convergence behavior and that IWA produces a distinct loss landscape.

**Data source**: `trainer_state.json` from 7 existing checkpoints — zero compute cost.

| Curve | Checkpoint | Log entries |
|-------|-----------|-------------|
| SD-RPN only | ddp-verify | 1450 |
| + Confluent α=2/3 | selfdistill-a0.5 | 1450 |
| + Confluent + IWA | run7b-sdpa-iwa | 1450 |

**Figure 2a — Training Loss Curves** (line plot)
- x-axis: training step (0–1450)
- y-axis: loss
- 3 main curves: SD-RPN, +Confluent, +Confluent+IWA
- Inset/zoom: steps 0–50 showing early convergence differences
- Key observation: IWA loss is higher (~0.50) than SD-RPN (~0.01) because KL term dominates — this is expected and shows the distillation is active
- Color palette: 3 distinct colors from academic palette
- Plot type: solid lines, 1.5pt width, subtle grid

**Figure 2b — Gradient Norm Evolution** (line plot)
- x-axis: training step
- y-axis: gradient L2 norm (log scale)
- Same 4 curves
- Insight: IWA produces higher grad norm early (larger attention bias gradients), then stabilizes
- Convincing because: shows IWA fundamentally changes optimization dynamics, not just a passive bias

**Figure 2c — Learning Rate Schedule** (thin overlay, shared x-axis with 2a)
- Cosine schedule with warmup — single dashed gray curve
- Shared x-axis with loss plot for temporal alignment

**Implementation**: Single Python script reads `trainer_state.json` from each checkpoint, extracts `loss` and `grad_norm` arrays, plots with matplotlib.

---

### Ablation B: Distillation Weight α Sensitivity (CHEAP — eval existing checkpoints)

**Goal**: Show optimal α and demonstrate the method's sensitivity/robustness to this hyperparameter.

**Data**: Eval OCR+InfoVQA on 4 existing checkpoints (α=1.0, 10/11, 2/3, 1/2). Most results already exist.

| α | OCR Source | InfoVQA Source | Need Eval? |
|---|-----------|---------------|------------|
| 0.0 | sdrpn-mar30 | sdrpn-mar30 | NO |
| 0.1 | self-distill results | self-distill results | NO |
| 0.5 | selfdistill-v1-a0.5-mar30 | selfdistill-v1-a0.5-mar30 | NO (have full) |
| 1.0 | selfdistill-run5-a1.0 | selfdistill-run5-a1.0 | NO (have full) |

**Figure 3a — α vs OCRBench** (line plot with markers)
- x-axis: α (0.0, 0.1, 0.5, 1.0)
- y-axis: OCRBench score
- Single line with circular markers, error bands if we have multiple runs
- Dotted horizontal line: vanilla baseline (839)

**Figure 3b — α vs InfoVQA** (same layout, shared x-axis)
- Can be subplot beside 3a for compact layout

**Figure 3c — α vs Multiple Benchmarks** (grouped bar chart)
- x-axis: α values
- y-axis: normalized improvement over vanilla (%)
- Grouped bars: OCR, InfoVQA, DocVQA, V*Bench
- Shows α=2/3 is optimal across most benchmarks

**Convincing because**: 4 data points across the full α range with consistent trend. Peak at 0.5 with degradation at 1.0 tells a clean story.

---

### Ablation C: Additive Component Analysis (FREE — from existing results)

**Goal**: Show the additive benefit of each component: SD-RPN alone → + Confluent KL → + IWA.

**Data**: All results already exist.

| Method | Checkpoint | Result Source |
|--------|-----------|---------------|
| Vanilla (no training) | Qwen3-VL-4B-Instruct | baseline-mar30 |
| SD-RPN | ddp-verify | sdrpn-mar30 |
| + Confluent (α=2/3, τ=2) | selfdistill-a0.5 | selfdistill-v1-a0.5-mar30 |
| + Confluent + IWA | run7b-sdpa-iwa | selfdistill-run7b-sdpa-iwa |

**Figure 4 — Additive Component Gains** (grouped bar chart)
- x-axis: 4 methods (progressive addition)
- y-axis: score
- Grouped bars: OCR, InfoVQA, DocVQA, V*Bench
- Arrow annotations showing Δ between consecutive methods
- Shows each component adds independent value

**Convincing because**: Clean progressive story — each component is additive and none is redundant.

---

### Ablation D: Learned IWA γ per Layer (FREE — from checkpoint weights)

**Goal**: Show that IWA learns a meaningful, interpretable attention bias pattern — layer 24 (branch point) dominates.

**Data**: Load `iwa.gamma` from run7b checkpoint. Zero compute.

**Figure 5 — Learned γ Values** (bar chart)
- x-axis: layer index (24, 25, 26, ..., 35)
- y-axis: learned γ value
- Single bar chart, 12 bars
- Layer 24 bar visually dominant (~15.0 or large), others near zero
- Add annotation arrow: "Branch point layer — twig signal is freshest"

**Figure 5b — γ across different runs** (grouped bars, optional)
- Compare gamma from: run7b (original), run7b-det, run7-iwa (value-scale attempt)
- Shows convergence to similar pattern across runs (except the diverged one)

**Convincing because**: Highly interpretable — the model "discovers" that the branch point layer needs the most attention bias. This is a strong qualitative finding.

---

### Ablation E: Per-Benchmark Improvement Breakdown (FREE — from existing results)

**Goal**: Show which benchmark categories benefit most from each component.

**Data**: Compile existing results into delta tables.

**Figure 6 — Stacked Improvement over Vanilla** (stacked/grouped bar chart)
- x-axis: benchmarks (OCR, InfoVQA, DocVQA, ChartQA, TextVQA, V*Bench, HR4K, RWQA)
- y-axis: absolute improvement over vanilla
- 3 stacked components: SD-RPN gain, +Confluent gain, +IWA gain
- Shows SD-RPN helps OCR most, Confluent helps InfoVQA/V*Bench, IWA provides uniform boost

**Figure 6b — Radar/Spider Chart** (optional, for visual diversity)
- One polygon per method (Vanilla, SD-RPN, +Confluent, +Confluent+IWA)
- Axes: 8 benchmarks normalized to [0,1]
- Shows the expanding polygon as components are added

**Convincing because**: Comprehensive coverage of 8 benchmarks × 4 methods = 32 data points. Stacked format tells the "additive contribution" story cleanly.

---

### Ablation F: Convergence Speed Analysis (FREE — from existing logs)

**Goal**: Show that Confluent Distillation accelerates convergence to the same or better final performance.

**Data**: From trainer_state.json loss curves.

**Figure 7 — Steps to Reach Target Loss** (bar chart)
- For each method, compute: number of steps to reach loss < 0.1 (or first plateau)
- x-axis: methods
- y-axis: steps to convergence
- Shows Confluent+IWA reaches plateau faster despite higher absolute loss (because KL loss is added)

**Figure 7b — Loss at Fixed Steps** (grouped bar chart)
- x-axis: step milestones (50, 100, 200, 500, 1000)
- y-axis: loss value
- Grouped bars for each method
- Shows convergence trajectory at a glance

**Convincing because**: Training efficiency is a practical concern. Shows our method doesn't just improve final quality but also converges faster.

---

### Ablation G: ROI Threshold Sensitivity (FREE — have threshold_sweep results)

**Goal**: Show robustness to ROI confidence threshold at inference.

**Data**: `results/threshold_sweep/` already has results for thresh=0.05, 0.15, 0.30, 0.50.

**Figure 8 — Threshold Sensitivity** (line plot with markers)
- x-axis: ROI confidence threshold
- y-axis: OCRBench score
- Multiple lines if available for different methods
- Shows broad plateau around 0.10–0.20 → robust to threshold choice

**Convincing because**: Demonstrates the method doesn't require careful threshold tuning. Existing data, zero cost.

---

### Ablation H: Weight Divergence Analysis (FREE — from checkpoint comparison)

**Goal**: Quantify how much Confluent Distillation changes the twig weights relative to SD-RPN-only training. Shows the distillation signal meaningfully reshapes the learned representations.

**Data**: Load twig layer weights from SD-RPN checkpoint and Confluent+IWA checkpoint, compute per-layer L2 distance.

**Figure 9 — Twig Weight Drift per Layer** (bar chart)
- x-axis: twig layer (0, 1, 2) × sublayer (Q, K, V, O, MLP)
- y-axis: relative L2 distance ‖W_confluent - W_sdrpn‖ / ‖W_sdrpn‖
- Shows which twig components are most affected by distillation
- Hypothesis: Q/K projections (used for ROI scoring) show largest drift

**Convincing because**: Directly shows the KL distillation signal propagates into the twig weights, not just the backbone. Zero compute — just load and compare two safetensor files.

---

## 3. Visualization Figures

### Figure 1: Method Overview (architecture diagram)
- Schematic: twig branch at layer K, backbone continuation, IWA bias arrows, KL distillation loop
- Tool: TikZ or professional diagram tool
- Status: manual design needed

### Figure 10: Attention Map Comparison (qualitative)
- 3 example images × 4 columns: Input, Vanilla Attn, SD-RPN Attn, Confluent+IWA Attn
- Select examples where OCRBench improves (text-heavy images)
- Status: NEED to run attention extraction script on eval samples
- Cost: moderate (need forward pass with attention output, but single images)

### Figure 11: Twig Importance Heatmaps
- Overlay twig importance scores (from `compute_importance`) on input images
- Show: high importance on text regions, low on background
- Status: NEED lightweight script to extract importance from model forward pass
- Cost: low (single forward pass per image, no training)

---

## 4. Plot Style Guidelines

### Unified Color Palette

All figures use the following 8-color palette **exclusively** — no other colors for data series, bars, lines, or highlights. Assign colors semantically and keep assignments consistent across every figure in the paper.

```
#f57c6e  Coral Red        — Confluent+IWA (best method, always this color)
#f2b56e  Warm Amber       — Confluent (KL only)
#fbe79e  Soft Gold         — SD-RPN baseline
#84c3b7  Sage Teal        — Vanilla / base model
#88d7da  Light Cyan       — 8B scale results
#71b8ed  Sky Blue         — 30B scale results (or secondary accent)
#b8aeea  Soft Lavender    — ablation variants (α, τ sweep points)
#f2a8da  Rose Pink        — emphasis / annotation / delta highlights
```

Semantic assignment across all figures:

| Element | Color | Hex |
|---------|-------|-----|
| Vanilla (no training) | Sage Teal | `#84c3b7` |
| SD-RPN only | Soft Gold | `#fbe79e` |
| + Confluent (KL, α=2/3) | Warm Amber | `#f2b56e` |
| + Confluent + IWA (Run7b) | Coral Red | `#f57c6e` |
| α / τ sweep variants | Soft Lavender | `#b8aeea` |
| 8B scale | Light Cyan | `#88d7da` |
| 30B scale | Sky Blue | `#71b8ed` |
| Δ / annotation | Rose Pink | `#f2a8da` |

### rcParams Configuration

```python
import matplotlib.pyplot as plt
import matplotlib as mpl

# --- Unified palette ---
PALETTE = ['#f57c6e', '#f2b56e', '#fbe79e', '#84c3b7',
           '#88d7da', '#71b8ed', '#b8aeea', '#f2a8da']

COLORS = {
    'vanilla':    '#84c3b7',
    'sdrpn':      '#fbe79e',
    'confluent':  '#f2b56e',
    'iwa':        '#f57c6e',
    'sweep':      '#b8aeea',
    'scale_8b':   '#88d7da',
    'scale_30b':  '#71b8ed',
    'accent':     '#f2a8da',
}

plt.rcParams.update({
    # --- Resolution & export ---
    'figure.dpi': 300,
    'savefig.dpi': 300,
    'savefig.bbox': 'tight',
    'savefig.pad_inches': 0.02,

    # --- Typography (academic serif) ---
    'font.size': 9,
    'font.family': 'serif',
    'font.serif': ['Times New Roman', 'DejaVu Serif'],
    'mathtext.fontset': 'stix',
    'axes.labelsize': 10,
    'axes.titlesize': 11,
    'legend.fontsize': 8,
    'xtick.labelsize': 8,
    'ytick.labelsize': 8,

    # --- Grid & spines ---
    'axes.grid': True,
    'grid.alpha': 0.25,
    'grid.linestyle': '--',
    'grid.linewidth': 0.5,
    'axes.spines.top': False,
    'axes.spines.right': False,
    'axes.linewidth': 0.8,

    # --- Lines & markers ---
    'lines.linewidth': 1.5,
    'lines.markersize': 5,
    'lines.markeredgewidth': 0.8,

    # --- Color cycle (use our palette) ---
    'axes.prop_cycle': mpl.cycler(color=PALETTE),

    # --- Layout ---
    'figure.figsize': (3.5, 2.5),       # single-column
    'figure.constrained_layout.use': True,

    # --- Legend ---
    'legend.frameon': True,
    'legend.framealpha': 0.9,
    'legend.edgecolor': '#cccccc',
    'legend.fancybox': False,
    'legend.borderpad': 0.3,
    'legend.handlelength': 1.5,
})
```

### Layout & Rendering Rules

- Single-column: 3.5" wide, double-column: 7.0" wide
- Tight margins (`bbox_inches='tight'`, `pad_inches=0.02`)
- No unnecessary whitespace — every pixel carries information
- Legends inside plot area when possible (upper-right or lower-right)
- Axis labels with units, no redundant titles
- Consistent serif font (Times New Roman) across all figures
- Export as PDF for vector quality, PNG@300dpi for raster fallback
- Bar charts: use 0.6–0.8 bar width, thin dark edge (`edgecolor='#333'`, `linewidth=0.5`)
- Line plots: solid lines for main methods, dashed for baselines/references
- Markers: circles for main data points, triangles for ablation variants
- **All data series use only colors from the 8-color palette above — no exceptions**
- Grouped bars within a figure follow left-to-right semantic order: Vanilla → SD-RPN → Confluent → IWA
- Hatching patterns (e.g. `///`, `\\\\`) may be added for print accessibility but are optional

---

## 5. Execution Priority

| Priority | Ablation | Cost | Figures | Data Source |
|----------|----------|------|---------|-------------|
| **P0** | A: Training dynamics | FREE | Fig 2a,2b | trainer_state.json (7 runs) |
| **P0** | D: Learned γ | FREE | Fig 5 | checkpoint weights |
| **P0** | E: Per-benchmark breakdown | FREE | Fig 6 | existing eval results |
| **P0** | H: Weight divergence | FREE | Fig 9 | checkpoint weights |
| **P1** | B: α sensitivity | FREE/LOW | Fig 3 | existing eval results (may need 1-2 evals) |
| **P1** | C: Component ablation | FREE | Fig 4 | existing eval results |
| **P1** | F: Convergence speed | FREE | Fig 7 | trainer_state.json |
| **P1** | G: Threshold sensitivity | FREE | Fig 8 | threshold_sweep results |
| **P2** | Attention maps | MODERATE | Fig 10 | need forward pass extraction |
| **P2** | Importance heatmaps | LOW | Fig 11 | single forward pass per image |
| **P3** | Scaling plot | BLOCKED | Fig 6c | need 8B+30B results |
| **P3** | K/T ablation | EXPENSIVE | Table 5 | need new training runs |

**Immediate next steps (zero cost)**:
1. Generate Figures 2a, 2b, 5, 9 from existing data
2. Compile all existing eval results into master table
3. Generate Figures 3, 4, 6 from compiled results

**Low cost (eval only)**:
4. Run OCR+InfoVQA on remaining 4B checkpoints (running now)
5. Run full eval suite on 8B SD-RPN (after training finishes)
6. Run 8B Vanilla baseline eval (running now)

**Medium cost (new training)**:
7. Train 8B Confluent + 8B Confluent+IWA
8. Generate attention map visualizations