doc/experiment_plan.md

Experiment Plan: Confluent Distillation for Vision-Language Models

Design Principle

Maximize visual richness and experimental credibility while minimizing implementation cost. Prioritize: existing logs → existing checkpoints (eval only) → lightweight post-hoc analysis. Avoid: retraining, re-running full pipelines, expensive new computation.

---

1. Main Results Table (Table 1)

Full benchmark suite across model scales. Columns: OCR, InfoVQA, DocVQA, ChartQA, TextVQA, V*Bench, HR4K, HR8K, RWQA, MME-RW.

1.1 Qwen3-VL-4B

Method	Checkpoint	Eval Status
Vanilla	Qwen3-VL-4B-Instruct	HAVE (baseline-mar30)
SD-RPN	ddp-verify	HAVE (sdrpn-mar30)
+ Confluent (α=2/3)	selfdistill-a0.5	HAVE partial (selfdistill-v1-a0.5-mar30), running OCR+Info now
+ Confluent + IWA	run7b-sdpa-iwa	HAVE (selfdistill-run7b-sdpa-iwa)

1.2 Qwen3-VL-8B

Method	Checkpoint	Status
Vanilla	Qwen3-VL-8B-Instruct	Running OCR+Info eval now (GPU 3)
SD-RPN	8b-roi-K24T3-185k-sdrpn	Just finished training → NEED eval
+ Confluent	TODO	NEED train+eval
+ Confluent + IWA	TODO	NEED train+eval

1.3 Qwen3-VL-30B-A3B (MoE)

On hold per user — model on NAS (58GB, 48 layers). User will set up prerequisites.

---

2. Ablation Studies — Low-Cost, High-Impact

Ablation A: Training Dynamics Comparison (FREE — from existing logs)

Goal: Show that Confluent Distillation changes convergence behavior and that IWA produces a distinct loss landscape.

Data source: trainer_state.json from 7 existing checkpoints — zero compute cost.

Curve	Checkpoint	Log entries
SD-RPN only	ddp-verify	1450
+ Confluent α=2/3	selfdistill-a0.5	1450
+ Confluent + IWA	run7b-sdpa-iwa	1450

Figure 2a — Training Loss Curves (line plot)

• x-axis: training step (0–1450)
• y-axis: loss
• 3 main curves: SD-RPN, +Confluent, +Confluent+IWA
• Inset/zoom: steps 0–50 showing early convergence differences
• Key observation: IWA loss is higher (0.50) than SD-RPN (0.01) because KL term dominates — this is expected and shows the distillation is active
• Color palette: 3 distinct colors from academic palette
• Plot type: solid lines, 1.5pt width, subtle grid

Figure 2b — Gradient Norm Evolution (line plot)

• x-axis: training step
• y-axis: gradient L2 norm (log scale)
• Same 4 curves
• Insight: IWA produces higher grad norm early (larger attention bias gradients), then stabilizes
• Convincing because: shows IWA fundamentally changes optimization dynamics, not just a passive bias

Figure 2c — Learning Rate Schedule (thin overlay, shared x-axis with 2a)

• Cosine schedule with warmup — single dashed gray curve
• Shared x-axis with loss plot for temporal alignment

Implementation: Single Python script reads trainer_state.json from each checkpoint, extracts loss and grad_norm arrays, plots with matplotlib.

---

Ablation B: Distillation Weight α Sensitivity (CHEAP — eval existing checkpoints)

Goal: Show optimal α and demonstrate the method's sensitivity/robustness to this hyperparameter.

Data: Eval OCR+InfoVQA on 4 existing checkpoints (α=1.0, 10/11, 2/3, 1/2). Most results already exist.

α	OCR Source	InfoVQA Source	Need Eval?
0.0	sdrpn-mar30	sdrpn-mar30	NO
0.1	self-distill results	self-distill results	NO
0.5	selfdistill-v1-a0.5-mar30	selfdistill-v1-a0.5-mar30	NO (have full)
1.0	selfdistill-run5-a1.0	selfdistill-run5-a1.0	NO (have full)

Figure 3a — α vs OCRBench (line plot with markers)

• x-axis: α (0.0, 0.1, 0.5, 1.0)
• y-axis: OCRBench score
• Single line with circular markers, error bands if we have multiple runs
• Dotted horizontal line: vanilla baseline (839)

Figure 3b — α vs InfoVQA (same layout, shared x-axis)

• Can be subplot beside 3a for compact layout

Figure 3c — α vs Multiple Benchmarks (grouped bar chart)

• x-axis: α values
• y-axis: normalized improvement over vanilla (%)
• Grouped bars: OCR, InfoVQA, DocVQA, V*Bench
• Shows α=2/3 is optimal across most benchmarks

Convincing because: 4 data points across the full α range with consistent trend. Peak at 0.5 with degradation at 1.0 tells a clean story.

---

Ablation C: Additive Component Analysis (FREE — from existing results)

Goal: Show the additive benefit of each component: SD-RPN alone → + Confluent KL → + IWA.

Data: All results already exist.

Method	Checkpoint	Result Source
Vanilla (no training)	Qwen3-VL-4B-Instruct	baseline-mar30
SD-RPN	ddp-verify	sdrpn-mar30
+ Confluent (α=2/3, τ=2)	selfdistill-a0.5	selfdistill-v1-a0.5-mar30
+ Confluent + IWA	run7b-sdpa-iwa	selfdistill-run7b-sdpa-iwa

Figure 4 — Additive Component Gains (grouped bar chart)

• x-axis: 4 methods (progressive addition)
• y-axis: score
• Grouped bars: OCR, InfoVQA, DocVQA, V*Bench
• Arrow annotations showing Δ between consecutive methods
• Shows each component adds independent value

Convincing because: Clean progressive story — each component is additive and none is redundant.

---

Ablation D: Learned IWA γ per Layer (FREE — from checkpoint weights)

Goal: Show that IWA learns a meaningful, interpretable attention bias pattern — layer 24 (branch point) dominates.

Data: Load iwa.gamma from run7b checkpoint. Zero compute.

Figure 5 — Learned γ Values (bar chart)

• x-axis: layer index (24, 25, 26, ..., 35)
• y-axis: learned γ value
• Single bar chart, 12 bars
• Layer 24 bar visually dominant (~15.0 or large), others near zero
• Add annotation arrow: "Branch point layer — twig signal is freshest"

Figure 5b — γ across different runs (grouped bars, optional)

• Compare gamma from: run7b (original), run7b-det, run7-iwa (value-scale attempt)
• Shows convergence to similar pattern across runs (except the diverged one)

Convincing because: Highly interpretable — the model "discovers" that the branch point layer needs the most attention bias. This is a strong qualitative finding.

---

Ablation E: Per-Benchmark Improvement Breakdown (FREE — from existing results)

Goal: Show which benchmark categories benefit most from each component.

Data: Compile existing results into delta tables.

Figure 6 — Stacked Improvement over Vanilla (stacked/grouped bar chart)

• x-axis: benchmarks (OCR, InfoVQA, DocVQA, ChartQA, TextVQA, V*Bench, HR4K, RWQA)
• y-axis: absolute improvement over vanilla
• 3 stacked components: SD-RPN gain, +Confluent gain, +IWA gain
• Shows SD-RPN helps OCR most, Confluent helps InfoVQA/V*Bench, IWA provides uniform boost

Figure 6b — Radar/Spider Chart (optional, for visual diversity)

• One polygon per method (Vanilla, SD-RPN, +Confluent, +Confluent+IWA)
• Axes: 8 benchmarks normalized to [0,1]
• Shows the expanding polygon as components are added

Convincing because: Comprehensive coverage of 8 benchmarks × 4 methods = 32 data points. Stacked format tells the "additive contribution" story cleanly.

---

Ablation F: Convergence Speed Analysis (FREE — from existing logs)

Goal: Show that Confluent Distillation accelerates convergence to the same or better final performance.

Data: From trainer_state.json loss curves.

Figure 7 — Steps to Reach Target Loss (bar chart)

• For each method, compute: number of steps to reach loss < 0.1 (or first plateau)
• x-axis: methods
• y-axis: steps to convergence
• Shows Confluent+IWA reaches plateau faster despite higher absolute loss (because KL loss is added)

Figure 7b — Loss at Fixed Steps (grouped bar chart)

• x-axis: step milestones (50, 100, 200, 500, 1000)
• y-axis: loss value
• Grouped bars for each method
• Shows convergence trajectory at a glance

Convincing because: Training efficiency is a practical concern. Shows our method doesn't just improve final quality but also converges faster.

---

Ablation G: ROI Threshold Sensitivity (FREE — have threshold_sweep results)

Goal: Show robustness to ROI confidence threshold at inference.

Data: results/threshold_sweep/ already has results for thresh=0.05, 0.15, 0.30, 0.50.

Figure 8 — Threshold Sensitivity (line plot with markers)

• x-axis: ROI confidence threshold
• y-axis: OCRBench score
• Multiple lines if available for different methods
• Shows broad plateau around 0.10–0.20 → robust to threshold choice

Convincing because: Demonstrates the method doesn't require careful threshold tuning. Existing data, zero cost.

---

Ablation H: Weight Divergence Analysis (FREE — from checkpoint comparison)

Goal: Quantify how much Confluent Distillation changes the twig weights relative to SD-RPN-only training. Shows the distillation signal meaningfully reshapes the learned representations.

Data: Load twig layer weights from SD-RPN checkpoint and Confluent+IWA checkpoint, compute per-layer L2 distance.

Figure 9 — Twig Weight Drift per Layer (bar chart)

• x-axis: twig layer (0, 1, 2) × sublayer (Q, K, V, O, MLP)
• y-axis: relative L2 distance ‖W_confluent - W_sdrpn‖ / ‖W_sdrpn‖
• Shows which twig components are most affected by distillation
• Hypothesis: Q/K projections (used for ROI scoring) show largest drift

Convincing because: Directly shows the KL distillation signal propagates into the twig weights, not just the backbone. Zero compute — just load and compare two safetensor files.

---

3. Visualization Figures

Figure 1: Method Overview (architecture diagram)

• Schematic: twig branch at layer K, backbone continuation, IWA bias arrows, KL distillation loop
• Tool: TikZ or professional diagram tool
• Status: manual design needed

Figure 10: Attention Map Comparison (qualitative)

• 3 example images × 4 columns: Input, Vanilla Attn, SD-RPN Attn, Confluent+IWA Attn
• Select examples where OCRBench improves (text-heavy images)
• Status: NEED to run attention extraction script on eval samples
• Cost: moderate (need forward pass with attention output, but single images)

Figure 11: Twig Importance Heatmaps

• Overlay twig importance scores (from compute_importance) on input images
• Show: high importance on text regions, low on background
• Status: NEED lightweight script to extract importance from model forward pass
• Cost: low (single forward pass per image, no training)

---

4. Plot Style Guidelines

Unified Color Palette

All figures use the following 8-color palette exclusively — no other colors for data series, bars, lines, or highlights. Assign colors semantically and keep assignments consistent across every figure in the paper.

#f57c6e  Coral Red        — Confluent+IWA (best method, always this color)
#f2b56e  Warm Amber       — Confluent (KL only)
#fbe79e  Soft Gold         — SD-RPN baseline
#84c3b7  Sage Teal        — Vanilla / base model
#88d7da  Light Cyan       — 8B scale results
#71b8ed  Sky Blue         — 30B scale results (or secondary accent)
#b8aeea  Soft Lavender    — ablation variants (α, τ sweep points)
#f2a8da  Rose Pink        — emphasis / annotation / delta highlights

Semantic assignment across all figures:

Element	Color	Hex
Vanilla (no training)	Sage Teal	#84c3b7
SD-RPN only	Soft Gold	#fbe79e
+ Confluent (KL, α=2/3)	Warm Amber	#f2b56e
+ Confluent + IWA (Run7b)	Coral Red	#f57c6e
α / τ sweep variants	Soft Lavender	#b8aeea
8B scale	Light Cyan	#88d7da
30B scale	Sky Blue	#71b8ed
Δ / annotation	Rose Pink	#f2a8da

rcParams Configuration

import matplotlib.pyplot as plt
import matplotlib as mpl

# --- Unified palette ---
PALETTE = ['#f57c6e', '#f2b56e', '#fbe79e', '#84c3b7',
           '#88d7da', '#71b8ed', '#b8aeea', '#f2a8da']

COLORS = {
    'vanilla':    '#84c3b7',
    'sdrpn':      '#fbe79e',
    'confluent':  '#f2b56e',
    'iwa':        '#f57c6e',
    'sweep':      '#b8aeea',
    'scale_8b':   '#88d7da',
    'scale_30b':  '#71b8ed',
    'accent':     '#f2a8da',
}

plt.rcParams.update({
    # --- Resolution & export ---
    'figure.dpi': 300,
    'savefig.dpi': 300,
    'savefig.bbox': 'tight',
    'savefig.pad_inches': 0.02,

    # --- Typography (academic serif) ---
    'font.size': 9,
    'font.family': 'serif',
    'font.serif': ['Times New Roman', 'DejaVu Serif'],
    'mathtext.fontset': 'stix',
    'axes.labelsize': 10,
    'axes.titlesize': 11,
    'legend.fontsize': 8,
    'xtick.labelsize': 8,
    'ytick.labelsize': 8,

    # --- Grid & spines ---
    'axes.grid': True,
    'grid.alpha': 0.25,
    'grid.linestyle': '--',
    'grid.linewidth': 0.5,
    'axes.spines.top': False,
    'axes.spines.right': False,
    'axes.linewidth': 0.8,

    # --- Lines & markers ---
    'lines.linewidth': 1.5,
    'lines.markersize': 5,
    'lines.markeredgewidth': 0.8,

    # --- Color cycle (use our palette) ---
    'axes.prop_cycle': mpl.cycler(color=PALETTE),

    # --- Layout ---
    'figure.figsize': (3.5, 2.5),       # single-column
    'figure.constrained_layout.use': True,

    # --- Legend ---
    'legend.frameon': True,
    'legend.framealpha': 0.9,
    'legend.edgecolor': '#cccccc',
    'legend.fancybox': False,
    'legend.borderpad': 0.3,
    'legend.handlelength': 1.5,
})

Layout & Rendering Rules

• Single-column: 3.5" wide, double-column: 7.0" wide
• Tight margins (bbox_inches='tight', pad_inches=0.02)
• No unnecessary whitespace — every pixel carries information
• Legends inside plot area when possible (upper-right or lower-right)
• Axis labels with units, no redundant titles
• Consistent serif font (Times New Roman) across all figures
• Export as PDF for vector quality, PNG@300dpi for raster fallback
• Bar charts: use 0.6–0.8 bar width, thin dark edge (edgecolor='#333', linewidth=0.5)
• Line plots: solid lines for main methods, dashed for baselines/references
• Markers: circles for main data points, triangles for ablation variants
• All data series use only colors from the 8-color palette above — no exceptions
• Grouped bars within a figure follow left-to-right semantic order: Vanilla → SD-RPN → Confluent → IWA
• Hatching patterns (e.g. ///, \\\\) may be added for print accessibility but are optional

---

5. Execution Priority

Priority	Ablation	Cost	Figures	Data Source
P0	A: Training dynamics	FREE	Fig 2a,2b	trainer_state.json (7 runs)
P0	D: Learned γ	FREE	Fig 5	checkpoint weights
P0	E: Per-benchmark breakdown	FREE	Fig 6	existing eval results
P0	H: Weight divergence	FREE	Fig 9	checkpoint weights
P1	B: α sensitivity	FREE/LOW	Fig 3	existing eval results (may need 1-2 evals)
P1	C: Component ablation	FREE	Fig 4	existing eval results
P1	F: Convergence speed	FREE	Fig 7	trainer_state.json
P1	G: Threshold sensitivity	FREE	Fig 8	threshold_sweep results
P2	Attention maps	MODERATE	Fig 10	need forward pass extraction
P2	Importance heatmaps	LOW	Fig 11	single forward pass per image
P3	Scaling plot	BLOCKED	Fig 6c	need 8B+30B results
P3	K/T ablation	EXPENSIVE	Table 5	need new training runs

Immediate next steps (zero cost):

1. Generate Figures 2a, 2b, 5, 9 from existing data
2. Compile all existing eval results into master table
3. Generate Figures 3, 4, 6 from compiled results

Low cost (eval only): 4. Run OCR+InfoVQA on remaining 4B checkpoints (running now) 5. Run full eval suite on 8B SD-RPN (after training finishes) 6. Run 8B Vanilla baseline eval (running now)

Medium cost (new training): 7. Train 8B Confluent + 8B Confluent+IWA 8. Generate attention map visualizations