| # Experiment Plan: Confluent Distillation for Vision-Language Models |
|
|
| ## Design Principle |
| Maximize visual richness and experimental credibility while minimizing implementation cost. |
| Prioritize: existing logs β existing checkpoints (eval only) β lightweight post-hoc analysis. |
| Avoid: retraining, re-running full pipelines, expensive new computation. |
|
|
| --- |
|
|
| ## 1. Main Results Table (Table 1) |
|
|
| Full benchmark suite across model scales. Columns: OCR, InfoVQA, DocVQA, ChartQA, TextVQA, V*Bench, HR4K, HR8K, RWQA, MME-RW. |
| |
| ### 1.1 Qwen3-VL-4B |
| |
| | Method | Checkpoint | Eval Status | |
| |--------|------------|-------------| |
| | Vanilla | Qwen3-VL-4B-Instruct | HAVE (baseline-mar30) | |
| | SD-RPN | ddp-verify | HAVE (sdrpn-mar30) | |
| | + Confluent (Ξ±=2/3) | selfdistill-a0.5 | HAVE partial (selfdistill-v1-a0.5-mar30), running OCR+Info now | |
| | + Confluent + IWA | run7b-sdpa-iwa | HAVE (selfdistill-run7b-sdpa-iwa) | |
| |
| ### 1.2 Qwen3-VL-8B |
| |
| | Method | Checkpoint | Status | |
| |--------|------------|--------| |
| | Vanilla | Qwen3-VL-8B-Instruct | Running OCR+Info eval now (GPU 3) | |
| | SD-RPN | 8b-roi-K24T3-185k-sdrpn | Just finished training β NEED eval | |
| | + Confluent | TODO | NEED train+eval | |
| | + Confluent + IWA | TODO | NEED train+eval | |
| |
| ### 1.3 Qwen3-VL-30B-A3B (MoE) |
| |
| On hold per user β model on NAS (58GB, 48 layers). User will set up prerequisites. |
| |
| --- |
| |
| ## 2. Ablation Studies β Low-Cost, High-Impact |
| |
| ### Ablation A: Training Dynamics Comparison (FREE β from existing logs) |
| |
| **Goal**: Show that Confluent Distillation changes convergence behavior and that IWA produces a distinct loss landscape. |
| |
| **Data source**: `trainer_state.json` from 7 existing checkpoints β zero compute cost. |
| |
| | Curve | Checkpoint | Log entries | |
| |-------|-----------|-------------| |
| | SD-RPN only | ddp-verify | 1450 | |
| | + Confluent Ξ±=2/3 | selfdistill-a0.5 | 1450 | |
| | + Confluent + IWA | run7b-sdpa-iwa | 1450 | |
| |
| **Figure 2a β Training Loss Curves** (line plot) |
| - x-axis: training step (0β1450) |
| - y-axis: loss |
| - 3 main curves: SD-RPN, +Confluent, +Confluent+IWA |
| - Inset/zoom: steps 0β50 showing early convergence differences |
| - Key observation: IWA loss is higher (~0.50) than SD-RPN (~0.01) because KL term dominates β this is expected and shows the distillation is active |
| - Color palette: 3 distinct colors from academic palette |
| - Plot type: solid lines, 1.5pt width, subtle grid |
| |
| **Figure 2b β Gradient Norm Evolution** (line plot) |
| - x-axis: training step |
| - y-axis: gradient L2 norm (log scale) |
| - Same 4 curves |
| - Insight: IWA produces higher grad norm early (larger attention bias gradients), then stabilizes |
| - Convincing because: shows IWA fundamentally changes optimization dynamics, not just a passive bias |
| |
| **Figure 2c β Learning Rate Schedule** (thin overlay, shared x-axis with 2a) |
| - Cosine schedule with warmup β single dashed gray curve |
| - Shared x-axis with loss plot for temporal alignment |
| |
| **Implementation**: Single Python script reads `trainer_state.json` from each checkpoint, extracts `loss` and `grad_norm` arrays, plots with matplotlib. |
| |
| --- |
| |
| ### Ablation B: Distillation Weight Ξ± Sensitivity (CHEAP β eval existing checkpoints) |
| |
| **Goal**: Show optimal Ξ± and demonstrate the method's sensitivity/robustness to this hyperparameter. |
| |
| **Data**: Eval OCR+InfoVQA on 4 existing checkpoints (Ξ±=1.0, 10/11, 2/3, 1/2). Most results already exist. |
| |
| | Ξ± | OCR Source | InfoVQA Source | Need Eval? | |
| |---|-----------|---------------|------------| |
| | 0.0 | sdrpn-mar30 | sdrpn-mar30 | NO | |
| | 0.1 | self-distill results | self-distill results | NO | |
| | 0.5 | selfdistill-v1-a0.5-mar30 | selfdistill-v1-a0.5-mar30 | NO (have full) | |
| | 1.0 | selfdistill-run5-a1.0 | selfdistill-run5-a1.0 | NO (have full) | |
| |
| **Figure 3a β Ξ± vs OCRBench** (line plot with markers) |
| - x-axis: Ξ± (0.0, 0.1, 0.5, 1.0) |
| - y-axis: OCRBench score |
| - Single line with circular markers, error bands if we have multiple runs |
| - Dotted horizontal line: vanilla baseline (839) |
| |
| **Figure 3b β Ξ± vs InfoVQA** (same layout, shared x-axis) |
| - Can be subplot beside 3a for compact layout |
| |
| **Figure 3c β Ξ± vs Multiple Benchmarks** (grouped bar chart) |
| - x-axis: Ξ± values |
| - y-axis: normalized improvement over vanilla (%) |
| - Grouped bars: OCR, InfoVQA, DocVQA, V*Bench |
| - Shows Ξ±=2/3 is optimal across most benchmarks |
|
|
| **Convincing because**: 4 data points across the full Ξ± range with consistent trend. Peak at 0.5 with degradation at 1.0 tells a clean story. |
|
|
| --- |
|
|
| ### Ablation C: Additive Component Analysis (FREE β from existing results) |
|
|
| **Goal**: Show the additive benefit of each component: SD-RPN alone β + Confluent KL β + IWA. |
|
|
| **Data**: All results already exist. |
|
|
| | Method | Checkpoint | Result Source | |
| |--------|-----------|---------------| |
| | Vanilla (no training) | Qwen3-VL-4B-Instruct | baseline-mar30 | |
| | SD-RPN | ddp-verify | sdrpn-mar30 | |
| | + Confluent (Ξ±=2/3, Ο=2) | selfdistill-a0.5 | selfdistill-v1-a0.5-mar30 | |
| | + Confluent + IWA | run7b-sdpa-iwa | selfdistill-run7b-sdpa-iwa | |
|
|
| **Figure 4 β Additive Component Gains** (grouped bar chart) |
| - x-axis: 4 methods (progressive addition) |
| - y-axis: score |
| - Grouped bars: OCR, InfoVQA, DocVQA, V*Bench |
| - Arrow annotations showing Ξ between consecutive methods |
| - Shows each component adds independent value |
| |
| **Convincing because**: Clean progressive story β each component is additive and none is redundant. |
| |
| --- |
| |
| ### Ablation D: Learned IWA Ξ³ per Layer (FREE β from checkpoint weights) |
| |
| **Goal**: Show that IWA learns a meaningful, interpretable attention bias pattern β layer 24 (branch point) dominates. |
| |
| **Data**: Load `iwa.gamma` from run7b checkpoint. Zero compute. |
| |
| **Figure 5 β Learned Ξ³ Values** (bar chart) |
| - x-axis: layer index (24, 25, 26, ..., 35) |
| - y-axis: learned Ξ³ value |
| - Single bar chart, 12 bars |
| - Layer 24 bar visually dominant (~15.0 or large), others near zero |
| - Add annotation arrow: "Branch point layer β twig signal is freshest" |
| |
| **Figure 5b β Ξ³ across different runs** (grouped bars, optional) |
| - Compare gamma from: run7b (original), run7b-det, run7-iwa (value-scale attempt) |
| - Shows convergence to similar pattern across runs (except the diverged one) |
| |
| **Convincing because**: Highly interpretable β the model "discovers" that the branch point layer needs the most attention bias. This is a strong qualitative finding. |
| |
| --- |
| |
| ### Ablation E: Per-Benchmark Improvement Breakdown (FREE β from existing results) |
| |
| **Goal**: Show which benchmark categories benefit most from each component. |
| |
| **Data**: Compile existing results into delta tables. |
| |
| **Figure 6 β Stacked Improvement over Vanilla** (stacked/grouped bar chart) |
| - x-axis: benchmarks (OCR, InfoVQA, DocVQA, ChartQA, TextVQA, V*Bench, HR4K, RWQA) |
| - y-axis: absolute improvement over vanilla |
| - 3 stacked components: SD-RPN gain, +Confluent gain, +IWA gain |
| - Shows SD-RPN helps OCR most, Confluent helps InfoVQA/V*Bench, IWA provides uniform boost |
| |
| **Figure 6b β Radar/Spider Chart** (optional, for visual diversity) |
| - One polygon per method (Vanilla, SD-RPN, +Confluent, +Confluent+IWA) |
| - Axes: 8 benchmarks normalized to [0,1] |
| - Shows the expanding polygon as components are added |
| |
| **Convincing because**: Comprehensive coverage of 8 benchmarks Γ 4 methods = 32 data points. Stacked format tells the "additive contribution" story cleanly. |
| |
| --- |
| |
| ### Ablation F: Convergence Speed Analysis (FREE β from existing logs) |
| |
| **Goal**: Show that Confluent Distillation accelerates convergence to the same or better final performance. |
| |
| **Data**: From trainer_state.json loss curves. |
| |
| **Figure 7 β Steps to Reach Target Loss** (bar chart) |
| - For each method, compute: number of steps to reach loss < 0.1 (or first plateau) |
| - x-axis: methods |
| - y-axis: steps to convergence |
| - Shows Confluent+IWA reaches plateau faster despite higher absolute loss (because KL loss is added) |
| |
| **Figure 7b β Loss at Fixed Steps** (grouped bar chart) |
| - x-axis: step milestones (50, 100, 200, 500, 1000) |
| - y-axis: loss value |
| - Grouped bars for each method |
| - Shows convergence trajectory at a glance |
| |
| **Convincing because**: Training efficiency is a practical concern. Shows our method doesn't just improve final quality but also converges faster. |
| |
| --- |
| |
| ### Ablation G: ROI Threshold Sensitivity (FREE β have threshold_sweep results) |
| |
| **Goal**: Show robustness to ROI confidence threshold at inference. |
| |
| **Data**: `results/threshold_sweep/` already has results for thresh=0.05, 0.15, 0.30, 0.50. |
| |
| **Figure 8 β Threshold Sensitivity** (line plot with markers) |
| - x-axis: ROI confidence threshold |
| - y-axis: OCRBench score |
| - Multiple lines if available for different methods |
| - Shows broad plateau around 0.10β0.20 β robust to threshold choice |
| |
| **Convincing because**: Demonstrates the method doesn't require careful threshold tuning. Existing data, zero cost. |
| |
| --- |
| |
| ### Ablation H: Weight Divergence Analysis (FREE β from checkpoint comparison) |
| |
| **Goal**: Quantify how much Confluent Distillation changes the twig weights relative to SD-RPN-only training. Shows the distillation signal meaningfully reshapes the learned representations. |
| |
| **Data**: Load twig layer weights from SD-RPN checkpoint and Confluent+IWA checkpoint, compute per-layer L2 distance. |
| |
| **Figure 9 β Twig Weight Drift per Layer** (bar chart) |
| - x-axis: twig layer (0, 1, 2) Γ sublayer (Q, K, V, O, MLP) |
| - y-axis: relative L2 distance βW_confluent - W_sdrpnβ / βW_sdrpnβ |
| - Shows which twig components are most affected by distillation |
| - Hypothesis: Q/K projections (used for ROI scoring) show largest drift |
| |
| **Convincing because**: Directly shows the KL distillation signal propagates into the twig weights, not just the backbone. Zero compute β just load and compare two safetensor files. |
| |
| --- |
| |
| ## 3. Visualization Figures |
| |
| ### Figure 1: Method Overview (architecture diagram) |
| - Schematic: twig branch at layer K, backbone continuation, IWA bias arrows, KL distillation loop |
| - Tool: TikZ or professional diagram tool |
| - Status: manual design needed |
| |
| ### Figure 10: Attention Map Comparison (qualitative) |
| - 3 example images Γ 4 columns: Input, Vanilla Attn, SD-RPN Attn, Confluent+IWA Attn |
| - Select examples where OCRBench improves (text-heavy images) |
| - Status: NEED to run attention extraction script on eval samples |
| - Cost: moderate (need forward pass with attention output, but single images) |
| |
| ### Figure 11: Twig Importance Heatmaps |
| - Overlay twig importance scores (from `compute_importance`) on input images |
| - Show: high importance on text regions, low on background |
| - Status: NEED lightweight script to extract importance from model forward pass |
| - Cost: low (single forward pass per image, no training) |
| |
| --- |
| |
| ## 4. Plot Style Guidelines |
| |
| ### Unified Color Palette |
| |
| All figures use the following 8-color palette **exclusively** β no other colors for data series, bars, lines, or highlights. Assign colors semantically and keep assignments consistent across every figure in the paper. |
| |
| ``` |
| #f57c6e Coral Red β Confluent+IWA (best method, always this color) |
| #f2b56e Warm Amber β Confluent (KL only) |
| #fbe79e Soft Gold β SD-RPN baseline |
| #84c3b7 Sage Teal β Vanilla / base model |
| #88d7da Light Cyan β 8B scale results |
| #71b8ed Sky Blue β 30B scale results (or secondary accent) |
| #b8aeea Soft Lavender β ablation variants (Ξ±, Ο sweep points) |
| #f2a8da Rose Pink β emphasis / annotation / delta highlights |
| ``` |
| |
| Semantic assignment across all figures: |
| |
| | Element | Color | Hex | |
| |---------|-------|-----| |
| | Vanilla (no training) | Sage Teal | `#84c3b7` | |
| | SD-RPN only | Soft Gold | `#fbe79e` | |
| | + Confluent (KL, Ξ±=2/3) | Warm Amber | `#f2b56e` | |
| | + Confluent + IWA (Run7b) | Coral Red | `#f57c6e` | |
| | Ξ± / Ο sweep variants | Soft Lavender | `#b8aeea` | |
| | 8B scale | Light Cyan | `#88d7da` | |
| | 30B scale | Sky Blue | `#71b8ed` | |
| | Ξ / annotation | Rose Pink | `#f2a8da` | |
| |
| ### rcParams Configuration |
| |
| ```python |
| import matplotlib.pyplot as plt |
| import matplotlib as mpl |
| |
| # --- Unified palette --- |
| PALETTE = ['#f57c6e', '#f2b56e', '#fbe79e', '#84c3b7', |
| '#88d7da', '#71b8ed', '#b8aeea', '#f2a8da'] |
| |
| COLORS = { |
| 'vanilla': '#84c3b7', |
| 'sdrpn': '#fbe79e', |
| 'confluent': '#f2b56e', |
| 'iwa': '#f57c6e', |
| 'sweep': '#b8aeea', |
| 'scale_8b': '#88d7da', |
| 'scale_30b': '#71b8ed', |
| 'accent': '#f2a8da', |
| } |
| |
| plt.rcParams.update({ |
| # --- Resolution & export --- |
| 'figure.dpi': 300, |
| 'savefig.dpi': 300, |
| 'savefig.bbox': 'tight', |
| 'savefig.pad_inches': 0.02, |
| |
| # --- Typography (academic serif) --- |
| 'font.size': 9, |
| 'font.family': 'serif', |
| 'font.serif': ['Times New Roman', 'DejaVu Serif'], |
| 'mathtext.fontset': 'stix', |
| 'axes.labelsize': 10, |
| 'axes.titlesize': 11, |
| 'legend.fontsize': 8, |
| 'xtick.labelsize': 8, |
| 'ytick.labelsize': 8, |
| |
| # --- Grid & spines --- |
| 'axes.grid': True, |
| 'grid.alpha': 0.25, |
| 'grid.linestyle': '--', |
| 'grid.linewidth': 0.5, |
| 'axes.spines.top': False, |
| 'axes.spines.right': False, |
| 'axes.linewidth': 0.8, |
| |
| # --- Lines & markers --- |
| 'lines.linewidth': 1.5, |
| 'lines.markersize': 5, |
| 'lines.markeredgewidth': 0.8, |
| |
| # --- Color cycle (use our palette) --- |
| 'axes.prop_cycle': mpl.cycler(color=PALETTE), |
| |
| # --- Layout --- |
| 'figure.figsize': (3.5, 2.5), # single-column |
| 'figure.constrained_layout.use': True, |
| |
| # --- Legend --- |
| 'legend.frameon': True, |
| 'legend.framealpha': 0.9, |
| 'legend.edgecolor': '#cccccc', |
| 'legend.fancybox': False, |
| 'legend.borderpad': 0.3, |
| 'legend.handlelength': 1.5, |
| }) |
| ``` |
| |
| ### Layout & Rendering Rules |
| |
| - Single-column: 3.5" wide, double-column: 7.0" wide |
| - Tight margins (`bbox_inches='tight'`, `pad_inches=0.02`) |
| - No unnecessary whitespace β every pixel carries information |
| - Legends inside plot area when possible (upper-right or lower-right) |
| - Axis labels with units, no redundant titles |
| - Consistent serif font (Times New Roman) across all figures |
| - Export as PDF for vector quality, PNG@300dpi for raster fallback |
| - Bar charts: use 0.6β0.8 bar width, thin dark edge (`edgecolor='#333'`, `linewidth=0.5`) |
| - Line plots: solid lines for main methods, dashed for baselines/references |
| - Markers: circles for main data points, triangles for ablation variants |
| - **All data series use only colors from the 8-color palette above β no exceptions** |
| - Grouped bars within a figure follow left-to-right semantic order: Vanilla β SD-RPN β Confluent β IWA |
| - Hatching patterns (e.g. `///`, `\\\\`) may be added for print accessibility but are optional |
| |
| --- |
| |
| ## 5. Execution Priority |
| |
| | Priority | Ablation | Cost | Figures | Data Source | |
| |----------|----------|------|---------|-------------| |
| | **P0** | A: Training dynamics | FREE | Fig 2a,2b | trainer_state.json (7 runs) | |
| | **P0** | D: Learned Ξ³ | FREE | Fig 5 | checkpoint weights | |
| | **P0** | E: Per-benchmark breakdown | FREE | Fig 6 | existing eval results | |
| | **P0** | H: Weight divergence | FREE | Fig 9 | checkpoint weights | |
| | **P1** | B: Ξ± sensitivity | FREE/LOW | Fig 3 | existing eval results (may need 1-2 evals) | |
| | **P1** | C: Component ablation | FREE | Fig 4 | existing eval results | |
| | **P1** | F: Convergence speed | FREE | Fig 7 | trainer_state.json | |
| | **P1** | G: Threshold sensitivity | FREE | Fig 8 | threshold_sweep results | |
| | **P2** | Attention maps | MODERATE | Fig 10 | need forward pass extraction | |
| | **P2** | Importance heatmaps | LOW | Fig 11 | single forward pass per image | |
| | **P3** | Scaling plot | BLOCKED | Fig 6c | need 8B+30B results | |
| | **P3** | K/T ablation | EXPENSIVE | Table 5 | need new training runs | |
| |
| **Immediate next steps (zero cost)**: |
| 1. Generate Figures 2a, 2b, 5, 9 from existing data |
| 2. Compile all existing eval results into master table |
| 3. Generate Figures 3, 4, 6 from compiled results |
| |
| **Low cost (eval only)**: |
| 4. Run OCR+InfoVQA on remaining 4B checkpoints (running now) |
| 5. Run full eval suite on 8B SD-RPN (after training finishes) |
| 6. Run 8B Vanilla baseline eval (running now) |
| |
| **Medium cost (new training)**: |
| 7. Train 8B Confluent + 8B Confluent+IWA |
| 8. Generate attention map visualizations |
| |
