analysis/intervention_analysis/readme.md

Baseline-intervention analysis

The premise being tested, from the top-level readme.md: if the visual tower modelled the visual world perfectly, a small stereo-baseline intervention should not change the VLM's output. Every model answers the same question about the same scene at six baselines (0.00 = the untouched original, up to 0.25), so the answers line up per sample and any change across baselines is the intervention's effect — not sampling noise, since decoding is greedy throughout.

baseline_intervention.ipynb is the visualisation layer. This file explains what the numbers mean, how to regenerate them, and what not to read into them.

Contents

File	What it is
baseline_intervention.ipynb	The notebook. Run top to bottom; no arguments.
readme.md	This file.
analysis_summary.json	All 16 runs (4 models × 4 splits): accuracy, flip rate, stability, pairwise overlap. Generated.
<model>__<task>_failure_overlap.csv	The 15 baseline pairs for one run, failure and success overlap. Generated.
<model>__<task>_per_sample.csv	Per-sample answer + correctness at every baseline, plus its stability class. Generated.
notebook_summary.csv	The notebook's final one-row-per-run table. Written by the last cell.

Only the notebook and this readme are tracked; everything else is regenerated by

python analysis/analyze_baseline_overlap.py --result_dir result --dump_per_sample
# or
./sh/run_analysis.sh

Re-run it after any inference re-run — the CSVs carry no provenance stamp, so a stale set looks exactly like a fresh one.

What each measure means

accuracy per baseline — the headline, but the least informative here: a model can hold accuracy flat while answering differently on half the samples, because wins and losses cancel.

flip rate — the fraction of samples whose answer changes relative to baseline 0.00, right or wrong. This is the direct test of the premise. Read its shape, not its level:

• rising with baseline ⇒ a real intervention effect, and bigger interventions move more samples.
• high but flat from the first step ⇒ the model is near chance and the "flips" are its own indifference, not the intervention. Level 3 does this for every model (≈0.30 at b=0.05, unchanged through b=0.25).

stability — each sample is always_correct / always_wrong / unstable across the six baselines. unstable is the population the intervention actually reaches; a high rate paired with a flat flip curve means chance-level churn.

failure overlap — for each baseline pair, Jaccard over the sets of samples each one gets wrong. High ⇒ both baselines fail on the same items, so failure is a property of the item (shared difficulty). Low ⇒ each baseline breaks a different subset, so failure is driven by the intervention.

success overlap — the same Jaccard over the sets each one gets right. Not redundant: with 4-way answers the success sets are small and the failure sets large, so the two move independently, and a run can share its successes while its failures scatter. Report both.

answer agreement — the fraction of samples where two baselines give the identical answer. The pairwise generalisation of flip rate.

Reading the results

Chance is 0.500 on VSR (True/False) and 0.250 on the three Youtube levels (4-way). Raw accuracy is not comparable across splits without that, and it is not comparable across models within a split either, because all four models carry a heavy option-position bias in different directions. The notebook therefore also reports Cohen's κ, which nets out each model's own answer marginal: a model that always answers "B" scores κ = 0 regardless of how often "B" happens to be right.

Two things κ does not fix, both worth remembering before drawing conclusions:

• κ corrects for the model's bias, not for an item's effective option count. If a split ships a distractor that any model can reject without looking at the image, its true chance level is above 1/k and every κ on it is inflated. The level-2 v1 annotations had exactly this problem (a blind text heuristic scored 0.392 against a nominal 0.250); the current v2 annotations fixed it — the same heuristic now scores 0.250.
• κ says nothing about how a model got there. A model can post a respectable κ while its accuracy is completely unresponsive to the underlying depth differences. Check responsiveness directly before claiming a model "can do" a level.

Caveats attached to the current result set

1. Evaluation modes are not uniform. The two Qwen2.5-VL models use the split's default answer instruction; Qwen3.5-4B uses a stricter one (Respond with exactly one character…) because it otherwise narrates and gets truncated; Qwen3.5-4B-Base uses likelihood scoring over the answer options, since it has no chat template. Measured cost of generate-vs-likelihood on a matched subset is ≈0.03 accuracy — smaller than the cross-family gaps on L1 and L3, but not negligible on L2, where the whole spread is 0.074.
2. Level 2 is v2 data. Results predating 2026-08-29 12:00 used the v1 annotations and are not comparable. The v1 runs are kept under result/_archive_l2_v1/.
3. Level 3 sits at chance for both Qwen2.5-VL models (κ 0.035 / 0.068). Its overlap and flip numbers are dominated by chance churn for those two, so it does not support intervention claims about them.
4. VSR reuses an image across several statements, so its sample_id repeats (2195 rows over 1874 ids). The alignment key is (sample_id, question); keying on sample_id alone silently drops 321 rows.
5. Level 2 still has a first-element shortcut. Knowing only which point is nearest narrows the four orderings to ~1.35 on average (blind ceiling 0.743), so L2 partly re-measures level 1a rather than full ordering ability.