analysis/intervention_analysis/readme.md

# Baseline-intervention analysis

The premise being tested, from the top-level `readme.md`: *if the visual tower
modelled the visual world perfectly, a small stereo-baseline intervention should
not change the VLM's output.* Every model answers the **same question about the
same scene** at six baselines (0.00 = the untouched original, up to 0.25), so the
answers line up per sample and any change across baselines is the intervention's
effect — not sampling noise, since decoding is greedy throughout.

`baseline_intervention.ipynb` is the visualisation layer. This file explains what
the numbers mean, how to regenerate them, and what not to read into them.

## Contents

| File | What it is |
|---|---|
| `baseline_intervention.ipynb` | The notebook. Run top to bottom; no arguments. |
| `readme.md` | This file. |
| `analysis_summary.json` | All 16 runs (4 models × 4 splits): accuracy, flip rate, stability, pairwise overlap. Generated. |
| `<model>__<task>_failure_overlap.csv` | The 15 baseline pairs for one run, failure **and** success overlap. Generated. |
| `<model>__<task>_per_sample.csv` | Per-sample answer + correctness at every baseline, plus its stability class. Generated. |
| `notebook_summary.csv` | The notebook's final one-row-per-run table. Written by the last cell. |

Only the notebook and this readme are tracked; everything else is regenerated by

```bash
python analysis/analyze_baseline_overlap.py --result_dir result --dump_per_sample
# or
./sh/run_analysis.sh
```

Re-run it after **any** inference re-run — the CSVs carry no provenance stamp, so
a stale set looks exactly like a fresh one.

## What each measure means

**accuracy per baseline** — the headline, but the least informative here: a model
can hold accuracy flat while answering differently on half the samples, because
wins and losses cancel.

**flip rate** — the fraction of samples whose *answer* changes relative to
baseline 0.00, right or wrong. This is the direct test of the premise. Read its
**shape**, not its level:

- rising with baseline ⇒ a real intervention effect, and bigger interventions
  move more samples.
- high but flat from the first step ⇒ the model is near chance and the "flips"
  are its own indifference, not the intervention. Level 3 does this for every
  model (≈0.30 at b=0.05, unchanged through b=0.25).

**stability** — each sample is `always_correct` / `always_wrong` / `unstable`
across the six baselines. `unstable` is the population the intervention actually
reaches; a high rate paired with a flat flip curve means chance-level churn.

**failure overlap** — for each baseline pair, Jaccard over the sets of samples
each one gets wrong. High ⇒ both baselines fail on the *same* items, so failure
is a property of the item (shared difficulty). Low ⇒ each baseline breaks a
different subset, so failure is driven by the intervention.

**success overlap** — the same Jaccard over the sets each one gets *right*. Not
redundant: with 4-way answers the success sets are small and the failure sets
large, so the two move independently, and a run can share its successes while its
failures scatter. Report both.

**answer agreement** — the fraction of samples where two baselines give the
identical answer. The pairwise generalisation of flip rate.

## Reading the results

Chance is **0.500** on VSR (True/False) and **0.250** on the three Youtube levels
(4-way). Raw accuracy is not comparable across splits without that, and it is not
comparable *across models within a split* either, because all four models carry a
heavy option-position bias in different directions. The notebook therefore also
reports **Cohen's κ**, which nets out each model's own answer marginal: a model
that always answers "B" scores κ = 0 regardless of how often "B" happens to be
right.

Two things κ does **not** fix, both worth remembering before drawing conclusions:

- κ corrects for the *model's* bias, not for an item's **effective** option
  count. If a split ships a distractor that any model can reject without looking
  at the image, its true chance level is above 1/k and every κ on it is inflated.
  The level-2 v1 annotations had exactly this problem (a blind text heuristic
  scored 0.392 against a nominal 0.250); the current v2 annotations fixed it —
  the same heuristic now scores 0.250.
- κ says nothing about *how* a model got there. A model can post a respectable κ
  while its accuracy is completely unresponsive to the underlying depth
  differences. Check responsiveness directly before claiming a model "can do" a
  level.

## Caveats attached to the current result set

1. **Evaluation modes are not uniform.** The two Qwen2.5-VL models use the split's
   default answer instruction; Qwen3.5-4B uses a stricter one (`Respond with
   exactly one character…`) because it otherwise narrates and gets truncated;
   Qwen3.5-4B-Base uses likelihood scoring over the answer options, since it has
   no chat template. Measured cost of generate-vs-likelihood on a matched subset
   is ≈0.03 accuracy — smaller than the cross-family gaps on L1 and L3, but not
   negligible on L2, where the whole spread is 0.074.
2. **Level 2 is v2 data.** Results predating 2026-08-29 12:00 used the v1
   annotations and are not comparable. The v1 runs are kept under
   `result/_archive_l2_v1/`.
3. **Level 3 sits at chance for both Qwen2.5-VL models** (κ 0.035 / 0.068). Its
   overlap and flip numbers are dominated by chance churn for those two, so it
   does not support intervention claims about them.
4. **VSR reuses an image across several statements**, so its `sample_id` repeats
   (2195 rows over 1874 ids). The alignment key is `(sample_id, question)`; keying
   on `sample_id` alone silently drops 321 rows.
5. **Level 2 still has a first-element shortcut.** Knowing only which point is
   nearest narrows the four orderings to ~1.35 on average (blind ceiling 0.743),
   so L2 partly re-measures level 1a rather than full ordering ability.