analysis/dataset_level_analysis/readme.md

# Dataset-level difficulty

How hard each split is for each model, once the split's own guessing floor is
taken out, and whether the models find the same items hard.

```
dataset_difficulty.ipynb    the analysis, 8 figures, executed with outputs
build_notebook.py           regenerates the notebook from source (nbformat)
analyze_dataset_level.py    the same numbers as CSVs, no notebook needed
figures/*.png               every figure, also written by the notebook
```

Run either entry point from the experiment root:

```bash
python analysis/dataset_level_analysis/analyze_dataset_level.py     # CSVs + terminal tables
/home/disheng/miniconda3/envs/Spatial_VLM/bin/python \
    analysis/dataset_level_analysis/build_notebook.py               # rebuild the .ipynb
```

## Why not just compare accuracy

The four splits do not share a scale. VSR is a two-way true/false question, so
guessing scores 0.500; the three Youtube levels are four-way, so guessing scores
0.250. Level 2's three distractors are drawn uniformly from the 23 wrong
orderings, so nothing in the option text helps and its floor is plain chance —
measured on the split, the strongest text-only heuristic scores 0.252.

Everything is therefore reported as **headroom captured**,
`(acc - floor) / (1 - floor)`: the share of the distance from guessing to
perfect that the model actually covers. It is negative when a model does worse
than free.

| split | n | choices | guessing | floor used |
|---|---|---|---|---|
| VSR | 2195 | 2 | 0.500 | 0.500 |
| YT level 1 | 821 | 4 | 0.250 | 0.250 |
| YT level 2 | 764 | 4 | 0.250 | 0.250 |
| YT level 3 | 836 | 4 | 0.250 | 0.250 |

## Results

Accuracy, mean over the six baselines:

| model | VSR | YT L1 | YT L2 | YT L3 |
|---|---|---|---|---|
| Qwen2.5-VL-3B | 0.759 | 0.405 | 0.374 | 0.255 |
| Qwen2.5-VL-7B | 0.805 | 0.452 | 0.392 | 0.278 |
| Qwen3.5-4B | 0.813 | 0.530 | 0.386 | 0.296 |
| Qwen3.5-4B-Base | 0.827 | 0.557 | 0.448 | 0.279 |

Headroom captured:

| model | VSR | YT L1 | YT L2 | YT L3 |
|---|---|---|---|---|
| Qwen2.5-VL-3B | 0.519 | 0.206 | 0.166 | 0.006 |
| Qwen2.5-VL-7B | 0.610 | 0.269 | 0.189 | 0.037 |
| Qwen3.5-4B | 0.627 | 0.373 | 0.181 | 0.061 |
| Qwen3.5-4B-Base | 0.655 | 0.409 | 0.264 | 0.038 |

1. **All four models now agree on the ordering**: VSR > L1 > L2 > L3, easiest to
   hardest, with Spearman 1.00 for all six model pairs. The earlier split — where
   Qwen2.5-VL found L3 hardest and Qwen3.5 found L2 hardest — was a property of
   the level-3 v1 annotations, and did not survive their rebuild (see the level-3
   caveat below).
2. **Only VSR is comfortably solved** (0.52-0.66 of headroom). It is also the
   easiest question type — a binary relation often inferable from object
   identity without metric depth.
3. **Level 3 is now at the floor for everyone**: 0.006-0.061 of headroom, a
   0.054 spread that is the narrowest of the four splits. With each path given
   its own start point the split no longer separates the models; it only shows
   that none of them can trace depth monotonicity along a path.
4. **Level 1 is the sharpest discriminator**, 0.206 to 0.409, a 0.203 spread.
   It is the only Youtube level where model strength still shows through.
5. **Level 2 sits at 0.17-0.26 for everyone**, a narrow band that tracks neither
   size nor family. Nothing in its option text helps, so this is what four-way
   depth ranking actually costs these models: clearly above guessing, nowhere
   near solved.
6. **The base model wins on three splits of four.** Qwen3.5-4B-Base (likelihood
   scoring) beats Qwen3.5-4B (generation) on VSR, L1 and L2, but not on L3
   (0.038 vs 0.061) — where both are close enough to the floor that the ordering
   carries little weight. 4B-Instruct is also the only model producing
   unparsable answers (up to 0.7% on L1), which are scored wrong.
7. **The stereo intervention barely moves the aggregate.** Accuracy ranges over
   the six baselines are 0.012-0.036 with no consistent sign. Whether the same
   *items* flip is a separate question — see `../intervention_analysis`.
8. **Agreement on item difficulty is highest on L1, lowest on L3.** Excess mass
   at "0 or 4 models correct" over the independent null: +0.358 (L1), +0.267
   (VSR), +0.234 (L2), +0.160 (L3). On L3 the failures are close to independent
   noise, which is what near-chance accuracy looks like.

## Caveat on the level-2 numbers

Level 2 was rebuilt three times before this run, because two option
constructions leaked and one of them also leaked through the letters:

| construction | strongest text-only heuristic |
|---|---|
| distractors at Kendall distance 1 / 2 / >=3 from the gold | 0.388 (pick the medoid) |
| all three at the same distance | 0.750 |
| prefix ladder, distractors assigned to letters in fixed order | 0.759 (run the ladder, avoid D) |
| prefix ladder, letters shuffled | 0.500 (run the ladder) |
| **random distractors (current)** | **0.252** |

Under the leaking constructions these same models scored 0.40-0.47 on level 2;
with random distractors they score 0.37-0.45 against a real 0.25 floor. The
0.02-0.10 drop is the part of the old score that came from the option text
rather than the picture. Any level-2 result from before `rerun_l2_v5.log` is
contaminated and has been deleted.

## Caveat on the level-3 numbers

Level 3 was rebuilt on 2026-08-31 (`rerun_l3_v2.log`), and unlike level 2 the
change was to the task, not to a leak:

| version | n | paths | these models scored |
|---|---|---|---|
| v1 | 795 | all four start at a shared reference point R | 0.27 - 0.50 |
| **v2 (current)** | **836** | **each path has its own start point** | **0.26 - 0.30** |

Removing the shared start removed whatever local cue the models were reading off
it, and every model fell to the floor. The v1 runs are kept under
`result/_archive_l3_v1/`; they are not comparable to anything current.

The v2 annotations check out — all 836 gold paths are monotone (`worst_step` >=
0), every distractor violates monotonicity (worst is -0.0007), and the median
margin between the gold and the strongest distractor is 0.069 — so the collapse
is difficulty, not a broken split. A little signal survives: accuracy over
quartiles of that margin rises by +0.088 (7B), +0.083 (3.5-4B) and +0.044
(Base), reaching 0.30-0.345 on the most separable quarter. Only 2.5-VL-3B is
flat (-0.005), i.e. reading nothing at all. Both subtypes are hard, with
`near_to_far` (0.280-0.334) consistently easier than `far_to_near`
(0.233-0.264).