analysis/dataset_level_analysis/readme.md

Dataset-level difficulty

How hard each split is for each model, once the split's own guessing floor is taken out, and whether the models find the same items hard.

dataset_difficulty.ipynb    the analysis, 8 figures, executed with outputs
build_notebook.py           regenerates the notebook from source (nbformat)
analyze_dataset_level.py    the same numbers as CSVs, no notebook needed
figures/*.png               every figure, also written by the notebook

Run either entry point from the experiment root:

python analysis/dataset_level_analysis/analyze_dataset_level.py     # CSVs + terminal tables
/home/disheng/miniconda3/envs/Spatial_VLM/bin/python \
    analysis/dataset_level_analysis/build_notebook.py               # rebuild the .ipynb

Why not just compare accuracy

The four splits do not share a scale. VSR is a two-way true/false question, so guessing scores 0.500; the three Youtube levels are four-way, so guessing scores 0.250. Level 2's three distractors are drawn uniformly from the 23 wrong orderings, so nothing in the option text helps and its floor is plain chance — measured on the split, the strongest text-only heuristic scores 0.252.

Everything is therefore reported as headroom captured, (acc - floor) / (1 - floor): the share of the distance from guessing to perfect that the model actually covers. It is negative when a model does worse than free.

split	n	choices	guessing	floor used
VSR	2195	2	0.500	0.500
YT level 1	821	4	0.250	0.250
YT level 2	764	4	0.250	0.250
YT level 3	836	4	0.250	0.250

Results

Accuracy, mean over the six baselines:

model	VSR	YT L1	YT L2	YT L3
Qwen2.5-VL-3B	0.759	0.405	0.374	0.255
Qwen2.5-VL-7B	0.805	0.452	0.392	0.278
Qwen3.5-4B	0.813	0.530	0.386	0.296
Qwen3.5-4B-Base	0.827	0.557	0.448	0.279

Headroom captured:

model	VSR	YT L1	YT L2	YT L3
Qwen2.5-VL-3B	0.519	0.206	0.166	0.006
Qwen2.5-VL-7B	0.610	0.269	0.189	0.037
Qwen3.5-4B	0.627	0.373	0.181	0.061
Qwen3.5-4B-Base	0.655	0.409	0.264	0.038

1. All four models now agree on the ordering: VSR > L1 > L2 > L3, easiest to hardest, with Spearman 1.00 for all six model pairs. The earlier split — where Qwen2.5-VL found L3 hardest and Qwen3.5 found L2 hardest — was a property of the level-3 v1 annotations, and did not survive their rebuild (see the level-3 caveat below).
2. Only VSR is comfortably solved (0.52-0.66 of headroom). It is also the easiest question type — a binary relation often inferable from object identity without metric depth.
3. Level 3 is now at the floor for everyone: 0.006-0.061 of headroom, a 0.054 spread that is the narrowest of the four splits. With each path given its own start point the split no longer separates the models; it only shows that none of them can trace depth monotonicity along a path.
4. Level 1 is the sharpest discriminator, 0.206 to 0.409, a 0.203 spread. It is the only Youtube level where model strength still shows through.
5. Level 2 sits at 0.17-0.26 for everyone, a narrow band that tracks neither size nor family. Nothing in its option text helps, so this is what four-way depth ranking actually costs these models: clearly above guessing, nowhere near solved.
6. The base model wins on three splits of four. Qwen3.5-4B-Base (likelihood scoring) beats Qwen3.5-4B (generation) on VSR, L1 and L2, but not on L3 (0.038 vs 0.061) — where both are close enough to the floor that the ordering carries little weight. 4B-Instruct is also the only model producing unparsable answers (up to 0.7% on L1), which are scored wrong.
7. The stereo intervention barely moves the aggregate. Accuracy ranges over the six baselines are 0.012-0.036 with no consistent sign. Whether the same items flip is a separate question — see ../intervention_analysis.
8. Agreement on item difficulty is highest on L1, lowest on L3. Excess mass at "0 or 4 models correct" over the independent null: +0.358 (L1), +0.267 (VSR), +0.234 (L2), +0.160 (L3). On L3 the failures are close to independent noise, which is what near-chance accuracy looks like.

Caveat on the level-2 numbers

Level 2 was rebuilt three times before this run, because two option constructions leaked and one of them also leaked through the letters:

construction	strongest text-only heuristic
distractors at Kendall distance 1 / 2 / >=3 from the gold	0.388 (pick the medoid)
all three at the same distance	0.750
prefix ladder, distractors assigned to letters in fixed order	0.759 (run the ladder, avoid D)
prefix ladder, letters shuffled	0.500 (run the ladder)
random distractors (current)	0.252

Under the leaking constructions these same models scored 0.40-0.47 on level 2; with random distractors they score 0.37-0.45 against a real 0.25 floor. The 0.02-0.10 drop is the part of the old score that came from the option text rather than the picture. Any level-2 result from before rerun_l2_v5.log is contaminated and has been deleted.

Caveat on the level-3 numbers

Level 3 was rebuilt on 2026-08-31 (rerun_l3_v2.log), and unlike level 2 the change was to the task, not to a leak:

version	n	paths	these models scored
v1	795	all four start at a shared reference point R	0.27 - 0.50
v2 (current)	836	each path has its own start point	0.26 - 0.30

Removing the shared start removed whatever local cue the models were reading off it, and every model fell to the floor. The v1 runs are kept under result/_archive_l3_v1/; they are not comparable to anything current.

The v2 annotations check out — all 836 gold paths are monotone (worst_step >= 0), every distractor violates monotonicity (worst is -0.0007), and the median margin between the gold and the strongest distractor is 0.069 — so the collapse is difficulty, not a broken split. A little signal survives: accuracy over quartiles of that margin rises by +0.088 (7B), +0.083 (3.5-4B) and +0.044 (Base), reaching 0.30-0.345 on the most separable quarter. Only 2.5-VL-3B is flat (-0.005), i.e. reading nothing at all. Both subtypes are hard, with near_to_far (0.280-0.334) consistently easier than far_to_near (0.233-0.264).