| # Dataset-level difficulty |
|
|
| How hard each split is for each model, once the split's own guessing floor is |
| taken out, and whether the models find the same items hard. |
|
|
| ``` |
| dataset_difficulty.ipynb the analysis, 8 figures, executed with outputs |
| build_notebook.py regenerates the notebook from source (nbformat) |
| analyze_dataset_level.py the same numbers as CSVs, no notebook needed |
| figures/*.png every figure, also written by the notebook |
| ``` |
|
|
| Run either entry point from the experiment root: |
|
|
| ```bash |
| python analysis/dataset_level_analysis/analyze_dataset_level.py # CSVs + terminal tables |
| /home/disheng/miniconda3/envs/Spatial_VLM/bin/python \ |
| analysis/dataset_level_analysis/build_notebook.py # rebuild the .ipynb |
| ``` |
|
|
| ## Why not just compare accuracy |
|
|
| The four splits do not share a scale. VSR is a two-way true/false question, so |
| guessing scores 0.500; the three Youtube levels are four-way, so guessing scores |
| 0.250. Level 2's three distractors are drawn uniformly from the 23 wrong |
| orderings, so nothing in the option text helps and its floor is plain chance — |
| measured on the split, the strongest text-only heuristic scores 0.252. |
|
|
| Everything is therefore reported as **headroom captured**, |
| `(acc - floor) / (1 - floor)`: the share of the distance from guessing to |
| perfect that the model actually covers. It is negative when a model does worse |
| than free. |
|
|
| | split | n | choices | guessing | floor used | |
| |---|---|---|---|---| |
| | VSR | 2195 | 2 | 0.500 | 0.500 | |
| | YT level 1 | 821 | 4 | 0.250 | 0.250 | |
| | YT level 2 | 764 | 4 | 0.250 | 0.250 | |
| | YT level 3 | 836 | 4 | 0.250 | 0.250 | |
|
|
| ## Results |
|
|
| Accuracy, mean over the six baselines: |
|
|
| | model | VSR | YT L1 | YT L2 | YT L3 | |
| |---|---|---|---|---| |
| | Qwen2.5-VL-3B | 0.759 | 0.405 | 0.374 | 0.255 | |
| | Qwen2.5-VL-7B | 0.805 | 0.452 | 0.392 | 0.278 | |
| | Qwen3.5-4B | 0.813 | 0.530 | 0.386 | 0.296 | |
| | Qwen3.5-4B-Base | 0.827 | 0.557 | 0.448 | 0.279 | |
|
|
| Headroom captured: |
|
|
| | model | VSR | YT L1 | YT L2 | YT L3 | |
| |---|---|---|---|---| |
| | Qwen2.5-VL-3B | 0.519 | 0.206 | 0.166 | 0.006 | |
| | Qwen2.5-VL-7B | 0.610 | 0.269 | 0.189 | 0.037 | |
| | Qwen3.5-4B | 0.627 | 0.373 | 0.181 | 0.061 | |
| | Qwen3.5-4B-Base | 0.655 | 0.409 | 0.264 | 0.038 | |
|
|
| 1. **All four models now agree on the ordering**: VSR > L1 > L2 > L3, easiest to |
| hardest, with Spearman 1.00 for all six model pairs. The earlier split — where |
| Qwen2.5-VL found L3 hardest and Qwen3.5 found L2 hardest — was a property of |
| the level-3 v1 annotations, and did not survive their rebuild (see the level-3 |
| caveat below). |
| 2. **Only VSR is comfortably solved** (0.52-0.66 of headroom). It is also the |
| easiest question type — a binary relation often inferable from object |
| identity without metric depth. |
| 3. **Level 3 is now at the floor for everyone**: 0.006-0.061 of headroom, a |
| 0.054 spread that is the narrowest of the four splits. With each path given |
| its own start point the split no longer separates the models; it only shows |
| that none of them can trace depth monotonicity along a path. |
| 4. **Level 1 is the sharpest discriminator**, 0.206 to 0.409, a 0.203 spread. |
| It is the only Youtube level where model strength still shows through. |
| 5. **Level 2 sits at 0.17-0.26 for everyone**, a narrow band that tracks neither |
| size nor family. Nothing in its option text helps, so this is what four-way |
| depth ranking actually costs these models: clearly above guessing, nowhere |
| near solved. |
| 6. **The base model wins on three splits of four.** Qwen3.5-4B-Base (likelihood |
| scoring) beats Qwen3.5-4B (generation) on VSR, L1 and L2, but not on L3 |
| (0.038 vs 0.061) — where both are close enough to the floor that the ordering |
| carries little weight. 4B-Instruct is also the only model producing |
| unparsable answers (up to 0.7% on L1), which are scored wrong. |
| 7. **The stereo intervention barely moves the aggregate.** Accuracy ranges over |
| the six baselines are 0.012-0.036 with no consistent sign. Whether the same |
| *items* flip is a separate question — see `../intervention_analysis`. |
| 8. **Agreement on item difficulty is highest on L1, lowest on L3.** Excess mass |
| at "0 or 4 models correct" over the independent null: +0.358 (L1), +0.267 |
| (VSR), +0.234 (L2), +0.160 (L3). On L3 the failures are close to independent |
| noise, which is what near-chance accuracy looks like. |
|
|
| ## Caveat on the level-2 numbers |
|
|
| Level 2 was rebuilt three times before this run, because two option |
| constructions leaked and one of them also leaked through the letters: |
|
|
| | construction | strongest text-only heuristic | |
| |---|---| |
| | distractors at Kendall distance 1 / 2 / >=3 from the gold | 0.388 (pick the medoid) | |
| | all three at the same distance | 0.750 | |
| | prefix ladder, distractors assigned to letters in fixed order | 0.759 (run the ladder, avoid D) | |
| | prefix ladder, letters shuffled | 0.500 (run the ladder) | |
| | **random distractors (current)** | **0.252** | |
|
|
| Under the leaking constructions these same models scored 0.40-0.47 on level 2; |
| with random distractors they score 0.37-0.45 against a real 0.25 floor. The |
| 0.02-0.10 drop is the part of the old score that came from the option text |
| rather than the picture. Any level-2 result from before `rerun_l2_v5.log` is |
| contaminated and has been deleted. |
|
|
| ## Caveat on the level-3 numbers |
|
|
| Level 3 was rebuilt on 2026-08-31 (`rerun_l3_v2.log`), and unlike level 2 the |
| change was to the task, not to a leak: |
|
|
| | version | n | paths | these models scored | |
| |---|---|---|---| |
| | v1 | 795 | all four start at a shared reference point R | 0.27 - 0.50 | |
| | **v2 (current)** | **836** | **each path has its own start point** | **0.26 - 0.30** | |
|
|
| Removing the shared start removed whatever local cue the models were reading off |
| it, and every model fell to the floor. The v1 runs are kept under |
| `result/_archive_l3_v1/`; they are not comparable to anything current. |
|
|
| The v2 annotations check out — all 836 gold paths are monotone (`worst_step` >= |
| 0), every distractor violates monotonicity (worst is -0.0007), and the median |
| margin between the gold and the strongest distractor is 0.069 — so the collapse |
| is difficulty, not a broken split. A little signal survives: accuracy over |
| quartiles of that margin rises by +0.088 (7B), +0.083 (3.5-4B) and +0.044 |
| (Base), reaching 0.30-0.345 on the most separable quarter. Only 2.5-VL-3B is |
| flat (-0.005), i.e. reading nothing at all. Both subtypes are hard, with |
| `near_to_far` (0.280-0.334) consistently easier than `far_to_near` |
| (0.233-0.264). |
|
|
