Papers
arxiv:2611.20385

DominoBench: Evaluating Multi-Step Physical Reasoning from Still Images

Published on Nov 16
· Submitted by
Emilio Ferrara
on Nov 17
Authors:

Abstract

A new benchmark, DominoBench, shows that current vision-language and multimodal models cannot reliably predict physical outcomes from still images unless the critical calculation and necessary quantities are provided.

Can a vision-language model look at a picture of a chain reaction and tell what will happen? Recent multimodal models describe scenes fluently and solve textbook physics problems, which suggests they should also predict how a scene of ramps, ledges, pendulums, seesaws and dominoes plays out once a ball is pushed. We tested this claim on 30 models from 10 families and reached a clear conclusion: unaided, no model predicts the outcome reliably. We built DominoBench, a benchmark of still images of 2D rigid-body scenes whose outcome a physics simulator computes. It holds 854 items in 427 minimal pairs, in which a change of one to three grid steps in one quantity decides whether one person survives, and every item comes with a four-rung hint ladder ending in the exact calculation that decides the outcome. Unaided, frontier models from Anthropic, Google and OpenAI are right about the person whose fate flips in at most 72% of items and answer both twins of a pair correctly in at most 34%. Pointing at the deciding step helps little, and exact geometry given as text barely helps, yet once the deciding calculation and the quantities it needs, including speeds that a still image cannot show, are written out, the same models reach 82-92% on the deciding person, while small open-weight models stay near chance. Physical prediction from a still image therefore fails at the quantitative step: the strongest models use the deciding quantity when they are given it, but do not produce it on their own. For models that plan or act in the physical world, a fluent account of a scene is thus no evidence of a correct prediction: such models must find and compute the deciding quantity unprompted, and DominoBench tests whether they do.

Community

Can a vision-language model look at a still image of a chain reaction and tell who gets hit? DominoBench tests exactly that: 854 simulator-labelled scenes in minimal pairs, where one quantity moves one grid step and one person's fate flips.

What I found across 30 models:

  • Unaided, no model is reliable. The best are right about the person whose fate flips in at most 75% of items, and get both twins of a pair right in at most 40%.
  • Pointing at the deciding step helps little. Writing out the calculation helps a lot: the same models reach 80–92%.
  • Two families behave very differently. The GPT-6 models are the most accurate unaided but barely gain from the calculation, while GPT-5.6-Sol starts low and ends first.

Everything is open: images, simulator trajectories, the hint ladder, the generator and the evaluation code.
Dataset: https://huggingface.co/datasets/emilioferrara/DominoBench

I'd love to hear whether your models do better, and which mechanism trips them up.

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.20385
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.20385 in a model README.md to link it from this page.

Datasets citing this paper 1

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.20385 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.