TuringSpace: Grounding Spatial Intelligence in Action with Interactive CAPTCHAs
Abstract
Humans act in the visual world by inferring the 3D layout behind a 2D view, predicting how an action changes it, and correcting themselves after seeing the result. Can multimodal large language models (MLLMs) do the same? CAPTCHAs offer a natural test, since humans solve them through this very loop, clicking and dragging on the screen and adjusting after each result. Yet existing CAPTCHA benchmarks are largely solved and rarely spatial, while spatial benchmarks rarely require action. We therefore present TuringSpace, a synthetic web environment of 125 interactive CAPTCHAs that span five spatial abilities and must be solved through clicks and drags on the screen. Every task is a program that generates new instances with a checker and an oracle, so the tasks also form a scalable training gym. Humans solve 94.1% of the tasks, whereas the best model, GPT-6 Astra, still fails 25.9% of them, and the best open-weight model, Kimi K3, fails 68.0%. Our further analysis shows that most failures come from misreading the screenshot, and that model reasoning rarely corrects a wrong belief but hallucinates evidence to defend it. To test whether such spatial abilities can be learned, we fine-tune Qwen3.5-9B on this gym. Its success rate rises from 1.6% to 33.1%, above Kimi K3, and it improves on all seven out-of-distribution benchmarks, including a 9.9-point gain on the real-world indoor videos of VSI-Bench. These results show that grounding spatial intelligence in screen actions exposes what MLLMs lack and offers a scalable way to teach it, with gains that transfer to real-world scenes. We will release TuringSpace to support further study.
Community
MLLMs perform well on 2D perception but remain far behind humans in 3D spatial understanding, which makes space a natural basis for CAPTCHAs now that computer-use agents solve conventional ones. Yet existing spatial CAPTCHAs and benchmarks are mostly multiple choice, and where actions exist they are text commands or labeled DOM elements. Spatial reasoning is a loop: read the 3D scene, act, and adjust after seeing the result. TuringSpace tests this loop with 125 interactive spatial CAPTCHAs across five abilities, all solved by clicking and dragging in a real browser. The best model, GPT-6 Astra, solves 74.1% versus 94.1% for humans, and most failures come from misreading the screen, even when it shows the feedback of the model's own action. Each task is a program that generates fresh instances and replays its oracle solution into a verified trajectory, so the same tasks form a training gym. Fine-tuning Qwen3.5-9B on 55K such trajectories lifts it from 1.6% to 33.1% and transfers to other CAPTCHA, spatial understanding and robotic pointing benchmarks, including +9.9 points on real-world videos from VSI-Bench.
Get this paper in your agent:
hf papers read 2611.25387 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper