Papers
arxiv:2611.11166

DexForesight: Grounding Dexterous Policies with Implicit Spatial Foresight

Published on Nov 10
· Submitted by
ChenZhongxi
on Nov 11
Authors:
,
,
,
,
,
,
,
,

Abstract

Dexterous manipulation requires rich spatial foresight over both the surrounding scene and the robot hand. Existing world-action models often acquire such predictive knowledge through explicit future prediction. We introduce DexForesight, a framework that learns spatial foresight implicitly by injecting dynamics-aware world representations into the latent space of a dexterous policy. During training, DexForesight learns an action-conditioned spatial world model from offline scene depth, kinematic hand geometry, and demonstration trajectories. The world model predicts 3D hand-point motion and visual feature changes, capturing both articulated hand dynamics and scene changes under demonstrated actions, including the motion of fingers occluded from the cameras. We then freeze the world model and align policy queries with its predictive world queries, transferring the learned spatial dynamics into policy latents and using them to condition action generation. At deployment, only the policy is retained, operating on RGB observations, language, and robot state without depth input or explicit future prediction. On 11 DexJoCo simulation tasks, DexForesight improves the overall success rate of the base policy from 49.4% to 60.5%. On three real-robot dexterous manipulation tasks, DexForesight achieves 58.3% success, compared with 50.0% for the strongest baseline. Project page https://dexforesight.github.io/

Community

Paper submitter

Excited to share DexForesight, our latest work from the Sharpa Foundation Model Team!

Project Page: https://dexforesight.github.io/

Most existing dexterous manipulation models rely primarily on RGB observations. However, even in simple pick-and-place tasks, the robot's fingers—the parts that actually interact with objects—are often occluded by the object or the back of the hand, making them difficult to observe from both third-person and wrist-mounted cameras.

Even with depth observations, it remains challenging for the model to understand the hand's physical structure, geometry, and how it interacts with the environment.

DexForesight addresses this by injecting hand geometry into a world model, enabling it to explicitly imagine the future motion of the dexterous hand and its interactions with the environment.

More importantly, DexForesight transfers this capability to a standard VLA policy through learned World Queries.

This allows a policy that takes only images and language as input to develop hand-aware spatial imagination during high-level planning, without explicitly generating future observations at inference time!

Check out our project!

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.11166
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.11166 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.11166 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.11166 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.