DexForesight: Grounding Dexterous Policies with Implicit Spatial Foresight
Abstract
Dexterous manipulation requires rich spatial foresight over both the surrounding scene and the robot hand. Existing world-action models often acquire such predictive knowledge through explicit future prediction. We introduce DexForesight, a framework that learns spatial foresight implicitly by injecting dynamics-aware world representations into the latent space of a dexterous policy. During training, DexForesight learns an action-conditioned spatial world model from offline scene depth, kinematic hand geometry, and demonstration trajectories. The world model predicts 3D hand-point motion and visual feature changes, capturing both articulated hand dynamics and scene changes under demonstrated actions, including the motion of fingers occluded from the cameras. We then freeze the world model and align policy queries with its predictive world queries, transferring the learned spatial dynamics into policy latents and using them to condition action generation. At deployment, only the policy is retained, operating on RGB observations, language, and robot state without depth input or explicit future prediction. On 11 DexJoCo simulation tasks, DexForesight improves the overall success rate of the base policy from 49.4% to 60.5%. On three real-robot dexterous manipulation tasks, DexForesight achieves 58.3% success, compared with 50.0% for the strongest baseline. Project page https://dexforesight.github.io/
Community
Excited to share DexForesight, our latest work from the Sharpa Foundation Model Team!
Project Page: https://dexforesight.github.io/
Most existing dexterous manipulation models rely primarily on RGB observations. However, even in simple pick-and-place tasks, the robot's fingers—the parts that actually interact with objects—are often occluded by the object or the back of the hand, making them difficult to observe from both third-person and wrist-mounted cameras.
Even with depth observations, it remains challenging for the model to understand the hand's physical structure, geometry, and how it interacts with the environment.
DexForesight addresses this by injecting hand geometry into a world model, enabling it to explicitly imagine the future motion of the dexterous hand and its interactions with the environment.
More importantly, DexForesight transfers this capability to a standard VLA policy through learned World Queries.
This allows a policy that takes only images and language as input to develop hand-aware spatial imagination during high-level planning, without explicitly generating future observations at inference time!
Check out our project!
Get this paper in your agent:
hf papers read 2611.11166 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper