Grasping the Future

📄 Paper (arXiv:2611.12879) · 🌐 Project page · 🛠️ Code: RustinS/grasping-the-future

Weights of Grasping the Future (GTF), a generative model of 3D hand–object motion from the paper Grasping the Future: Joint 3D Hand–Object Forecasting from Internet Videos. GTF predicts both MANO hands (wrist pose and finger articulation) and the 6-DoF pose of the object they manipulate. One model, trained with flow matching on masked spatiotemporal completion, covers seven tasks: forecasting, hands from object motion, object from hand motion, temporal infilling and three goal-conditioned variants. It was trained on eight motion-capture datasets and on trajectories that our extraction pipeline recovers from internet videos (Action100M).

One model for five kinds of hand–object prediction

Model

Model overview

An interaction is a grid of tokens: one column per frame, rows for the two hands, the object and the two hand–object relations. Observed cells are given and the others start as noise; the transformer predicts the flow velocity of the noise cells, and ten Euler steps turn them into a sample. The observation mask chooses the task.

Backbone Transformer over the token grid, 12 blocks (9 factorized over time and across tokens, 3 over the whole grid), width 512, 8 heads, AdaLN-Zero conditioning on the flow time and the frame interval
Conditioning Object shape (Uni3D embedding of the mesh), scene (LitePT over a depth point cloud with DINOv3 features), MANO hand shapes, scale
Input Any observed part of a 5 s window (up to 150 frames) of both hands and the object, plus the clip's frames for the scene
Output N samples of the full window: per frame and hand, MANO global orientation, 15-joint finger pose and translation; per frame, the object's rotation and translation
Sampling Flow matching, 10 Euler steps per sample
Training data DexYCB, HOT3D, TACO, OakInk2, HO-3D, H2O, HO-Cap, SHOW3D and 62,562 clips reconstructed in 3D from internet videos (Action100M)
Training 40K steps, 16 GPUs × 32 windows (global batch 512)
Parameters 116M

Results (from the paper)

Comparison with released forecasters: final error after 4.5 s in mm (lower is better). A method that draws N samples is scored with the energy score of all of them, which equals its error when N = 1. GTF uses estimated depth here.

Method N Left wrist Right wrist Object
USST 1 196.3 182.3 –
ForeHand4D 20 122.8 133.6 –
ObjectForesight 20 – – 130.4
MotionForesight 1 – – 142.9
GTF, one sample 1 170.3 156.1 150.9
GTF 20 103.4 92.4 91.1

Retargeted to simulated robot hands with SPIDER, 73.8% of GTF's forecasts move the object along the predicted path (14.6% with kinematic retargeting).

Files

File Size Description
gtf_main.ckpt 466 MB The paper's main model in one file: weights, configuration, the normalization statistics it was trained with and the PCA basis of its DINOv3 scene features
teaser.jpg, model.png The figures above

Usage

Install the code (see its README; MANO must be downloaded separately from mano.is.tue.mpg.de), then, from the repository root:

hf download raivn/grasping-the-future gtf_main.ckpt --local-dir checkpoints

# 20 futures per 5 s window of your own clips (extracted from video by the pipeline, or in the format of docs/data.md)
python -m gtf.predict --segments my_segments --out predictions --task future_prediction
# Other tasks: hand_to_object, object_to_hand, temporal_infilling, goal_infilling, goal_hand_only, goal_object_only

# The paper's benchmark (Tables 1 and 2)
python -m gtf.eval.benchmark run --checkpoint checkpoints/gtf_main.ckpt --population mocap --out results/benchmark

Each window is written to predictions/<segment>/<task>_<first frame>.npz, in the camera frame of the clip. Posing one sample as meshes:

import numpy as np
from scipy.spatial.transform import Rotation
from gtf.paths import MANO_DIR
from gtf.utils.canonical import mano_fk_verts

p = np.load("predictions/my_clip/future_prediction_0.npz")    # N samples x T frames
k = 0                                                          # first sample
hands = {side: mano_fk_verts(p[f"hand_{side}_orient"][k], p[f"hand_{side}_pose"][k], p[f"hand_{side}_betas"],
                             p[f"hand_{side}_cam_t"][k], side, MANO_DIR)    # (T, 778, 3) MANO vertices, meters
         for side in ("left", "right")}
R = Rotation.from_rotvec(p["obj_rotation"][k]).as_matrix()                  # (T, 3, 3)
obj = np.einsum("tij,vj->tvi", R, p["obj_canonical_verts"] * p["obj_scale"]) + p["obj_translation"][k][:, None]
present = p["present"]      # (T, 3): left hand, right hand, object; absent entities are zero-filled
observed = p["observed"]    # (T, 3): the frames given to the model (they hold the input)

License

Released for non-commercial research use. The weights derive from the motion-capture datasets DexYCB, HOT3D, TACO, OakInk2, HO-3D, H2O, HO-Cap and SHOW3D and from internet videos; use them under the terms of those datasets. Using the model requires MANO (its own license) and, for new data, DINOv3, MoGe-2 and Uni3D under their licenses.

Citation

@article{soraki2026grasping,
  title={Grasping the Future: Joint 3D Hand-Object Forecasting From Internet Videos},
  author={Soraki, Rustin and Wallingford, Matthew and Aby, Ryan and Chee, Brandon and Ren, Jason and Farhadi, Ali and Bharadhwaj, Homanga and Mottaghi, Roozbeh},
  journal={arXiv preprint arXiv:2611.12879},
  year={2026}
}

Built with (please also cite): MANO, LitePT, Uni3D, DINOv3 and MoGe-2 for the model; WiLoR, Molmo2, SAM 3, SAM 3D Objects, CoTracker and MegaSaM for the extraction pipeline; SPIDER for the robot retargeting.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Paper for raivn/grasping-the-future