Grasping the Future
📄 Paper (arXiv:2611.12879) · 🌐 Project page · 🛠️ Code: RustinS/grasping-the-future
Weights of Grasping the Future (GTF), a generative model of 3D hand–object motion from the paper Grasping the Future: Joint 3D Hand–Object Forecasting from Internet Videos. GTF predicts both MANO hands (wrist pose and finger articulation) and the 6-DoF pose of the object they manipulate. One model, trained with flow matching on masked spatiotemporal completion, covers seven tasks: forecasting, hands from object motion, object from hand motion, temporal infilling and three goal-conditioned variants. It was trained on eight motion-capture datasets and on trajectories that our extraction pipeline recovers from internet videos (Action100M).
Model
An interaction is a grid of tokens: one column per frame, rows for the two hands, the object and the two hand–object relations. Observed cells are given and the others start as noise; the transformer predicts the flow velocity of the noise cells, and ten Euler steps turn them into a sample. The observation mask chooses the task.
| Backbone | Transformer over the token grid, 12 blocks (9 factorized over time and across tokens, 3 over the whole grid), width 512, 8 heads, AdaLN-Zero conditioning on the flow time and the frame interval |
| Conditioning | Object shape (Uni3D embedding of the mesh), scene (LitePT over a depth point cloud with DINOv3 features), MANO hand shapes, scale |
| Input | Any observed part of a 5 s window (up to 150 frames) of both hands and the object, plus the clip's frames for the scene |
| Output | N samples of the full window: per frame and hand, MANO global orientation, 15-joint finger pose and translation; per frame, the object's rotation and translation |
| Sampling | Flow matching, 10 Euler steps per sample |
| Training data | DexYCB, HOT3D, TACO, OakInk2, HO-3D, H2O, HO-Cap, SHOW3D and 62,562 clips reconstructed in 3D from internet videos (Action100M) |
| Training | 40K steps, 16 GPUs × 32 windows (global batch 512) |
| Parameters | 116M |
Results (from the paper)
Comparison with released forecasters: final error after 4.5 s in mm (lower is better). A method that draws N samples is scored with the energy score of all of them, which equals its error when N = 1. GTF uses estimated depth here.
| Method | N | Left wrist | Right wrist | Object |
|---|---|---|---|---|
| USST | 1 | 196.3 | 182.3 | – |
| ForeHand4D | 20 | 122.8 | 133.6 | – |
| ObjectForesight | 20 | – | – | 130.4 |
| MotionForesight | 1 | – | – | 142.9 |
| GTF, one sample | 1 | 170.3 | 156.1 | 150.9 |
| GTF | 20 | 103.4 | 92.4 | 91.1 |
Retargeted to simulated robot hands with SPIDER, 73.8% of GTF's forecasts move the object along the predicted path (14.6% with kinematic retargeting).
Files
| File | Size | Description |
|---|---|---|
gtf_main.ckpt |
466 MB | The paper's main model in one file: weights, configuration, the normalization statistics it was trained with and the PCA basis of its DINOv3 scene features |
teaser.jpg, model.png |
The figures above |
Usage
Install the code (see its README; MANO must be downloaded separately from mano.is.tue.mpg.de), then, from the repository root:
hf download raivn/grasping-the-future gtf_main.ckpt --local-dir checkpoints
# 20 futures per 5 s window of your own clips (extracted from video by the pipeline, or in the format of docs/data.md)
python -m gtf.predict --segments my_segments --out predictions --task future_prediction
# Other tasks: hand_to_object, object_to_hand, temporal_infilling, goal_infilling, goal_hand_only, goal_object_only
# The paper's benchmark (Tables 1 and 2)
python -m gtf.eval.benchmark run --checkpoint checkpoints/gtf_main.ckpt --population mocap --out results/benchmark
Each window is written to predictions/<segment>/<task>_<first frame>.npz, in the camera frame of the clip. Posing one
sample as meshes:
import numpy as np
from scipy.spatial.transform import Rotation
from gtf.paths import MANO_DIR
from gtf.utils.canonical import mano_fk_verts
p = np.load("predictions/my_clip/future_prediction_0.npz") # N samples x T frames
k = 0 # first sample
hands = {side: mano_fk_verts(p[f"hand_{side}_orient"][k], p[f"hand_{side}_pose"][k], p[f"hand_{side}_betas"],
p[f"hand_{side}_cam_t"][k], side, MANO_DIR) # (T, 778, 3) MANO vertices, meters
for side in ("left", "right")}
R = Rotation.from_rotvec(p["obj_rotation"][k]).as_matrix() # (T, 3, 3)
obj = np.einsum("tij,vj->tvi", R, p["obj_canonical_verts"] * p["obj_scale"]) + p["obj_translation"][k][:, None]
present = p["present"] # (T, 3): left hand, right hand, object; absent entities are zero-filled
observed = p["observed"] # (T, 3): the frames given to the model (they hold the input)
License
Released for non-commercial research use. The weights derive from the motion-capture datasets DexYCB, HOT3D, TACO, OakInk2, HO-3D, H2O, HO-Cap and SHOW3D and from internet videos; use them under the terms of those datasets. Using the model requires MANO (its own license) and, for new data, DINOv3, MoGe-2 and Uni3D under their licenses.
Citation
@article{soraki2026grasping,
title={Grasping the Future: Joint 3D Hand-Object Forecasting From Internet Videos},
author={Soraki, Rustin and Wallingford, Matthew and Aby, Ryan and Chee, Brandon and Ren, Jason and Farhadi, Ali and Bharadhwaj, Homanga and Mottaghi, Roozbeh},
journal={arXiv preprint arXiv:2611.12879},
year={2026}
}
Built with (please also cite): MANO, LitePT, Uni3D, DINOv3 and MoGe-2 for the model; WiLoR, Molmo2, SAM 3, SAM 3D Objects, CoTracker and MegaSaM for the extraction pipeline; SPIDER for the robot retargeting.

