| --- |
| license: other |
| license_name: fair-noncommercial-research-license-v1 |
| license_link: https://huggingface.co/Rice-RobotPI-Lab/robotok-public/blob/main/LICENSE-Action100M |
| library_name: pytorch |
| tags: |
| - robotics |
| - video-retrieval |
| - hand-pose |
| - trajectory |
| - dynamic-time-warping |
| - computer-vision |
| - action100m |
| --- |
| |
| # RoboTok — A Scalable Data Engine for Internet Demonstration Video Retrieval and Dexterous Manipulation Learning |
|
|
| 🌐 **Project website:** [rice-robotpi-lab.github.io/RoboTok](https://rice-robotpi-lab.github.io/RoboTok/) |
|
|
|  |
|
|
| Released checkpoints and evaluation keypoints for **RoboTok**, a model that |
| retrieves web video clips by 3D hand-motion similarity. Similarity is defined by |
| DTW over torso-relative 3D hand keypoints; the encoder is trained to reproduce |
| that DTW ranking in a fast embedding space. |
|
|
| Training and evaluation code is in the accompanying source release. |
|
|
| ## Files |
|
|
| | File | Size | Description | |
| | --- | --- | --- | |
| | `models/best_abs_retrieval_model.pt` | 3.8 MB | Retrieval encoder. Cross-attention head over `[T, 126]` hand-trajectory features (21 joints x 3 coords x 2 hands, `T_max = 42`): 1 learned query token, 256-d input projection, 1 cross-attention layer (4 heads, sinusoidal PE), 2-layer MLP to a 256-d embedding. DTW design `abs_21j_coords`. | |
| | `models/best_abs_retrieval_model.yaml` | 2 KB | Minimal config to reload the encoder for inference. | |
| | `models/body_pose_est.pt` | 9.9 MB | Vector-neuron torso/body-frame estimator: 4-layer rotation-equivariant transformer mapping two-hand trajectories to a torso frame, with separate rotation and translation heads. | |
| | `eval_data/torso_relative_clip_keypoints.pt` | 6.5 GB | Torso-relative 3D hand keypoints per clip. Each entry has `video_number`, `node_number`, `node_uid`, `keypoints_per_frame` (`kpts_2d`, `kpts_3d`), and `infilled` / `depth_grounded` flags. | |
|
|
| ## Loading |
|
|
| ```python |
| import torch |
| |
| ckpt = torch.load("models/best_abs_retrieval_model.pt", map_location="cpu", weights_only=True) |
| ckpt["head_state_dict"] # encoder weights |
| ckpt["config"] # full training configuration |
| |
| vn = torch.load("models/body_pose_est.pt", map_location="cpu", weights_only=True) |
| vn["model"] # torso estimator weights |
| ``` |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{qian2026robotok, |
| title = {RoboTok: A Scalable Data Engine for Internet |
| Demonstration Video Retrieval and Dexterous Manipulation |
| Learning}, |
| author = {Qian, Howard and Chen, Yiting and Xie, Yunfei and |
| Ren, Kejia and Chanrungmaneekul, Podshara and Wang, Gaotian and |
| Wen, Bowen and Wei, Chen and Hang, Kaiyu}, |
| journal = {arXiv preprint arXiv:2609.03199}, |
| year = {2026} |
| } |
| ``` |
|
|
| ## License |
|
|
| **FAIR Noncommercial Research License v1** (see |
| [LICENSE-Action100M](LICENSE-Action100M)). Noncommercial research only. |
|
|
| The released checkpoints (`models/*.pt`) and evaluation keypoints |
| (`eval_data/torso_relative_clip_keypoints.pt`) are derivative works of |
| Action100M (Meta FAIR) clips and are governed by that license. It covers |
| trained model weights as "Research Materials", and restricts both those |
| materials and any outputs or results obtained from them to noncommercial |
| research use. If you publish results obtained using these materials, the |
| license requires you to acknowledge that use. |
|
|
| MIT ([LICENSE](LICENSE)) covers only `models/best_abs_retrieval_model.yaml` |
| and the accompanying source release. |
|
|
| The MANO / SMPL-H body models required by parts of the pipeline are **not** |
| included and remain under their own MPI-IS license terms — register at |
| https://mano.is.tue.mpg.de to obtain them. |
|
|
