Egocentric two-hand pose estimation pipeline
Offline AnyHand-WiLoR + YOLO-World inference for egocentric H.264 video stored in MCAP. Both hands are estimated from the left camera view, with CSRT tracking and per-hand detection recovery across numbered recordings.
This repository holds the source only. Model assets (AnyHand-WiLoR, YOLO-World, MANO) are not redistributed here; see "Assets" below.
Pipeline
| stage | module | what it does |
|---|---|---|
| video | mcap_video.py |
decodes the H.264 stream from an MCAP file, yields the left stereo view at requested timestamps |
| detection | WiLoR/detectors/yolo_world/ |
open-vocabulary hand detection on the full view, with half-view fallback |
| tracking | tracking.py |
CSRT trackers per hand, scheduled detector refresh, gated recovery, merge/jump rejection |
| pose | WiLoR/ |
ViT backbone + refinement head producing MANO parameters, 21 3-D joints and a hand mesh |
| sequence | sequence.py |
carries tracker state, hand IDs and the detector schedule across numbered recordings |
| filter | pose_filter.py |
optional causal MANO error-state EKF |
| output | process_poses.py, render.py |
JSONL rows, optional overlay frames and preview video |
Output
One JSONL row per exported frame:
offset_s, frame_index, packet_index, log_time_ns, image_width, image_height, num_hands,
hands: [ { right, detector_score, bbox, joints_3d[21,3], mano_global_orient[3,3],
mano_hand_pose, mano_betas, cam_t, vertices } ],
sequence_id, tracking_segment_id, sequence_frame_index, sequence_offset_s, recording_number
joints_3d is root-relative in metres. cam_t is a weak-perspective translation whose scale
depends on the assumed focal length; use it for relative geometry, not absolute depth.
Usage
pip install -r requirements.txt # needs opencv-contrib-python (CSRT)
python prepare_weights.py --assets /path/to/model-assets
python run.py /path/to/session --output results --sample-hz 10 --device cuda --fp16
A directory input enables sequence mode: recordings are ordered numerically and tracker state is
carried across file boundaries unless a real discontinuity is detected (timestamp gap beyond
--track-max-gap, missing file numbers, or a camera configuration change).
Useful flags: --all-frames (export every decoded picture), --sample-hz (resample),
--pose-filter ekf (causal smoothing), --max-frames, --duration, --threads, --video.
Environment switches: HANDPOSE_TRACKER (CSRT or KCF), HANDPOSE_TRACKER_SCALE,
HANDPOSE_PREVIEW=1 (write overlay frames; off by default).
Assets
Seven files are required under weights/, listed with their SHA-256 in ASSETS.json:
AnyHand-WiLoR checkpoint and config, YOLO-World checkpoint and CLIP vocabulary, and the MANO
right-hand model. Obtain each from its own source under its own terms. MANO in particular
requires registration at https://mano.is.tue.mpg.de and may not be redistributed.
mano_pkl_to_pt.py converts the registered MANO pickle into the tensor file this code loads.
Licensing
Component notices are preserved in licenses/: AnyHand (CC0 1.0), WiLoR (CC BY-NC-ND 4.0),
YOLO-World (GPLv3), MANO (its own terms). No additional rights are granted by this port.