Egocentric two-hand pose estimation pipeline

Offline AnyHand-WiLoR + YOLO-World inference for egocentric H.264 video stored in MCAP. Both hands are estimated from the left camera view, with CSRT tracking and per-hand detection recovery across numbered recordings.

This repository holds the source only. Model assets (AnyHand-WiLoR, YOLO-World, MANO) are not redistributed here; see "Assets" below.

Pipeline

stage module what it does
video mcap_video.py decodes the H.264 stream from an MCAP file, yields the left stereo view at requested timestamps
detection WiLoR/detectors/yolo_world/ open-vocabulary hand detection on the full view, with half-view fallback
tracking tracking.py CSRT trackers per hand, scheduled detector refresh, gated recovery, merge/jump rejection
pose WiLoR/ ViT backbone + refinement head producing MANO parameters, 21 3-D joints and a hand mesh
sequence sequence.py carries tracker state, hand IDs and the detector schedule across numbered recordings
filter pose_filter.py optional causal MANO error-state EKF
output process_poses.py, render.py JSONL rows, optional overlay frames and preview video

Output

One JSONL row per exported frame:

offset_s, frame_index, packet_index, log_time_ns, image_width, image_height, num_hands,
hands: [ { right, detector_score, bbox, joints_3d[21,3], mano_global_orient[3,3],
            mano_hand_pose, mano_betas, cam_t, vertices } ],
sequence_id, tracking_segment_id, sequence_frame_index, sequence_offset_s, recording_number

joints_3d is root-relative in metres. cam_t is a weak-perspective translation whose scale depends on the assumed focal length; use it for relative geometry, not absolute depth.

Usage

pip install -r requirements.txt            # needs opencv-contrib-python (CSRT)
python prepare_weights.py --assets /path/to/model-assets
python run.py /path/to/session --output results --sample-hz 10 --device cuda --fp16

A directory input enables sequence mode: recordings are ordered numerically and tracker state is carried across file boundaries unless a real discontinuity is detected (timestamp gap beyond --track-max-gap, missing file numbers, or a camera configuration change).

Useful flags: --all-frames (export every decoded picture), --sample-hz (resample), --pose-filter ekf (causal smoothing), --max-frames, --duration, --threads, --video.

Environment switches: HANDPOSE_TRACKER (CSRT or KCF), HANDPOSE_TRACKER_SCALE, HANDPOSE_PREVIEW=1 (write overlay frames; off by default).

Assets

Seven files are required under weights/, listed with their SHA-256 in ASSETS.json: AnyHand-WiLoR checkpoint and config, YOLO-World checkpoint and CLIP vocabulary, and the MANO right-hand model. Obtain each from its own source under its own terms. MANO in particular requires registration at https://mano.is.tue.mpg.de and may not be redistributed. mano_pkl_to_pt.py converts the registered MANO pickle into the tensor file this code loads.

Licensing

Component notices are preserved in licenses/: AnyHand (CC0 1.0), WiLoR (CC BY-NC-ND 4.0), YOLO-World (GPLv3), MANO (its own terms). No additional rights are granted by this port.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support