EgoHand-v1: mirror of nvidia/EgoHand-v1-GA

Co-authored-by: yuhan210 <yuhan210@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,134 @@
+---
+license: openmdw-1.1
+---
+
+# EgoHand-1.0 Overview
+
+## Description:  
+EgoHand predicts SOMA-hand parameters (pose, 27D MHR shape/scale coefficients) and a full-perspective camera translation (conditioned on input camera intrinsics) from a single image cropped from a 2D hand bounding box and resized to 256×256, using a DINOv3 vision backbone (initialized from [SAM 3D Body](https://huggingface.co/facebook/sam-3d-body-dinov3)) feeding a TransformerDecoder head (SOMAHandTransformerDecoderHeadV2) with learnable init pose/shape/camera queries. The predicted shape/scale coefficients parameterize MHR's rest-pose hand identity, which SOMA's rig then deforms via forward kinematics using the predicted pose to produce the final 3D hand mesh. It is trained on synthetic and real world hand data with keypoint-2D, keypoint-3D, and MHR/SOMA parameter losses computed in the crop frame, and is evaluated on the HOT3D real-world egocentric benchmark.  
+EgoHand-1.0 was developed by NVIDIA as a part of EgoHand.  
+
+_This model is ready for commercial or non-commercial use._
+
+### License/Terms of Use:
+Use of the associated model is governed by the [OpenMDW License Agreement, version 1.1](https://github.com/OpenMDW/OpenMDW/blob/main/1.1/LICENSE.OpenMDW-1.1).  
+
+ADDITIONAL INFORMATION: [Apache License, Version 2.0.](https://www.apache.org/licenses/LICENSE-2.0)
+
+### Deployment Geography:  
+Global
+
+### Use Case:  
+EgoHand could be used by researchers and engineers working on egocentric hand pose estimation to produce accurate 3D hand keypoints and MHR/SOMA-hand mesh parameters from single RGB crops. The resulting 3D hand pose/keypoint data serves as a training signal for downstream robotics applications — particularly robotic foundation models and manipulation policy training, where accurate hand trajectories human demonstration video are needed to supervise or bootstrap policy learning.  
+  
+
+## Model Architecture:   
+**Architecture Type:** Transformer
+**Network Architecture:** DINOv3 (Vision Transformer backbone) + TransformerDecoder head  
+**This model was developed based on DINOv3.**  
+**Number of model parameters:** 883.5M  
+
+ 
+## Input:  
+**Input Type(s):**
+
+* Image
+* Other: Camera intrinsics
+
+**Input Format(s):**
+
+* Image: Red, Green, Blue (RGB)
+* Camera intrinsics: Float
+
+**Input Parameters:**
+
+* Image: Two-Dimensional (2D)
+* Camera intrinsics: Other: 3x3 matrix (focal length + principal point)
+
+**Other Properties Related to Input:** Image cropped from 2D hand bounding box (squared, with padding) and resized to 256×256; camera intrinsics (focal length + principal point)  
+  
+
+## Output:  
+**Output Type(s):**
+
+* Other: SOMA parameters (hand pose and shape)
+* Other: Camera pose
+
+**Output Format:**
+
+* SOMA parameters: Float
+* Camera pose: Float
+
+**Output Parameters:**
+
+* SOMA parameters: Other: parameter vector (pose rotations + shape/scale coefficients)
+* Camera pose: Three-Dimensional (3D)
+
+**Other Properties Related to Output:** 3D hand joints and mesh  
+   
+
+Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
+
+## Software Integration:
+**Runtime Engine(s):** PyTorch 
+**Supported Hardware Microarchitecture Compatibility:** NVIDIA Ampere
+**Supported Operating System(s):** Linux  
+
+The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.  
+
+
+## Model Version(s): 
+EgoHand v1.0 is a monocular model that takes a single RGB image as input. Users can post-train the model on their own data, and the resulting 3D hand joint outputs can be used to train robot foundation models.
+ 
+### Training Dataset:
+
+**Data Modality:**
+* Image  
+* Video  
+
+**Image Training Data Size:** \~14M images (\~9M in-the-wild and MOCAP data; \~5M synthetic)  
+**Video Training Data Size:** \~126 hours  
+**Data count:** \~14M images  
+**Data Collection Method by dataset:** Synthetic  
+**Labeling Method by dataset:** Synthetic  
+**Properties (Quantity, Dataset Descriptions, Sensor(s)):** The model is trained on a mix of synthetic and real-world hand motion image/video data, annotated with 3D hand keypoints and camera pose.
+* In-house purchased real-world egocentric hand data: RGB image/video of hand motion captured with head-mounted egocentric cameras, sourced from 3rd-party data vendors. 
+* Synthetic data generated via the in-house RigPlay/DigitalHuman rendering pipeline: photorealistic renders of hand motion with exact ground-truth 3D keypoints, MHR/SOMA parameters, and camera pose.
+* In-house purchased motion capture data: real hand motion captured with a multi-camera, marker-based (reflective-marker) motion-capture rig for high-precision ground-truth 3D hand keypoints and pose. \~16 hours total across two capture formats: 1360×1360 at 30 FPS and 2464×2064 at 60 FPS.
+
+### Testing Dataset:
+
+**Data Collection Method by dataset:** Manually-Collected
+**Labeling Method by dataset:** Automatic/Sensors
+**Image Evaluation Data Size:** 100K Images  
+**Properties (Quantity, Dataset Descriptions, Sensor(s)):** 
+* We tested on videos of hand movements and gestures captured in realistic scenes. Videos are collected by cameras or XR devices (e.g., Apple Vision Pro) and chunked into 30 secs long at 30 FPS. We only used the data to evaluate model qualitative performance.
+* EgoDex: 829 hours of 1080p, 30 FPS egocentric video (338K task-demonstration episodes, ~90M frames) across 194 diverse tabletop manipulation tasks, collected on Apple Vision Pro. Paired 3D hand/finger and upper-body joint pose annotations (25 joints per hand) and camera extrinsics are estimated on-device via ARKit/SLAM at recording time; we only used the RGB frames for qualitative evaluation.
+
+
+### Evaluation Dataset:
+**Benchmark Score:** Undisclosed
+
+**Data Collection Method by dataset:** Manually-Collected
+**Labeling Method by dataset:** Automatic/Sensors  
+**Image Evaluation Data Size:** 231K Images  
+**Properties (Quantity, Dataset Descriptions, Sensor(s)):** 
+* HOT3D: 9 subjects, 1.5K clips, 227K frames. Real-world egocentric dataset for 3D hand and object tracking, recorded with Meta Project Aria glasses (RGB + monochrome cameras) and Meta Quest 3 headsets (stereo monochrome cameras). Ground-truth 3D hand poses (UmeTrack) were captured with a marker-based motion-capture system and a custom 3D hand scanner per subject.
+* H2O: 4 subjects, 184 clips, 114K frames. Real-world egocentric dataset of two hands manipulating objects, recorded indoors with synchronized multi-view RGB-D cameras including a head-mounted egocentric camera. Ground-truth 3D hand poses (with MANO fits for left/right hands) were derived via markerless multi-view triangulation.
+
+
+
+## Inference:
+**Acceleration Engine:** Other: None
+**Test Hardware:**
+* NVIDIA Lovelace L40
+* NVIDIA Lovelace L40s
+
+
+## Ethical Considerations:
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications.  Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.  
+Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.
+
+For more detailed information on ethical considerations for this model, please see the Model Card++ [Bias](egohand-subcards/bias.md), [Explainability](egohand-subcards/explainability.md), [Safety & Security](egohand-subcards/safety_security.md), and [Privacy](egohand-subcards/privacy.md) Subcards.  
+
+Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).

egohand-subcards/bias.md

@@ -0,0 +1,14 @@
+# Bias Subcard
+
+## Participation considerations from adversely impacted groups [protected classes](https://www.senate.ca.gov/content/protected-classes) in model design and testing:
+
+Age, Ethnicity, Gender, Skin Color
+
+## Measures taken to mitigate against unwanted bias:
+
+- For real-world data, data was sourced from multiple vendors and curated to achieve diverse distribution across hand size, pose, and data collectors.
+- For synthetic data, virtual characters with varying age, ethnicity, and skin color were used to ensure demographic diversity in training samples.
+
+## Bias Metric (If Measured):
+
+None

egohand-subcards/explainability.md

@@ -0,0 +1,48 @@
+# Explainability Subcard
+
+## Intended Task/Domain:
+
+3D Hand Keypoint/Mesh Prediction - Robotics 
+
+## Model Type:
+
+Transformer
+
+## Intended Users:
+
+Researchers and developers working on egocentric hand pose estimation, particularly for applications involving 3D hand modeling, robot imitation learning, and manipulation policy training.
+
+## Output:
+
+Array (Floating Point) which includes SOMA hand parameters, camera translation, and derived 3D hand joints and mesh vertices.
+
+## Describe how the model works:
+
+EgoHand takes a single 256×256 RGB crop of a hand (extracted from a 2D bounding box detection) and camera intrinsics as input. A DINOv3 Vision Transformer backbone extracts image features, which are fed into a TransformerDecoder head with learnable pose, shape, and camera queries. The decoder predicts SOMA-hand parameters — joint rotations and 27D MHR shape/scale coefficients — along with a full-perspective camera translation conditioned on the input intrinsics. The SOMA rig then applies forward kinematics to the predicted parameters to produce the final 3D hand joints and mesh.
+
+## List the groups (or group characteristics) for which this was tested to produce comparable outcomes:
+
+Age, Ethnicity, Gender, Skin Color
+
+## Technical Limitations and Mitigation:
+
+The model may not generalize or perform well in challenging scenes, such as low lighting, heavy occlusion (e.g., hands partially blocked by objects, the other hand, or the frame edge), motion blur from fast hand movement, or unusual hand poses not well represented in training data. Accuracy depends on the quality of the upstream 2D bounding box detection — a missed or inaccurate bbox will degrade 3D pose output for that frame. Mitigation: fine-tuning on additional real-world egocentric data covering these conditions, and applying corresponding data augmentation during training. Upgrading or fine-tuning the upstream 2D hand detector on domain-specific data can also significantly improve overall 3D pose quality by providing more accurate and consistent bounding box inputs.
+
+## Verified to have met prescribed NVIDIA quality standards:
+
+Yes
+
+## Performance Metrics:
+
+Mean Per Joint Position Error (MPJPE)
+Root Translation Error (RTE)
+Acceleration Error
+Throughput & Latency
+
+## Potential Known Risks:
+
+The model may produce inaccurate 3D hand pose or mesh for frames with heavy occlusion, fast motion blur, or poses underrepresented in training data. Errors in predicted joint positions or mesh vertices may propagate to downstream applications (e.g., robot policy training), potentially degrading the quality of learned manipulation behaviors.
+
+## Terms of Use/Licensing:
+
+Use of the associated model is governed by the [OpenMDW License Agreement, version 1.1](https://github.com/OpenMDW/OpenMDW/blob/main/1.1/LICENSE.OpenMDW-1.1).

egohand-subcards/privacy.md

@@ -0,0 +1,53 @@
+# Privacy Subcard
+
+## Generatable or reverse engineerable personal data?
+
+No
+
+## Personal data used to create this model?
+
+Yes
+
+## Was consent obtained for any personal data used?
+
+Yes
+
+## How often is dataset reviewed?
+
+Dataset is initially reviewed upon addition, and subsequent reviews are conducted as needed or upon request for changes.
+
+## Is a mechanism in place to honor data subject right of access or deletion of personal data?
+
+Yes.
+
+## If personal data was collected for the development of the model, was it collected directly by NVIDIA?
+
+No
+
+## If personal data was collected for the development of the model by NVIDIA, do you maintain or have access to disclosures made to data subjects?
+
+Not Applicable 
+
+## If personal data was collected for the development of this AI model, was it minimized to only what was required?
+
+Yes
+
+## Was data from user interactions with the AI model (e.g. user input and prompts) used to train the model?
+
+No
+
+## Is there provenance for all datasets used in training?
+
+Yes
+
+## Does data labeling (annotation, metadata) comply with privacy laws?
+
+Yes
+
+## Is data compliant with data subject requests for data correction or removal, if such a request was made?
+
+No, not possible for externally sourced data.
+
+## Applicable Privacy Policy
+
+https://www.nvidia.com/en-us/about-nvidia/privacy-policy/

egohand-subcards/safety_security.md

@@ -0,0 +1,17 @@
+# Safety & Security Subcard
+
+## Model Application Field(s):
+
+3D Hand Pose Estimation and Mesh Prediction for Robotics Applications
+
+## Describe the life critical impact (if present).
+
+Not Applicable - This model performs 3D hand pose estimation and mesh prediction from egocentric RGB images, intended for robotics use cases where accurate hand keypoints and mesh parameters serve as a training signal for robot foundation models and manipulation policies. It is not designed or validated for safety-critical applications.
+
+## Use Case Restrictions:
+
+Abide by the [OpenMDW License Agreement, version 1.1](https://github.com/OpenMDW/OpenMDW/blob/main/1.1/LICENSE.OpenMDW-1.1)
+
+## Model and dataset restrictions:
+
+The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development. Restrictions enforce dataset access during training, and dataset license constraints adhered to.

egohand_s060000.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f940ecb6e3b73d5bd9e562cff232550e1c072f6c063eb4f705951122b15d8a75
+size 10657169753