Update README.md
Browse files
README.md
CHANGED
|
@@ -70,10 +70,8 @@ EgoHand could be used by researchers and engineers working on egocentric hand po
|
|
| 70 |
Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
|
| 71 |
|
| 72 |
## Software Integration:
|
| 73 |
-
**Runtime Engine(s):** PyTorch
|
| 74 |
-
|
| 75 |
-
**Supported Hardware Microarchitecture Compatibility:** NVIDIA Ampere
|
| 76 |
-
|
| 77 |
**Supported Operating System(s):** Linux
|
| 78 |
|
| 79 |
The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
|
|
@@ -93,25 +91,25 @@ EgoHand v1.0 is a monocular model that takes a single RGB image as input. Users
|
|
| 93 |
**Data count:** \~14M images
|
| 94 |
**Data Collection Method by dataset:** Synthetic
|
| 95 |
**Labeling Method by dataset:** Synthetic
|
| 96 |
-
**Properties (Quantity, Dataset Descriptions, Sensor(s)):** The model is trained on a mix of synthetic and real-world hand motion image/video data, annotated with 3D hand keypoints and camera pose.
|
| 97 |
* In-house purchased real-world egocentric hand data: RGB image/video of hand motion captured with head-mounted egocentric cameras, sourced from 3rd-party data vendors.
|
| 98 |
-
* Synthetic data generated via the in-house RigPlay/DigitalHuman rendering pipeline: photorealistic renders of hand motion with exact ground-truth 3D keypoints, MHR/SOMA parameters, and camera pose.
|
| 99 |
* In-house purchased motion capture data: real hand motion captured with a multi-camera, marker-based (reflective-marker) motion-capture rig for high-precision ground-truth 3D hand keypoints and pose. \~16 hours total across two capture formats: 1360×1360 at 30 FPS and 2464×2064 at 60 FPS.
|
| 100 |
|
| 101 |
### Testing Dataset:
|
| 102 |
|
| 103 |
-
**Data Collection Method by dataset:** Manually-Collected
|
| 104 |
-
**Labeling Method by dataset:** Automatic/Sensors
|
| 105 |
**Image Evaluation Data Size:** 100K Images
|
| 106 |
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**
|
| 107 |
-
* We tested on videos of hand movements and gestures captured in realistic scenes. Videos are collected by cameras or XR devices (e.g., Apple Vision Pro) and chunked into 30 secs long at 30 FPS. We only used the data to evaluate model qualitative performance.
|
| 108 |
-
* EgoDex: 829 hours of 1080p, 30 FPS egocentric video (338K task-demonstration episodes, ~90M frames) across 194 diverse tabletop manipulation tasks, collected on Apple Vision Pro. Paired 3D hand/finger and upper-body joint pose annotations (25 joints per hand) and camera extrinsics are estimated on-device via ARKit/SLAM at recording time; we only used the RGB frames for qualitative evaluation.
|
| 109 |
|
| 110 |
|
| 111 |
### Evaluation Dataset:
|
| 112 |
-
**Benchmark Score:** Undisclosed
|
| 113 |
|
| 114 |
-
**Data Collection Method by dataset:** Manually-Collected
|
| 115 |
**Labeling Method by dataset:** Automatic/Sensors
|
| 116 |
**Image Evaluation Data Size:** 231K Images
|
| 117 |
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**
|
|
@@ -122,10 +120,10 @@ EgoHand v1.0 is a monocular model that takes a single RGB image as input. Users
|
|
| 122 |
|
| 123 |
## Inference:
|
| 124 |
**Acceleration Engine:**
|
| 125 |
-
Other: None
|
| 126 |
|
| 127 |
**Test Hardware:**
|
| 128 |
-
* NVIDIA Lovelace L40
|
| 129 |
* NVIDIA Lovelace L40s
|
| 130 |
|
| 131 |
|
|
|
|
| 70 |
Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
|
| 71 |
|
| 72 |
## Software Integration:
|
| 73 |
+
**Runtime Engine(s):** PyTorch
|
| 74 |
+
**Supported Hardware Microarchitecture Compatibility:** NVIDIA Ampere
|
|
|
|
|
|
|
| 75 |
**Supported Operating System(s):** Linux
|
| 76 |
|
| 77 |
The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
|
|
|
|
| 91 |
**Data count:** \~14M images
|
| 92 |
**Data Collection Method by dataset:** Synthetic
|
| 93 |
**Labeling Method by dataset:** Synthetic
|
| 94 |
+
**Properties (Quantity, Dataset Descriptions, Sensor(s)):** The model is trained on a mix of synthetic and real-world hand motion image/video data, annotated with 3D hand keypoints and camera pose.
|
| 95 |
* In-house purchased real-world egocentric hand data: RGB image/video of hand motion captured with head-mounted egocentric cameras, sourced from 3rd-party data vendors.
|
| 96 |
+
* Synthetic data generated via the in-house RigPlay/DigitalHuman rendering pipeline: photorealistic renders of hand motion with exact ground-truth 3D keypoints, MHR/SOMA parameters, and camera pose.
|
| 97 |
* In-house purchased motion capture data: real hand motion captured with a multi-camera, marker-based (reflective-marker) motion-capture rig for high-precision ground-truth 3D hand keypoints and pose. \~16 hours total across two capture formats: 1360×1360 at 30 FPS and 2464×2064 at 60 FPS.
|
| 98 |
|
| 99 |
### Testing Dataset:
|
| 100 |
|
| 101 |
+
**Data Collection Method by dataset:** Manually-Collected
|
| 102 |
+
**Labeling Method by dataset:** Automatic/Sensors
|
| 103 |
**Image Evaluation Data Size:** 100K Images
|
| 104 |
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**
|
| 105 |
+
* We tested on videos of hand movements and gestures captured in realistic scenes. Videos are collected by cameras or XR devices (e.g., Apple Vision Pro) and chunked into 30 secs long at 30 FPS. We only used the data to evaluate model qualitative performance.
|
| 106 |
+
* EgoDex: 829 hours of 1080p, 30 FPS egocentric video (338K task-demonstration episodes, ~90M frames) across 194 diverse tabletop manipulation tasks, collected on Apple Vision Pro. Paired 3D hand/finger and upper-body joint pose annotations (25 joints per hand) and camera extrinsics are estimated on-device via ARKit/SLAM at recording time; we only used the RGB frames for qualitative evaluation.
|
| 107 |
|
| 108 |
|
| 109 |
### Evaluation Dataset:
|
| 110 |
+
**Benchmark Score:** Undisclosed
|
| 111 |
|
| 112 |
+
**Data Collection Method by dataset:** Manually-Collected
|
| 113 |
**Labeling Method by dataset:** Automatic/Sensors
|
| 114 |
**Image Evaluation Data Size:** 231K Images
|
| 115 |
**Properties (Quantity, Dataset Descriptions, Sensor(s)):**
|
|
|
|
| 120 |
|
| 121 |
## Inference:
|
| 122 |
**Acceleration Engine:**
|
| 123 |
+
Other: None
|
| 124 |
|
| 125 |
**Test Hardware:**
|
| 126 |
+
* NVIDIA Lovelace L40
|
| 127 |
* NVIDIA Lovelace L40s
|
| 128 |
|
| 129 |
|