Update GEM model card and Model Card++ subcards

5b9a69d verified 27 days ago

5.36 kB

# Explainability Subcard

## Intended Task/Domain:

3D Human Body Pose Estimation from monocular video — recovering full-body pose (77 joints, body + hands) and world-space global motion trajectories from unconstrained video with dynamic cameras.

## Model Type:

Regression-based Transformer. Pure feed-forward architecture; no iterative sampling or diffusion steps at inference.

## Intended Users:

Computer vision researchers; graphics and animation engineers; machine learning engineers; game developers; biomechanics and sports science, and robotics researchers.

## Output:

Per-frame SOMA body parameters describing the pose of 77 joints (body and hands), global root orientation, world-space root translation, identity (body-shape) coefficients, and body-part scale parameters. The SOMA body model decodes these parameters into 3D joint positions and mesh vertices.

## Describe how the model works:

GEM reconstructs human motion from a monocular RGB video through four stages:

1. **Pre-processing:** The inference pipeline localizes and tracks the target person, obtains per-frame bounding boxes, and uses supplied camera intrinsics or estimates them. A 2D observation model extracts 77-joint keypoints. Camera rotation estimates and frame timestamps provide motion and timing conditions.
2. **Feature extraction:** A frozen Body Pose model with a DINOv3 image backbone processes person crops independently and supplies per-frame Pose query features. Temporal context is modeled in the Motion network rather than in this feature extractor.
3. **Regression transformer:** A 12-layer temporal Transformer with encoder-style self-attention and rotary positional embeddings combines the Pose features, 2D observations, bounding-box/intrinsics conditioning, camera-motion conditioning, and physical-time inputs. It predicts a 655-dimensional SOMA motion representation together with camera and contact outputs.
4. **Decoding:** The predicted representation is decoded into body pose, global orientation, world-space translation, body shape, and scale parameters. The SOMA body model produces the corresponding 3D joints and mesh. Optional contact/IK postprocessing corrects world-space translation and pose using predicted contacts; raw predictions are retained separately.

## List the groups (or group characteristics) for which this was tested to produce comparable outcomes:

No formal demographic subgroup evaluation is reported for this release. The reported evaluation measures aggregate pose error on an internal held-out synthetic dataset; it does not establish comparable outcomes across demographic groups or group characteristics.

## Technical Limitations and Mitigation:

- **Occlusion and visual ambiguity:** Heavy occlusion, motion blur, poor lighting, unusual clothing, or unfamiliar poses may produce inaccurate body or hand estimates. Use clear footage where possible and inspect the reconstructed motion against the input video.
- **Camera estimation errors:** Errors in camera intrinsics or the estimated camera trajectory can affect reconstructed pose and world-space motion. Check camera estimates and use the static-camera inference mode only when the input camera is stationary.
- **Person detection and tracking errors:** Incorrect bounding boxes or changes in the tracked identity can corrupt the estimated motion. Inspect the tracked subject and bounding boxes, particularly in scenes with multiple people or substantial occlusion.
- **Generalization beyond the evaluation set:** Performance on held-out synthetic videos does not establish accuracy for all real-world body proportions, appearances, activities, or recording conditions. Evaluate representative data from the intended application before deployment.

## Verified to have met prescribed NVIDIA quality standards:

Not confirmed for this release.

## Performance Metrics:

World-space mean per-joint position error (W-MPJPE), measured in millimeters; lower values indicate lower pose error. The reported GEM v1.1 result is **100.2 mm** on the internal evaluation set of **5,000 held-out synthetically rendered video clips**, as described in the [GEM model overview](../README.md#evaluation-dataset).

## Potential Known Risks:

- **Incorrect motion estimates:** Pose and trajectory errors may lead to incorrect animation or misleading downstream motion analysis. Validate predictions against the input video and application requirements before relying on them.
- **Privacy:** Reconstructed body shape and motion describe the people in the input video. Geometric motion output should not be treated as inherently anonymous; users should obtain appropriate authorization for the input footage and handle the resulting motion data appropriately.
- **Non-consensual or misleading use:** Recovered motion could be used to animate avatars or create representations of people without their consent. Use the model only with appropriate rights and permissions, and avoid presenting reconstructed motion as verified evidence of a person's actions.

## Terms of Use/Licensing:

Use of the source code is governed by the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0). Use of the associated model is governed by the [NVIDIA Open Model License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/).

	# Explainability Subcard

	## Intended Task/Domain:

	3D Human Body Pose Estimation from monocular video — recovering full-body pose (77 joints, body + hands) and world-space global motion trajectories from unconstrained video with dynamic cameras.

	## Model Type:

	Regression-based Transformer. Pure feed-forward architecture; no iterative sampling or diffusion steps at inference.

	## Intended Users:

	Computer vision researchers; graphics and animation engineers; machine learning engineers; game developers; biomechanics and sports science, and robotics researchers.

	## Output:

	Per-frame SOMA body parameters describing the pose of 77 joints (body and hands), global root orientation, world-space root translation, identity (body-shape) coefficients, and body-part scale parameters. The SOMA body model decodes these parameters into 3D joint positions and mesh vertices.

	## Describe how the model works:

	GEM reconstructs human motion from a monocular RGB video through four stages:

	1. Pre-processing: The inference pipeline localizes and tracks the target person, obtains per-frame bounding boxes, and uses supplied camera intrinsics or estimates them. A 2D observation model extracts 77-joint keypoints. Camera rotation estimates and frame timestamps provide motion and timing conditions.
	2. Feature extraction: A frozen Body Pose model with a DINOv3 image backbone processes person crops independently and supplies per-frame Pose query features. Temporal context is modeled in the Motion network rather than in this feature extractor.
	3. Regression transformer: A 12-layer temporal Transformer with encoder-style self-attention and rotary positional embeddings combines the Pose features, 2D observations, bounding-box/intrinsics conditioning, camera-motion conditioning, and physical-time inputs. It predicts a 655-dimensional SOMA motion representation together with camera and contact outputs.
	4. Decoding: The predicted representation is decoded into body pose, global orientation, world-space translation, body shape, and scale parameters. The SOMA body model produces the corresponding 3D joints and mesh. Optional contact/IK postprocessing corrects world-space translation and pose using predicted contacts; raw predictions are retained separately.

	## List the groups (or group characteristics) for which this was tested to produce comparable outcomes:

	No formal demographic subgroup evaluation is reported for this release. The reported evaluation measures aggregate pose error on an internal held-out synthetic dataset; it does not establish comparable outcomes across demographic groups or group characteristics.

	## Technical Limitations and Mitigation:

	- Occlusion and visual ambiguity: Heavy occlusion, motion blur, poor lighting, unusual clothing, or unfamiliar poses may produce inaccurate body or hand estimates. Use clear footage where possible and inspect the reconstructed motion against the input video.
	- Camera estimation errors: Errors in camera intrinsics or the estimated camera trajectory can affect reconstructed pose and world-space motion. Check camera estimates and use the static-camera inference mode only when the input camera is stationary.
	- Person detection and tracking errors: Incorrect bounding boxes or changes in the tracked identity can corrupt the estimated motion. Inspect the tracked subject and bounding boxes, particularly in scenes with multiple people or substantial occlusion.
	- Generalization beyond the evaluation set: Performance on held-out synthetic videos does not establish accuracy for all real-world body proportions, appearances, activities, or recording conditions. Evaluate representative data from the intended application before deployment.

	## Verified to have met prescribed NVIDIA quality standards:

	Not confirmed for this release.

	## Performance Metrics:

	World-space mean per-joint position error (W-MPJPE), measured in millimeters; lower values indicate lower pose error. The reported GEM v1.1 result is 100.2 mm on the internal evaluation set of 5,000 held-out synthetically rendered video clips, as described in the [GEM model overview](../README.md#evaluation-dataset).

	## Potential Known Risks:

	- Incorrect motion estimates: Pose and trajectory errors may lead to incorrect animation or misleading downstream motion analysis. Validate predictions against the input video and application requirements before relying on them.
	- Privacy: Reconstructed body shape and motion describe the people in the input video. Geometric motion output should not be treated as inherently anonymous; users should obtain appropriate authorization for the input footage and handle the resulting motion data appropriately.
	- Non-consensual or misleading use: Recovered motion could be used to animate avatars or create representations of people without their consent. Use the model only with appropriate rights and permissions, and avoid presenting reconstructed motion as verified evidence of a person's actions.

	## Terms of Use/Licensing:

	Use of the source code is governed by the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0). Use of the associated model is governed by the [NVIDIA Open Model License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/).