Release unified Body Pose, Hand Pose and native77 Motion (2026-10-20)
#1
by jeffffffli - opened
- .gitattributes +0 -5
- README.md +44 -4
- body_config.json +42 -0
- gem_smpl.ckpt → body_s240000.ckpt +2 -2
- body_video_native77.yaml +48 -0
- config.json +0 -40
- gem_smpl/missing_hmr4d_support/inputs/EMDB/hmr4d_support/emdb_slam_traj.pt → egohand_s060000.ckpt +2 -2
- gem_smpl/missing_hmr4d_support/README_GEM_missing_files.txt +0 -19
- gem_smpl/missing_hmr4d_support/inputs/3DPW/hmr4d_support/3dpw_test_slam_traj.pt +0 -3
- gem_smpl/missing_hmr4d_support/inputs/EMDB/hmr4d_support/emdb_vimo.pt +0 -3
- gem_smpl/missing_hmr4d_support/inputs/RICH/hmr4d_support/rich_test_vimo_preproc.pt +0 -3
- gem_smpl/onnx/gem_smpl_denoiser.onnx +0 -3
- gem_smpl/onnx/gem_smpl_denoiser_no_imgfeat.onnx +0 -3
- gem_smpl/onnx/hmr2.onnx +0 -3
- gem_smpl/onnx/hmr2.onnx.data +0 -3
- gem_smpl/onnx/vitpose_coco17.onnx +0 -3
- gem_smpl/onnx/vitpose_coco17.onnx.data +0 -3
- gem_smpl_config.json +0 -33
- gem_soma.ckpt +0 -3
- hand_config.json +64 -0
- mhr_model.pt +0 -3
- model_config.yaml +0 -73
- motion_native77.yaml +31 -0
- gem_smpl/missing_hmr4d_support/inputs/3DPW/hmr4d_support/test_3dpw_vimo_labels.pt → motion_native77_e774_s292175.ckpt +2 -2
- scale_mean.pth → motion_stats_raw.npz +2 -2
- onnx/gem_denoiser.onnx +0 -3
- onnx/gem_denoiser.onnx.data +0 -3
- onnx/gem_denoiser_no_imgfeat.onnx +0 -3
- onnx/gem_denoiser_no_imgfeat.onnx.data +0 -3
- onnx/sam3db_backbone.onnx +0 -3
- onnx/sam3db_backbone.onnx.data +0 -3
- sam3d_body.ckpt +0 -3
- scale_comps.pth +0 -3
.gitattributes
CHANGED
|
@@ -33,9 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
-
onnx/gem_denoiser.onnx.data filter=lfs diff=lfs merge=lfs -text
|
| 37 |
-
onnx/gem_denoiser_no_imgfeat.onnx.data filter=lfs diff=lfs merge=lfs -text
|
| 38 |
onnx/vitpose.onnx.data filter=lfs diff=lfs merge=lfs -text
|
| 39 |
-
onnx/sam3db_backbone.onnx.data filter=lfs diff=lfs merge=lfs -text
|
| 40 |
-
gem_smpl/onnx/vitpose_coco17.onnx.data filter=lfs diff=lfs merge=lfs -text
|
| 41 |
-
gem_smpl/onnx/hmr2.onnx.data filter=lfs diff=lfs merge=lfs -text
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
| 36 |
onnx/vitpose.onnx.data filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
README.md
CHANGED
|
@@ -22,6 +22,10 @@ All training data and the SOMA parametric body model are owned by NVIDIA or rele
|
|
| 22 |
GEM was developed by NVIDIA.
|
| 23 |
*This model is ready for commercial or non-commercial use.*
|
| 24 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
### License/Terms of Use:
|
| 26 |
|
| 27 |
Use of the source code is governed by the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0). Use of the associated model is governed by the [NVIDIA Open Model License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/).
|
|
@@ -53,9 +57,9 @@ GEM is intended for use by individuals and professionals in fields such as machi
|
|
| 53 |
## Model Architecture:
|
| 54 |
|
| 55 |
**Architecture Type:** Transformer
|
| 56 |
-
**Network Architecture:** 12-layer Transformer
|
| 57 |
**This model was developed based on Not Applicable (N/A).**
|
| 58 |
-
**Number of model parameters:**
|
| 59 |
|
| 60 |
## Computational Load
|
| 61 |
|
|
@@ -68,7 +72,7 @@ GEM is intended for use by individuals and professionals in fields such as machi
|
|
| 68 |
|
| 69 |
**Input Format(s):**
|
| 70 |
|
| 71 |
-
- Video: RGB tensor `(T, H, W, 3)`, any resolution, internally processed via the frozen
|
| 72 |
- Other — person bounding box: `(T, 4)` float array in `xyxy` pixel coordinates
|
| 73 |
- Other — camera intrinsics: `(T, 3, 3)` float array of per-frame camera intrinsic matrices containing focal lengths and principal-point coordinates
|
| 74 |
|
|
@@ -108,6 +112,42 @@ GEM v1.1
|
|
| 108 |
An installable Python package provides the `gem-infer`, `gem-render`, `gem-train`, and `gem-eval` entry points. Body Motion inference uses `gem-infer motion-video --task body` for RGB video or `gem-infer motion-cache --task body` for prepared conditions. The package requires Python 3.10 or newer. The validated CUDA runtime uses PyTorch 2.10 and SOMA 0.3.3 with separately supplied model weights and geometry assets. Mesh/video rendering uses the documented x86/Open3D 0.19 environment.
|
| 109 |
|
| 110 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 111 |
## Training, Testing, and Evaluation Datasets:
|
| 112 |
|
| 113 |
|
|
@@ -147,7 +187,7 @@ An installable Python package provides the `gem-infer`, `gem-render`, `gem-train
|
|
| 147 |
|
| 148 |
## Inference:
|
| 149 |
|
| 150 |
-
**Acceleration Engine:**
|
| 151 |
**Hardware Requirements (GPU Architecture, Model):** NVIDIA Ampere A100 80GB
|
| 152 |
|
| 153 |
## Ethical Considerations:
|
|
|
|
| 22 |
GEM was developed by NVIDIA.
|
| 23 |
*This model is ready for commercial or non-commercial use.*
|
| 24 |
|
| 25 |
+
This card describes temporal Body Motion. The repository also distributes
|
| 26 |
+
separate Body Pose and Hand Pose checkpoints for image inputs; the checkpoint
|
| 27 |
+
bundle below identifies their configurations and roles.
|
| 28 |
+
|
| 29 |
### License/Terms of Use:
|
| 30 |
|
| 31 |
Use of the source code is governed by the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0). Use of the associated model is governed by the [NVIDIA Open Model License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/).
|
|
|
|
| 57 |
## Model Architecture:
|
| 58 |
|
| 59 |
**Architecture Type:** Transformer
|
| 60 |
+
**Network Architecture:** 12-layer temporal Transformer, hidden size 512, 8 attention heads<br>
|
| 61 |
**This model was developed based on Not Applicable (N/A).**
|
| 62 |
+
**Number of model parameters:** 46.8M total temporal Motion parameters (46,762,136), including 44.2M trainable parameters (44,202,136). This excludes the separate Body Pose feature extractor, Hand Pose model, and detectors.
|
| 63 |
|
| 64 |
## Computational Load
|
| 65 |
|
|
|
|
| 72 |
|
| 73 |
**Input Format(s):**
|
| 74 |
|
| 75 |
+
- Video: RGB tensor `(T, H, W, 3)`, any resolution, internally processed via the frozen Body Pose image encoder
|
| 76 |
- Other — person bounding box: `(T, 4)` float array in `xyxy` pixel coordinates
|
| 77 |
- Other — camera intrinsics: `(T, 3, 3)` float array of per-frame camera intrinsic matrices containing focal lengths and principal-point coordinates
|
| 78 |
|
|
|
|
| 112 |
An installable Python package provides the `gem-infer`, `gem-render`, `gem-train`, and `gem-eval` entry points. Body Motion inference uses `gem-infer motion-video --task body` for RGB video or `gem-infer motion-cache --task body` for prepared conditions. The package requires Python 3.10 or newer. The validated CUDA runtime uses PyTorch 2.10 and SOMA 0.3.3 with separately supplied model weights and geometry assets. Mesh/video rendering uses the documented x86/Open3D 0.19 environment.
|
| 113 |
|
| 114 |
|
| 115 |
+
## Pretrained Checkpoint Bundle
|
| 116 |
+
|
| 117 |
+
The unified release provides three model checkpoints with matching public
|
| 118 |
+
configurations. Body and Hand exports contain complete model weights for
|
| 119 |
+
inference and fine-tuning initialization; optimizer and training-loop state are
|
| 120 |
+
not included, so these files are not full training-resume checkpoints.
|
| 121 |
+
|
| 122 |
+
| Component | Weights | Configuration / companion files |
|
| 123 |
+
| --- | --- | --- |
|
| 124 |
+
| Body Pose | [body_s240000.ckpt](https://huggingface.co/nvidia/GEM-X/resolve/main/body_s240000.ckpt) | [body_config.json](https://huggingface.co/nvidia/GEM-X/blob/main/body_config.json); preset `body` |
|
| 125 |
+
| Hand Pose | [egohand_s060000.ckpt](https://huggingface.co/nvidia/GEM-X/resolve/main/egohand_s060000.ckpt) | [hand_config.json](https://huggingface.co/nvidia/GEM-X/blob/main/hand_config.json); preset `hand`; includes the learned RLE density used for fine-tuning |
|
| 126 |
+
| Body Motion native77 | [motion_native77_e774_s292175.ckpt](https://huggingface.co/nvidia/GEM-X/resolve/main/motion_native77_e774_s292175.ckpt) | [motion_stats_raw.npz](https://huggingface.co/nvidia/GEM-X/resolve/main/motion_stats_raw.npz) and [motion_native77.yaml](https://huggingface.co/nvidia/GEM-X/blob/main/motion_native77.yaml) |
|
| 127 |
+
| Original native77 ViTPose | [vitpose.pth](https://huggingface.co/nvidia/GEM-X/resolve/main/vitpose.pth) | Used by [body_video_native77.yaml](https://huggingface.co/nvidia/GEM-X/blob/main/body_video_native77.yaml) |
|
| 128 |
+
|
| 129 |
+
The video recipe pairs the original final775 Motion weights with Body Pose
|
| 130 |
+
s240000 features and the existing original ViTPose detector. The experimental
|
| 131 |
+
random-blur epoch10 detector is not included in this release bundle.
|
| 132 |
+
|
| 133 |
+
Download the model files together, preserving their relative placement:
|
| 134 |
+
|
| 135 |
+
```bash
|
| 136 |
+
hf download nvidia/GEM-X \
|
| 137 |
+
body_s240000.ckpt body_config.json \
|
| 138 |
+
egohand_s060000.ckpt hand_config.json \
|
| 139 |
+
motion_native77_e774_s292175.ckpt motion_stats_raw.npz \
|
| 140 |
+
motion_native77.yaml body_video_native77.yaml vitpose.pth \
|
| 141 |
+
--local-dir weights
|
| 142 |
+
```
|
| 143 |
+
|
| 144 |
+
The video YAML resolves checkpoint paths relative to its own directory. Obtain
|
| 145 |
+
the YOLOX HumanArt person detector separately as `weights/yolox_humanart.onnx`,
|
| 146 |
+
or change that path in the YAML. SOMA 0.3.3 geometry assets and an optional
|
| 147 |
+
RT-DETR Hand detector are also supplied separately. Follow the
|
| 148 |
+
[GEM-X installation and inference guides](https://github.com/NVlabs/GEM-X)
|
| 149 |
+
for these prerequisites and the unified command-line interface.
|
| 150 |
+
|
| 151 |
## Training, Testing, and Evaluation Datasets:
|
| 152 |
|
| 153 |
|
|
|
|
| 187 |
|
| 188 |
## Inference:
|
| 189 |
|
| 190 |
+
**Acceleration Engine:** PyTorch with CUDA
|
| 191 |
**Hardware Requirements (GPU Architecture, Model):** NVIDIA Ampere A100 80GB
|
| 192 |
|
| 193 |
## Ethical Considerations:
|
body_config.json
ADDED
|
@@ -0,0 +1,42 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"task": "body",
|
| 3 |
+
"head": {
|
| 4 |
+
"num_body_joints": 76,
|
| 5 |
+
"num_identity_coeffs": 128,
|
| 6 |
+
"num_scale_params": 57,
|
| 7 |
+
"scale_comps_dim": 0,
|
| 8 |
+
"scale_param_mode": "direct",
|
| 9 |
+
"scale_parameterization": "packed",
|
| 10 |
+
"fixed_global_scale": 1.0,
|
| 11 |
+
"transformer_dim": 1024,
|
| 12 |
+
"transformer_depth": 6,
|
| 13 |
+
"transformer_heads": 8,
|
| 14 |
+
"transformer_dim_head": 64,
|
| 15 |
+
"transformer_mlp_dim": 1024,
|
| 16 |
+
"transformer_context_dim": 1280,
|
| 17 |
+
"transformer_dropout": 0.0,
|
| 18 |
+
"transformer_emb_dropout": 0.0,
|
| 19 |
+
"ief_iters": 1,
|
| 20 |
+
"learnable_init": true,
|
| 21 |
+
"use_init_to_token": true,
|
| 22 |
+
"two_layer_heads": true,
|
| 23 |
+
"per_layer_predictions": true,
|
| 24 |
+
"cliff_cond_dim": 3,
|
| 25 |
+
"camera_scale_mode": "ste_clamp"
|
| 26 |
+
},
|
| 27 |
+
"backbone": {
|
| 28 |
+
"name": "dinov3_vith16plus",
|
| 29 |
+
"drop_path_rate": 0.1
|
| 30 |
+
},
|
| 31 |
+
"conditioning": {
|
| 32 |
+
"camera_mode": "sam3db_perspective"
|
| 33 |
+
},
|
| 34 |
+
"backbone_input_crop_w": null,
|
| 35 |
+
"ray_pe_num_bands": null,
|
| 36 |
+
"kp2d_translation": null,
|
| 37 |
+
"pose_release_id": "gem-pose-body-s240000-v1",
|
| 38 |
+
"inference": {
|
| 39 |
+
"camera_representation": "legacy",
|
| 40 |
+
"minimum_depth": 0.05
|
| 41 |
+
}
|
| 42 |
+
}
|
gem_smpl.ckpt → body_s240000.ckpt
RENAMED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e244d27f2834eb6382c5f6b9cbf2a38681b685514a1b2e86f8f3d2592d614959
|
| 3 |
+
size 3538753341
|
body_video_native77.yaml
ADDED
|
@@ -0,0 +1,48 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Original native77 final775 Motion with original ViTPose and Body Pose features.
|
| 2 |
+
# Checkpoint and statistics paths resolve relative to this file.
|
| 3 |
+
# Obtain yolox_humanart.onnx separately and place it here or change its path.
|
| 4 |
+
# Keep motion_stats_raw.npz; the codec uses its trained 0.01 std floor.
|
| 5 |
+
pose:
|
| 6 |
+
checkpoint: body_s240000.ckpt
|
| 7 |
+
release: body
|
| 8 |
+
batch_size: 4
|
| 9 |
+
observations:
|
| 10 |
+
checkpoint: vitpose.pth
|
| 11 |
+
release_id: vitpose77-dinov3-huge-metrosim-256x192-v1
|
| 12 |
+
batch_size: 4
|
| 13 |
+
detector:
|
| 14 |
+
checkpoint: yolox_humanart.onnx
|
| 15 |
+
device: cpu
|
| 16 |
+
motion:
|
| 17 |
+
checkpoint: motion_native77_e774_s292175.ckpt
|
| 18 |
+
stats: motion_stats_raw.npz
|
| 19 |
+
network:
|
| 20 |
+
output_dim: 655
|
| 21 |
+
max_len: 120
|
| 22 |
+
cliffcam_dim: 3
|
| 23 |
+
cam_angvel_dim: 6
|
| 24 |
+
imgseq_dim: 1024
|
| 25 |
+
latent_dim: 512
|
| 26 |
+
num_layers: 12
|
| 27 |
+
num_heads: 8
|
| 28 |
+
mlp_ratio: 4.0
|
| 29 |
+
pred_cam_dim: 3
|
| 30 |
+
static_conf_dim: 6
|
| 31 |
+
dropout: 0.1
|
| 32 |
+
avgbeta: true
|
| 33 |
+
obs_num_joints: 77
|
| 34 |
+
sequence_static_slices:
|
| 35 |
+
- - 456
|
| 36 |
+
- 584
|
| 37 |
+
- - 584
|
| 38 |
+
- 640
|
| 39 |
+
condition_spec:
|
| 40 |
+
pose_release_id: gem-pose-body-s240000-v1:pose_tokens
|
| 41 |
+
observation_source: vitpose77-dinov3-huge-metrosim-256x192-v1
|
| 42 |
+
observation_joint_set: soma77_native_v1
|
| 43 |
+
time_options:
|
| 44 |
+
reference_fps: 30.0
|
| 45 |
+
default_fps: 30.0
|
| 46 |
+
fps_by_data_name: {}
|
| 47 |
+
camera:
|
| 48 |
+
mode: background
|
config.json
DELETED
|
@@ -1,40 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"model_type": "gem",
|
| 3 |
-
"body_model": "soma",
|
| 4 |
-
"denoiser": {
|
| 5 |
-
"latent_dim": 1024,
|
| 6 |
-
"num_layers": 16,
|
| 7 |
-
"num_heads": 8,
|
| 8 |
-
"mlp_ratio": 4,
|
| 9 |
-
"scale_params_dim": 69,
|
| 10 |
-
"scale_comps_dim": 28,
|
| 11 |
-
"fix_scale_comps": true,
|
| 12 |
-
"encode_text": false,
|
| 13 |
-
"encoded_text_dim": 4096,
|
| 14 |
-
"text_encoder_cfg": {
|
| 15 |
-
"mode": "none",
|
| 16 |
-
"cross_attn_type": "mha"
|
| 17 |
-
},
|
| 18 |
-
"text_mask_prob": 0.1,
|
| 19 |
-
"music_mask_prob": 0.1,
|
| 20 |
-
"speech_mask_prob": 0.1,
|
| 21 |
-
"use_text_pos_enc": true
|
| 22 |
-
},
|
| 23 |
-
"endecoder": {
|
| 24 |
-
"stats_name": "MM_V2_SOMA_METROSIM",
|
| 25 |
-
"encode_type": "soma_v2",
|
| 26 |
-
"feat_dim": 585,
|
| 27 |
-
"clip_std": true
|
| 28 |
-
},
|
| 29 |
-
"diffusion": {
|
| 30 |
-
"sampler": "ddim",
|
| 31 |
-
"noise_schedule": "cosine",
|
| 32 |
-
"test_timestep_respacing": "50",
|
| 33 |
-
"ddim_eta": 0.0,
|
| 34 |
-
"guidance_param": 2.5
|
| 35 |
-
},
|
| 36 |
-
"pipeline": {
|
| 37 |
-
"remove_fingers": true,
|
| 38 |
-
"extract_features": false
|
| 39 |
-
}
|
| 40 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
gem_smpl/missing_hmr4d_support/inputs/EMDB/hmr4d_support/emdb_slam_traj.pt → egohand_s060000.ckpt
RENAMED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:af56925fa08b7736787346c95b6bfe4fe7b857cb81e5803c336f72fb114eb900
|
| 3 |
+
size 3552688469
|
gem_smpl/missing_hmr4d_support/README_GEM_missing_files.txt
DELETED
|
@@ -1,19 +0,0 @@
|
|
| 1 |
-
GEM missing hmr4d_support files
|
| 2 |
-
|
| 3 |
-
This folder contains supplemental EMDB / RICH / 3DPW support files for GEM.
|
| 4 |
-
|
| 5 |
-
Copy or merge the `inputs/` directory in this folder into the root of the
|
| 6 |
-
GEM repository. After copying, the files should be located at:
|
| 7 |
-
|
| 8 |
-
inputs/RICH/hmr4d_support/rich_test_vimo_preproc.pt
|
| 9 |
-
inputs/EMDB/hmr4d_support/emdb_vimo.pt
|
| 10 |
-
inputs/EMDB/hmr4d_support/emdb_slam_traj.pt
|
| 11 |
-
inputs/3DPW/hmr4d_support/test_3dpw_vimo_labels.pt
|
| 12 |
-
inputs/3DPW/hmr4d_support/3dpw_test_slam_traj.pt
|
| 13 |
-
|
| 14 |
-
These paths are the ones used by:
|
| 15 |
-
|
| 16 |
-
gem/datasets/rich/rich_motion_test.py
|
| 17 |
-
gem/datasets/emdb/emdb_motion_test.py
|
| 18 |
-
gem/datasets/threedpw/threedpw_motion_test.py
|
| 19 |
-
gem/datasets/threedpw/threedpw_occ_motion_test.py
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
gem_smpl/missing_hmr4d_support/inputs/3DPW/hmr4d_support/3dpw_test_slam_traj.pt
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:16340d0f2cf1453762163082f327d0a3cebd1488399dff71e60e1d001e53d249
|
| 3 |
-
size 3271056
|
|
|
|
|
|
|
|
|
|
|
|
gem_smpl/missing_hmr4d_support/inputs/EMDB/hmr4d_support/emdb_vimo.pt
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:c40713cf7b57e76a645edaa43c32d46aa6814daedaafbadaf1db16f5881c06e7
|
| 3 |
-
size 999364858
|
|
|
|
|
|
|
|
|
|
|
|
gem_smpl/missing_hmr4d_support/inputs/RICH/hmr4d_support/rich_test_vimo_preproc.pt
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:4506ad4b0663b7933e4679f367b23067a06671f35c0c59a0bf1a9418a894296d
|
| 3 |
-
size 1653984970
|
|
|
|
|
|
|
|
|
|
|
|
gem_smpl/onnx/gem_smpl_denoiser.onnx
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:f174f92a07c40626e953bfec8c3c9c44ae94e2f68203348969b06654e69cf4f3
|
| 3 |
-
size 1734170359
|
|
|
|
|
|
|
|
|
|
|
|
gem_smpl/onnx/gem_smpl_denoiser_no_imgfeat.onnx
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:ad2255e3ec64f799a0191b4d92c45d5af21014ea71e3cff7923e3cb8312df914
|
| 3 |
-
size 1734171175
|
|
|
|
|
|
|
|
|
|
|
|
gem_smpl/onnx/hmr2.onnx
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:4f78023fd2abbaf26805b440c7e3c7a28187c920af389ddffc12e91c57a99f19
|
| 3 |
-
size 724939
|
|
|
|
|
|
|
|
|
|
|
|
gem_smpl/onnx/hmr2.onnx.data
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:9ae90b247a5b5bbd1f4fba781a67eab03aa62ae6331d2ca55d7363b1106612b1
|
| 3 |
-
size 2681192448
|
|
|
|
|
|
|
|
|
|
|
|
gem_smpl/onnx/vitpose_coco17.onnx
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:0c8d1cbf52ab358e5152b595b5ec0fe848d75c7b156291bc7af85d5f18c2bbcc
|
| 3 |
-
size 341207
|
|
|
|
|
|
|
|
|
|
|
|
gem_smpl/onnx/vitpose_coco17.onnx.data
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:6521118c8c1d9deb29285ffd5a06bf6ab140a33d93c82daab6de843f236c6286
|
| 3 |
-
size 2548839424
|
|
|
|
|
|
|
|
|
|
|
|
gem_smpl_config.json
DELETED
|
@@ -1,33 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"model_type": "gem",
|
| 3 |
-
"body_model": "smplx",
|
| 4 |
-
"denoiser": {
|
| 5 |
-
"latent_dim": 1024,
|
| 6 |
-
"num_layers": 16,
|
| 7 |
-
"num_heads": 8,
|
| 8 |
-
"mlp_ratio": 4,
|
| 9 |
-
"encode_text": false,
|
| 10 |
-
"encoded_text_dim": 1024,
|
| 11 |
-
"text_mask_prob": 0.1,
|
| 12 |
-
"music_mask_prob": 0.1,
|
| 13 |
-
"speech_mask_prob": 0.1,
|
| 14 |
-
"use_text_pos_enc": true
|
| 15 |
-
},
|
| 16 |
-
"endecoder": {
|
| 17 |
-
"stats_name": "MM_V1_AMASS_LOCAL_BEDLAM_CAM",
|
| 18 |
-
"encode_type": "gvhmr",
|
| 19 |
-
"feat_dim": 151,
|
| 20 |
-
"clip_std": true
|
| 21 |
-
},
|
| 22 |
-
"diffusion": {
|
| 23 |
-
"sampler": "ddim",
|
| 24 |
-
"noise_schedule": "cosine",
|
| 25 |
-
"test_timestep_respacing": "50",
|
| 26 |
-
"ddim_eta": 0.0,
|
| 27 |
-
"guidance_param": 2.5
|
| 28 |
-
},
|
| 29 |
-
"pipeline": {
|
| 30 |
-
"remove_fingers": false,
|
| 31 |
-
"extract_features": false
|
| 32 |
-
}
|
| 33 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
gem_soma.ckpt
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:4c1f85ca8c1e11e6588aead49fbc024bf660708def670043e0b537c101ee298e
|
| 3 |
-
size 541758499
|
|
|
|
|
|
|
|
|
|
|
|
hand_config.json
ADDED
|
@@ -0,0 +1,64 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"task": "hand",
|
| 3 |
+
"head": {
|
| 4 |
+
"num_finger_joints": 24,
|
| 5 |
+
"num_identity_coeffs": 5,
|
| 6 |
+
"num_scale_params": 27,
|
| 7 |
+
"transformer_dim": 1024,
|
| 8 |
+
"transformer_depth": 6,
|
| 9 |
+
"transformer_heads": 8,
|
| 10 |
+
"transformer_dim_head": 64,
|
| 11 |
+
"transformer_mlp_dim": 1024,
|
| 12 |
+
"transformer_context_dim": 1280,
|
| 13 |
+
"transformer_dropout": 0.0,
|
| 14 |
+
"transformer_emb_dropout": 0.0,
|
| 15 |
+
"ief_iters": 1,
|
| 16 |
+
"learnable_init": true,
|
| 17 |
+
"use_init_to_token": true,
|
| 18 |
+
"two_layer_heads": true,
|
| 19 |
+
"per_layer_predictions": true,
|
| 20 |
+
"cliff_cond_dim": 3,
|
| 21 |
+
"ray_cond_dim": 32,
|
| 22 |
+
"camera_scale_mode": "ste_clamp",
|
| 23 |
+
"camera_depth_mode": "log_lambda",
|
| 24 |
+
"init_global_scale": 1.0,
|
| 25 |
+
"init_camera_scalar": -1.8199350824986398,
|
| 26 |
+
"camera_scalar_std": 0.3735116511306386
|
| 27 |
+
},
|
| 28 |
+
"backbone": {
|
| 29 |
+
"name": "dinov3_vith16plus",
|
| 30 |
+
"drop_path_rate": 0.0
|
| 31 |
+
},
|
| 32 |
+
"conditioning": {
|
| 33 |
+
"camera_mode": "sam3db_perspective"
|
| 34 |
+
},
|
| 35 |
+
"backbone_input_crop_w": null,
|
| 36 |
+
"ray_pe_num_bands": 8,
|
| 37 |
+
"kp2d_translation": {
|
| 38 |
+
"enabled": true,
|
| 39 |
+
"topology": "separate",
|
| 40 |
+
"num_joints": 25,
|
| 41 |
+
"d_model": 256,
|
| 42 |
+
"feat_hw": [
|
| 43 |
+
16,
|
| 44 |
+
16
|
| 45 |
+
],
|
| 46 |
+
"decoder_depth": 3,
|
| 47 |
+
"num_heads": 8,
|
| 48 |
+
"presence_threshold": 0.5,
|
| 49 |
+
"negative_presence_weight": 1.0,
|
| 50 |
+
"min_points": 4,
|
| 51 |
+
"min_depth": 0.0001,
|
| 52 |
+
"damping": 1e-06,
|
| 53 |
+
"max_condition_number": 1000000.0
|
| 54 |
+
},
|
| 55 |
+
"pose_release_id": "gem-pose-hand-ga-s060000",
|
| 56 |
+
"inference": {
|
| 57 |
+
"hand_mirror": "full_image",
|
| 58 |
+
"camera_representation": "log_lambda",
|
| 59 |
+
"camera_log_value_bounds": [
|
| 60 |
+
-3.5,
|
| 61 |
+
3.0
|
| 62 |
+
]
|
| 63 |
+
}
|
| 64 |
+
}
|
mhr_model.pt
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:352e271a6c42729c68554ceaea0c955e866970160c31e35506d782dc0f7377bc
|
| 3 |
-
size 696110248
|
|
|
|
|
|
|
|
|
|
|
|
model_config.yaml
DELETED
|
@@ -1,73 +0,0 @@
|
|
| 1 |
-
TRAIN:
|
| 2 |
-
USE_FP16: true
|
| 3 |
-
FP16_TYPE: bfloat16
|
| 4 |
-
MODEL:
|
| 5 |
-
IMAGE_SIZE:
|
| 6 |
-
- 512
|
| 7 |
-
- 512
|
| 8 |
-
IMAGE_MEAN:
|
| 9 |
-
- 0.485
|
| 10 |
-
- 0.456
|
| 11 |
-
- 0.406
|
| 12 |
-
IMAGE_STD:
|
| 13 |
-
- 0.229
|
| 14 |
-
- 0.224
|
| 15 |
-
- 0.225
|
| 16 |
-
BACKBONE:
|
| 17 |
-
TYPE: dinov3_vith16plus
|
| 18 |
-
PRETRAINED_WEIGHTS: ''
|
| 19 |
-
FROZEN_STAGES: -1
|
| 20 |
-
DROP_PATH_RATE: 0.1
|
| 21 |
-
DECODER:
|
| 22 |
-
TYPE: sam
|
| 23 |
-
DIM: 1024
|
| 24 |
-
DEPTH: 6
|
| 25 |
-
HEADS: 8
|
| 26 |
-
MLP_DIM: 1024
|
| 27 |
-
DIM_HEAD: 64
|
| 28 |
-
LAYER_SCALE_INIT: 0.0
|
| 29 |
-
DROP_RATE: 0.0
|
| 30 |
-
ATTN_DROP_RATE: 0.0
|
| 31 |
-
DROP_PATH_RATE: 0.0
|
| 32 |
-
FFN_TYPE: origin
|
| 33 |
-
ENABLE_TWOWAY: false
|
| 34 |
-
REPEAT_PE: true
|
| 35 |
-
FROZEN: false
|
| 36 |
-
CONDITION_TYPE: cliff
|
| 37 |
-
USE_INTRIN_CENTER: true
|
| 38 |
-
DO_INTERM_PREDS: true
|
| 39 |
-
DO_INTERM_SUP: true
|
| 40 |
-
DO_KEYPOINT_TOKENS: true
|
| 41 |
-
DO_HAND_DETECT_TOKENS: true
|
| 42 |
-
KEYPOINT_TOKEN_UPDATE: v2
|
| 43 |
-
KEYPOINT_TOKEN_UPDATE_COORD_EMB_USE_MLP: true
|
| 44 |
-
DO_KEYPOINT3D_TOKENS: true
|
| 45 |
-
PROMPT_ENCODER:
|
| 46 |
-
ENABLE: true
|
| 47 |
-
MAX_NUM_CLICKS: 2
|
| 48 |
-
PROMPT_KEYPOINTS: mhr70
|
| 49 |
-
FROZEN: false
|
| 50 |
-
KEYPOINT_SAMPLER:
|
| 51 |
-
TYPE: v1
|
| 52 |
-
WORST_RATIO: 0.8
|
| 53 |
-
KEYBODY_RATIO: 0.8
|
| 54 |
-
NEGATIVE_RATIO: 0.1
|
| 55 |
-
DUMMY_RATIO: 0.1
|
| 56 |
-
DISTANCE_THRESH: 0.0001
|
| 57 |
-
MASK_EMBED_TYPE: v2
|
| 58 |
-
MASK_PROMPT: v1
|
| 59 |
-
PERSON_HEAD:
|
| 60 |
-
POSE_TYPE: mhr
|
| 61 |
-
CAMERA_ENABLE: true
|
| 62 |
-
CAMERA_TYPE: perspective
|
| 63 |
-
ZERO_POSE_INIT: true
|
| 64 |
-
ZERO_POSE_INIT_BODY_FACTOR: 1
|
| 65 |
-
MHR_HEAD:
|
| 66 |
-
MLP_DEPTH: 2
|
| 67 |
-
MLP_CHANNEL_DIV_FACTOR: 1
|
| 68 |
-
CAMERA_HEAD:
|
| 69 |
-
MLP_DEPTH: 2
|
| 70 |
-
MLP_CHANNEL_DIV_FACTOR: 1
|
| 71 |
-
DEFAULT_SCALE_FACTOR_HAND: 10
|
| 72 |
-
ENABLE_BODY: true
|
| 73 |
-
ENABLE_HAND: true
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
motion_native77.yaml
ADDED
|
@@ -0,0 +1,31 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Original native77 final775 Motion with original ViTPose and Body Pose features.
|
| 2 |
+
# For motion-cache, pass motion_native77_e774_s292175.ckpt with --checkpoint
|
| 3 |
+
# and motion_stats_raw.npz with --stats. The codec uses its trained 0.01 std floor.
|
| 4 |
+
network:
|
| 5 |
+
output_dim: 655
|
| 6 |
+
max_len: 120
|
| 7 |
+
cliffcam_dim: 3
|
| 8 |
+
cam_angvel_dim: 6
|
| 9 |
+
imgseq_dim: 1024
|
| 10 |
+
latent_dim: 512
|
| 11 |
+
num_layers: 12
|
| 12 |
+
num_heads: 8
|
| 13 |
+
mlp_ratio: 4.0
|
| 14 |
+
pred_cam_dim: 3
|
| 15 |
+
static_conf_dim: 6
|
| 16 |
+
dropout: 0.1
|
| 17 |
+
avgbeta: true
|
| 18 |
+
obs_num_joints: 77
|
| 19 |
+
sequence_static_slices:
|
| 20 |
+
- - 456
|
| 21 |
+
- 584
|
| 22 |
+
- - 584
|
| 23 |
+
- 640
|
| 24 |
+
condition_spec:
|
| 25 |
+
pose_release_id: gem-pose-body-s240000-v1:pose_tokens
|
| 26 |
+
observation_source: vitpose77-dinov3-huge-metrosim-256x192-v1
|
| 27 |
+
observation_joint_set: soma77_native_v1
|
| 28 |
+
time_options:
|
| 29 |
+
reference_fps: 30.0
|
| 30 |
+
default_fps: 30.0
|
| 31 |
+
fps_by_data_name: {}
|
gem_smpl/missing_hmr4d_support/inputs/3DPW/hmr4d_support/test_3dpw_vimo_labels.pt → motion_native77_e774_s292175.ckpt
RENAMED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c7790f4587e518bcea40cb8281c9be2233d37f6caa668835462542760dcf82aa
|
| 3 |
+
size 187134331
|
scale_mean.pth → motion_stats_raw.npz
RENAMED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b7719b634e0fc1f001005605d117d77448c8d4a110923149df58529c68112ed6
|
| 3 |
+
size 21320
|
onnx/gem_denoiser.onnx
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:20aab83c01bbd909a258ad0fa465458eda382c63dc80d56952b2ec952d6e192c
|
| 3 |
-
size 1508810
|
|
|
|
|
|
|
|
|
|
|
|
onnx/gem_denoiser.onnx.data
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:720c2400281b730574bf2a446c33ecd77bf494ac8161b081b09bb122b9fad2f8
|
| 3 |
-
size 199114752
|
|
|
|
|
|
|
|
|
|
|
|
onnx/gem_denoiser_no_imgfeat.onnx
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:65c576968bc9548b5c485522930b7985097b70a5e0154e5cd01241d6c806415f
|
| 3 |
-
size 187936462
|
|
|
|
|
|
|
|
|
|
|
|
onnx/gem_denoiser_no_imgfeat.onnx.data
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:5ca91bdf443dc5139335ecef83139572d24ca7765e9cc6dfdd6441ed44bd6278
|
| 3 |
-
size 1156259576
|
|
|
|
|
|
|
|
|
|
|
|
onnx/sam3db_backbone.onnx
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:e03fe953ff9762c3e96b67dcd78e52769e67798cd1f8bd82c4ef5dd7bde48b22
|
| 3 |
-
size 878918
|
|
|
|
|
|
|
|
|
|
|
|
onnx/sam3db_backbone.onnx.data
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:479e7da52d115344a3fd3e7711dae58176f68c723486e80ba2c3efc046f4dbc6
|
| 3 |
-
size 3362053120
|
|
|
|
|
|
|
|
|
|
|
|
sam3d_body.ckpt
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:b5a2f9d305dd02626b967aa2e86021fba07065df66ce7a7e00ffb9664f150abf
|
| 3 |
-
size 2109129346
|
|
|
|
|
|
|
|
|
|
|
|
scale_comps.pth
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:4c4861384b176fbe3d3bd7480e87ff8161f19553bf6ec1b718bc9b6bbb559cd6
|
| 3 |
-
size 8816
|
|
|
|
|
|
|
|
|
|
|
|