Release unified Body Pose, Hand Pose and native77 Motion (2026-10-20)

#1
.gitattributes CHANGED
@@ -33,9 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
- onnx/gem_denoiser.onnx.data filter=lfs diff=lfs merge=lfs -text
37
- onnx/gem_denoiser_no_imgfeat.onnx.data filter=lfs diff=lfs merge=lfs -text
38
  onnx/vitpose.onnx.data filter=lfs diff=lfs merge=lfs -text
39
- onnx/sam3db_backbone.onnx.data filter=lfs diff=lfs merge=lfs -text
40
- gem_smpl/onnx/vitpose_coco17.onnx.data filter=lfs diff=lfs merge=lfs -text
41
- gem_smpl/onnx/hmr2.onnx.data filter=lfs diff=lfs merge=lfs -text
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
36
  onnx/vitpose.onnx.data filter=lfs diff=lfs merge=lfs -text
 
 
 
README.md CHANGED
@@ -22,6 +22,10 @@ All training data and the SOMA parametric body model are owned by NVIDIA or rele
22
  GEM was developed by NVIDIA.
23
  *This model is ready for commercial or non-commercial use.*
24
 
 
 
 
 
25
  ### License/Terms of Use:
26
 
27
  Use of the source code is governed by the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0). Use of the associated model is governed by the [NVIDIA Open Model License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/).
@@ -53,9 +57,9 @@ GEM is intended for use by individuals and professionals in fields such as machi
53
  ## Model Architecture:
54
 
55
  **Architecture Type:** Transformer
56
- **Network Architecture:** 12-layer Transformer Decoder
57
  **This model was developed based on Not Applicable (N/A).**
58
- **Number of model parameters:** 520M (5.2*10^8)
59
 
60
  ## Computational Load
61
 
@@ -68,7 +72,7 @@ GEM is intended for use by individuals and professionals in fields such as machi
68
 
69
  **Input Format(s):**
70
 
71
- - Video: RGB tensor `(T, H, W, 3)`, any resolution, internally processed via the frozen video encoder
72
  - Other — person bounding box: `(T, 4)` float array in `xyxy` pixel coordinates
73
  - Other — camera intrinsics: `(T, 3, 3)` float array of per-frame camera intrinsic matrices containing focal lengths and principal-point coordinates
74
 
@@ -108,6 +112,42 @@ GEM v1.1
108
  An installable Python package provides the `gem-infer`, `gem-render`, `gem-train`, and `gem-eval` entry points. Body Motion inference uses `gem-infer motion-video --task body` for RGB video or `gem-infer motion-cache --task body` for prepared conditions. The package requires Python 3.10 or newer. The validated CUDA runtime uses PyTorch 2.10 and SOMA 0.3.3 with separately supplied model weights and geometry assets. Mesh/video rendering uses the documented x86/Open3D 0.19 environment.
109
 
110
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
111
  ## Training, Testing, and Evaluation Datasets:
112
 
113
 
@@ -147,7 +187,7 @@ An installable Python package provides the `gem-infer`, `gem-render`, `gem-train
147
 
148
  ## Inference:
149
 
150
- **Acceleration Engine:** TensorRT, PyTorch
151
  **Hardware Requirements (GPU Architecture, Model):** NVIDIA Ampere A100 80GB
152
 
153
  ## Ethical Considerations:
 
22
  GEM was developed by NVIDIA.
23
  *This model is ready for commercial or non-commercial use.*
24
 
25
+ This card describes temporal Body Motion. The repository also distributes
26
+ separate Body Pose and Hand Pose checkpoints for image inputs; the checkpoint
27
+ bundle below identifies their configurations and roles.
28
+
29
  ### License/Terms of Use:
30
 
31
  Use of the source code is governed by the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0). Use of the associated model is governed by the [NVIDIA Open Model License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/).
 
57
  ## Model Architecture:
58
 
59
  **Architecture Type:** Transformer
60
+ **Network Architecture:** 12-layer temporal Transformer, hidden size 512, 8 attention heads<br>
61
  **This model was developed based on Not Applicable (N/A).**
62
+ **Number of model parameters:** 46.8M total temporal Motion parameters (46,762,136), including 44.2M trainable parameters (44,202,136). This excludes the separate Body Pose feature extractor, Hand Pose model, and detectors.
63
 
64
  ## Computational Load
65
 
 
72
 
73
  **Input Format(s):**
74
 
75
+ - Video: RGB tensor `(T, H, W, 3)`, any resolution, internally processed via the frozen Body Pose image encoder
76
  - Other — person bounding box: `(T, 4)` float array in `xyxy` pixel coordinates
77
  - Other — camera intrinsics: `(T, 3, 3)` float array of per-frame camera intrinsic matrices containing focal lengths and principal-point coordinates
78
 
 
112
  An installable Python package provides the `gem-infer`, `gem-render`, `gem-train`, and `gem-eval` entry points. Body Motion inference uses `gem-infer motion-video --task body` for RGB video or `gem-infer motion-cache --task body` for prepared conditions. The package requires Python 3.10 or newer. The validated CUDA runtime uses PyTorch 2.10 and SOMA 0.3.3 with separately supplied model weights and geometry assets. Mesh/video rendering uses the documented x86/Open3D 0.19 environment.
113
 
114
 
115
+ ## Pretrained Checkpoint Bundle
116
+
117
+ The unified release provides three model checkpoints with matching public
118
+ configurations. Body and Hand exports contain complete model weights for
119
+ inference and fine-tuning initialization; optimizer and training-loop state are
120
+ not included, so these files are not full training-resume checkpoints.
121
+
122
+ | Component | Weights | Configuration / companion files |
123
+ | --- | --- | --- |
124
+ | Body Pose | [body_s240000.ckpt](https://huggingface.co/nvidia/GEM-X/resolve/main/body_s240000.ckpt) | [body_config.json](https://huggingface.co/nvidia/GEM-X/blob/main/body_config.json); preset `body` |
125
+ | Hand Pose | [egohand_s060000.ckpt](https://huggingface.co/nvidia/GEM-X/resolve/main/egohand_s060000.ckpt) | [hand_config.json](https://huggingface.co/nvidia/GEM-X/blob/main/hand_config.json); preset `hand`; includes the learned RLE density used for fine-tuning |
126
+ | Body Motion native77 | [motion_native77_e774_s292175.ckpt](https://huggingface.co/nvidia/GEM-X/resolve/main/motion_native77_e774_s292175.ckpt) | [motion_stats_raw.npz](https://huggingface.co/nvidia/GEM-X/resolve/main/motion_stats_raw.npz) and [motion_native77.yaml](https://huggingface.co/nvidia/GEM-X/blob/main/motion_native77.yaml) |
127
+ | Original native77 ViTPose | [vitpose.pth](https://huggingface.co/nvidia/GEM-X/resolve/main/vitpose.pth) | Used by [body_video_native77.yaml](https://huggingface.co/nvidia/GEM-X/blob/main/body_video_native77.yaml) |
128
+
129
+ The video recipe pairs the original final775 Motion weights with Body Pose
130
+ s240000 features and the existing original ViTPose detector. The experimental
131
+ random-blur epoch10 detector is not included in this release bundle.
132
+
133
+ Download the model files together, preserving their relative placement:
134
+
135
+ ```bash
136
+ hf download nvidia/GEM-X \
137
+ body_s240000.ckpt body_config.json \
138
+ egohand_s060000.ckpt hand_config.json \
139
+ motion_native77_e774_s292175.ckpt motion_stats_raw.npz \
140
+ motion_native77.yaml body_video_native77.yaml vitpose.pth \
141
+ --local-dir weights
142
+ ```
143
+
144
+ The video YAML resolves checkpoint paths relative to its own directory. Obtain
145
+ the YOLOX HumanArt person detector separately as `weights/yolox_humanart.onnx`,
146
+ or change that path in the YAML. SOMA 0.3.3 geometry assets and an optional
147
+ RT-DETR Hand detector are also supplied separately. Follow the
148
+ [GEM-X installation and inference guides](https://github.com/NVlabs/GEM-X)
149
+ for these prerequisites and the unified command-line interface.
150
+
151
  ## Training, Testing, and Evaluation Datasets:
152
 
153
 
 
187
 
188
  ## Inference:
189
 
190
+ **Acceleration Engine:** PyTorch with CUDA
191
  **Hardware Requirements (GPU Architecture, Model):** NVIDIA Ampere A100 80GB
192
 
193
  ## Ethical Considerations:
body_config.json ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "task": "body",
3
+ "head": {
4
+ "num_body_joints": 76,
5
+ "num_identity_coeffs": 128,
6
+ "num_scale_params": 57,
7
+ "scale_comps_dim": 0,
8
+ "scale_param_mode": "direct",
9
+ "scale_parameterization": "packed",
10
+ "fixed_global_scale": 1.0,
11
+ "transformer_dim": 1024,
12
+ "transformer_depth": 6,
13
+ "transformer_heads": 8,
14
+ "transformer_dim_head": 64,
15
+ "transformer_mlp_dim": 1024,
16
+ "transformer_context_dim": 1280,
17
+ "transformer_dropout": 0.0,
18
+ "transformer_emb_dropout": 0.0,
19
+ "ief_iters": 1,
20
+ "learnable_init": true,
21
+ "use_init_to_token": true,
22
+ "two_layer_heads": true,
23
+ "per_layer_predictions": true,
24
+ "cliff_cond_dim": 3,
25
+ "camera_scale_mode": "ste_clamp"
26
+ },
27
+ "backbone": {
28
+ "name": "dinov3_vith16plus",
29
+ "drop_path_rate": 0.1
30
+ },
31
+ "conditioning": {
32
+ "camera_mode": "sam3db_perspective"
33
+ },
34
+ "backbone_input_crop_w": null,
35
+ "ray_pe_num_bands": null,
36
+ "kp2d_translation": null,
37
+ "pose_release_id": "gem-pose-body-s240000-v1",
38
+ "inference": {
39
+ "camera_representation": "legacy",
40
+ "minimum_depth": 0.05
41
+ }
42
+ }
gem_smpl.ckpt → body_s240000.ckpt RENAMED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:1d15cbe2864d6de61a75e83fdbfe83bec3c7b183eee3d3dcdbd9107e4456454a
3
- size 5518923941
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e244d27f2834eb6382c5f6b9cbf2a38681b685514a1b2e86f8f3d2592d614959
3
+ size 3538753341
body_video_native77.yaml ADDED
@@ -0,0 +1,48 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Original native77 final775 Motion with original ViTPose and Body Pose features.
2
+ # Checkpoint and statistics paths resolve relative to this file.
3
+ # Obtain yolox_humanart.onnx separately and place it here or change its path.
4
+ # Keep motion_stats_raw.npz; the codec uses its trained 0.01 std floor.
5
+ pose:
6
+ checkpoint: body_s240000.ckpt
7
+ release: body
8
+ batch_size: 4
9
+ observations:
10
+ checkpoint: vitpose.pth
11
+ release_id: vitpose77-dinov3-huge-metrosim-256x192-v1
12
+ batch_size: 4
13
+ detector:
14
+ checkpoint: yolox_humanart.onnx
15
+ device: cpu
16
+ motion:
17
+ checkpoint: motion_native77_e774_s292175.ckpt
18
+ stats: motion_stats_raw.npz
19
+ network:
20
+ output_dim: 655
21
+ max_len: 120
22
+ cliffcam_dim: 3
23
+ cam_angvel_dim: 6
24
+ imgseq_dim: 1024
25
+ latent_dim: 512
26
+ num_layers: 12
27
+ num_heads: 8
28
+ mlp_ratio: 4.0
29
+ pred_cam_dim: 3
30
+ static_conf_dim: 6
31
+ dropout: 0.1
32
+ avgbeta: true
33
+ obs_num_joints: 77
34
+ sequence_static_slices:
35
+ - - 456
36
+ - 584
37
+ - - 584
38
+ - 640
39
+ condition_spec:
40
+ pose_release_id: gem-pose-body-s240000-v1:pose_tokens
41
+ observation_source: vitpose77-dinov3-huge-metrosim-256x192-v1
42
+ observation_joint_set: soma77_native_v1
43
+ time_options:
44
+ reference_fps: 30.0
45
+ default_fps: 30.0
46
+ fps_by_data_name: {}
47
+ camera:
48
+ mode: background
config.json DELETED
@@ -1,40 +0,0 @@
1
- {
2
- "model_type": "gem",
3
- "body_model": "soma",
4
- "denoiser": {
5
- "latent_dim": 1024,
6
- "num_layers": 16,
7
- "num_heads": 8,
8
- "mlp_ratio": 4,
9
- "scale_params_dim": 69,
10
- "scale_comps_dim": 28,
11
- "fix_scale_comps": true,
12
- "encode_text": false,
13
- "encoded_text_dim": 4096,
14
- "text_encoder_cfg": {
15
- "mode": "none",
16
- "cross_attn_type": "mha"
17
- },
18
- "text_mask_prob": 0.1,
19
- "music_mask_prob": 0.1,
20
- "speech_mask_prob": 0.1,
21
- "use_text_pos_enc": true
22
- },
23
- "endecoder": {
24
- "stats_name": "MM_V2_SOMA_METROSIM",
25
- "encode_type": "soma_v2",
26
- "feat_dim": 585,
27
- "clip_std": true
28
- },
29
- "diffusion": {
30
- "sampler": "ddim",
31
- "noise_schedule": "cosine",
32
- "test_timestep_respacing": "50",
33
- "ddim_eta": 0.0,
34
- "guidance_param": 2.5
35
- },
36
- "pipeline": {
37
- "remove_fingers": true,
38
- "extract_features": false
39
- }
40
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
gem_smpl/missing_hmr4d_support/inputs/EMDB/hmr4d_support/emdb_slam_traj.pt → egohand_s060000.ckpt RENAMED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:0f760c4327b2cec66dae849b6345f2730f66f338285609c831d45d74560aa2b6
3
- size 5042492
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:af56925fa08b7736787346c95b6bfe4fe7b857cb81e5803c336f72fb114eb900
3
+ size 3552688469
gem_smpl/missing_hmr4d_support/README_GEM_missing_files.txt DELETED
@@ -1,19 +0,0 @@
1
- GEM missing hmr4d_support files
2
-
3
- This folder contains supplemental EMDB / RICH / 3DPW support files for GEM.
4
-
5
- Copy or merge the `inputs/` directory in this folder into the root of the
6
- GEM repository. After copying, the files should be located at:
7
-
8
- inputs/RICH/hmr4d_support/rich_test_vimo_preproc.pt
9
- inputs/EMDB/hmr4d_support/emdb_vimo.pt
10
- inputs/EMDB/hmr4d_support/emdb_slam_traj.pt
11
- inputs/3DPW/hmr4d_support/test_3dpw_vimo_labels.pt
12
- inputs/3DPW/hmr4d_support/3dpw_test_slam_traj.pt
13
-
14
- These paths are the ones used by:
15
-
16
- gem/datasets/rich/rich_motion_test.py
17
- gem/datasets/emdb/emdb_motion_test.py
18
- gem/datasets/threedpw/threedpw_motion_test.py
19
- gem/datasets/threedpw/threedpw_occ_motion_test.py
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
gem_smpl/missing_hmr4d_support/inputs/3DPW/hmr4d_support/3dpw_test_slam_traj.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:16340d0f2cf1453762163082f327d0a3cebd1488399dff71e60e1d001e53d249
3
- size 3271056
 
 
 
 
gem_smpl/missing_hmr4d_support/inputs/EMDB/hmr4d_support/emdb_vimo.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:c40713cf7b57e76a645edaa43c32d46aa6814daedaafbadaf1db16f5881c06e7
3
- size 999364858
 
 
 
 
gem_smpl/missing_hmr4d_support/inputs/RICH/hmr4d_support/rich_test_vimo_preproc.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:4506ad4b0663b7933e4679f367b23067a06671f35c0c59a0bf1a9418a894296d
3
- size 1653984970
 
 
 
 
gem_smpl/onnx/gem_smpl_denoiser.onnx DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:f174f92a07c40626e953bfec8c3c9c44ae94e2f68203348969b06654e69cf4f3
3
- size 1734170359
 
 
 
 
gem_smpl/onnx/gem_smpl_denoiser_no_imgfeat.onnx DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:ad2255e3ec64f799a0191b4d92c45d5af21014ea71e3cff7923e3cb8312df914
3
- size 1734171175
 
 
 
 
gem_smpl/onnx/hmr2.onnx DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:4f78023fd2abbaf26805b440c7e3c7a28187c920af389ddffc12e91c57a99f19
3
- size 724939
 
 
 
 
gem_smpl/onnx/hmr2.onnx.data DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:9ae90b247a5b5bbd1f4fba781a67eab03aa62ae6331d2ca55d7363b1106612b1
3
- size 2681192448
 
 
 
 
gem_smpl/onnx/vitpose_coco17.onnx DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:0c8d1cbf52ab358e5152b595b5ec0fe848d75c7b156291bc7af85d5f18c2bbcc
3
- size 341207
 
 
 
 
gem_smpl/onnx/vitpose_coco17.onnx.data DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:6521118c8c1d9deb29285ffd5a06bf6ab140a33d93c82daab6de843f236c6286
3
- size 2548839424
 
 
 
 
gem_smpl_config.json DELETED
@@ -1,33 +0,0 @@
1
- {
2
- "model_type": "gem",
3
- "body_model": "smplx",
4
- "denoiser": {
5
- "latent_dim": 1024,
6
- "num_layers": 16,
7
- "num_heads": 8,
8
- "mlp_ratio": 4,
9
- "encode_text": false,
10
- "encoded_text_dim": 1024,
11
- "text_mask_prob": 0.1,
12
- "music_mask_prob": 0.1,
13
- "speech_mask_prob": 0.1,
14
- "use_text_pos_enc": true
15
- },
16
- "endecoder": {
17
- "stats_name": "MM_V1_AMASS_LOCAL_BEDLAM_CAM",
18
- "encode_type": "gvhmr",
19
- "feat_dim": 151,
20
- "clip_std": true
21
- },
22
- "diffusion": {
23
- "sampler": "ddim",
24
- "noise_schedule": "cosine",
25
- "test_timestep_respacing": "50",
26
- "ddim_eta": 0.0,
27
- "guidance_param": 2.5
28
- },
29
- "pipeline": {
30
- "remove_fingers": false,
31
- "extract_features": false
32
- }
33
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
gem_soma.ckpt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:4c1f85ca8c1e11e6588aead49fbc024bf660708def670043e0b537c101ee298e
3
- size 541758499
 
 
 
 
hand_config.json ADDED
@@ -0,0 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "task": "hand",
3
+ "head": {
4
+ "num_finger_joints": 24,
5
+ "num_identity_coeffs": 5,
6
+ "num_scale_params": 27,
7
+ "transformer_dim": 1024,
8
+ "transformer_depth": 6,
9
+ "transformer_heads": 8,
10
+ "transformer_dim_head": 64,
11
+ "transformer_mlp_dim": 1024,
12
+ "transformer_context_dim": 1280,
13
+ "transformer_dropout": 0.0,
14
+ "transformer_emb_dropout": 0.0,
15
+ "ief_iters": 1,
16
+ "learnable_init": true,
17
+ "use_init_to_token": true,
18
+ "two_layer_heads": true,
19
+ "per_layer_predictions": true,
20
+ "cliff_cond_dim": 3,
21
+ "ray_cond_dim": 32,
22
+ "camera_scale_mode": "ste_clamp",
23
+ "camera_depth_mode": "log_lambda",
24
+ "init_global_scale": 1.0,
25
+ "init_camera_scalar": -1.8199350824986398,
26
+ "camera_scalar_std": 0.3735116511306386
27
+ },
28
+ "backbone": {
29
+ "name": "dinov3_vith16plus",
30
+ "drop_path_rate": 0.0
31
+ },
32
+ "conditioning": {
33
+ "camera_mode": "sam3db_perspective"
34
+ },
35
+ "backbone_input_crop_w": null,
36
+ "ray_pe_num_bands": 8,
37
+ "kp2d_translation": {
38
+ "enabled": true,
39
+ "topology": "separate",
40
+ "num_joints": 25,
41
+ "d_model": 256,
42
+ "feat_hw": [
43
+ 16,
44
+ 16
45
+ ],
46
+ "decoder_depth": 3,
47
+ "num_heads": 8,
48
+ "presence_threshold": 0.5,
49
+ "negative_presence_weight": 1.0,
50
+ "min_points": 4,
51
+ "min_depth": 0.0001,
52
+ "damping": 1e-06,
53
+ "max_condition_number": 1000000.0
54
+ },
55
+ "pose_release_id": "gem-pose-hand-ga-s060000",
56
+ "inference": {
57
+ "hand_mirror": "full_image",
58
+ "camera_representation": "log_lambda",
59
+ "camera_log_value_bounds": [
60
+ -3.5,
61
+ 3.0
62
+ ]
63
+ }
64
+ }
mhr_model.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:352e271a6c42729c68554ceaea0c955e866970160c31e35506d782dc0f7377bc
3
- size 696110248
 
 
 
 
model_config.yaml DELETED
@@ -1,73 +0,0 @@
1
- TRAIN:
2
- USE_FP16: true
3
- FP16_TYPE: bfloat16
4
- MODEL:
5
- IMAGE_SIZE:
6
- - 512
7
- - 512
8
- IMAGE_MEAN:
9
- - 0.485
10
- - 0.456
11
- - 0.406
12
- IMAGE_STD:
13
- - 0.229
14
- - 0.224
15
- - 0.225
16
- BACKBONE:
17
- TYPE: dinov3_vith16plus
18
- PRETRAINED_WEIGHTS: ''
19
- FROZEN_STAGES: -1
20
- DROP_PATH_RATE: 0.1
21
- DECODER:
22
- TYPE: sam
23
- DIM: 1024
24
- DEPTH: 6
25
- HEADS: 8
26
- MLP_DIM: 1024
27
- DIM_HEAD: 64
28
- LAYER_SCALE_INIT: 0.0
29
- DROP_RATE: 0.0
30
- ATTN_DROP_RATE: 0.0
31
- DROP_PATH_RATE: 0.0
32
- FFN_TYPE: origin
33
- ENABLE_TWOWAY: false
34
- REPEAT_PE: true
35
- FROZEN: false
36
- CONDITION_TYPE: cliff
37
- USE_INTRIN_CENTER: true
38
- DO_INTERM_PREDS: true
39
- DO_INTERM_SUP: true
40
- DO_KEYPOINT_TOKENS: true
41
- DO_HAND_DETECT_TOKENS: true
42
- KEYPOINT_TOKEN_UPDATE: v2
43
- KEYPOINT_TOKEN_UPDATE_COORD_EMB_USE_MLP: true
44
- DO_KEYPOINT3D_TOKENS: true
45
- PROMPT_ENCODER:
46
- ENABLE: true
47
- MAX_NUM_CLICKS: 2
48
- PROMPT_KEYPOINTS: mhr70
49
- FROZEN: false
50
- KEYPOINT_SAMPLER:
51
- TYPE: v1
52
- WORST_RATIO: 0.8
53
- KEYBODY_RATIO: 0.8
54
- NEGATIVE_RATIO: 0.1
55
- DUMMY_RATIO: 0.1
56
- DISTANCE_THRESH: 0.0001
57
- MASK_EMBED_TYPE: v2
58
- MASK_PROMPT: v1
59
- PERSON_HEAD:
60
- POSE_TYPE: mhr
61
- CAMERA_ENABLE: true
62
- CAMERA_TYPE: perspective
63
- ZERO_POSE_INIT: true
64
- ZERO_POSE_INIT_BODY_FACTOR: 1
65
- MHR_HEAD:
66
- MLP_DEPTH: 2
67
- MLP_CHANNEL_DIV_FACTOR: 1
68
- CAMERA_HEAD:
69
- MLP_DEPTH: 2
70
- MLP_CHANNEL_DIV_FACTOR: 1
71
- DEFAULT_SCALE_FACTOR_HAND: 10
72
- ENABLE_BODY: true
73
- ENABLE_HAND: true
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
motion_native77.yaml ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Original native77 final775 Motion with original ViTPose and Body Pose features.
2
+ # For motion-cache, pass motion_native77_e774_s292175.ckpt with --checkpoint
3
+ # and motion_stats_raw.npz with --stats. The codec uses its trained 0.01 std floor.
4
+ network:
5
+ output_dim: 655
6
+ max_len: 120
7
+ cliffcam_dim: 3
8
+ cam_angvel_dim: 6
9
+ imgseq_dim: 1024
10
+ latent_dim: 512
11
+ num_layers: 12
12
+ num_heads: 8
13
+ mlp_ratio: 4.0
14
+ pred_cam_dim: 3
15
+ static_conf_dim: 6
16
+ dropout: 0.1
17
+ avgbeta: true
18
+ obs_num_joints: 77
19
+ sequence_static_slices:
20
+ - - 456
21
+ - 584
22
+ - - 584
23
+ - 640
24
+ condition_spec:
25
+ pose_release_id: gem-pose-body-s240000-v1:pose_tokens
26
+ observation_source: vitpose77-dinov3-huge-metrosim-256x192-v1
27
+ observation_joint_set: soma77_native_v1
28
+ time_options:
29
+ reference_fps: 30.0
30
+ default_fps: 30.0
31
+ fps_by_data_name: {}
gem_smpl/missing_hmr4d_support/inputs/3DPW/hmr4d_support/test_3dpw_vimo_labels.pt → motion_native77_e774_s292175.ckpt RENAMED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:8fd8a8d1a5a10804257c4fac30c0215c32757aa995d5576b73380eda29f3fe35
3
- size 466754526
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c7790f4587e518bcea40cb8281c9be2233d37f6caa668835462542760dcf82aa
3
+ size 187134331
scale_mean.pth → motion_stats_raw.npz RENAMED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:f696a0793217f27fcf23fa1fd63e8a34aa74ef64580078698fb9ea7322ce2891
3
- size 1451
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b7719b634e0fc1f001005605d117d77448c8d4a110923149df58529c68112ed6
3
+ size 21320
onnx/gem_denoiser.onnx DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:20aab83c01bbd909a258ad0fa465458eda382c63dc80d56952b2ec952d6e192c
3
- size 1508810
 
 
 
 
onnx/gem_denoiser.onnx.data DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:720c2400281b730574bf2a446c33ecd77bf494ac8161b081b09bb122b9fad2f8
3
- size 199114752
 
 
 
 
onnx/gem_denoiser_no_imgfeat.onnx DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:65c576968bc9548b5c485522930b7985097b70a5e0154e5cd01241d6c806415f
3
- size 187936462
 
 
 
 
onnx/gem_denoiser_no_imgfeat.onnx.data DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:5ca91bdf443dc5139335ecef83139572d24ca7765e9cc6dfdd6441ed44bd6278
3
- size 1156259576
 
 
 
 
onnx/sam3db_backbone.onnx DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:e03fe953ff9762c3e96b67dcd78e52769e67798cd1f8bd82c4ef5dd7bde48b22
3
- size 878918
 
 
 
 
onnx/sam3db_backbone.onnx.data DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:479e7da52d115344a3fd3e7711dae58176f68c723486e80ba2c3efc046f4dbc6
3
- size 3362053120
 
 
 
 
sam3d_body.ckpt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:b5a2f9d305dd02626b967aa2e86021fba07065df66ce7a7e00ffb9664f150abf
3
- size 2109129346
 
 
 
 
scale_comps.pth DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:4c4861384b176fbe3d3bd7480e87ff8161f19553bf6ec1b718bc9b6bbb559cd6
3
- size 8816