Publish hftrainer artifact and model card
Browse files- Mean.npy +3 -0
- README.md +139 -0
- Std.npy +3 -0
- denoiser.safetensors +3 -0
- mld_config.json +52 -0
- model_index.json +17 -0
- vae.safetensors +3 -0
Mean.npy
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f23187256a29c715a228b4e3439c14e308ec861759e320428f60fb26ae4b8a9e
|
| 3 |
+
size 1180
|
README.md
ADDED
|
@@ -0,0 +1,139 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
library_name: hftrainer
|
| 3 |
+
pipeline_tag: other
|
| 4 |
+
tags:
|
| 5 |
+
- motion-generation
|
| 6 |
+
- text-to-motion
|
| 7 |
+
- humanml3d
|
| 8 |
+
- mld
|
| 9 |
+
- latent-diffusion
|
| 10 |
+
license: other
|
| 11 |
+
---
|
| 12 |
+
|
| 13 |
+
<!-- This model card is synchronized from docs/model_zoo/mld.md by tools/sync_model_zoo_cards.py. -->
|
| 14 |
+
|
| 15 |
+
# MLD - Motion Latent Diffusion
|
| 16 |
+
|
| 17 |
+
Text-to-motion baseline integrated into the hftrainer Model Zoo. The
|
| 18 |
+
reproduction keeps the MLD motion VAE, latent denoiser, DDIM scheduler wiring,
|
| 19 |
+
and SentenceT5 text wrapper in the native `hftrainer` runtime, so inference no
|
| 20 |
+
longer imports the upstream repository.
|
| 21 |
+
|
| 22 |
+
| | |
|
| 23 |
+
|---|---|
|
| 24 |
+
| **Task** | Text-to-Motion (T2M) |
|
| 25 |
+
| **Bundle / Pipeline** | `MLDBundle` / `MLDPipeline` |
|
| 26 |
+
| **Motion representation** | **HumanML3D-263** (263-dim, 20 fps, 22 joints) |
|
| 27 |
+
| **Backbone** | MLD VAE + latent diffusion denoiser, default 50 DDIM steps |
|
| 28 |
+
| **Text encoder** | SentenceT5-Large (`sentence-transformers/sentence-t5-large`, frozen) |
|
| 29 |
+
| **Paper** | *Executing Your Commands via Motion Diffusion in Latent Space*, Chen et al., CVPR 2023 |
|
| 30 |
+
| **Original code** | https://github.com/ChenFengYe/motion-latent-diffusion |
|
| 31 |
+
|
| 32 |
+
---
|
| 33 |
+
|
| 34 |
+
## Weights
|
| 35 |
+
|
| 36 |
+
Current hftrainer artifact:
|
| 37 |
+
|
| 38 |
+
| Artifact | Location | Contents | Status |
|
| 39 |
+
|---|---|---|---|
|
| 40 |
+
| MLD HumanML3D | [`ZeyuLing/hftrainer-mld-humanml3d`](https://huggingface.co/ZeyuLing/hftrainer-mld-humanml3d) / `checkpoints/mld/humanml3d` | `vae.safetensors` + `denoiser.safetensors` + `mld_config.json` + `Mean.npy` / `Std.npy` | hftrainer artifact |
|
| 41 |
+
|
| 42 |
+
Load through the same `from_pretrained` surface as the other reproduced
|
| 43 |
+
baselines:
|
| 44 |
+
|
| 45 |
+
```python
|
| 46 |
+
from hftrainer.pipelines.mld import MLDPipeline
|
| 47 |
+
|
| 48 |
+
pipe = MLDPipeline.from_pretrained(
|
| 49 |
+
"ZeyuLing/hftrainer-mld-humanml3d",
|
| 50 |
+
device="cuda",
|
| 51 |
+
)
|
| 52 |
+
motions = pipe.infer_t2m(
|
| 53 |
+
["a person walks forward then sits down"],
|
| 54 |
+
[120],
|
| 55 |
+
num_inference_steps=50,
|
| 56 |
+
)
|
| 57 |
+
```
|
| 58 |
+
|
| 59 |
+
Package the artifact from the upstream Lightning checkpoint:
|
| 60 |
+
|
| 61 |
+
```bash
|
| 62 |
+
python3 scripts/eval/convert_mld_checkpoint.py \
|
| 63 |
+
--model_ckpt ref_repo/MotionLCM/experiments_t2m/mld_humanml/mld_humanml_v1.ckpt \
|
| 64 |
+
--out_dir checkpoints/mld/humanml3d
|
| 65 |
+
```
|
| 66 |
+
|
| 67 |
+
The frozen SentenceT5-Large encoder is resolved by name rather than duplicated
|
| 68 |
+
inside the artifact. For fully offline use, snapshot the text encoder into the
|
| 69 |
+
local Hugging Face cache before calling `from_pretrained`.
|
| 70 |
+
|
| 71 |
+
---
|
| 72 |
+
|
| 73 |
+
## Motion Representation
|
| 74 |
+
|
| 75 |
+
**HumanML3D-263**, the standard redundant T2M feature (Guo et al.), 20 fps,
|
| 76 |
+
22-joint SMPL skeleton. Per frame (263 dims):
|
| 77 |
+
|
| 78 |
+
| Slice | Dim | Meaning |
|
| 79 |
+
|---|---:|---|
|
| 80 |
+
| `root_rot_vel` | 1 | root angular velocity (about Y) |
|
| 81 |
+
| `root_lin_vel` | 2 | root linear velocity (XZ plane) |
|
| 82 |
+
| `root_y` | 1 | root height |
|
| 83 |
+
| `ric_data` | 63 | local joint positions (21x3) |
|
| 84 |
+
| `rot_data` | 126 | local joint rotations (21x6, continuous 6D) |
|
| 85 |
+
| `local_vel` | 66 | local joint velocities (22x3) |
|
| 86 |
+
| `foot_contact` | 4 | binary foot-contact labels |
|
| 87 |
+
|
| 88 |
+
MLD samples in latent space and decodes directly back to HumanML3D-263. Convert
|
| 89 |
+
to SMPL or MotionStreamer-272 only when a cross-representation evaluator needs
|
| 90 |
+
that space.
|
| 91 |
+
|
| 92 |
+
---
|
| 93 |
+
|
| 94 |
+
## Evaluation
|
| 95 |
+
|
| 96 |
+
Generation follows the official HumanML3D protocol: standard test split,
|
| 97 |
+
native 263-dim at 20 fps, first caption, and one prediction per test id.
|
| 98 |
+
|
| 99 |
+
```bash
|
| 100 |
+
python3 scripts/eval/mld_t2m_h3d263.py \
|
| 101 |
+
--data_root ref_repo/CondMDI/dataset/HumanML3D \
|
| 102 |
+
--model_path checkpoints/mld/humanml3d \
|
| 103 |
+
--num_inference_steps 50 \
|
| 104 |
+
--out_dir outputs/evaluation/t2m/humanml3d_official_test/hml263/mld
|
| 105 |
+
```
|
| 106 |
+
|
| 107 |
+
The full reproduction pipeline writes the canonical outputs:
|
| 108 |
+
|
| 109 |
+
| Representation | Canonical path |
|
| 110 |
+
|---|---|
|
| 111 |
+
| HML263 | `outputs/evaluation/t2m/humanml3d_official_test/hml263/mld` |
|
| 112 |
+
| SMPL motion_135 | `outputs/evaluation/t2m/humanml3d_official_test/motion135/mld` |
|
| 113 |
+
| MotionStreamer-272 | `outputs/evaluation/t2m/humanml3d_official_test/ms272/mld` |
|
| 114 |
+
|
| 115 |
+
Run the Taiji wrapper for full generation, conversion, and evaluators:
|
| 116 |
+
|
| 117 |
+
```bash
|
| 118 |
+
python3 scripts/submit/submit_mld_standard_pipeline_taiji.py \
|
| 119 |
+
--gpu A100 \
|
| 120 |
+
--num-gpus 8
|
| 121 |
+
```
|
| 122 |
+
|
| 123 |
+
Report current metrics from the generated evaluator JSONs under
|
| 124 |
+
`outputs/evaluation/t2m/humanml3d_official_test/_runs/<run>/metrics/`.
|
| 125 |
+
|
| 126 |
+
---
|
| 127 |
+
|
| 128 |
+
## Implementation Notes
|
| 129 |
+
|
| 130 |
+
- **hftrainer-native runtime**: `hftrainer.models.motion.mld` wraps the shared
|
| 131 |
+
native MLD VAE / denoiser / SentenceT5 components and does not import
|
| 132 |
+
`ref_repo` at inference time.
|
| 133 |
+
- **Scheduler**: MLD uses `diffusers.DDIMScheduler` with 50 inference steps by
|
| 134 |
+
default (`eta=0.0`, `steps_offset=1`), matching the official inference config.
|
| 135 |
+
- **Classifier-free guidance**: the denoiser has no LCM `time_cond_proj`, so
|
| 136 |
+
guidance uses the standard unconditional/conditional two-pass batch.
|
| 137 |
+
- **Normalization travels with the checkpoint**: `Mean.npy` / `Std.npy` are
|
| 138 |
+
embedded in the artifact to avoid evaluator drift caused by mismatched
|
| 139 |
+
HumanML3D statistics.
|
Std.npy
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b7e5be77800bb2fd4da3badd5e7e018aaa654dc103f867c2a2300ba1cffada64
|
| 3 |
+
size 1180
|
denoiser.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f9dcdd5aca52421ed7b7a5dfb65f24fd18ac2d2222342c2187b94d58f399d4a3
|
| 3 |
+
size 32898016
|
mld_config.json
ADDED
|
@@ -0,0 +1,52 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model_type": "mld",
|
| 3 |
+
"text_encoder_name": "sentence-transformers/sentence-t5-large",
|
| 4 |
+
"guidance_scale": 7.5,
|
| 5 |
+
"num_inference_steps": 50,
|
| 6 |
+
"config": {
|
| 7 |
+
"vae": {
|
| 8 |
+
"nfeats": 263,
|
| 9 |
+
"latent_dim": [
|
| 10 |
+
1,
|
| 11 |
+
256
|
| 12 |
+
],
|
| 13 |
+
"ff_size": 1024,
|
| 14 |
+
"num_layers": 9,
|
| 15 |
+
"num_heads": 4,
|
| 16 |
+
"dropout": 0.1,
|
| 17 |
+
"arch": "encoder_decoder",
|
| 18 |
+
"normalize_before": false,
|
| 19 |
+
"activation": "gelu",
|
| 20 |
+
"position_embedding": "learned"
|
| 21 |
+
},
|
| 22 |
+
"denoiser": {
|
| 23 |
+
"latent_dim": [
|
| 24 |
+
1,
|
| 25 |
+
256
|
| 26 |
+
],
|
| 27 |
+
"ff_size": 1024,
|
| 28 |
+
"num_layers": 9,
|
| 29 |
+
"num_heads": 4,
|
| 30 |
+
"dropout": 0.1,
|
| 31 |
+
"normalize_before": false,
|
| 32 |
+
"activation": "gelu",
|
| 33 |
+
"flip_sin_to_cos": true,
|
| 34 |
+
"return_intermediate_dec": false,
|
| 35 |
+
"position_embedding": "learned",
|
| 36 |
+
"arch": "trans_enc",
|
| 37 |
+
"freq_shift": 0,
|
| 38 |
+
"text_encoded_dim": 768,
|
| 39 |
+
"time_cond_proj_dim": null
|
| 40 |
+
},
|
| 41 |
+
"scheduler": {
|
| 42 |
+
"num_train_timesteps": 1000,
|
| 43 |
+
"beta_start": 0.00085,
|
| 44 |
+
"beta_end": 0.012,
|
| 45 |
+
"beta_schedule": "scaled_linear",
|
| 46 |
+
"clip_sample": false,
|
| 47 |
+
"set_alpha_to_one": false,
|
| 48 |
+
"steps_offset": 1,
|
| 49 |
+
"eta": 0.0
|
| 50 |
+
}
|
| 51 |
+
}
|
| 52 |
+
}
|
model_index.json
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model_type": "mld",
|
| 3 |
+
"library_name": "hftrainer",
|
| 4 |
+
"pipeline": "hftrainer.pipelines.mld.MLDPipeline",
|
| 5 |
+
"bundle": "hftrainer.models.motion.mld.MLDBundle",
|
| 6 |
+
"task": "text-to-motion",
|
| 7 |
+
"dataset": "HumanML3D",
|
| 8 |
+
"representation": "HumanML3D-263",
|
| 9 |
+
"artifact_format": "hftrainer-mld-v1",
|
| 10 |
+
"files": [
|
| 11 |
+
"mld_config.json",
|
| 12 |
+
"vae.safetensors",
|
| 13 |
+
"denoiser.safetensors",
|
| 14 |
+
"Mean.npy",
|
| 15 |
+
"Std.npy"
|
| 16 |
+
]
|
| 17 |
+
}
|
vae.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e5bde62cadf3a05336f7671aad290a4a49cc34aab77efa36f7859a6a67c18600
|
| 3 |
+
size 72160708
|