ZeyuLing commited on
Commit
3dc059b
·
verified ·
1 Parent(s): 57720ef

Publish hftrainer artifact and model card

Browse files
Files changed (7) hide show
  1. Mean.npy +3 -0
  2. README.md +139 -0
  3. Std.npy +3 -0
  4. denoiser.safetensors +3 -0
  5. mld_config.json +52 -0
  6. model_index.json +17 -0
  7. vae.safetensors +3 -0
Mean.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f23187256a29c715a228b4e3439c14e308ec861759e320428f60fb26ae4b8a9e
3
+ size 1180
README.md ADDED
@@ -0,0 +1,139 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: hftrainer
3
+ pipeline_tag: other
4
+ tags:
5
+ - motion-generation
6
+ - text-to-motion
7
+ - humanml3d
8
+ - mld
9
+ - latent-diffusion
10
+ license: other
11
+ ---
12
+
13
+ <!-- This model card is synchronized from docs/model_zoo/mld.md by tools/sync_model_zoo_cards.py. -->
14
+
15
+ # MLD - Motion Latent Diffusion
16
+
17
+ Text-to-motion baseline integrated into the hftrainer Model Zoo. The
18
+ reproduction keeps the MLD motion VAE, latent denoiser, DDIM scheduler wiring,
19
+ and SentenceT5 text wrapper in the native `hftrainer` runtime, so inference no
20
+ longer imports the upstream repository.
21
+
22
+ | | |
23
+ |---|---|
24
+ | **Task** | Text-to-Motion (T2M) |
25
+ | **Bundle / Pipeline** | `MLDBundle` / `MLDPipeline` |
26
+ | **Motion representation** | **HumanML3D-263** (263-dim, 20 fps, 22 joints) |
27
+ | **Backbone** | MLD VAE + latent diffusion denoiser, default 50 DDIM steps |
28
+ | **Text encoder** | SentenceT5-Large (`sentence-transformers/sentence-t5-large`, frozen) |
29
+ | **Paper** | *Executing Your Commands via Motion Diffusion in Latent Space*, Chen et al., CVPR 2023 |
30
+ | **Original code** | https://github.com/ChenFengYe/motion-latent-diffusion |
31
+
32
+ ---
33
+
34
+ ## Weights
35
+
36
+ Current hftrainer artifact:
37
+
38
+ | Artifact | Location | Contents | Status |
39
+ |---|---|---|---|
40
+ | MLD HumanML3D | [`ZeyuLing/hftrainer-mld-humanml3d`](https://huggingface.co/ZeyuLing/hftrainer-mld-humanml3d) / `checkpoints/mld/humanml3d` | `vae.safetensors` + `denoiser.safetensors` + `mld_config.json` + `Mean.npy` / `Std.npy` | hftrainer artifact |
41
+
42
+ Load through the same `from_pretrained` surface as the other reproduced
43
+ baselines:
44
+
45
+ ```python
46
+ from hftrainer.pipelines.mld import MLDPipeline
47
+
48
+ pipe = MLDPipeline.from_pretrained(
49
+ "ZeyuLing/hftrainer-mld-humanml3d",
50
+ device="cuda",
51
+ )
52
+ motions = pipe.infer_t2m(
53
+ ["a person walks forward then sits down"],
54
+ [120],
55
+ num_inference_steps=50,
56
+ )
57
+ ```
58
+
59
+ Package the artifact from the upstream Lightning checkpoint:
60
+
61
+ ```bash
62
+ python3 scripts/eval/convert_mld_checkpoint.py \
63
+ --model_ckpt ref_repo/MotionLCM/experiments_t2m/mld_humanml/mld_humanml_v1.ckpt \
64
+ --out_dir checkpoints/mld/humanml3d
65
+ ```
66
+
67
+ The frozen SentenceT5-Large encoder is resolved by name rather than duplicated
68
+ inside the artifact. For fully offline use, snapshot the text encoder into the
69
+ local Hugging Face cache before calling `from_pretrained`.
70
+
71
+ ---
72
+
73
+ ## Motion Representation
74
+
75
+ **HumanML3D-263**, the standard redundant T2M feature (Guo et al.), 20 fps,
76
+ 22-joint SMPL skeleton. Per frame (263 dims):
77
+
78
+ | Slice | Dim | Meaning |
79
+ |---|---:|---|
80
+ | `root_rot_vel` | 1 | root angular velocity (about Y) |
81
+ | `root_lin_vel` | 2 | root linear velocity (XZ plane) |
82
+ | `root_y` | 1 | root height |
83
+ | `ric_data` | 63 | local joint positions (21x3) |
84
+ | `rot_data` | 126 | local joint rotations (21x6, continuous 6D) |
85
+ | `local_vel` | 66 | local joint velocities (22x3) |
86
+ | `foot_contact` | 4 | binary foot-contact labels |
87
+
88
+ MLD samples in latent space and decodes directly back to HumanML3D-263. Convert
89
+ to SMPL or MotionStreamer-272 only when a cross-representation evaluator needs
90
+ that space.
91
+
92
+ ---
93
+
94
+ ## Evaluation
95
+
96
+ Generation follows the official HumanML3D protocol: standard test split,
97
+ native 263-dim at 20 fps, first caption, and one prediction per test id.
98
+
99
+ ```bash
100
+ python3 scripts/eval/mld_t2m_h3d263.py \
101
+ --data_root ref_repo/CondMDI/dataset/HumanML3D \
102
+ --model_path checkpoints/mld/humanml3d \
103
+ --num_inference_steps 50 \
104
+ --out_dir outputs/evaluation/t2m/humanml3d_official_test/hml263/mld
105
+ ```
106
+
107
+ The full reproduction pipeline writes the canonical outputs:
108
+
109
+ | Representation | Canonical path |
110
+ |---|---|
111
+ | HML263 | `outputs/evaluation/t2m/humanml3d_official_test/hml263/mld` |
112
+ | SMPL motion_135 | `outputs/evaluation/t2m/humanml3d_official_test/motion135/mld` |
113
+ | MotionStreamer-272 | `outputs/evaluation/t2m/humanml3d_official_test/ms272/mld` |
114
+
115
+ Run the Taiji wrapper for full generation, conversion, and evaluators:
116
+
117
+ ```bash
118
+ python3 scripts/submit/submit_mld_standard_pipeline_taiji.py \
119
+ --gpu A100 \
120
+ --num-gpus 8
121
+ ```
122
+
123
+ Report current metrics from the generated evaluator JSONs under
124
+ `outputs/evaluation/t2m/humanml3d_official_test/_runs/<run>/metrics/`.
125
+
126
+ ---
127
+
128
+ ## Implementation Notes
129
+
130
+ - **hftrainer-native runtime**: `hftrainer.models.motion.mld` wraps the shared
131
+ native MLD VAE / denoiser / SentenceT5 components and does not import
132
+ `ref_repo` at inference time.
133
+ - **Scheduler**: MLD uses `diffusers.DDIMScheduler` with 50 inference steps by
134
+ default (`eta=0.0`, `steps_offset=1`), matching the official inference config.
135
+ - **Classifier-free guidance**: the denoiser has no LCM `time_cond_proj`, so
136
+ guidance uses the standard unconditional/conditional two-pass batch.
137
+ - **Normalization travels with the checkpoint**: `Mean.npy` / `Std.npy` are
138
+ embedded in the artifact to avoid evaluator drift caused by mismatched
139
+ HumanML3D statistics.
Std.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b7e5be77800bb2fd4da3badd5e7e018aaa654dc103f867c2a2300ba1cffada64
3
+ size 1180
denoiser.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f9dcdd5aca52421ed7b7a5dfb65f24fd18ac2d2222342c2187b94d58f399d4a3
3
+ size 32898016
mld_config.json ADDED
@@ -0,0 +1,52 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_type": "mld",
3
+ "text_encoder_name": "sentence-transformers/sentence-t5-large",
4
+ "guidance_scale": 7.5,
5
+ "num_inference_steps": 50,
6
+ "config": {
7
+ "vae": {
8
+ "nfeats": 263,
9
+ "latent_dim": [
10
+ 1,
11
+ 256
12
+ ],
13
+ "ff_size": 1024,
14
+ "num_layers": 9,
15
+ "num_heads": 4,
16
+ "dropout": 0.1,
17
+ "arch": "encoder_decoder",
18
+ "normalize_before": false,
19
+ "activation": "gelu",
20
+ "position_embedding": "learned"
21
+ },
22
+ "denoiser": {
23
+ "latent_dim": [
24
+ 1,
25
+ 256
26
+ ],
27
+ "ff_size": 1024,
28
+ "num_layers": 9,
29
+ "num_heads": 4,
30
+ "dropout": 0.1,
31
+ "normalize_before": false,
32
+ "activation": "gelu",
33
+ "flip_sin_to_cos": true,
34
+ "return_intermediate_dec": false,
35
+ "position_embedding": "learned",
36
+ "arch": "trans_enc",
37
+ "freq_shift": 0,
38
+ "text_encoded_dim": 768,
39
+ "time_cond_proj_dim": null
40
+ },
41
+ "scheduler": {
42
+ "num_train_timesteps": 1000,
43
+ "beta_start": 0.00085,
44
+ "beta_end": 0.012,
45
+ "beta_schedule": "scaled_linear",
46
+ "clip_sample": false,
47
+ "set_alpha_to_one": false,
48
+ "steps_offset": 1,
49
+ "eta": 0.0
50
+ }
51
+ }
52
+ }
model_index.json ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_type": "mld",
3
+ "library_name": "hftrainer",
4
+ "pipeline": "hftrainer.pipelines.mld.MLDPipeline",
5
+ "bundle": "hftrainer.models.motion.mld.MLDBundle",
6
+ "task": "text-to-motion",
7
+ "dataset": "HumanML3D",
8
+ "representation": "HumanML3D-263",
9
+ "artifact_format": "hftrainer-mld-v1",
10
+ "files": [
11
+ "mld_config.json",
12
+ "vae.safetensors",
13
+ "denoiser.safetensors",
14
+ "Mean.npy",
15
+ "Std.npy"
16
+ ]
17
+ }
vae.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e5bde62cadf3a05336f7671aad290a4a49cc34aab77efa36f7859a6a67c18600
3
+ size 72160708