Download README.md from Yakimask/PartLock-CMU: direct link, hf CLI and curl.
- Browser
- Download file 9.26 kB
-
https://huggingface.co/Yakimask/PartLock-CMU/resolve/main/README.md
- Command line
-
hf download hf://Yakimask/PartLock-CMU/README.md
-
curl -L -o README.md https://huggingface.co/Yakimask/PartLock-CMU/resolve/main/README.md
license: apache-2.0
language:
- en
library_name: pytorch
pipeline_tag: text-to-3d
tags:
- text-to-motion
- motion-generation
- human-motion
- motion-capture
- cmu-mocap
- real-time
PartLock-CMU
Real-time, endless text-to-motion trained only on the CMU Motion Capture Database, so the weights can be used in commercial products.
Type a short English caption. The model keeps generating full-body motion forever, and you can switch the caption at any moment. A new caption takes effect at the next step (4 frames = 0.13 s), and the motion so far is kept as context, so the transition is continuous.
Author: やきますく (Yakimasuku) · Code and weights: Apache License 2.0
One continuous take (seed 1, not cherry-picked). The caption changes every 4 s: walk → Wave → run/jog → jump → Standing still.
Seed 1, first take of each. The figure is a simple preview body drawn from the generated joint positions and rotations; it is not an SMPL mesh.
Real-time demo
pip install -r requirements.txt
python live/live_server.py # then open http://localhost:8765
On Windows you can also double-click live/start_windows.bat.
- Type a caption and press the green button, or click one of the example captions. The page is in Japanese: 生成 = generate, 一時停止 = pause, 最初から = restart (clear the context), 文章の強さ = text strength (classifier-free guidance scale).
- The server generates only 0.8 s ahead of playback, so a new caption appears about 0.6 s after you press the button.
- Needs an NVIDIA GPU with CUDA. On an RTX 3060 (12 GB), one step (4 frames) takes 35–38 ms. Real time needs 133 ms or less, so there is about 3.5× headroom.
- At start-up the server checks itself. The CUDA-graph path is compared with the plain forward pass, and the largest logit difference was 0.06 (bf16 rounding; logits are up to about 12).
Generate to a file
python generate.py --sec 4 --out motion.npz "walk" "Wave" "run/jog" "jump"
Each caption is used for --sec seconds, in order, in one continuous motion. The output .npz is at 30 fps:
| key | shape | content |
|---|---|---|
positions |
(T, 22, 3) | world joint positions in meters, y up, SMPL joint order |
rotations |
(T, 22, 3, 3) | world joint rotations |
root_xz |
(T, 2) | pelvis path on the floor |
prompt |
(T,) | caption used for each frame |
parents |
(22,) | skeleton parents |
Values are rounded to 0.001 (1 mm for positions).
Captions that work
The model only knows the CMU motion descriptions. Short captions in the CMU style work best, for example:
walk, run/jog, jump, Wave, kick, salsa dance, boxing, Standing still, sit on high stool, stand up, walk backwards and turn, cartwheel, Mickey sneaking walk
The full list is in data/cmu_files.csv (column desc). Case doesn't matter: in our tests Run and run gave identical results, because the CLIP tokenizer lower-cases the text.
Model
| Motion representation | 272 numbers per frame at 30 fps (root velocity and heading, joint positions, velocities and 6D rotations of 22 SMPL-order joints), following the 272-dim representation of MotionStreamer. CMU ASF/AMC is converted with our own converter (code/cmu_to272.py). |
| Tokenizer (7.5 M parameters) | Causal hierarchical VQ. Every 4 frames become 7 tokens: 1 global code (codebook 4096) plus 6 part codes (codebook 128 each) for torso, head, right arm, left arm, right leg and left leg. Encoder and decoder only look at the past, which allows streaming. |
| Generator (30.5 M parameters) | Decoder-only Transformer: 8 layers, width 512, 8 heads, RoPE. Context is 64 steps (8.5 s) and slides forever, which makes it endless. The text embedding (CLIP ViT-B/32) is added at every position, so the caption can be swapped mid-stream. Classifier-free guidance s = 4 (null text for 10% of training samples). |
Training data and results
- Data: CMU Motion Capture Database, 2,308 clips (9.9 h) with captions taken from the CMU motion descriptions. 179 clips were skipped (not in the CMU list or too short). Every clip is also mirrored left–right, which doubles the data.
- Split: by original clip, 90 / 5 / 5 (train 2,078, validation 115, test 115 clips). A mirrored copy always stays in the same split as its original.
- Hardware: one RTX 3060 (12 GB). Tokenizer 2 h, generator 75 min to its best validation loss. The whole pipeline, including conversion, took about 3 h 50 min.
Tokenizer reconstruction on the validation data (best step 18,000):
| part | position error (mm) | rotation error (°) |
|---|---|---|
| torso | 33.2 | 5.3 |
| head | 56.1 | 8.2 |
| right arm | 67.4 | 14.2 |
| left arm | 68.4 | 14.2 |
| right leg | 55.4 | 8.2 |
| left leg | 56.4 | 8.2 |
| all | 57.4 | 9.9 (global) |
Generator validation loss (cross-entropy per token) at the best step (9,500):
| global | torso | head | right arm | left arm | right leg | left leg | mean |
|---|---|---|---|---|---|---|---|
| 2.654 | 2.459 | 2.428 | 2.208 | 2.114 | 2.196 | 2.114 | 2.311 |
There is no standard text-to-motion benchmark for CMU-only models, so these numbers can't be compared directly with models trained on HumanML3D.
Limitations
- Only CMU motions. Motions that are not in CMU didn't work in our tests. All 20 novel actions we tried came out nearly stationary.
- Mixing two actions in one caption (for example "wave while walking") usually loses most of the movement. Switching captions over time works much better; see the demo above.
runalone runs for about 2 s and then slows to about 0.5 m/s. Userun/jog, which keeps running at about 2.2 m/s.- English only, because of CLIP.
- CMU subjects have different body sizes and the data isn't normalized to one body, so bone lengths can vary a little within a motion.
- The output is a 22-joint skeleton with rotations. Retargeting to a character rig is up to you.
Train from scratch
Download
allasfamc.zipfrom mocap.cs.cmu.edu and extract it. The folder containssubjects/.Run:
python code/run_cmu.py --amc <extracted folder> --work <empty work folder>This runs convert → tokenizer (2 h) → prep → generator (2 h). Checkpoints are written to
<work>/ckptas.ptfiles. To use them with the demo, convert them toweights/*.safetensors.
The released weights came from this pipeline. In the release copy, run_cmu.py was changed only to take the paths as arguments. That edited copy hasn't been run end to end again.
Files
| path | content |
|---|---|
weights/generator.safetensors |
generator |
weights/tokenizer.safetensors |
causal tokenizer (encoder and decoder) |
weights/Mean.npy, weights/Std.npy |
normalization of the 272-dim features |
live/ |
real-time server and page |
generate.py |
generate to .npz |
code/ |
model code and the training pipeline |
data/cmu_files.csv |
CMU clip list with motion descriptions (used as captions) |
code/tvq.py, code/tvq272.py and code/vq_mix.py are shared modules from earlier research code, and their comments mention HumanML3D experiments. Only their model classes and helper functions are used here. No HumanML3D or AMASS data or weights were used to train this model.
License and data
- Code and weights: Apache License 2.0 (
LICENSE). - Training data: the CMU Motion Capture Database. Its terms say: "This data is free for use in research projects. You may include this data in commercially-sold products, but you may not resell this data directly, even in converted form." This repository doesn't contain the motion data.
data/cmu_files.csvis only the list of clip names and descriptions. - The text encoder openai/clip-vit-base-patch32 is downloaded at run time under its own license.
Acknowledgment: The data used in this project was obtained from mocap.cs.cmu.edu. The database was created with funding from NSF EIA-0196217.
日本語
CMU のモーションキャプチャだけで学習した、文章から全身の動きを作るモデルです。止まらずに動きを作り続け、途中で文章を切り替えられます。商用でも使えます。
python live/live_server.pyを実行して http://localhost:8765 を開くと、文章を入れてリアルタイムに動かせます(NVIDIA の GPU が必要です)。- 学習データにある短い英語の文章(walk, run/jog, jump, Wave など)がよく効きます。学習データにない動きは、まだうまく作れません。
- 作者:やきますく








