PartLock-CMU / README.md
Yakimask's picture
Upload 31 files
c7ae99a verified
|
Raw History Blame Contribute Delete
9.26 kB
metadata
license: apache-2.0
language:
  - en
library_name: pytorch
pipeline_tag: text-to-3d
tags:
  - text-to-motion
  - motion-generation
  - human-motion
  - motion-capture
  - cmu-mocap
  - real-time

PartLock-CMU

Real-time, endless text-to-motion trained only on the CMU Motion Capture Database, so the weights can be used in commercial products.

Type a short English caption. The model keeps generating full-body motion forever, and you can switch the caption at any moment. A new caption takes effect at the next step (4 frames = 0.13 s), and the motion so far is kept as context, so the transition is continuous.

Author: やきますく (Yakimasuku) · Code and weights: Apache License 2.0

switching captions: walk → Wave → run/jog → jump → Standing still

One continuous take (seed 1, not cherry-picked). The caption changes every 4 s: walk → Wave → run/jog → jump → Standing still.

walk run/jog jump Wave
walk run/jog jump Wave
kick salsa dance boxing
kick salsa dance boxing

Seed 1, first take of each. The figure is a simple preview body drawn from the generated joint positions and rotations; it is not an SMPL mesh.

Real-time demo

real-time demo page

pip install -r requirements.txt
python live/live_server.py        # then open http://localhost:8765

On Windows you can also double-click live/start_windows.bat.

  • Type a caption and press the green button, or click one of the example captions. The page is in Japanese: 生成 = generate, 一時停止 = pause, 最初から = restart (clear the context), 文章の強さ = text strength (classifier-free guidance scale).
  • The server generates only 0.8 s ahead of playback, so a new caption appears about 0.6 s after you press the button.
  • Needs an NVIDIA GPU with CUDA. On an RTX 3060 (12 GB), one step (4 frames) takes 35–38 ms. Real time needs 133 ms or less, so there is about 3.5× headroom.
  • At start-up the server checks itself. The CUDA-graph path is compared with the plain forward pass, and the largest logit difference was 0.06 (bf16 rounding; logits are up to about 12).

Generate to a file

python generate.py --sec 4 --out motion.npz "walk" "Wave" "run/jog" "jump"

Each caption is used for --sec seconds, in order, in one continuous motion. The output .npz is at 30 fps:

key shape content
positions (T, 22, 3) world joint positions in meters, y up, SMPL joint order
rotations (T, 22, 3, 3) world joint rotations
root_xz (T, 2) pelvis path on the floor
prompt (T,) caption used for each frame
parents (22,) skeleton parents

Values are rounded to 0.001 (1 mm for positions).

Captions that work

The model only knows the CMU motion descriptions. Short captions in the CMU style work best, for example:

walk, run/jog, jump, Wave, kick, salsa dance, boxing, Standing still, sit on high stool, stand up, walk backwards and turn, cartwheel, Mickey sneaking walk

The full list is in data/cmu_files.csv (column desc). Case doesn't matter: in our tests Run and run gave identical results, because the CLIP tokenizer lower-cases the text.

Model

Motion representation 272 numbers per frame at 30 fps (root velocity and heading, joint positions, velocities and 6D rotations of 22 SMPL-order joints), following the 272-dim representation of MotionStreamer. CMU ASF/AMC is converted with our own converter (code/cmu_to272.py).
Tokenizer (7.5 M parameters) Causal hierarchical VQ. Every 4 frames become 7 tokens: 1 global code (codebook 4096) plus 6 part codes (codebook 128 each) for torso, head, right arm, left arm, right leg and left leg. Encoder and decoder only look at the past, which allows streaming.
Generator (30.5 M parameters) Decoder-only Transformer: 8 layers, width 512, 8 heads, RoPE. Context is 64 steps (8.5 s) and slides forever, which makes it endless. The text embedding (CLIP ViT-B/32) is added at every position, so the caption can be swapped mid-stream. Classifier-free guidance s = 4 (null text for 10% of training samples).

Training data and results

  • Data: CMU Motion Capture Database, 2,308 clips (9.9 h) with captions taken from the CMU motion descriptions. 179 clips were skipped (not in the CMU list or too short). Every clip is also mirrored left–right, which doubles the data.
  • Split: by original clip, 90 / 5 / 5 (train 2,078, validation 115, test 115 clips). A mirrored copy always stays in the same split as its original.
  • Hardware: one RTX 3060 (12 GB). Tokenizer 2 h, generator 75 min to its best validation loss. The whole pipeline, including conversion, took about 3 h 50 min.

Tokenizer reconstruction on the validation data (best step 18,000):

part position error (mm) rotation error (°)
torso 33.2 5.3
head 56.1 8.2
right arm 67.4 14.2
left arm 68.4 14.2
right leg 55.4 8.2
left leg 56.4 8.2
all 57.4 9.9 (global)

Generator validation loss (cross-entropy per token) at the best step (9,500):

global torso head right arm left arm right leg left leg mean
2.654 2.459 2.428 2.208 2.114 2.196 2.114 2.311

There is no standard text-to-motion benchmark for CMU-only models, so these numbers can't be compared directly with models trained on HumanML3D.

Limitations

  • Only CMU motions. Motions that are not in CMU didn't work in our tests. All 20 novel actions we tried came out nearly stationary.
  • Mixing two actions in one caption (for example "wave while walking") usually loses most of the movement. Switching captions over time works much better; see the demo above.
  • run alone runs for about 2 s and then slows to about 0.5 m/s. Use run/jog, which keeps running at about 2.2 m/s.
  • English only, because of CLIP.
  • CMU subjects have different body sizes and the data isn't normalized to one body, so bone lengths can vary a little within a motion.
  • The output is a 22-joint skeleton with rotations. Retargeting to a character rig is up to you.

Train from scratch

  1. Download allasfamc.zip from mocap.cs.cmu.edu and extract it. The folder contains subjects/.

  2. Run:

    python code/run_cmu.py --amc <extracted folder> --work <empty work folder>
    

    This runs convert → tokenizer (2 h) → prep → generator (2 h). Checkpoints are written to <work>/ckpt as .pt files. To use them with the demo, convert them to weights/*.safetensors.

The released weights came from this pipeline. In the release copy, run_cmu.py was changed only to take the paths as arguments. That edited copy hasn't been run end to end again.

Files

path content
weights/generator.safetensors generator
weights/tokenizer.safetensors causal tokenizer (encoder and decoder)
weights/Mean.npy, weights/Std.npy normalization of the 272-dim features
live/ real-time server and page
generate.py generate to .npz
code/ model code and the training pipeline
data/cmu_files.csv CMU clip list with motion descriptions (used as captions)

code/tvq.py, code/tvq272.py and code/vq_mix.py are shared modules from earlier research code, and their comments mention HumanML3D experiments. Only their model classes and helper functions are used here. No HumanML3D or AMASS data or weights were used to train this model.

License and data

  • Code and weights: Apache License 2.0 (LICENSE).
  • Training data: the CMU Motion Capture Database. Its terms say: "This data is free for use in research projects. You may include this data in commercially-sold products, but you may not resell this data directly, even in converted form." This repository doesn't contain the motion data. data/cmu_files.csv is only the list of clip names and descriptions.
  • The text encoder openai/clip-vit-base-patch32 is downloaded at run time under its own license.

Acknowledgment: The data used in this project was obtained from mocap.cs.cmu.edu. The database was created with funding from NSF EIA-0196217.


日本語

CMU のモーションキャプチャだけで学習した、文章から全身の動きを作るモデルです。止まらずに動きを作り続け、途中で文章を切り替えられます。商用でも使えます。

  • python live/live_server.py を実行して http://localhost:8765 を開くと、文章を入れてリアルタイムに動かせます(NVIDIA の GPU が必要です)。
  • 学習データにある短い英語の文章(walk, run/jog, jump, Wave など)がよく効きます。学習データにない動きは、まだうまく作れません。
  • 作者:やきますく