Towards 3D-Aware Video Diffusion Models: Render-Free Human Motion Control with Mesh Tokenization (MeshToken)

Jingyun Liang, Min Wei, Shikai Li, Yizeng Han, Hangjie Yuan, Lei Sun, Weihua Chen, Fan Wang

DAMO Academy, Alibaba

---

This repository is the official implementation of MeshToken: .

Gallery

Here are several example videos generated by MeshToken. Note that the GIFs shown here have some degree of visual quality degradation.

---

Diffusion models have shown remarkable success in video generation. However, whether such models are truly aware of the 3D structure underlying visual observations, rather than simply reproducing plausible 2D projections, remains an open question. In this work, we investigate this question through human motion control, a task that requires precise modelling of 3D human geometry, motion, camera viewpoint, and scene context. Unlike prior methods that rely on rendered 2D motion guidance videos, we propose a render-free framework that conditions video generation directly on compressed 3D human mesh tokens. This representation preserves full 3D geometric information while enabling a unified token-based generation pipeline that processes video tokens jointly with motion tokens in a DiT-based architecture. This design requires the model to reason jointly about appearance, 3D structure, and camera viewpoint during video generation. Experimental results demonstrate strong performance on human motion control benchmarks, while reducing artifacts induced by view-dependent 2D guidance and trajectory-pose mismatches during editing. These findings suggest that video diffusion models, when equipped with mesh tokenization, can better capture complex 3D human structures and their interactions with the surrounding environment.

Quick Start

We follow the same setup and file tree as our previous work RealisMotion.

1. Setup Repository and Environment

git clone https://github.com/jingyunliang/MeshToken.git
cd MeshToken

conda create -n MeshToken python=3.10
conda activate MeshToken

pip install -r requirements.txt

# install FA3
git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention
git checkout 0dfb28174333d9eefb7c1dd4292690a8458d1e89  # Important: using other FA3 might yield bad results on H20 GPUs
cd hopper
python setup.py install

cd ../../

2. Download Checkpoints

Please download the checkpoints as below. Use HF_ENDPOINT=https://hf-mirror.com huggingface-cli xxxx if you need to speed up downloading. By the way, put pretrained_models/ under a fast disk path (e.g., /tmp/) can reduce the model loading time significantly. Later, you can load the model by --ckpt /tmp/pretrained_models/MeshToken in that case.

huggingface-cli download --resume-download jingyunliang/MeshToken --local-dir pretrained_models/MeshToken

3. Install GVHMR

Please install GVHMR and DPVO following install GVHMR. The nvcc in third-party/DPVO/setup.py should be modified as ['-O3', '-gencode', 'arch=compute_90,code=sm_90'] for H20 GPUs. Please symlink the checkpoints by ln -s YOUR_PATH/GVHMR/inputs/checkpoints inputs/checkpoints.

Then, please install DepthPro for focal length calibration as follows (optional).

git clone https://github.com/apple/ml-depth-pro
cd ml-depth-pro
pip install .
source get_pretrained_models.sh 
cd ..

4. Install Mesh-VQ-VAE

Please first clone Mesh-VQ-VAE to ./third-party/Mesh-VQ-VAE.

git clone https://github.com/g-fiche/Mesh-VQ-VAE.git ./third-party/Mesh-VQ-VAE

Then, follow the install guidance to download the body models.

5. Quick Inference

• Inference with Single GPU

export PYTHONPATH="YOUR_PATH/GVHMR/hmr4d:$PYTHONPATH"

CUDA_VISIBLE_DEVICES=0 python inference_mt.py \
    --background_path inputs/demo/image.png \
    --motion_folder inputs/demo \
    --motion_start_index 0 \
    --motion_num_frames 97 \
    --prompt "A tense scene on a rooftop with a view of a bustling city and harbor, showcasing dramatic cityscape under a cloudy sky. Two figures are centered in the composition, one dressed in a dark outfit, extending their arm with a gun pointed at another figure wearing a navy suit. The urban backdrop features high-rise buildings and mountains in the distance, suggesting a metropolitan setting with a blend of natural elements." \
    --save-dir ./outputs \
    --ckpt pretrained_models/MeshToken

Note: add --enable-teacache to inference with TeaCache for acceleration (optional, may cause quality degradation); add --save-gpu-memory to inference with small GPU memory (optional, will be super slow. Can be used with TeaCache).

• Inference with multi GPUs (Optional. Can be used with TeaCache)

export PYTHONPATH="YOUR_PATH/GVHMR/hmr4d:$PYTHONPATH"

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --nproc_per_node=8 inference_mt.py \
    --background_path inputs/demo/image.png \
    --motion_folder inputs/demo \
    --motion_start_index 0 \
    --motion_num_frames 97 \
    --prompt "A woman in black athletic wear runs along a paved seaside path, wearing white headphones. The path is lined with evenly spaced concrete pillars and metal railings, leading her past them. The ocean waves crash against the rocks in the background, under a clear blue sky with scattered white clouds. The scene is serene, with the woman focused on her run, the calm sea extending to the horizon, and the gentle motion of the waves adding to the tranquility of the setting." \
    --save-dir ./outputs \
    --multi-gpu \
    --ckpt pretrained_models/MeshToken

Disclaimer

This project is released for academic use. We disclaim responsibility for user-generated content.

Citation

@article{liang2026towards,
    title={Towards 3D-Aware Video Diffusion Models: Render-Free Human Motion Control with Mesh Tokenization},
    author={Liang, Jingyun and Wei, Min and Li, Shikai and Han, Yizeng and Yuan, Hangjie and Sun, Lei and Chen, Weihua and Wang, Fan},
    journal={arXiv preprint arXiv:2605.00000},
    year={2026}
}

Acknowledgement

We thank the authors of GVHMR, Mesh-VQ-VAE and RealisMotion for their great works, without which our project/code would not be possible.