| --- |
| license: apache-2.0 |
| language: |
| - en |
| --- |
| |
| # MotionCtrl Model Card |
|
|
| <div align="center"> |
|
|
| [**Project Page**](https://wzhouxiff.github.io/projects/MotionCtrl/) **|** [**Paper (ArXiv)**](https://arxiv.org/pdf/2312.03641.pdf) **|** [**Code**](https://github.com/TencentARC/MotionCtrl) |
|
|
| [🤗 **Gradio demo (MotionCtrl+VideoCrafter)**](https://huggingface.co/spaces/TencentARC/MotionCtrl) **|** [🤗 **Gradio demo (MotionCtrl+SVD)**](https://huggingface.co/spaces/TencentARC/MotionCtrl_SVD) |
|
|
| </div> |
|
|
| ## Introduction |
|
|
| MotionCtrl, when provided with specific camera movements indicated by camera poses or several trajectories, |
| can independently control both the **camera** and **object** motion in a generated video. |
| It can be deployed on both Text-to-Video generation models, such as **LVDM/VideoCrafter** and **AnimateDiff**, |
| as well as Image-to-Video generation models like **SVD**. |
|
|
| ## MotionCtrl + SVD |
|
|
| - Model: [motionctrl.pth](https://huggingface.co/TencentARC/MotionCtrl/blob/main/motionctrl.pth) |
| - Results: |
|
|
| <div align="center"> |
| <img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/svd/00_ibzz5-dxv2h.gif", width="49.7%", height="100%", style="float: left"> |
| <img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/svd/01_5guvn-0x6v2.gif", width="49.7%", height="100%", style="float: right"> |
| </div> |
| |
|   |
|
|
| ## MotionCtrl + LVDM/VideoCrafter |
|
|
| - Model: |
| - [motionctrl_svd.ckpt](https://huggingface.co/TencentARC/MotionCtrl/blob/main/motionctrl_svd.ckpt) |
| - [motionctrl_videocrafter2_cmcm.ckpt](https://huggingface.co/TencentARC/MotionCtrl/blob/main/motionctrl_videocrafter2_cmcm.ckpt) |
| - Results: |
|
|
| <div align="center"> |
| <img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/hpxvu-3d8ym.gif", width="100%", height="100%", style="float: left"> |
| <img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/w3nb7-9vz5t.gif", width="100%", height="100%", style="float: right"> |
| <img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/62n2a-wuvsw.gif", width="100%", height="100%", style="float: left"> |
| <img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/ilw96-ak827.gif", width="100%", height="100%", style="float: right"> |
| </div> |
| |
|   |
|
|
| ## MotionCtrl + AnimateDiff |
|
|
| - Model: |
| - [motionctrl_animatediff_cmcm.ckpt](https://huggingface.co/TencentARC/MotionCtrl/blob/main/motionctrl_animatediff_cmcm.ckpt) |
| - [motionctrl_animatediff_omcm.ckpt](https://huggingface.co/TencentARC/MotionCtrl/blob/main/motionctrl_animatediff_omcm.ckpt) |
| - Results: |
|
|
| <div align="center"> |
| <img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/complex/1424acd0007d40b5.png", width="25%", height="100%", style="float: left"> |
| <img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/complex/1424acd0007d40b5.gif", width="25%", height="100%", style="float: left"> |
| <img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/complex/camera_Round-R_ZoomIn.png", width="25%", height="100%", style="float: left"> |
| <img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/complex/round_in.gif", width="25%", height="100%", style="float: left"> |
| </div> |
| |
| <div align="center"> |
| <img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/traj/traj_curve_1.gif", width="25%", height="100%", style="float: left"> |
| <img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/traj/curve_1_1.gif", width="25%", height="100%", style="float: left"> |
| <img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/traj/curve_1_2.gif", width="25%", height="100%", style="float: left"> |
| <img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/traj/curve_1.gif", width="25%", height="100%", style="float: left"> |
| </div> |
| |
|   |
|
|
| ## Usage |
|
|
| - Download directly in this repository OR Download with python script |
|
|
| ```python |
| from huggingface_hub import hf_hub_download |
| motionctrl_lvdm_ckpt = hf_hub_download(repo_id="TencentARC/MotionCtrl", filename="motionctrl.ckpt", repo_type="model") |
| motionctrl_svd_ckpt = hf_hub_download(repo_id="TencentARC/MotionCtrl", filename="motionctrl_svd.ckpt", repo_type="model") |
| ``` |
|
|
| - Then generate the controllable video following the instructions in our [GitHub repository](https://github.com/TencentARC/MotionCtrl/). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{wang2023motionctrl, |
| title={MotionCtrl: A Unified and Flexible Motion Controller for Video Generation}, |
| author={Wang, Zhouxia and Yuan, Ziyang and Wang, Xintao and Chen, Tianshui and Xia, Menghan and Luo, Ping and Shan, Yin}, |
| booktitle={arXiv preprint arXiv:2312.03641}, |
| year={2023} |
| } |
| ``` |
|
|
