## MotionCtrl: A Unified and Flexible Motion Controller for Video Generation #### [SIGGRAPH 2024 CONFERENCE PROCEEDINGS] ###

👉 MotionCtrl for [SVD], for [VideoCrafter], for [AnimateDiff]

[![Paper](https://img.shields.io/badge/Paper-gray)](https://wzhouxiff.github.io/projects/MotionCtrl/assets/paper/MotionCtrl.pdf) [![arXiv](https://img.shields.io/badge/arXiv-red)](https://arxiv.org/pdf/2312.03641.pdf) [![Project Page](https://img.shields.io/badge/Project%20Page-green )](https://wzhouxiff.github.io/projects/MotionCtrl/) 🤗 [![HF Demo](https://img.shields.io/static/v1?label=Demo&message=MotionCtrl%2BSVD&color=orange)](https://huggingface.co/spaces/TencentARC/MotionCtrl_SVD) 🤗 [![HF Demo](https://img.shields.io/static/v1?label=Demo&message=MotionCtrl%2BVideoCrafter&color=orange)](https://huggingface.co/spaces/TencentARC/MotionCtrl) More examples of MotionCtrl+SVD are in [showcase_svd](https://github.com/TencentARC/MotionCtrl/blob/svd/doc/showcase_svd.md)

--- https://github.com/TencentARC/MotionCtrl/assets/19488619/45d44bf5-d4bf-4e45-8628-2c8926b5954a --- 🔥🔥 We release the codes, [models](https://huggingface.co/TencentARC/MotionCtrl/tree/main) and [demos](https://huggingface.co/spaces/TencentARC/MotionCtrl_SVD) for MotionCtrl on [Stable Video Diffusion (SVD)](https://github.com/Stability-AI/generative-models). Official implementation of [MotionCtrl: A Unified and Flexible Motion Controller for Video Generation](https://arxiv.org/abs/2312.03641). MotionCtrl can Independently control **complex camera motion** and **object motion** of generated videos, with **only a unified** model. ### Results of MotionCtrl+SVD More results are in [showcase_svd](https://github.com/TencentARC/MotionCtrl/blob/svd/doc/showcase_svd.md) and our [Project Page](https://wzhouxiff.github.io/projects/MotionCtrl/).

## ⚙️ Environment conda create -n motionctrl python=3.10.6 conda activate motionctrl pip install -r requirements.txt ## 💫 Inference - #### Run local inference script 1. Download the weights of MotionCtrl [motionctrl_svd.pth](https://huggingface.co/TencentARC/MotionCtrl/blob/main/motionctrl_svd.ckpt) and put it to `./checkpoints`. 2. Running scripts: sh configs/inference/run.sh - #### Run local gradio demo python -m app --share ❗❗❗ **Noted** ❗❗❗ 1. If the motion control is not obvious, try to increase the `speed` in the run.sh or `Motion Speed` in the gradio demo. 2. If the generated videos are distored severely, try to descrease the `speed` in the run.sh or `Motion Speed` in the gradio demo. Or increase `FPS`. ## :books: Citation If you make use of our work, please cite our paper. ```bibtex @inproceedings{wang2024motionctrl, title={Motionctrl: A unified and flexible motion controller for video generation}, author={Wang, Zhouxia and Yuan, Ziyang and Wang, Xintao and Li, Yaowei and Chen, Tianshui and Xia, Menghan and Luo, Ping and Shan, Ying}, booktitle={ACM SIGGRAPH 2024 Conference Papers}, pages={1--11}, year={2024} } ``` ## 🤗 Acknowledgment The current version of **MotionCtrl** is built on [SVD](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid). We appreciate the authors for sharing their awesome codebase. ## ❓ Contact For any question, feel free to email `wzhoux@connect.hku.hk` or `zhouzi1212@gmail.com`.