| --- |
| license: apache-2.0 |
| library_name: transformers |
| pipeline_tag: video-text-to-text |
| base_model: Qwen/Qwen3-VL-8B-Instruct |
| base_model_relation: finetune |
| tags: |
| - qwen3-vl |
| - video |
| - motion-analysis |
| - motion-quality-assessment |
| - emnlp-2026-findings |
| - custom-code |
| --- |
| |
| <p align="center"> |
| <img src="assets/motioninsight-header.png" alt="MotionInsight" width="100%"> |
| </p> |
|
|
| <h1 align="center">MotionInsight: Diagnosing Object Motion<br>Deficiencies in Generated Videos</h1> |
| <p align="center"><strong>EMNLP 2026 Findings</strong></p> |
| <p align="center"> |
| Jiahao Zhan, Yongrui Ma, Qunliang Xing, Xuanyu Zhang,<br> |
| Jingqi Tong, Junlin Li, Li Zhang, Shijie Zhao, Tianfan Xue |
| </p> |
| <p align="center"> |
| <a href="https://github.com/JohnZhan2023/MotionInsight">Code</a> · |
| <a href="https://arxiv.org/abs/2609.37030">arXiv</a> · |
| <a href="#quick-start">Quick start</a> · |
| <a href="#citation">Citation</a> |
| </p> |
|
|
| ## MotionInsight-8B |
|
|
| MotionInsight diagnoses object motion in generated videos using RGB frames, |
| object tracks, and camera motion. It produces reasoning and three scores from |
| **1 (poor)** to **5 (excellent)**: |
|
|
| | Dimension | Output field | |
| |---|---| |
| | Object consistency | `structural_stability` | |
| | Motion continuity | `motion_coherence` | |
| | Physical plausibility | `physical_plausibility` | |
|
|
| The model is based on Qwen3-VL-8B-Instruct and uses four BF16 Safetensors shards |
| (approximately 17.7 GB). |
|
|
| ## Quick start |
|
|
| Use Python 3.10 and the custom Qwen3-VL patch from the code repository: |
|
|
| ```bash |
| git clone --recurse-submodules https://github.com/JohnZhan2023/MotionInsight.git |
| cd MotionInsight |
| python3.10 -m venv .venv-model |
| source .venv-model/bin/activate |
| python -m pip install --upgrade pip |
| python -m pip install huggingface-hub==0.36.2 |
| hf download JohnZhan/MotionInsight-8B --local-dir checkpoints/MotionInsight-8B |
| python -m pip install torch==2.11.0 torchvision==0.26.0 \ |
| --index-url https://download.pytorch.org/whl/cu126 |
| python -m pip install -r checkpoints/MotionInsight-8B/requirements/inference.txt |
| python scripts/patch_transformers.py |
| ``` |
|
|
| After [extracting motion features](https://github.com/JohnZhan2023/MotionInsight#extract-motion-features), run: |
|
|
| ```bash |
| python inference.py \ |
| --model_name_or_path checkpoints/MotionInsight-8B \ |
| --video_path /path/to/video.mp4 \ |
| --object_motion_path /path/to/object_motion.pt \ |
| --camera_motion_path /path/to/camera_motion.pt \ |
| --target "tennis ball" \ |
| --output_path outputs/prediction.jsonl |
| ``` |
|
|
| Example answer format: |
|
|
| ```text |
| <thinking>Diagnostic reasoning about the target object's motion.</thinking> |
| <answer>{"structural_stability": 3.0, "physical_plausibility": 4.5, "motion_coherence": 3.5}</answer> |
| ``` |
|
|
| ## GRPO |
|
|
| The code repository includes a [GRPO fine-tuning script](https://github.com/JohnZhan2023/MotionInsight/blob/main/training/run_grpo.sh): |
|
|
| ```bash |
| python -m pip install -r checkpoints/MotionInsight-8B/requirements/train.txt |
| ATTN_IMPLEMENTATION=sdpa bash training/run_grpo.sh \ |
| checkpoints/MotionInsight-8B /path/to/train.jsonl outputs/motioninsight-grpo |
| ``` |
|
|
| See the [repository README](https://github.com/JohnZhan2023/MotionInsight#grpo) |
| for the input format and preprocessing setup. |
|
|
| ## License |
|
|
| Model weights use [Apache-2.0](LICENSE). Preprocessing dependencies retain their |
| own licenses; see [NOTICE](NOTICE). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{zhan2026motioninsightdiagnosingobjectmotion, |
| title = {MotionInsight: Diagnosing Object Motion Deficiencies in Generated Videos}, |
| author = {Jiahao Zhan and Yongrui Ma and Qunliang Xing and Xuanyu Zhang and Jingqi Tong and Junlin Li and Li zhang and Shijie Zhao and Tianfan Xue}, |
| year = {2026}, |
| eprint = {2609.37030}, |
| archivePrefix = {arXiv}, |
| primaryClass = {cs.CV}, |
| url = {https://arxiv.org/abs/2609.37030}, |
| } |
| ``` |
|
|
