Instructions to use JinPLU/CaT-Movie with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Wan2.2
How to use JinPLU/CaT-Movie with Wan2.2:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
CaT-Movie-5B (Renderer Checkpoint)
Finetuned Wan2.2-TI2V-5B multi-shot renderer weights (step-3000.safetensors, 9.44 GiB, BF16) for CaT-Movie: Balancing Global Planning and Local Rendering for Long Multi-Shot Video Generation.
Architecture & Usage Contract
Important:
step-3000.safetensorsis not a drop-in checkpoint for stock Wan2.2generate.py. Run inference throughrun/render_story.pywithconfigs/infer/cat_movie_5b.pyso CaT-Movie's compiled prompts, window attention masks, and anchor recall slots are enabled.
- Backbone:
Wan-AI/Wan2.2-TI2V-5B(1280Γ704,24 fps,umT5-XXL, Wan2.2 16ch VAE; all5.0Btransformer blocks trained for3,000rectified-flow updates on8Γ A100). - Shot-Wise Cross-Attention: Each target shot in a window (
8 straining /7 stest cap) attends exclusively to its compiled per-shot promptc_k. - Reference-Routed Sparse Self-Attention (
holocine_first_frame+routed_self_only_v1): Each shot attends to itself, co-generated window first latent frames, and up to10storyboard-indexed clean anchor views (history_policy = plan_selection) placed at negative temporal RoPE offsets (Ξt = -5) with per-reference identity lines (frame_entities_v1). - Planner: Pairs cleanly with a training-free structured planner (Qwen3.8-27B via vLLM; setup in the GitHub repository).
Quickstart: Render a Story
export MSPC_SHARED_ROOT=/path/to/cat_movie_workspace
# 1. Download base Wan2.2-TI2V-5B and CaT-Movie-5B weights
hf download Wan-AI/Wan2.2-TI2V-5B --local-dir "$MSPC_SHARED_ROOT/checkpoints/Wan2.2-TI2V-5B"
hf download JinPLU/CaT-Movie step-3000.safetensors --local-dir "$MSPC_SHARED_ROOT/checkpoints/CaT-Movie"
# 2. From a clone of JinPLu/CaT-Movie:
export MSPC_CODE_ROOT="$(pwd)"
PYTHONPATH=.:src python run/render_story.py \
--config configs/infer/cat_movie_5b.py \
--storyboard examples/sample_storyboard.json \
--story-id cinedance-008935-b0001-shot01 \
--output outputs/demo/cinedance-008935-b0001-shot01 \
--seed 42 --segment-seed-stride 1009 --segment-max-shots none \
--segment-max-frames 112 --history-policy plan_selection
Files in This Repository
| File | Description | Size |
|---|---|---|
step-3000.safetensors |
CaT-Movie Wan2.2-TI2V-5B renderer weights (step 3,000, BF16) |
9.44 GiB |
LICENSE |
CC BY-NC-SA 4.0 (non-commercial research weights) | β |
Main Quantitative Results (CaT-MBench, 92 Test Stories)
Evaluated with seed 42 via TransNetV2 (Cut F1 @ 0.125 s, Execution over 674 setup-changing cuts), Gemini 3.8 Flash (6 native-res frames/pair for Consistency; 2 fps whole video for Establish/Persist under the cut gate), and ViCLIP (Text Alignment). Baselines use annotated reference storyboards in native prompt formats; CaT-Movie renders its Qwen3.8-27B planner's storyboard from the shots request (bold = best, italic = second best):
| Method | Cut F1 β | Exec. β | Char. β | Obj. β | Scene β | Est. β | Persist β | Align. β | Avg. β |
|---|---|---|---|---|---|---|---|---|---|
| Wan2.1, per shot (1.3B) | 0.998 | 0.994 | 0.219 | 0.169 | 0.310 | 0.274 | 0.694 | 0.181 | 0.474 |
| Wan2.2, per shot (5B) | 0.999 | 0.996 | 0.227 | 0.275 | 0.310 | 0.387 | 0.701 | 0.174 | 0.497 |
| StoryMem (A14B) | 0.928 | 0.719 | 0.631 | 0.453 | 0.687 | 0.484 | 0.545 | 0.179 | 0.527 |
| StoryDiffusion (A14B) | 0.905 | 0.825 | 0.300 | 0.265 | 0.245 | 0.597 | 0.535 | 0.171 | 0.468 |
| IC-LoRA (A14B) | 0.976 | 0.961 | 0.168 | 0.133 | 0.406 | 0.452 | 0.663 | 0.176 | 0.484 |
| MultiShotMaster (1.3B) | 0.994 | 0.990 | 0.538 | 0.254 | 0.582 | 0.371 | 0.587 | 0.178 | 0.527 |
| HoloCine (A14B) | 0.748 | 0.967 | 0.517 | 0.512 | 0.586 | 0.694 | 0.691 | 0.195 | 0.571 |
| CaT-Movie (Ours) (5B) | 0.995 | 0.974 | 0.651 | 0.497 | 0.630 | 0.468 | 0.785 | 0.186 | 0.597 |
(Backbones: Wan2.1-T2V-1.3B, Wan2.2-TI2V-5B, Wan2.2-I2V-A14B for StoryMem/StoryDiffusion/IC-LoRA, Wan2.2-T2V-A14B for HoloCine. Columns shorten Execution, Character, Object, Scene, Establish, Text Alignment.)
Compute & Limitations
- Hardware: Single GPU with β₯ 40 GB VRAM (BF16) for
1280Γ704, 50 flow UniPC steps (shift = 5,CFG = 5). - Trade-offs:
CaT-Movielifts within-shot change execution (Establish) over its 5B base (0.387 β 0.468) alongside0.785Persist, thoughEstablishtracks base model size across 5B vs. 14B models (0.45β0.69). Multi-window anchor recall assumes returning entities remain clear in an earlier keyframe.
CaT-Movie Release Ecosystem (GitHub & 8 HF Repositories)
| Repository | Tier | Contents |
|---|---|---|
JinPLu/CaT-Movie (GitHub) |
Core | Planner, training, inference, curation & CaT-MBench code (Apache-2.0) |
JinPLU/CaT-Movie (this repo) |
Core | Renderer weights (step-3000.safetensors) |
JinPLU/CaT-Movie-Data |
Core | 75,461 blocks, Qwen3.7-Flash schemas, grounding, 96,000 train windows & SAM 3.1 weights |
JinPLU/CaT-Movie-T5-Cache |
Core | Precomputed umT5-XXL BF16 caches (332,516 unique prompts) |
JinPLU/CaT-Movie-VAE-Caches |
Core | Precomputed VAE target latents (Wan2.2 704p + Wan2.1 480p) |
JinPLU/CaT-Movie-Source-Videos |
Supporting | Complete 18,173 block-yielding source MP4 clips |
JinPLU/CaT-MBench |
Core | 122 stories, cut-gated GPT-6.1-sol VQA bank, Gemini 3.8 Flash verdicts & 557 human labels |
JinPLU/CaT-MBench-Videos |
Supporting | 736 generated test videos (92 test stories Γ 8 methods) |
JinPLU/CaT-MBench-Author-Labels |
Archive | Early Nov 6 single-author calibration archive (superseded by CaT-MBench/human/) |
Licensing & Citation
Released under CC BY-NC-SA 4.0 strictly for non-commercial academic research (derived from Wan-AI/Wan2.2-TI2V-5B, Apache-2.0; codebase Apache-2.0).
@misc{catmovie2026,
title = {CaT-Movie: Balancing Global Planning and Local Rendering for Long Multi-Shot Video Generation},
author = {Jinpeng Lu and Haoyuan Shi and Yuanshen Guan and Jiyuan Xia and Bin Wu and Yinda Chen and Zhiwei Xiong},
year = {2026},
url = {https://jinplu.github.io/CaT-Movie/}
}
Model tree for JinPLU/CaT-Movie
Base model
Wan-AI/Wan2.2-TI2V-5B