license: apache-2.0
language:
- en
- zh
pipeline_tag: video-to-video
JoyAI-Video-Edit
Real-Time Open-Ended Video Editing with Autoregressive Diffusion
🐶 JoyAI-Video-Edit
JoyAI-Video-Edit is a real-time, instruction-guided video editing system for open-ended video streams. Given a live camera stream or uploaded video and a natural-language edit instruction, it edits frames causally as they arrive, without waiting for the full video, requiring a predefined video length, or revisiting future frames. In our deployment benchmark, the full end-to-end pipeline reaches 30.19 FPS at 720x1280, pushing video editing from offline batch processing toward interactive streaming generation.
The system combines an MLLM-based condition encoder, a causal video VAE, and a 16B-parameter multimodal diffusion transformer. It is trained and deployed as an autoregressive diffusion editor, then accelerated with aligned autoregressive distribution matching distillation, long-horizon optimization, bounded KV-state inference, and deployment-oriented scheduling to sustain high-throughput 720p editing while reducing train-inference mismatch and accumulated temporal drift.
💎 Highlights
- Real-time open-ended editing. Edits live or uploaded videos as frames arrive, without requiring the full sequence upfront.
- Diverse instruction control. Supports subject edits, local edits, background changes, style transfer, motion changes, and reference-guided editing.
- Autoregressive diffusion design. Combines an MLLM condition encoder, causal video VAE, and MMDiT backbone for streaming video editing.
- High-throughput 720p deployment. Reaches 30.19 FPS end-to-end throughput at 720x1280 with bounded KV-state inference and stable per-chunk compute.
🚀 Quick Start
1. Install
conda create -n joyai-video-edit python=3.10 -y
conda activate joyai-video-edit
python -m pip install -r requirements.txt
2. Prepare Checkpoints
Download the released weights from the Hugging Face link above. MiMo-VL and the ONNX detector files are external runtime dependencies; see DEPLOYMENT.md for deployment details.
3. Launch
cd deploy
bash run_server.sh
Then open:
http://localhost:8080
For remote machines, bind the server to 0.0.0.0 and open the selected port, or use SSH port forwarding.
📦 Model Download
Download the released JoyAI-Video-Edit weights from Hugging Face, then place them under:
deploy/deps/checkpoints/JoyAI-Video-Edit/
|-- dit/
| `-- joyai_video_edit_dit_0804.pth
`-- vae/
|-- config.json
`-- diffusion_pytorch_model.safetensors
License Agreement
JoyAI-Video-Edit is licensed under Apache 2.0.