README.md

---
license: apache-2.0
language:
  - en
  - zh
pipeline_tag: video-to-video
tags:
  - video-editing
  - video-to-video
  - streaming-video
  - real-time-video
  - autoregressive-diffusion
  - multimodal
  - joyai
---

<h1 align="center">JoyAI-Video-Edit</h1>

<h3 align="center">
Real-Time Open-Ended Video Editing with Autoregressive Diffusion
</h3>

<p align="center">
  <a href="https://arxiv.org/abs/2608.03974">
    <img src="https://img.shields.io/badge/Paper-arXiv-red" alt="Paper">
  </a>
  <a href="https://github.com/jd-opensource/JoyAI-Video-Edit">
    <img src="https://img.shields.io/badge/Code-GitHub-black" alt="GitHub">
  </a>
  <a href="https://huggingface.co/jdopensource/JoyAI-Video-Edit">
    <img src="https://img.shields.io/badge/%F0%9F%A4%97%20Model-JoyAI--Video--Edit-yellow" alt="Hugging Face">
  </a>
  <a href="https://joyai-labs.jd.com/v2v/">
    <img src="https://img.shields.io/badge/%F0%9F%9A%80%20Demo-Streaming--V2V-orange" alt="Demo">
  </a>
  <a href="https://github.com/jd-opensource/JoyAI-Video-Edit/blob/main/LICENSE">
    <img src="https://img.shields.io/badge/License-Apache_2.0-blue.svg" alt="License">
  </a>
</p>

<p align="center">
  <img
    src="https://raw.githubusercontent.com/jd-opensource/JoyAI-Video-Edit/main/assets/teaser.jpg"
    width="96%"
    alt="JoyAI-Video-Edit teaser"
  >
</p>

## 🐶 JoyAI-Video-Edit

**JoyAI-Video-Edit** is a real-time, instruction-guided video editing framework designed for open-ended video streams.

Unlike conventional offline video editing models that require a complete video before inference, JoyAI-Video-Edit processes frames **causally as they arrive**. It can continuously edit a live camera stream or an uploaded video according to natural-language instructions without accessing future frames or requiring a predefined video length.

The framework combines:

- an **MLLM-based condition encoder** for understanding editing instructions,
- a **causal video VAE** for streaming visual representation,
- and a **16B multimodal diffusion transformer** for high-quality video editing.

Through autoregressive diffusion modeling, aligned autoregressive distribution matching distillation, long-horizon optimization, bounded KV-state inference, and deployment-oriented scheduling, JoyAI-Video-Edit achieves stable long-range editing while substantially reducing inference latency.

Our deployment system reaches **30.19 FPS end-to-end at 720 × 1280 resolution**, enabling real-time streaming video editing.

---

## 🔥 News

- **2026.08.06** — 🎉 We released the live demo of JoyAI-Video-Edit.  
  Try it here: **[Streaming V2V Demo](https://joyai-labs.jd.com/v2v/)**

- **2026.08.05** — 🎉 We released the model checkpoints, deployment code, online demo, and technical report.

---

## 💎 Highlights

### ⚡ Real-Time Streaming Editing

JoyAI-Video-Edit edits frames continuously as they arrive instead of waiting for the entire video.

It supports open-ended streams and does not require a predefined video duration.

### 🎯 Instruction-Guided Video Editing

The model supports a broad range of editing instructions, including:

- Subject modification
- Subject addition
- Subject removal
- Subject replacement
- Local object editing
- Background replacement
- Style transfer
- Appearance modification
- Motion editing
- Reference-guided video editing

### 🔄 Autoregressive Diffusion

JoyAI-Video-Edit introduces an autoregressive diffusion framework consisting of:

- MLLM condition encoder
- Causal video VAE
- 16B multimodal diffusion transformer
- Chunk-wise autoregressive generation
- Long-horizon temporal optimization
- Bounded KV-state inference

### 🚀 Real-Time 720p Deployment

The full deployment pipeline achieves:

**30.19 FPS @ 720 × 1280**

with end-to-end streaming inference.

---

## 🎬 Demo

Try our online real-time video editing demo:

👉 **https://joyai-labs.jd.com/v2v/**

Project repository:

👉 **https://github.com/jd-opensource/JoyAI-Video-Edit**

Technical report:

👉 **https://arxiv.org/abs/2608.03974**

---

## 📦 Model Download

The released JoyAI-Video-Edit checkpoints are hosted on Hugging Face.

### Hugging Face CLI

First install the Hugging Face CLI:

```bash
pip install -U huggingface_hub
```

Then download the complete model repository:

```bash
hf download jdopensource/JoyAI-Video-Edit \
    --local-dir ./deploy/deps/checkpoints/JoyAI-Video-Edit
```

Alternatively, you can use:

```bash
huggingface-cli download jdopensource/JoyAI-Video-Edit \
    --local-dir ./deploy/deps/checkpoints/JoyAI-Video-Edit
```

The downloaded checkpoint directory should look like:

```text
deploy/deps/checkpoints/JoyAI-Video-Edit/
├── config.json
├── dit/
│   └── joyai_video_edit_dit_0804.pth
└── vae/
    ├── config.json
    └── diffusion_pytorch_model.safetensors
```

> **Note**
>
> JoyAI-Video-Edit currently uses its own deployment and inference pipeline.
> The released checkpoint should **not** be loaded directly with
> `diffusers.DiffusionPipeline.from_pretrained()`.

MiMo-VL and the ONNX detector files are additional runtime dependencies. Please refer to the deployment documentation in the GitHub repository for details.

---

## 🚀 Quick Start

### 1. Clone the Repository

```bash
git clone https://github.com/jd-opensource/JoyAI-Video-Edit.git
cd JoyAI-Video-Edit
```

### 2. Create the Environment

```bash
conda create -n joyai-video-edit python=3.10 -y
conda activate joyai-video-edit

python -m pip install -r requirements.txt
```

### 3. Download JoyAI-Video-Edit

```bash
hf download jdopensource/JoyAI-Video-Edit \
    --local-dir ./deploy/deps/checkpoints/JoyAI-Video-Edit
```

### 4. Prepare Runtime Dependencies

MiMo-VL and the required ONNX detector files are external runtime dependencies.

Please follow:

**[DEPLOYMENT.md](https://github.com/jd-opensource/JoyAI-Video-Edit/blob/main/DEPLOYMENT.md)**

for detailed checkpoint preparation and deployment instructions.

### 5. Launch

```bash
cd deploy
bash run_server.sh
```

Then open:

```text
http://localhost:8080
```

For deployment on a remote server, bind the service to `0.0.0.0` and expose the selected port or use SSH port forwarding.

---

## 🛠️ Custom Deployment

For custom deployment settings, edit:

```text
deploy/run_server.sh
```

You can configure:

- JoyAI-Video-Edit checkpoint paths
- CUDA device placement
- Server host
- Server port
- Runtime cache directories

The default deployment script uses persistent TorchInductor, Triton, and CUDA cache directories so that compiled artifacts can be reused across launches.

`TORCHINDUCTOR_AUTOGRAD_CACHE` is not required for inference-only serving.

---

## 🚧 Roadmap

- [ ] **Consumer GPU support**  
  Optimize deployment for consumer GPUs such as NVIDIA GeForce RTX 5090.

- [ ] **Stronger JoyAI-Video-Edit model**  
  Improve overall editing quality with a particular focus on reference-image-guided video editing (RV2V).

- [ ] **Training framework**  
  Release the complete training pipeline.

- [ ] **Data pipeline**  
  Release the data generation and processing framework.

---

## 📚 Citation

If JoyAI-Video-Edit is useful for your research or project, please cite:

```bibtex
@article{xiao2026joyai,
  title={JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion},
  author={Xiao, Yicheng and Dai, Wenxun and Qin, Xinran and Song, Lin and Zhang, Maoquan and Xu, Hang and Chen, Yukang and Li, Yitong and Zhang, Guohui and Zhang, Yuan and Zhang, Xuying and Zhang, Tommy and Yuan, Jianlong and Li, Peihao and Lu, Shuai and Fu, Siming and Zhao, Chuyang and Han, Xin and Huang, Jie and Li, Wenbo and Ma, Guoqing and Huang, Wei and Qi, Xiaojuan and Huang, Haoyang and Duan, Nan},
  journal={arXiv preprint arXiv:2608.03974},
  year={2026}
}
```

---

## 📄 License

JoyAI-Video-Edit is released under the **Apache License 2.0**.

Please refer to the project repository for the complete license:

https://github.com/jd-opensource/JoyAI-Video-Edit/blob/main/LICENSE