docs: trim model card (deploy/quickstart moved to GitHub); fix figure to 30 FPS @ 720x1248; condense highlights
Browse files
README.md
CHANGED
|
@@ -61,7 +61,7 @@ The framework combines:
|
|
| 61 |
|
| 62 |
Through autoregressive diffusion modeling, aligned autoregressive distribution matching distillation, long-horizon optimization, bounded KV-state inference, and deployment-oriented scheduling, JoyAI-Video-Edit achieves stable long-range editing while substantially reducing inference latency.
|
| 63 |
|
| 64 |
-
Our deployment system reaches **30
|
| 65 |
|
| 66 |
---
|
| 67 |
|
|
@@ -78,9 +78,7 @@ Our deployment system reaches **30.19 FPS end-to-end at 720 × 1280 resolution**
|
|
| 78 |
|
| 79 |
### ⚡ Real-Time Streaming Editing
|
| 80 |
|
| 81 |
-
JoyAI-Video-Edit edits frames continuously as they arrive instead of waiting for the entire video.
|
| 82 |
-
|
| 83 |
-
It supports open-ended streams and does not require a predefined video duration.
|
| 84 |
|
| 85 |
### 🎯 Instruction-Guided Video Editing
|
| 86 |
|
|
@@ -110,11 +108,7 @@ JoyAI-Video-Edit introduces an autoregressive diffusion framework consisting of:
|
|
| 110 |
|
| 111 |
### 🚀 Real-Time 720p Deployment
|
| 112 |
|
| 113 |
-
The full deployment pipeline achieves
|
| 114 |
-
|
| 115 |
-
**30.19 FPS @ 720 × 1280**
|
| 116 |
-
|
| 117 |
-
with end-to-end streaming inference.
|
| 118 |
|
| 119 |
---
|
| 120 |
|
|
@@ -134,144 +128,6 @@ Technical report:
|
|
| 134 |
|
| 135 |
---
|
| 136 |
|
| 137 |
-
## 📦 Model Download
|
| 138 |
-
|
| 139 |
-
The released JoyAI-Video-Edit checkpoints are hosted on Hugging Face.
|
| 140 |
-
|
| 141 |
-
### Hugging Face CLI
|
| 142 |
-
|
| 143 |
-
First install the Hugging Face CLI:
|
| 144 |
-
|
| 145 |
-
```bash
|
| 146 |
-
pip install -U huggingface_hub
|
| 147 |
-
```
|
| 148 |
-
|
| 149 |
-
Then download the complete model repository:
|
| 150 |
-
|
| 151 |
-
```bash
|
| 152 |
-
hf download jdopensource/JoyAI-Video-Edit \
|
| 153 |
-
--local-dir ./deploy/deps/checkpoints/JoyAI-Video-Edit
|
| 154 |
-
```
|
| 155 |
-
|
| 156 |
-
Alternatively, you can use:
|
| 157 |
-
|
| 158 |
-
```bash
|
| 159 |
-
huggingface-cli download jdopensource/JoyAI-Video-Edit \
|
| 160 |
-
--local-dir ./deploy/deps/checkpoints/JoyAI-Video-Edit
|
| 161 |
-
```
|
| 162 |
-
|
| 163 |
-
The downloaded checkpoint directory should look like:
|
| 164 |
-
|
| 165 |
-
```text
|
| 166 |
-
deploy/deps/checkpoints/JoyAI-Video-Edit/
|
| 167 |
-
├── config.json
|
| 168 |
-
├── dit/
|
| 169 |
-
│ └── joyai_video_edit_dit_0804.pth
|
| 170 |
-
└── vae/
|
| 171 |
-
├── config.json
|
| 172 |
-
└── diffusion_pytorch_model.safetensors
|
| 173 |
-
```
|
| 174 |
-
|
| 175 |
-
> **Note**
|
| 176 |
-
>
|
| 177 |
-
> JoyAI-Video-Edit currently uses its own deployment and inference pipeline.
|
| 178 |
-
> The released checkpoint should **not** be loaded directly with
|
| 179 |
-
> `diffusers.DiffusionPipeline.from_pretrained()`.
|
| 180 |
-
|
| 181 |
-
MiMo-VL and the ONNX detector files are additional runtime dependencies. Please refer to the deployment documentation in the GitHub repository for details.
|
| 182 |
-
|
| 183 |
-
---
|
| 184 |
-
|
| 185 |
-
## 🚀 Quick Start
|
| 186 |
-
|
| 187 |
-
### 1. Clone the Repository
|
| 188 |
-
|
| 189 |
-
```bash
|
| 190 |
-
git clone https://github.com/jd-opensource/JoyAI-Video-Edit.git
|
| 191 |
-
cd JoyAI-Video-Edit
|
| 192 |
-
```
|
| 193 |
-
|
| 194 |
-
### 2. Create the Environment
|
| 195 |
-
|
| 196 |
-
```bash
|
| 197 |
-
conda create -n joyai-video-edit python=3.10 -y
|
| 198 |
-
conda activate joyai-video-edit
|
| 199 |
-
|
| 200 |
-
python -m pip install -r requirements.txt
|
| 201 |
-
```
|
| 202 |
-
|
| 203 |
-
### 3. Download JoyAI-Video-Edit
|
| 204 |
-
|
| 205 |
-
```bash
|
| 206 |
-
hf download jdopensource/JoyAI-Video-Edit \
|
| 207 |
-
--local-dir ./deploy/deps/checkpoints/JoyAI-Video-Edit
|
| 208 |
-
```
|
| 209 |
-
|
| 210 |
-
### 4. Prepare Runtime Dependencies
|
| 211 |
-
|
| 212 |
-
MiMo-VL and the required ONNX detector files are external runtime dependencies.
|
| 213 |
-
|
| 214 |
-
Please follow:
|
| 215 |
-
|
| 216 |
-
**[DEPLOYMENT.md](https://github.com/jd-opensource/JoyAI-Video-Edit/blob/main/DEPLOYMENT.md)**
|
| 217 |
-
|
| 218 |
-
for detailed checkpoint preparation and deployment instructions.
|
| 219 |
-
|
| 220 |
-
### 5. Launch
|
| 221 |
-
|
| 222 |
-
```bash
|
| 223 |
-
cd deploy
|
| 224 |
-
bash run_server.sh
|
| 225 |
-
```
|
| 226 |
-
|
| 227 |
-
Then open:
|
| 228 |
-
|
| 229 |
-
```text
|
| 230 |
-
http://localhost:8080
|
| 231 |
-
```
|
| 232 |
-
|
| 233 |
-
For deployment on a remote server, bind the service to `0.0.0.0` and expose the selected port or use SSH port forwarding.
|
| 234 |
-
|
| 235 |
-
---
|
| 236 |
-
|
| 237 |
-
## 🛠️ Custom Deployment
|
| 238 |
-
|
| 239 |
-
For custom deployment settings, edit:
|
| 240 |
-
|
| 241 |
-
```text
|
| 242 |
-
deploy/run_server.sh
|
| 243 |
-
```
|
| 244 |
-
|
| 245 |
-
You can configure:
|
| 246 |
-
|
| 247 |
-
- JoyAI-Video-Edit checkpoint paths
|
| 248 |
-
- CUDA device placement
|
| 249 |
-
- Server host
|
| 250 |
-
- Server port
|
| 251 |
-
- Runtime cache directories
|
| 252 |
-
|
| 253 |
-
The default deployment script uses persistent TorchInductor, Triton, and CUDA cache directories so that compiled artifacts can be reused across launches.
|
| 254 |
-
|
| 255 |
-
`TORCHINDUCTOR_AUTOGRAD_CACHE` is not required for inference-only serving.
|
| 256 |
-
|
| 257 |
-
---
|
| 258 |
-
|
| 259 |
-
## 🚧 Roadmap
|
| 260 |
-
|
| 261 |
-
- [ ] **Consumer GPU support**
|
| 262 |
-
Optimize deployment for consumer GPUs such as NVIDIA GeForce RTX 5090.
|
| 263 |
-
|
| 264 |
-
- [ ] **Stronger JoyAI-Video-Edit model**
|
| 265 |
-
Improve overall editing quality with a particular focus on reference-image-guided video editing (RV2V).
|
| 266 |
-
|
| 267 |
-
- [ ] **Training framework**
|
| 268 |
-
Release the complete training pipeline.
|
| 269 |
-
|
| 270 |
-
- [ ] **Data pipeline**
|
| 271 |
-
Release the data generation and processing framework.
|
| 272 |
-
|
| 273 |
-
---
|
| 274 |
-
|
| 275 |
## 📚 Citation
|
| 276 |
|
| 277 |
If JoyAI-Video-Edit is useful for your research or project, please cite:
|
|
|
|
| 61 |
|
| 62 |
Through autoregressive diffusion modeling, aligned autoregressive distribution matching distillation, long-horizon optimization, bounded KV-state inference, and deployment-oriented scheduling, JoyAI-Video-Edit achieves stable long-range editing while substantially reducing inference latency.
|
| 63 |
|
| 64 |
+
Our deployment system reaches **30 FPS end-to-end at 720 × 1248 resolution**, enabling real-time streaming video editing.
|
| 65 |
|
| 66 |
---
|
| 67 |
|
|
|
|
| 78 |
|
| 79 |
### ⚡ Real-Time Streaming Editing
|
| 80 |
|
| 81 |
+
JoyAI-Video-Edit edits frames continuously as they arrive instead of waiting for the entire video. It supports open-ended streams and does not require a predefined video duration.
|
|
|
|
|
|
|
| 82 |
|
| 83 |
### 🎯 Instruction-Guided Video Editing
|
| 84 |
|
|
|
|
| 108 |
|
| 109 |
### 🚀 Real-Time 720p Deployment
|
| 110 |
|
| 111 |
+
The full deployment pipeline achieves **30 FPS @ 720 × 1248** with end-to-end streaming inference.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 112 |
|
| 113 |
---
|
| 114 |
|
|
|
|
| 128 |
|
| 129 |
---
|
| 130 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 131 |
## 📚 Citation
|
| 132 |
|
| 133 |
If JoyAI-Video-Edit is useful for your research or project, please cite:
|