wxDai commited on
Commit
ee6ed2c
·
verified ·
1 Parent(s): 6a0a0c3

docs: trim model card (deploy/quickstart moved to GitHub); fix figure to 30 FPS @ 720x1248; condense highlights

Browse files
Files changed (1) hide show
  1. README.md +3 -147
README.md CHANGED
@@ -61,7 +61,7 @@ The framework combines:
61
 
62
  Through autoregressive diffusion modeling, aligned autoregressive distribution matching distillation, long-horizon optimization, bounded KV-state inference, and deployment-oriented scheduling, JoyAI-Video-Edit achieves stable long-range editing while substantially reducing inference latency.
63
 
64
- Our deployment system reaches **30.19 FPS end-to-end at 720 × 1280 resolution**, enabling real-time streaming video editing.
65
 
66
  ---
67
 
@@ -78,9 +78,7 @@ Our deployment system reaches **30.19 FPS end-to-end at 720 × 1280 resolution**
78
 
79
  ### ⚡ Real-Time Streaming Editing
80
 
81
- JoyAI-Video-Edit edits frames continuously as they arrive instead of waiting for the entire video.
82
-
83
- It supports open-ended streams and does not require a predefined video duration.
84
 
85
  ### 🎯 Instruction-Guided Video Editing
86
 
@@ -110,11 +108,7 @@ JoyAI-Video-Edit introduces an autoregressive diffusion framework consisting of:
110
 
111
  ### 🚀 Real-Time 720p Deployment
112
 
113
- The full deployment pipeline achieves:
114
-
115
- **30.19 FPS @ 720 × 1280**
116
-
117
- with end-to-end streaming inference.
118
 
119
  ---
120
 
@@ -134,144 +128,6 @@ Technical report:
134
 
135
  ---
136
 
137
- ## 📦 Model Download
138
-
139
- The released JoyAI-Video-Edit checkpoints are hosted on Hugging Face.
140
-
141
- ### Hugging Face CLI
142
-
143
- First install the Hugging Face CLI:
144
-
145
- ```bash
146
- pip install -U huggingface_hub
147
- ```
148
-
149
- Then download the complete model repository:
150
-
151
- ```bash
152
- hf download jdopensource/JoyAI-Video-Edit \
153
- --local-dir ./deploy/deps/checkpoints/JoyAI-Video-Edit
154
- ```
155
-
156
- Alternatively, you can use:
157
-
158
- ```bash
159
- huggingface-cli download jdopensource/JoyAI-Video-Edit \
160
- --local-dir ./deploy/deps/checkpoints/JoyAI-Video-Edit
161
- ```
162
-
163
- The downloaded checkpoint directory should look like:
164
-
165
- ```text
166
- deploy/deps/checkpoints/JoyAI-Video-Edit/
167
- ├── config.json
168
- ├── dit/
169
- │ └── joyai_video_edit_dit_0804.pth
170
- └── vae/
171
- ├── config.json
172
- └── diffusion_pytorch_model.safetensors
173
- ```
174
-
175
- > **Note**
176
- >
177
- > JoyAI-Video-Edit currently uses its own deployment and inference pipeline.
178
- > The released checkpoint should **not** be loaded directly with
179
- > `diffusers.DiffusionPipeline.from_pretrained()`.
180
-
181
- MiMo-VL and the ONNX detector files are additional runtime dependencies. Please refer to the deployment documentation in the GitHub repository for details.
182
-
183
- ---
184
-
185
- ## 🚀 Quick Start
186
-
187
- ### 1. Clone the Repository
188
-
189
- ```bash
190
- git clone https://github.com/jd-opensource/JoyAI-Video-Edit.git
191
- cd JoyAI-Video-Edit
192
- ```
193
-
194
- ### 2. Create the Environment
195
-
196
- ```bash
197
- conda create -n joyai-video-edit python=3.10 -y
198
- conda activate joyai-video-edit
199
-
200
- python -m pip install -r requirements.txt
201
- ```
202
-
203
- ### 3. Download JoyAI-Video-Edit
204
-
205
- ```bash
206
- hf download jdopensource/JoyAI-Video-Edit \
207
- --local-dir ./deploy/deps/checkpoints/JoyAI-Video-Edit
208
- ```
209
-
210
- ### 4. Prepare Runtime Dependencies
211
-
212
- MiMo-VL and the required ONNX detector files are external runtime dependencies.
213
-
214
- Please follow:
215
-
216
- **[DEPLOYMENT.md](https://github.com/jd-opensource/JoyAI-Video-Edit/blob/main/DEPLOYMENT.md)**
217
-
218
- for detailed checkpoint preparation and deployment instructions.
219
-
220
- ### 5. Launch
221
-
222
- ```bash
223
- cd deploy
224
- bash run_server.sh
225
- ```
226
-
227
- Then open:
228
-
229
- ```text
230
- http://localhost:8080
231
- ```
232
-
233
- For deployment on a remote server, bind the service to `0.0.0.0` and expose the selected port or use SSH port forwarding.
234
-
235
- ---
236
-
237
- ## 🛠️ Custom Deployment
238
-
239
- For custom deployment settings, edit:
240
-
241
- ```text
242
- deploy/run_server.sh
243
- ```
244
-
245
- You can configure:
246
-
247
- - JoyAI-Video-Edit checkpoint paths
248
- - CUDA device placement
249
- - Server host
250
- - Server port
251
- - Runtime cache directories
252
-
253
- The default deployment script uses persistent TorchInductor, Triton, and CUDA cache directories so that compiled artifacts can be reused across launches.
254
-
255
- `TORCHINDUCTOR_AUTOGRAD_CACHE` is not required for inference-only serving.
256
-
257
- ---
258
-
259
- ## 🚧 Roadmap
260
-
261
- - [ ] **Consumer GPU support**
262
- Optimize deployment for consumer GPUs such as NVIDIA GeForce RTX 5090.
263
-
264
- - [ ] **Stronger JoyAI-Video-Edit model**
265
- Improve overall editing quality with a particular focus on reference-image-guided video editing (RV2V).
266
-
267
- - [ ] **Training framework**
268
- Release the complete training pipeline.
269
-
270
- - [ ] **Data pipeline**
271
- Release the data generation and processing framework.
272
-
273
- ---
274
-
275
  ## 📚 Citation
276
 
277
  If JoyAI-Video-Edit is useful for your research or project, please cite:
 
61
 
62
  Through autoregressive diffusion modeling, aligned autoregressive distribution matching distillation, long-horizon optimization, bounded KV-state inference, and deployment-oriented scheduling, JoyAI-Video-Edit achieves stable long-range editing while substantially reducing inference latency.
63
 
64
+ Our deployment system reaches **30 FPS end-to-end at 720 × 1248 resolution**, enabling real-time streaming video editing.
65
 
66
  ---
67
 
 
78
 
79
  ### ⚡ Real-Time Streaming Editing
80
 
81
+ JoyAI-Video-Edit edits frames continuously as they arrive instead of waiting for the entire video. It supports open-ended streams and does not require a predefined video duration.
 
 
82
 
83
  ### 🎯 Instruction-Guided Video Editing
84
 
 
108
 
109
  ### 🚀 Real-Time 720p Deployment
110
 
111
+ The full deployment pipeline achieves **30 FPS @ 720 × 1248** with end-to-end streaming inference.
 
 
 
 
112
 
113
  ---
114
 
 
128
 
129
  ---
130
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
131
  ## 📚 Citation
132
 
133
  If JoyAI-Video-Edit is useful for your research or project, please cite: