π Introducing VideoCoF: Unified Video Editing with a Temporal Reasoner (Chain-of-Frames)!
Weβre excited to introduce VideoCoF, a unified framework for instruction-based video editing that enables temporal reasoning and ~4Γ video length extrapolation, trained with only 50k video pairs. π₯
π What makes VideoCoF different? π§ Chain-of-Frames reasoning , mimic human thinking process like Seeing β Reasoning β Editing to apply edits accurately over time without external masks, ensuring physically plausible results. π Strong length generalization β trained on 33-frame clips, yet supports multi-shot editing and long-video extrapolation (~4Γ). π― Unified fine-grained editing β Object Removal, Addition, Swap, and Local Style Transfer, with instance-level & part-level, spatial-aware control.
β‘ Fast inference update π H100: ~20s / video with 4-step inference, making high-quality video editing far more practical for real-world use.
Excited to share our Unified Multimodal Models new work Reconstruction Alignment (RecA)! π Just 6 Γ 80GB A100s Γ 4.5 hours to boost BAGEL performance across all tasks! Outperforms FLUX-Kontext in image editing capabilities!
β RecA trains UMMs to reconstruct images from their own visual understanding encoder embeddings β big gains in image generation π¨ & editing βοΈ.
π₯ Why itβs cool: - Achieves high-quality, multi-task image editing. - Uses only 1% of the training parameters and 0.1% of the training data compared to existing methods β extremely efficient - Beats several commercial models on background preservation, ID control, and consistency - Open-source, low-cost, faster, and stronger β think of it as the βDeepSeek of image editingβ π
We also implemented a Gradio demo app, available directly in our GitHub repo! And we made a flashy demo video β happy to send it your way!
π₯ Why 3DIS & 3DIS-FLUX? Current SOTA multi-instance generation methods are typically adapter-based, requiring additional control modules trained on pre-trained models for layout and instance attribute control. However, with the emergence of more powerful models like FLUX and SD3.5, these methods demand constant retraining and extensive resources.
β¨ Our Solution: 3DIS We introduce a decoupled approach that only requires training a low-resolution Layout-to-Depth model to convert layouts into coarse-grained scene depth maps. Leveraging community and company pre-trained models like ControlNet + SAM2, we enable training-free controllable image generation on high-resolution models such as SDXL and FLUX.
π Benefits of Our Decoupled Multi-Instance Generation: 1. Enhanced Control: By constructing scenes using depth maps in the first stage, the model focuses on coarse-grained scene layout, improving control over instance placement. 2. Flexibility & Preservation: The second stage employs training-free rendering methods, allowing seamless integration with various models (e.g., fine-tuned weights, LoRA) while maintaining the generative capabilities of pre-trained models.
Join us in advancing Layout-to-Image Generation! Follow and star our repository to stay updated! β