| --- |
| license: apache-2.0 |
| library_name: diffusers |
| pipeline_tag: image-to-image |
| tags: |
| - optical-flow prediction |
| - motion prediction |
| - diffusion |
| --- |
| |
| # FOFPred: Language-Driven Future Optical Flow Prediction |
|
|
|
|
| [Paper](https://arxiv.org/abs/2601.10781) | [Project Site](https://fofpred.github.io) | [Demo](https://fofpred.salesforceresearch.ai) |
|
|
| **FOFPred** is a diffusion-based model that predicts future optical flow from a single image guided by natural language instructions. Given an input image and a text prompt describing a desired action (e.g., *"Moving the water bottle from right to left"*), FOFPred generates 4 sequential optical flow frames showing how objects would move. |
|
|
| ## Usage |
|
|
| ```python |
| import einops |
| import numpy as np |
| import torch |
| from diffusers import DiffusionPipeline |
| from PIL import Image |
| |
| # Load pipeline with trust_remote_code |
| pipeline = DiffusionPipeline.from_pretrained( |
| "Salesforce/FOFPred", |
| torch_dtype=torch.bfloat16, |
| trust_remote_code=True, |
| ).to("cuda") |
| |
| # Run inference |
| results = pipeline( |
| prompt="Moving the water bottle from right to left.", |
| input_images=[Image.open("your_image.jpg")], |
| width=256, |
| height=256, |
| num_inference_steps=1, |
| num_images_per_prompt=4, |
| frame_count=4, |
| generator=torch.Generator(device="cuda").manual_seed(42), |
| output_type="pt", |
| ) |
| |
| flow_frames = results.images # [B, F, C, H, W] |
| |
| output_tensor = flow_frames[0] # [F, C, H, W] |
| output_np = pipeline.image_processor.pt_to_numpy(output_tensor) # [F, H, W, C] |
| reshaped = einops.rearrange(output_np, "f h w c -> h (f w) c") |
| img = Image.fromarray((reshaped * 255).astype(np.uint8)) |
| img.save("output_combined.png") |
| ``` |
|
|
| ## Architecture |
|
|
| | Component | Model | |
| |-----------|-------| |
| | **V-LLM** | Qwen2.5-VL-3B-Instruct | |
| | **DiT** | OmniGen2Transformer3DModel | |
| | **VAE** | FLUX.1-dev AutoencoderKL | |
| | **Scheduler** | FlowMatchEulerDiscreteScheduler | |
|
|
| ## Acknowledgements |
|
|
| - [OmniGen2](https://github.com/VectorSpaceLab/OmniGen2) |
| - [Qwen2.5-VL](https://huggingface.co/Qwen/Qwen2.5-VL) |
| - [Flux VAE](https://huggingface.co/black-forest-labs/FLUX.1-dev) |
|
|
| ## License |
|
|
| Our code and weights are released under the [CC by-NC 4.0 license](https://creativecommons.org/licenses/by-nc/4.0/deed.en). |
|
|
