| --- |
| license: other |
| license_name: sample-code-license |
| license_link: https://github.com/EPFL-VILAB/Video-4M/blob/main/LICENSE_WEIGHTS |
| pipeline_tag: image-to-video |
| library_name: pytorch |
| tags: |
| - video-generation |
| - any-to-any |
| - multimodal |
| --- |
| |
| # Any-to-Any Video Modeling: Modeling the World by Traversing Time and Modalities |
|
|
| [Muhammad Uzair Khattak](https://muzairkhattak.github.io/), [Won Jun Kim](https://one-june.github.io/), [Reza Abbassi](https://abbasireza.github.io/), [Albias Havolli](https://www.linkedin.com/in/albias-havolli-a9065924b/), [Michael Murphy](https://www.linkedin.com/in/m1chaelmurphy/), [Amir Zadeh](https://www.linkedin.com/in/amir-zadeh-894921320/), [Chuan Li](https://www.linkedin.com/in/chuan-li-a9474b63/), [Muhammad Ferjad Naeem*](https://ferjad.github.io/), [Oğuzhan Fatih Kar*](https://ofkar.github.io/), [Roman Bachmann*](https://roman-bachmann.github.io/), [Andrei Atanov*](https://andrewatanov.github.io/), [Federico Tombari](https://federicotombari.github.io/), [Amir Zamir](https://vilab.epfl.ch/zamir/) |
|
|
| EPFL, Google, Lambda |
|
|
| *Official pre-trained checkpoints for Video-4M, an any-to-any multimodal model for the video domain that allows flexible traversal across both modality and time axes.* |
|
|
| [`🌐 Project page`](https://video-4m.epfl.ch/) | [`📄 Paper`](https://video-4m.epfl.ch/assets/Video_4M.pdf) | [`💻 Code`](https://github.com/EPFL-VILAB/Video-4M) | [`🤗 Tokenizers`](https://huggingface.co/EPFL-VILAB/Video-4M-tokenizers) | [`🤗 Demo`](https://huggingface.co/spaces/EPFL-VILAB/Video-4M) | [`BibTeX`](#citation) |
|
|
| ## Model checkpoints |
|
|
| This repository hosts the main Video-4M model checkpoints. We release two checkpoint versions, both pretrained on the [K600-MM dataset](https://huggingface.co/datasets/EPFL-VILAB/K600-MM): |
|
|
| | Model | Params | File | Config | |
| |------------------|---|---|---| |
| | Video-4M-L | 705M | [`main_model/Video-4M-L.pth`](main_model/Video-4M-L.pth) | [Config](https://github.com/EPFL-VILAB/Video-4M/blob/main/cfgs/default/video-4m/models/main/Video-4M_L_pretraining_config.yaml) | |
| | Video-4M-Pred-L | 705M | [`main_model/Video-4M-Pred-L.pth`](main_model/Video-4M-Pred-L.pth) | [Config](https://github.com/EPFL-VILAB/Video-4M/blob/main/cfgs/default/video-4m/models/main/Video-4M_Pred_L_pretraining_config.yaml) | |
|
|
| Video-4M-Pred is a Video-4M model whose pretraining input-output mixture is biased toward predicting video from sparse modalities (e.g., caption, transcription) and toward forecasting tasks. It can still perform all other any-to-any tasks. |
|
|
| ## Usage |
|
|
| We provide two ways to get started with Video-4M: the [🤗 Space demo](https://huggingface.co/spaces/EPFL-VILAB/Video-4M), which lets you test Video-4M without any local setup, and the [notebooks](https://github.com/EPFL-VILAB/Video-4M/tree/main/notebooks) in our main GitHub code repository. Both download the model checkpoints and the [tokenizers](https://huggingface.co/EPFL-VILAB/Video-4M-tokenizers) automatically. |
|
|
| Below, we provide a minimal example of performing inference with Video-4M using our notebook pipeline. Here, we give an RGB video as input and generate all other modalities via chained generation: |
|
|
| ```python |
| from notebooks import pipeline |
| from notebooks.display_utils import video_grid |
| |
| tokenizers = pipeline.load_tokenizers() # downloads the tokenizers from the Hub on first call |
| model = pipeline.load_model(tokenizers, model_name="Video-4M_Pred_L") # downloads the model from the Hub on first call |
| |
| example = pipeline.load_example(pipeline.list_examples()[0]) # downloads pre-tokenized video examples from the Hub, and selects the first one |
| |
| results = pipeline.generate( |
| example, input_modalities=["rgb"], |
| chain=pipeline.CHAIN_PRESETS["rgb_to_others"]["chain"], # you can also define a custom chain, e.g. ['caption', 'depth'] |
| tokenizers=tokenizers, model=model, |
| ) |
| video_grid(results, cols=5, width=170, title="rgb -> everything else") # for visualizing the generation results |
| ``` |
|
|
| For installation, training, and evaluation instructions, please see the [README](https://github.com/EPFL-VILAB/Video-4M) of our main GitHub code repository. |
|
|
| ## License |
|
|
| The model weights are released under the Sample Code license as found in [LICENSE_WEIGHTS](https://github.com/EPFL-VILAB/Video-4M/blob/main/LICENSE_WEIGHTS). |
|
|
| ## Citation |
|
|
| If you find this repository useful, please consider citing: |
|
|
| ``` |
| @article{video4m2026, |
| title={{Any-to-Any Video Modeling: Modeling the World by Traversing Time and Modalities}}, |
| author={Khattak, Muhammad Uzair and Kim, Won Jun and Abbassi, Reza and Havolli, Albias and Murphy, Michael and Zadeh, Amir and Li, Chuan and Naeem, Muhammad Ferjad and Kar, O\u{g}uzhan Fatih and Bachmann, Roman and Atanov, Andrei and Tombari, Federico and Zamir, Amir}, |
| journal={arXiv preprint}, |
| year={2026}, |
| } |
| ``` |
|
|
