README.md

---
license: mit
library_name: multimodal-flow
tags:
- multimodal
- flow-matching
- text-to-image
- image-to-text
language:
- en
pipeline_tag: any-to-any
---

# Multimodal Flow

**Unified Flow Modeling of Language and Vision in Embedding Spaces**

Multimodal Flow (MF) is a fully continuous framework for multimodal
understanding and generation. Language and vision remain continuous states,
organized as ordered hyperchunks in a shared causal stream. The same model
interface supports text continuation, image understanding, and image
generation.

<p align="center">
  <img src="https://raw.githubusercontent.com/hustvl/Multimodal-Flow/main/docs/assets/multimodal-flow-architecture.png"
       alt="Multimodal Flow architecture" width="100%">
</p>

## Authors

[Hongyuan Tao](https://github.com/Hongyuan-Tao), [Xinggang Wang](https://xwcv.github.io/),
[Lianghui Zhu](https://lh-zhu.github.io/index.html), [Yongkang Li](https://scholar.google.com/citations?user=F7hPv1QAAAAJ&hl=en),
[Yunchao Wei](https://weiyc.github.io/),
[Bin Feng](https://scholar.google.com/citations?user=nRc8u6gAAAAJ&hl=zh-CN),
[Shaoyu Chen](https://scholar.google.com/citations?user=PIeNN2gAAAAJ&hl=en),
[Qian Zhang](https://scholar.google.com/citations?user=pCY-bikAAAAJ&hl=zh-CN),
[Chang Huang](https://scholar.google.com/citations?user=IyyEKyIAAAAJ&hl=en), and
[Kai Yu](https://scholar.google.com/citations?user=y5zkBeMAAAAJ&hl=en).

Huazhong University of Science and Technology · Beijing Jiaotong University ·
Horizon Robotics

## Models and assets

| Path | Description |
| --- | --- |
| `MF/pretrain` | 1.6B pretraining model |
| `MF/sft` | 1.6B supervised fine-tuned model |
| `Text Decoder/` | Shared BF16 text decoder |
| `Vision statistics/` | Shared FP32 vision normalization statistics |

The model weights are BF16. The release contains inference weights and
configuration only; training states and optimizer states are not included.

## Usage

Use the models with the open-source code:

[Multimodal-Flow](https://github.com/hustvl/Multimodal-Flow)

```bash
# Text continuation
mf infer --checkpoint ./MF/pretrain text \
  --prompt "A short language model can"

# Image understanding
mf infer --checkpoint ./MF/sft caption \
  --image /path/to/image.jpg \
  --prompt "Describe this image."

# Text-to-image generation
mf infer --checkpoint ./MF/sft image \
  --prompt "A quiet observatory above the clouds." \
  --output outputs/sample.png
```

Text inference uses the model package directly. Image understanding and image
generation additionally require the compatible Scale RAE decoder assets; set
`MF_ASSETS_ROOT` to a directory containing `scale_rae_decoder/`.

## License

MIT License. Please also review the licenses of external encoder, tokenizer,
and decoder assets used with the models.

## Citation

```bibtex
@article{tao2026multimodalflow,
  title={Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces},
  author={Tao, Hongyuan and Wang, Xinggang and Zhu, Lianghui and Li, Yongkang and Wei, Yunchao and Feng, Bin and Chen, Shaoyu and Zhang, Qian and Huang, Chang and Yu, Kai},
  journal={arXiv preprint arXiv:2609.40362},
  year={2026},
  url={https://arxiv.org/abs/2609.40362}
}
```