File size: 3,214 Bytes
67debf8 193555e 60ae51c 67debf8 193555e a44fc74 d7de27d 193555e 13818c6 193555e 60ae51c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 | ---
license: mit
library_name: multimodal-flow
tags:
- multimodal
- flow-matching
- text-to-image
- image-to-text
language:
- en
pipeline_tag: any-to-any
---
# Multimodal Flow
**Unified Flow Modeling of Language and Vision in Embedding Spaces**
Multimodal Flow (MF) is a fully continuous framework for multimodal
understanding and generation. Language and vision remain continuous states,
organized as ordered hyperchunks in a shared causal stream. The same model
interface supports text continuation, image understanding, and image
generation.
<p align="center">
<img src="https://raw.githubusercontent.com/hustvl/Multimodal-Flow/main/docs/assets/multimodal-flow-architecture.png"
alt="Multimodal Flow architecture" width="100%">
</p>
## Authors
[Hongyuan Tao](https://github.com/Hongyuan-Tao), [Xinggang Wang](https://xwcv.github.io/),
[Lianghui Zhu](https://lh-zhu.github.io/index.html), [Yongkang Li](https://scholar.google.com/citations?user=F7hPv1QAAAAJ&hl=en),
[Yunchao Wei](https://weiyc.github.io/),
[Bin Feng](https://scholar.google.com/citations?user=nRc8u6gAAAAJ&hl=zh-CN),
[Shaoyu Chen](https://scholar.google.com/citations?user=PIeNN2gAAAAJ&hl=en),
[Qian Zhang](https://scholar.google.com/citations?user=pCY-bikAAAAJ&hl=zh-CN),
[Chang Huang](https://scholar.google.com/citations?user=IyyEKyIAAAAJ&hl=en), and
[Kai Yu](https://scholar.google.com/citations?user=y5zkBeMAAAAJ&hl=en).
Huazhong University of Science and Technology · Beijing Jiaotong University ·
Horizon Robotics
## Models and assets
| Path | Description |
| --- | --- |
| `MF/pretrain` | 1.6B pretraining model |
| `MF/sft` | 1.6B supervised fine-tuned model |
| `Text Decoder/` | Shared BF16 text decoder |
| `Vision statistics/` | Shared FP32 vision normalization statistics |
The model weights are BF16. The release contains inference weights and
configuration only; training states and optimizer states are not included.
## Usage
Use the models with the open-source code:
[Multimodal-Flow](https://github.com/hustvl/Multimodal-Flow)
```bash
# Text continuation
mf infer --checkpoint ./MF/pretrain text \
--prompt "A short language model can"
# Image understanding
mf infer --checkpoint ./MF/sft caption \
--image /path/to/image.jpg \
--prompt "Describe this image."
# Text-to-image generation
mf infer --checkpoint ./MF/sft image \
--prompt "A quiet observatory above the clouds." \
--output outputs/sample.png
```
Text inference uses the model package directly. Image understanding and image
generation additionally require the compatible Scale RAE decoder assets; set
`MF_ASSETS_ROOT` to a directory containing `scale_rae_decoder/`.
## License
MIT License. Please also review the licenses of external encoder, tokenizer,
and decoder assets used with the models.
## Citation
```bibtex
@article{tao2026multimodalflow,
title={Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces},
author={Tao, Hongyuan and Wang, Xinggang and Zhu, Lianghui and Li, Yongkang and Wei, Yunchao and Feng, Bin and Chen, Shaoyu and Zhang, Qian and Huang, Chang and Yu, Kai},
journal={arXiv preprint arXiv:2609.40362},
year={2026},
url={https://arxiv.org/abs/2609.40362}
}
``` |