README.md

---
license: apache-2.0
tags:
- multimodal
- discrete-diffusion
- vision-language
- unified-model
pipeline_tag: any-to-any
library_name: transformers
---

<p align="center">
  <img src="assets/gestalt-logo.svg" alt="Gestalt" width="360"/>
</p>

<h1 align="center">Gestalt: Large Multimodal Interplay Model</h1>

<p align="center">
  <a href="https://gewu-lab.github.io/Gestalt/"><img src="https://img.shields.io/badge/Gestalt-Website-0A66C2?logo=safari&logoColor=white" alt="Website"/></a>
  <a href="https://arxiv.org/pdf/2610.00576"><img src="https://img.shields.io/badge/Gestalt-Paper-red?logo=arxiv&logoColor=red" alt="Paper"/></a>
  <a href="https://github.com/GeWu-Lab/Gestalt"><img src="https://img.shields.io/badge/Gestalt-Code-181717?logo=github" alt="Code"/></a>
  <a href="https://huggingface.co/GeWuLab/Gestalt"><img src="https://img.shields.io/badge/Gestalt-Model-yellow?logo=huggingface&logoColor=yellow" alt="Model"/></a>
</p>

<p align="center">
  <a href="https://bjlfzs.github.io/">Zequn Yang</a><sup>†</sup>&emsp;
  <a href="https://github.com/fishmeoww">Yu Miao</a><sup>†</sup>&emsp;
  <a href="https://github.com/nihaotian1">Haotian Ni</a><sup>†</sup>&emsp;
  <a href="https://github.com/Gutsczh">Ziheng Chen</a><sup>†</sup>&emsp;
  <a href="https://scholar.google.com/citations?user=YJWX9e0AAAAJ">Chengxiang Huang</a><sup>†</sup>
  <br/>
  <a href="https://scholar.google.com/citations?user=Ox6SxpoAAAAJ">Dongzhan Zhou</a>&emsp;
  <a href="https://chenkai.site/">Kai Chen</a>&emsp;
  <a href="http://qizhang.info/">Qi Zhang</a>&emsp;
  <a href="https://gsai.ruc.edu.cn/english/jrwen">Ji-Rong Wen</a>&emsp;
  <a href="https://echo0409.github.io/">Yake Wei</a><sup>‡</sup>&emsp;
  <a href="https://dtaoo.github.io/">Di Hu</a><sup>‡,✉</sup>
</p>

<p align="center">
  <sub><sup>†</sup> Equal contribution&emsp;<sup>‡</sup> Team leader&emsp;<sup>✉</sup> Corresponding author</sub>
</p>

> **Gestalt** is a new paradigm of large multimodal model built around **multimodal interplay**. Guided by a multimodal interplay pyramid — from modality-specific modeling, through cross-modal alignment, to multimodal synergy — Gestalt adopts a unified **discrete diffusion** framework with an interplay-partitioned architecture, where learnable interplay tokens mediate cross-modal exchange and integration. The name is inspired by Gestalt psychology: *the whole is greater than the sum of its parts.*

<p align="center">
  <img src="assets/teaser.png" alt="Gestalt teaser" width="95%"/>
</p>

## Model Description

**Gestalt** is an interplay-centric large multimodal model built on a unified discrete diffusion framework. Through multimodal pretraining, continual pretraining, and interplay-oriented supervised fine-tuning,Gestalt supports both understanding and generation.

| Property | Value |
|:--|:--|
| Architecture | GestaltModelLM (discrete diffusion with interplay tokens) |
| Parameters | 8B + 0.6B |
| Hidden size | 4096 |
| Layers | 32 |
| Attention heads | 32 |
| Vocab size | 142,848 |
| Max sequence length | 4,096 |
| Precision | bfloat16 |

## Download

```bash
# Gestalt model
huggingface-cli download GeWuLab/Gestalt --local-dir /path/to/gestalt

# IBQ vision tokenizer (required for encoding/decoding images)
huggingface-cli download TencentARC/IBQ-Tokenizer-16384 --local-dir /path/to/ibq
```

The IBQ tokenizer is from [TencentARC/IBQ-Tokenizer-16384](https://huggingface.co/TencentARC/IBQ-Tokenizer-16384), used to convert between images and discrete visual tokens.

## Training Pipeline

| Stage | Data | Objective |
|:--|:--:|:--|
| **Multimodal Pretraining** | 70M | Joint masked prediction |
| **Continual Pretraining** ← this checkpoint | 8M | Conditional masked prediction |
| **Supervised Fine-Tuning** | 13.7M (+≈2.5M T2I) | Four interplay categories, three-phase curriculum |

## Citation

If you find Gestalt useful for your research, please cite:

```bibtex
@article{gestalt2026,
  title   = {Gestalt: Large Multimodal Interplay Model},
  author  = {Yang, Zequn and Miao, Yu and Ni, Haotian and Chen, Ziheng and
             Huang, Chengxiang and Zhou, Dongzhan and Chen, Kai and Zhang, Qi and
             Wen, Ji-Rong and Wei, Yake and Hu, Di},
  year    = {2026},
  url     = {https://github.com/GeWu-Lab/Gestalt}
}
```

## License

This project is released under the Apache 2.0 license.

## Author Contributions

Zequn Yang, Yake Wei, and Di Hu drove the overall advancement of the project. Zequn Yang, Yu Miao, Haotian Ni, Ziheng Chen, and Chengxiang Huang contributed equally to this work. Zequn Yang conducted model pretraining and supervised fine-tuning. Zequn Yang, Yu Miao, and Haotian Ni developed the model architecture and conducted the core experiments. Yu Miao, Ziheng Chen, and Chengxiang Huang contributed to data processing and organization. Yu Miao conducted the evaluation of image generation capabilities, Haotian Ni conducted the multimodal understanding evaluation, and Ziheng Chen conducted the text-only evaluation. Dongzhan Zhou, Kai Chen, Qi Zhang, and Ji-Rong Wen contributed to discussions on the technical design and methodology. Yake Wei, Zequn Yang, Yu Miao, Haotian Ni, Ziheng Chen, and Di Hu contributed to writing and revising the manuscript. Di Hu initiated the project. Yake Wei and Di Hu supervised and advised the project.