Upload README.md with huggingface_hub

README.md

@@ -1,112 +1,85 @@
 ---
 license: apache-2.0
-pipeline_tag: any-to-any
-library_name: transformers
 tags:
-- multimodal
-- unified-model
-- discrete-diffusion
-- text-to-image
-- image-to-text
-language:
-- en
+  - multimodal
+  - discrete-diffusion
+  - vision-language
+  - continual-pretraining
 ---
 
 <p align="center">
-  <img src="assets/gestalt-logo.svg" alt="Gestalt" width="360"/>
+  <img src="https://raw.githubusercontent.com/GeWu-Lab/Gestalt/main/assets/gestalt-logo.svg" alt="Gestalt" width="360"/>
 </p>
 
 <h1 align="center">Gestalt: Large Multimodal Interplay Model</h1>
 
 <p align="center">
   <a href="https://gewu-lab.github.io/Gestalt/"><img src="https://img.shields.io/badge/Gestalt-Website-0A66C2?logo=safari&logoColor=white" alt="Website"/></a>
-  <a href="#"><img src="https://img.shields.io/badge/Gestalt-Paper-red?logo=arxiv&logoColor=red" alt="Paper"/></a>
+  <a href="https://github.com/GeWu-Lab/Gestalt/blob/main/archive/Gestalt.pdf"><img src="https://img.shields.io/badge/Gestalt-Paper-red?logo=arxiv&logoColor=red" alt="Paper"/></a>
   <a href="https://github.com/GeWu-Lab/Gestalt"><img src="https://img.shields.io/badge/Gestalt-Code-181717?logo=github" alt="Code"/></a>
-  <a href="https://huggingface.co/GeWu-Lab/Gestalt"><img src="https://img.shields.io/badge/Gestalt-Model-yellow?logo=huggingface&logoColor=yellow" alt="Model"/></a>
+  <a href="https://huggingface.co/GeWuLab/Gestalt"><img src="https://img.shields.io/badge/Gestalt-Model-yellow?logo=huggingface&logoColor=yellow" alt="Model"/></a>
 </p>
 
-> **Gestalt** is a new paradigm of large multimodal model built around **multimodal interplay**. Guided by a multimodal interplay pyramid — from modality-specific modeling, through cross-modal alignment, to multimodal synergy — Gestalt adopts a unified **discrete diffusion** framework with an interplay-partitioned architecture, where learnable interplay tokens mediate cross-modal exchange and integration. The name is inspired by Gestalt psychology: *the whole is greater than the sum of its parts.*
-
 <p align="center">
-  <img src="assets/teaser.png" alt="Gestalt teaser" width="95%"/>
+  <a href="https://bjlfzs.github.io/">Zequn Yang</a><sup>†</sup>&emsp;
+  <a href="https://github.com/fishmeoww">Yu Miao</a><sup>†</sup>&emsp;
+  <a href="https://github.com/nihaotian1">Haotian Ni</a><sup>†</sup>&emsp;
+  <a href="https://github.com/Gutsczh">Ziheng Chen</a><sup>†</sup>&emsp;
+  <a href="https://scholar.google.com/citations?user=YJWX9e0AAAAJ">Chengxiang Huang</a><sup>†</sup>
+  <br/>
+  <a href="https://scholar.google.com/citations?user=Ox6SxpoAAAAJ">Dongzhan Zhou</a>&emsp;
+  <a href="https://chenkai.site/">Kai Chen</a>&emsp;
+  <a href="http://qizhang.info/">Qi Zhang</a>&emsp;
+  <a href="https://gsai.ruc.edu.cn/english/jrwen">Ji-Rong Wen</a>&emsp;
+  <a href="https://echo0409.github.io/">Yake Wei</a><sup>‡</sup>&emsp;
+  <a href="https://dtaoo.github.io/">Di Hu</a><sup>‡,✉</sup>
 </p>
 
-## 🏗️ Architecture
-
-Gestalt brings vision and language into a shared discrete diffusion framework. The Transformer is divided into two successive zones, with **learnable interplay tokens** mediating cross-modal information flow throughout:
-
 <p align="center">
-  <img src="assets/method.png" alt="Gestalt architecture" width="95%"/>
+  <sub><sup>†</sup> Equal contribution&emsp;<sup>‡</sup> Team leader&emsp;<sup>✉</sup> Corresponding author</sub>
 </p>
 
-- **Interplay Zone I — Bottleneck Interplay.** Separate visual and textual FFNs preserve each modality's information; direct cross-modal attention is restricted, and exchange is mediated by the interplay tokens.
-- **Interplay Zone II — Full Multimodal Interplay.** The restriction is removed: image, text, and interplay tokens interact through full multimodal self-attention and a shared FFN, integrating complementary information across all tokens.
-
-## 🔥 Quick Start
-
-Please refer to the [GitHub repository](https://github.com/GeWu-Lab/Gestalt) for the full training and inference code.
-
-```bash
-git clone https://github.com/GeWu-Lab/Gestalt.git
-cd Gestalt
-pip install -r requirements.txt
-
-huggingface-cli download GeWu-Lab/Gestalt --local-dir /path/to/checkpoint
-```
-
-**Multimodal understanding (MMU)** — image tokens + question → text:
-
-```bash
-python scripts/inference/run_mmu.py \
-  --model /path/to/checkpoint \
-  --image-tokens image_tokens.npy \
-  --question "What is in this image?" \
-  --token-h 32 --token-w 32
-```
-
-**Text-to-image (T2I)** — text → VQVAE tokens:
-
-```bash
-python scripts/inference/run_t2i.py \
-  --model /path/to/checkpoint \
-  --prompt "A cat sitting on a windowsill" \
-  --output output_tokens.npy \
-  --lat-h 32 --lat-w 32
-```
-
-## 📊 Benchmarks
-
-### 🎨 Fine-Grained Text-to-Image Generation
-
-**UniGenBench** — Gestalt achieves the strongest overall result among the evaluated models, combining fine-grained semantic alignment with compositional generation.
+> **Gestalt** is a new paradigm of large multimodal model built around **multimodal interplay**. Guided by a multimodal interplay pyramid — from modality-specific modeling, through cross-modal alignment, to multimodal synergy — Gestalt adopts a unified **discrete diffusion** framework with an interplay-partitioned architecture, where learnable interplay tokens mediate cross-modal exchange and integration. The name is inspired by Gestalt psychology: *the whole is greater than the sum of its parts.*
 
 <p align="center">
-  <img src="assets/unigenbench.png" alt="UniGenBench results" width="95%"/>
+  <img src="https://raw.githubusercontent.com/GeWu-Lab/Gestalt/main/assets/gestalt_1.png" alt="Gestalt teaser" width="95%"/>
 </p>
 
-**TIIF-Bench** — Gestalt leads across short and long instructions, with a larger advantage when longer prompts introduce more interdependent requirements.
+## Model Description
 
-<p align="center">
-  <img src="assets/tiifbench.png" alt="TIIF-Bench results" width="95%"/>
-</p>
+This is the **Continual Pretraining (CPT)** checkpoint of Gestalt, trained with conditional masked prediction on 8M samples. For the full SFT model, see [GeWuLab/Gestalt-SFT](https://huggingface.co/GeWuLab/Gestalt-SFT).
 
-### 👁️ Multimodal Understanding
+| Property | Value |
+|:--|:--|
+| Architecture | GestaltModelLM (discrete diffusion with interplay tokens) |
+| Parameters | ~8B |
+| Hidden size | 4096 |
+| Layers | 32 |
+| Attention heads | 32 |
+| Vocab size | 142,848 |
+| Max sequence length | 4,096 |
+| Precision | bfloat16 |
 
-The gains on vision-centric tasks reflect the value of preserving modality-specific information within a unified model.
+## Download
 
-<p align="center">
-  <img src="assets/mmu.png" alt="Multimodal understanding results" width="95%"/>
-</p>
+```bash
+# Using huggingface-cli
+huggingface-cli download GeWuLab/Gestalt --local-dir /path/to/checkpoint
 
-### 📝 Language
+# Or using git
+git lfs install
+git clone https://huggingface.co/GeWuLab/Gestalt
+```
 
-Gestalt leads the evaluated diffusion-based unified models and achieves language performance comparable to autoregressive models such as Janus-Pro.
+## Training Pipeline
 
-<p align="center">
-  <img src="assets/text.png" alt="Language benchmark results" width="80%"/>
-</p>
+| Stage | Data | Objective |
+|:--|:--:|:--|
+| **Multimodal Pretraining** | 70M | Joint masked prediction |
+| **Continual Pretraining** ← this checkpoint | 8M | Conditional masked prediction |
+| **Supervised Fine-Tuning** | 13.7M (+≈2.5M T2I) | Four interplay categories, three-phase curriculum |
 
-## ✍️ Citation
+## Citation
 
 If you find Gestalt useful for your research, please cite:
 
@@ -121,10 +94,10 @@ If you find Gestalt useful for your research, please cite:
 }
 ```
 
-## 📜 License
+## License
 
-This model is released under the Apache 2.0 license.
+This project is released under the Apache 2.0 license.
 
-## 🙏 Acknowledgements
+## Acknowledgements
 
-Gestalt is developed by [GeWu-Lab](https://gewu-lab.github.io/) at the Gaoling School of Artificial Intelligence, Renmin University of China, in collaboration with Shanghai Artificial Intelligence Laboratory, Imperial College London, Beijing University of Posts and Telecommunications, and AresoX.
+This work is supported in part by the Beijing Natural Science Foundation under Grant No. 4262050, and by the Beijing Nova Program under Grant No. 202604841277.