UniMore / README.md
HuggingSelf's picture
Remove displayed file checksums
3940779 verified
|
Raw History Blame Contribute Delete
2.44 kB
---
license: apache-2.0
base_model: ByteDance-Seed/BAGEL-7B-MoT
pipeline_tag: any-to-any
language:
- en
tags:
- unimore
- bagel
- multimodal
- text-to-image
datasets:
- HuggingSelf/UniMore
---
# UniMore-BAGEL
The final UniMore checkpoint based on BAGEL-7B-MoT, supporting multimodal
understanding and image generation. This repository contains the merged BF16
model, its VAE, tokenizer, and configuration files.
## Method
Two branches are trained independently from the same pretrained BAGEL model:
- Generation: grounded phrase-to-crop supervision with the native generation loss.
- Understanding: full-image caption supervision with positive cross-entropy and
full-caption ranking against detail-omission, incorrect-detail, and random negatives.
The branches are combined using
```text
W = W_base + ((W_t2i - W_base) + lambda * (W_i2t - W_base))
```
Here, `lambda` controls the understanding branch's contribution. Merge arithmetic
uses BF16, including intermediate differences and additions.
## Download and Use
```bash
hf download HuggingSelf/UniMore --local-dir /data/models/UniMore-BAGEL
```
Use the BAGEL adapter in [TorchUMM](https://github.com/AIFrontierLab/TorchUMM),
setting `model_path` to the downloaded directory. Installation, training, and
inference instructions are in the
[UniMore package](https://github.com/AIFrontierLab/TorchUMM/tree/main/src/umm/post_training/UniMore).
These weights use BAGEL's native `ema.safetensors` format, not a Transformers
`AutoModel` checkpoint.
The generation evaluation configuration uses 512px images, 50 sampling steps,
text/image CFG 4/1, timestep shift 3, and seed 42.
## Training
| Setting | Generation | Understanding |
|---|---:|---:|
| Training records | 2,460 | 4,000 |
| Optimizer updates | 1,000 | 600 |
| Learning rate | 5e-5 | 1.8e-5 |
| Warmup updates | 50 | 60 |
| Training image size | 256 | 112 |
| LoRA rank / alpha | 16 / 32 | 16 / 32 |
Training data and a preparation script are available in the
[UniMore dataset repository](https://huggingface.co/datasets/HuggingSelf/UniMore).
## Files and Attribution
- `ema.safetensors`: final merged model.
- `ae.safetensors`: the original BAGEL VAE.
- Tokenizer and JSON files: the original BAGEL inference configuration.
- `release.json`: file sizes and fixed source revisions.
The base model is [BAGEL-7B-MoT](https://huggingface.co/ByteDance-Seed/BAGEL-7B-MoT).
The Apache 2.0 license is included in `LICENSE`.