File size: 2,443 Bytes
5b016d7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3940779
5b016d7
 
3940779
5b016d7
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
---
license: apache-2.0
base_model: ByteDance-Seed/BAGEL-7B-MoT
pipeline_tag: any-to-any
language:
- en
tags:
- unimore
- bagel
- multimodal
- text-to-image
datasets:
- HuggingSelf/UniMore
---

# UniMore-BAGEL

The final UniMore checkpoint based on BAGEL-7B-MoT, supporting multimodal
understanding and image generation. This repository contains the merged BF16
model, its VAE, tokenizer, and configuration files.

## Method

Two branches are trained independently from the same pretrained BAGEL model:

- Generation: grounded phrase-to-crop supervision with the native generation loss.
- Understanding: full-image caption supervision with positive cross-entropy and
  full-caption ranking against detail-omission, incorrect-detail, and random negatives.

The branches are combined using

```text
W = W_base + ((W_t2i - W_base) + lambda * (W_i2t - W_base))
```

Here, `lambda` controls the understanding branch's contribution. Merge arithmetic
uses BF16, including intermediate differences and additions.

## Download and Use

```bash
hf download HuggingSelf/UniMore --local-dir /data/models/UniMore-BAGEL
```

Use the BAGEL adapter in [TorchUMM](https://github.com/AIFrontierLab/TorchUMM),
setting `model_path` to the downloaded directory. Installation, training, and
inference instructions are in the
[UniMore package](https://github.com/AIFrontierLab/TorchUMM/tree/main/src/umm/post_training/UniMore).
These weights use BAGEL's native `ema.safetensors` format, not a Transformers
`AutoModel` checkpoint.

The generation evaluation configuration uses 512px images, 50 sampling steps,
text/image CFG 4/1, timestep shift 3, and seed 42.

## Training

| Setting | Generation | Understanding |
|---|---:|---:|
| Training records | 2,460 | 4,000 |
| Optimizer updates | 1,000 | 600 |
| Learning rate | 5e-5 | 1.8e-5 |
| Warmup updates | 50 | 60 |
| Training image size | 256 | 112 |
| LoRA rank / alpha | 16 / 32 | 16 / 32 |

Training data and a preparation script are available in the
[UniMore dataset repository](https://huggingface.co/datasets/HuggingSelf/UniMore).

## Files and Attribution

- `ema.safetensors`: final merged model.
- `ae.safetensors`: the original BAGEL VAE.
- Tokenizer and JSON files: the original BAGEL inference configuration.
- `release.json`: file sizes and fixed source revisions.

The base model is [BAGEL-7B-MoT](https://huggingface.co/ByteDance-Seed/BAGEL-7B-MoT).
The Apache 2.0 license is included in `LICENSE`.