|
Download README.md from HuggingSelf/UniMore: direct link, hf CLI and curl.
- Browser
- Download file 2.44 kB
-
https://huggingface.co/HuggingSelf/UniMore/resolve/main/README.md
- Command line
-
hf download hf://HuggingSelf/UniMore/README.md
-
curl -L -o README.md https://huggingface.co/HuggingSelf/UniMore/resolve/main/README.md
2.44 kB
| license: apache-2.0 | |
| base_model: ByteDance-Seed/BAGEL-7B-MoT | |
| pipeline_tag: any-to-any | |
| language: | |
| - en | |
| tags: | |
| - unimore | |
| - bagel | |
| - multimodal | |
| - text-to-image | |
| datasets: | |
| - HuggingSelf/UniMore | |
| # UniMore-BAGEL | |
| The final UniMore checkpoint based on BAGEL-7B-MoT, supporting multimodal | |
| understanding and image generation. This repository contains the merged BF16 | |
| model, its VAE, tokenizer, and configuration files. | |
| ## Method | |
| Two branches are trained independently from the same pretrained BAGEL model: | |
| - Generation: grounded phrase-to-crop supervision with the native generation loss. | |
| - Understanding: full-image caption supervision with positive cross-entropy and | |
| full-caption ranking against detail-omission, incorrect-detail, and random negatives. | |
| The branches are combined using | |
| ```text | |
| W = W_base + ((W_t2i - W_base) + lambda * (W_i2t - W_base)) | |
| ``` | |
| Here, `lambda` controls the understanding branch's contribution. Merge arithmetic | |
| uses BF16, including intermediate differences and additions. | |
| ## Download and Use | |
| ```bash | |
| hf download HuggingSelf/UniMore --local-dir /data/models/UniMore-BAGEL | |
| ``` | |
| Use the BAGEL adapter in [TorchUMM](https://github.com/AIFrontierLab/TorchUMM), | |
| setting `model_path` to the downloaded directory. Installation, training, and | |
| inference instructions are in the | |
| [UniMore package](https://github.com/AIFrontierLab/TorchUMM/tree/main/src/umm/post_training/UniMore). | |
| These weights use BAGEL's native `ema.safetensors` format, not a Transformers | |
| `AutoModel` checkpoint. | |
| The generation evaluation configuration uses 512px images, 50 sampling steps, | |
| text/image CFG 4/1, timestep shift 3, and seed 42. | |
| ## Training | |
| | Setting | Generation | Understanding | | |
| |---|---:|---:| | |
| | Training records | 2,460 | 4,000 | | |
| | Optimizer updates | 1,000 | 600 | | |
| | Learning rate | 5e-5 | 1.8e-5 | | |
| | Warmup updates | 50 | 60 | | |
| | Training image size | 256 | 112 | | |
| | LoRA rank / alpha | 16 / 32 | 16 / 32 | | |
| Training data and a preparation script are available in the | |
| [UniMore dataset repository](https://huggingface.co/datasets/HuggingSelf/UniMore). | |
| ## Files and Attribution | |
| - `ema.safetensors`: final merged model. | |
| - `ae.safetensors`: the original BAGEL VAE. | |
| - Tokenizer and JSON files: the original BAGEL inference configuration. | |
| - `release.json`: file sizes and fixed source revisions. | |
| The base model is [BAGEL-7B-MoT](https://huggingface.co/ByteDance-Seed/BAGEL-7B-MoT). | |
| The Apache 2.0 license is included in `LICENSE`. | |