Spaces:
Sleeping
Sleeping
File size: 1,335 Bytes
e8fb64b 2267636 e8fb64b a5b5245 e8fb64b 2267636 a07f22a e8fb64b 2267636 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | ---
title: MMDiff
emoji: 🎨
colorFrom: blue
colorTo: yellow
sdk: gradio
sdk_version: "5.23.0"
app_file: app.py
short_description: Multi-modal generation with diffusion transformers
python_version: "3.10"
startup_duration_timeout: "10m"
---
# MMDiff: Extending Diffusion Transformers for Multi-Modal Generation
This Space demonstrates MMDiff, a method that extends frozen diffusion transformers (FLUX.1-dev) to generate images alongside dense predictions (saliency maps, segmentation maps, depth maps) in a single forward pass.
## How it works
1. A text prompt is used to generate an image with FLUX.1-dev
2. During denoising, intermediate transformer features and concept attention maps are captured
3. Lightweight trained decoder heads (DPT, DeepLabV3+) decode these features into dense predictions:
- **Saliency** (DUTS): Binary foreground/background segmentation
- **Segmentation** (Pascal VOC): 21-class semantic segmentation
- **Depth** (NYU Depth V2): Monocular depth estimation
## Model
- **Backbone**: [black-forest-labs/FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) (frozen)
- **Decoder weights**: [yagmurakarken/mmdiff](https://huggingface.co/yagmurakarken/mmdiff)
- **Paper**: [MMDiff: Extending Diffusion Transformers for Multi-Modal Generation](https://huggingface.co/papers/2606.16673) |