| --- |
| license: apache-2.0 |
| --- |
| |
| # AnyControl: Create Your Artwork with Versatile Control on Text-to-Image Generation |
| [Yanan Sun](https://scholar.google.com/citations?user=6TA1oPkAAAAJ&hl=en), Yanchen Liu, Yinhao Tang, [Wenjie Pei](https://wenjiepei.github.io/) and [Kai Chen*](https://chenkai.site/) |
|
|
| **Shanghai AI Laboratory** |
|
|
|  |
|
|
|
|
| ## Overview |
| The field of text-to-image (T2I) generation has made significant progress in recent years, |
| largely driven by advancements in diffusion models. |
| Linguistic control enables effective content creation, but struggles with fine-grained control over image generation. |
| This challenge has been explored, to a great extent, by incorporating additional usersupplied spatial conditions, |
| such as depth maps and edge maps, into pre-trained T2I models through extra encoding. |
| However, multi-control image synthesis still faces several challenges. |
| Specifically, current approaches are limited in handling free combinations of diverse input control signals, |
| overlook the complex relationships among multiple spatial conditions, and often fail to maintain semantic alignment with provided textual prompts. |
| This can lead to suboptimal user experiences. To address these challenges, we propose AnyControl, |
| a multi-control image synthesis framework that supports arbitrary combinations of diverse control signals. |
| AnyControl develops a novel Multi-Control Encoder that extracts a unified multi-modal embedding to guide the generation process. |
| This approach enables a holistic understanding of user inputs, and produces high-quality, |
| faithful results under versatile control signals, as demonstrated by extensive quantitative and qualitative evaluations. |
|
|
|
|
| ## Model Card |
| AnyControl for SD 1.5 |
| - `ckpts/anycontrol_15.ckpt`: weights for AnyControl. |
| - `ckpts/init_local.ckpt`: initial weights of AnyControl during training, generated following [Uni-ControlNet](https://github.com/ShihaoZhaoZSH/Uni-ControlNet). |
| - `ckpts/blip2_pretrained.pth`: third-party model. |
| - `annotator/ckpts`: third-party models used in annotators. |
|
|
|
|
| ## License and Citation |
|
|
| All models and assets are under the [Apache 2.0 license](./LICENSE) unless specified otherwise. |
|
|
| If this work is helpful for your research, please consider citing the following BibTeX entry. |
|
|
| ``` bibtex |
| @misc{sun2024anycontrol, |
| title={AnyControl: Create your artwork with versatile control on text-to-image generation}, |
| author={Sun, Yanan and Liu, Yanchen and Tang, Yinhao and Pei, Wenjie and Chen, Kai}, |
| booktitle={ECCV}, |
| year={2024} |
| |
| } |
| ``` |
