| --- |
| library_name: diffusers |
| license: creativeml-openrail-m |
| base_model: stable-diffusion-v1-5/stable-diffusion-v1-5 |
| pipeline_tag: text-to-image |
| tags: |
| - stable-diffusion |
| - text-to-image |
| - procreate |
| datasets: |
| - Jacklu0831/few-shot-creative-generation-8 |
| --- |
| |
| # ProCreate checkpoints |
|
|
| Eight Stable Diffusion 1.5 UNets, each fine-tuned for 2,000 steps on one category of [FSCG-8](https://huggingface.co/datasets/Jacklu0831/few-shot-creative-generation-8) (50 caption-image pairs per category), released with **ProCreate, Don't Reproduce! Propulsive Energy Diffusion for Creative Generation** (ECCV 2024). |
|
|
| ProCreate is a sampling method that pushes generated images away from the reference images, so a model fine-tuned on a few examples produces novel samples in the same style instead of replicating its training data. |
|
|
| [Paper](https://arxiv.org/abs/2408.02226) 路 [Project page](https://agenticlearning.ai/procreate/) 路 [Code](https://github.com/agentic-learning-ai-lab/procreate-diffusion) 路 [Dataset](https://huggingface.co/datasets/Jacklu0831/few-shot-creative-generation-8) |
|
|
| ## Checkpoints |
|
|
| One folder per FSCG-8 category, each holding a fine-tuned UNet (`UNet2DConditionModel`, diffusers format): |
|
|
| | Folder | Fine-tuned on | Sample prompt | |
| | --- | --- | --- | |
| | `amedeo_modigliani` | Amedeo Modigliani paintings | a Amedeo Modigliani painting of a boy in a suit and hat | |
| | `apple` | Apple product designs | an Apple VR headset | |
| | `burberry` | Burberry designs | a Burberry stuffed bear | |
| | `frank_gehry` | Frank Gehry architecture | a twisting tall apartment building, designed by Frank Gehry | |
| | `nouns` | Nouns characters | a mountain Nouns character | |
| | `one_piece` | One Piece characters | a One Piece man in a purple robe | |
| | `pokemon` | Pokemon | a jellyfish Pokemon | |
| | `rococo` | Rococo style art and furniture | a Rococo style bed | |
|
|
| ## Usage |
|
|
| Load a category's UNet into a Stable Diffusion 1.5 pipeline: |
|
|
| ```python |
| import torch |
| from diffusers import StableDiffusionPipeline, UNet2DConditionModel |
| |
| pipe = StableDiffusionPipeline.from_pretrained( |
| "stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.float16 |
| ).to("cuda") |
| pipe.unet = UNet2DConditionModel.from_pretrained( |
| "Jacklu0831/procreate-diffusion", subfolder="apple", torch_dtype=torch.float16 |
| ).to("cuda") |
| image = pipe("an Apple VR headset").images[0] |
| ``` |
|
|
| This samples from the fine-tuned model directly. To sample with ProCreate, use `src/inference.py` in the [code repository](https://github.com/agentic-learning-ai-lab/procreate-diffusion), which downloads the checkpoint automatically: |
|
|
| ```bash |
| python src/inference.py \ |
| --dataset_dir few-shot-creative-generation-8/apple \ |
| --unet_ckpt_dir Jacklu0831/procreate-diffusion \ |
| --unet_subfolder apple \ |
| --prompt "an Apple VR headset" |
| ``` |
|
|
| ## License |
|
|
| The weights are a derivative of Stable Diffusion 1.5 and are released under the [CreativeML Open RAIL-M license](https://huggingface.co/spaces/CompVis/stable-diffusion-license). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{lu2024procreate, |
| title = {ProCreate, Don't Reproduce! Propulsive Energy Diffusion for Creative Generation}, |
| author = {Lu, Jack and Teehan, Ryan and Ren, Mengye}, |
| booktitle = {European Conference on Computer Vision (ECCV)}, |
| year = {2024} |
| } |
| ``` |
|
|
