File size: 3,289 Bytes
40d98a7 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 | ---
library_name: diffusers
license: creativeml-openrail-m
base_model: stable-diffusion-v1-5/stable-diffusion-v1-5
pipeline_tag: text-to-image
tags:
- stable-diffusion
- text-to-image
- procreate
datasets:
- Jacklu0831/few-shot-creative-generation-8
---
# ProCreate checkpoints
Eight Stable Diffusion 1.5 UNets, each fine-tuned for 2,000 steps on one category of [FSCG-8](https://huggingface.co/datasets/Jacklu0831/few-shot-creative-generation-8) (50 caption-image pairs per category), released with **ProCreate, Don't Reproduce! Propulsive Energy Diffusion for Creative Generation** (ECCV 2024).
ProCreate is a sampling method that pushes generated images away from the reference images, so a model fine-tuned on a few examples produces novel samples in the same style instead of replicating its training data.
[Paper](https://arxiv.org/abs/2408.02226) 路 [Project page](https://agenticlearning.ai/procreate/) 路 [Code](https://github.com/agentic-learning-ai-lab/procreate-diffusion) 路 [Dataset](https://huggingface.co/datasets/Jacklu0831/few-shot-creative-generation-8)
## Checkpoints
One folder per FSCG-8 category, each holding a fine-tuned UNet (`UNet2DConditionModel`, diffusers format):
| Folder | Fine-tuned on | Sample prompt |
| --- | --- | --- |
| `amedeo_modigliani` | Amedeo Modigliani paintings | a Amedeo Modigliani painting of a boy in a suit and hat |
| `apple` | Apple product designs | an Apple VR headset |
| `burberry` | Burberry designs | a Burberry stuffed bear |
| `frank_gehry` | Frank Gehry architecture | a twisting tall apartment building, designed by Frank Gehry |
| `nouns` | Nouns characters | a mountain Nouns character |
| `one_piece` | One Piece characters | a One Piece man in a purple robe |
| `pokemon` | Pokemon | a jellyfish Pokemon |
| `rococo` | Rococo style art and furniture | a Rococo style bed |
## Usage
Load a category's UNet into a Stable Diffusion 1.5 pipeline:
```python
import torch
from diffusers import StableDiffusionPipeline, UNet2DConditionModel
pipe = StableDiffusionPipeline.from_pretrained(
"stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.float16
).to("cuda")
pipe.unet = UNet2DConditionModel.from_pretrained(
"Jacklu0831/procreate-diffusion", subfolder="apple", torch_dtype=torch.float16
).to("cuda")
image = pipe("an Apple VR headset").images[0]
```
This samples from the fine-tuned model directly. To sample with ProCreate, use `src/inference.py` in the [code repository](https://github.com/agentic-learning-ai-lab/procreate-diffusion), which downloads the checkpoint automatically:
```bash
python src/inference.py \
--dataset_dir few-shot-creative-generation-8/apple \
--unet_ckpt_dir Jacklu0831/procreate-diffusion \
--unet_subfolder apple \
--prompt "an Apple VR headset"
```
## License
The weights are a derivative of Stable Diffusion 1.5 and are released under the [CreativeML Open RAIL-M license](https://huggingface.co/spaces/CompVis/stable-diffusion-license).
## Citation
```bibtex
@inproceedings{lu2024procreate,
title = {ProCreate, Don't Reproduce! Propulsive Energy Diffusion for Creative Generation},
author = {Lu, Jack and Teehan, Ryan and Ren, Mengye},
booktitle = {European Conference on Computer Vision (ECCV)},
year = {2024}
}
```
|