README.md

---
license: creativeml-openrail-m
base_model: runwayml/stable-diffusion-v1-5
tags:
- stable-diffusion
- stable-diffusion-diffusers
- text-to-image
- diffusers
- lora
- controlnet
- control-lora
inference: true
---
# ControlLoRA text2image fine-tuning - Official Model Repository
These are ControlLoRA adaption weights for [runwayml/stable-diffusion-v1-5](https://huggingface.co/runwayml/stable-diffusion-v1-5). You can find the code repository in [HighCWu/ControlLoRA](https://github.com/HighCWu/ControlLoRA).

With ControlLoRA, a simple and small (~7M parameters, ~25M storage space) network, you could control the spatial information of stable diffusion.

Each of the weights is fine-tuned on the [diffusiondb_2m_first_5k_canny](https://huggingface.co/datasets/HighCWu/diffusiondb_2m_first_5k_canny) and [mpii_100_openpose](https://huggingface.co/datasets/HighCWu/mpii_100_openpose) datasets with 30k steps on RTX4080 in 3 hours. 

You can find some example images in the following. 

![img_{i}](./docs/imgs/validation_25100_fe0b49075b45efdc311c.png)
*boys are playing with a frisbee in a field, 
2009 cinematography, trending on artforum, running pose, 
bruce springsteen, connected to heart machines, with tattoos, 
beautiful - n 9, by Eric Dinyer, young child, midlands*

![img_{i}](./docs/imgs/validation_25000_6b3f8b37f2268ec0c09e.png)
*portrait of a dancing eagle woman, "
"beautiful blonde haired lakota sioux goddess, "
"intricate, highly detailed art by james jean, "
"ray tracing, digital painting, artstation, "
"concept art, smooth, sharp focus, illustration, "
"artgerm and greg rutkowski and alphonse mucha, "
"vladimir kush, giger, roger dean, 8 k*

I also uploaded a lora model fine-tuned on my selfies with 2k steps which could be use in the experiment of mixing LoRA and ControlLoRA.

![img_{i}](./docs/imgs/7.png)
*portrait of female HighCWu as a cute pink hair girl*