Spaces:
Running on Zero
Running on Zero
| title: DiffusionDB SD 1.5 LoRA | |
| emoji: 🦉 | |
| colorFrom: indigo | |
| colorTo: gray | |
| sdk: gradio | |
| sdk_version: 6.25.0 | |
| python_version: '3.12' | |
| app_file: app.py | |
| pinned: false | |
| license: creativeml-openrail-m | |
| short_description: SD 1.5 style LoRA with a live before/after slider | |
| tags: | |
| - text-to-image | |
| - stable-diffusion | |
| - lora | |
| - diffusers | |
| - gradio | |
| models: | |
| - whosouravsharma/diffusiondb-sd15-lora | |
| datasets: | |
| - whosouravsharma/text-to-image-diffusiondb-2M | |
| # DiffusionDB SD 1.5 LoRA | |
| Generate images with Stable Diffusion 1.5 plus a rank-32 **style LoRA** | |
| fine-tuned on 13,598 prompt–image pairs from | |
| [DiffusionDB](https://huggingface.co/datasets/poloclub/diffusiondb). The | |
| adapter gives SD 1.5 a **more saturated, higher-contrast, illustrative look**, | |
| and it follows prompts just as well as the base model. | |
|  | |
| **Set LoRA strength to 0** to render plain SD 1.5 at the same seed. At | |
| strength 0 the output is pixel-identical to the base model, so the slider is | |
| a true before/after comparison. | |
| > This Space records visitor IP addresses. See [Privacy](#privacy). | |
| This Space is the **frontend** of a two-Space system. Image generation runs | |
| on a GPU in the | |
| [`diffusiondb-sd15-lora-inference`](https://huggingface.co/spaces/whosouravsharma/diffusiondb-sd15-lora-inference) | |
| Space. | |
| ## How it works | |
| 1. You enter a prompt and, optionally, a negative prompt, step count, | |
| guidance, LoRA strength and seed. | |
| 2. This Space connects to the inference Space with `gradio_client`. If the | |
| inference Space is asleep, that connection wakes it (see | |
| [Cold starts](#cold-starts)). | |
| 3. The inference Space loads SD 1.5 (fp16) and the LoRA adapter from the | |
| Hub, applies the requested LoRA strength, and generates one 512×512 image | |
| with the DPM-Solver++ scheduler. | |
| 4. The image and the seed used come back here and are shown. | |
| ``` | |
| Browser ──▶ UI Space (this, Gradio) ──gradio_client /generate──▶ Inference Space (T4 GPU) | |
| │ SD 1.5 fp16 | |
| │ + LoRA checkpoint-4240 | |
| │ DPM-Solver++ scheduler | |
| ◀──────────────── image + seed ────────────────────────┘ | |
| ``` | |
| ## Components | |
| | Component | Role | | |
| |---|---| | |
| | **This Space** | Gradio interface. It runs no model code, holds no weights and doesn't install torch | | |
| | [Inference Space](https://huggingface.co/spaces/whosouravsharma/diffusiondb-sd15-lora-inference) | Loads the pipeline and serves one `/generate` endpoint. Runs on a T4 GPU | | |
| | [Model repo](https://huggingface.co/whosouravsharma/diffusiondb-sd15-lora) | LoRA checkpoints, training and evaluation code, and the full model card | | |
| | [Dataset repo](https://huggingface.co/datasets/whosouravsharma/text-to-image-diffusiondb-2M) | The cleaned training data, the cached latents, and the 50 held-out evaluation prompts | | |
| Keeping the interface separate means this Space doesn't need a GPU. Only the | |
| inference Space costs GPU time. | |
| ## Model | |
| | | | | |
| |---|---| | |
| | Base | [`stable-diffusion-v1-5/stable-diffusion-v1-5`](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5) | | |
| | Adapter | LoRA rank 32 on the UNet attention layers, `checkpoint-4240` (the final step) | | |
| | Training | 10 epochs on 13,598 images at 512×512, with the text encoder frozen, in 2.7 hours on one A10G | | |
| | Defaults here | 25 steps, guidance 7.5, LoRA strength 1.0 (adjustable from 0 to 1.5) | | |
| The example prompts in the UI are real prompts from the held-out evaluation | |
| set. The model never trained on them. | |
| ## Evaluation at a glance | |
| The model was evaluated on 506 held-out prompts, against the real DiffusionDB | |
| images for those prompts: | |
| | | KID ×10³ ↓ | CLIP score | | |
| |---|---|---| | |
| | SD 1.5 base | 0.14 ± 0.32 | 28.94 | | |
| | + LoRA `checkpoint-4240` | 2.88 ± 0.61 | 28.98 | | |
| - **Prompt-following is unchanged.** The two CLIP scores are within noise of | |
| each other. | |
| - **The LoRA adds its own style; it doesn't reproduce DiffusionDB's.** | |
| DiffusionDB is itself SD 1.x output, and plain SD 1.5 already matches it | |
| (KID ≈ 0). The adapter moves outputs away from it, toward the saturated, | |
| high-contrast look in the gallery. | |
| - **No extra risk measured.** The safety-checker flag rate is the same as | |
| base (2.6%), and a nearest-neighbour check finds no copying of training | |
| images. | |
| The [model card](https://huggingface.co/whosouravsharma/diffusiondb-sd15-lora) | |
| has the full protocol, the loss curve, the training progression, failure cases | |
| and every figure. | |
| ## Intended use | |
| **Use it for** giving SD 1.5 images a punchier, illustrative finish, and for | |
| seeing what a LoRA fine-tune changes compared with its base model. | |
| **Don't use it for** muted or pastel palettes (the adapter overrides them), | |
| resolutions other than 512×512, or photorealistic images of real people. | |
| ## API | |
| The inference Space can be called directly: | |
| ```python | |
| from gradio_client import Client | |
| client = Client("whosouravsharma/diffusiondb-sd15-lora-inference") | |
| image_path, seed = client.predict( | |
| "a steampunk owl inside a glass jar, intricate detail", # prompt | |
| "", # negative prompt | |
| 25, # steps (at most 50) | |
| 7.5, # guidance | |
| 1.0, # LoRA strength; 0 = plain SD 1.5 | |
| -1, # seed; -1 = random | |
| api_name="/generate", | |
| ) | |
| ``` | |
| ## Cold starts | |
| The inference Space goes to sleep after 5 minutes without requests. Waking | |
| it means scheduling a GPU, starting the container and loading about 2 GB of | |
| weights, so the first image after a wake is slow (the app estimates about | |
| 40 seconds). Later images take a few seconds. | |
| This Space treats that as a wait, not an error. It retries the connection | |
| every 5 seconds for up to 10 minutes and shows **Starting the server…** | |
| with the time elapsed. It shows an error only when waiting can't help: when | |
| the inference Space is `PAUSED`, or in `RUNTIME_ERROR`, `BUILD_ERROR` or | |
| `CONFIG_ERROR`, it has to be restarted from its Settings page. | |
| ## Privacy | |
| For each generation, this Space records: | |
| - the visitor's **IP address**, from the first entry of `x-forwarded-for` | |
| - browser **language**, **user agent** and Gradio session ID | |
| - the generation settings and the seed (never the image) | |
| With LangSmith tracing on, which it is in this deployment, this is stored as | |
| run metadata in LangSmith. It is also written to the Space's container logs, | |
| which only the owner can see. An IP address counts as personal data under | |
| GDPR and UK GDPR, and under several US state laws. | |
| | Variable | Default | Effect | | |
| |---|---|---| | |
| | `LANGSMITH_LOG_IP` | `on` | `off` stops recording the raw IP address | | |
| | `LANGSMITH_TRACING` | unset | `true` sends the data above to LangSmith | | |
| ## Setup (for forks / redeploys) | |
| | Name | Type | Purpose | | |
| |---|---|---| | |
| | `HF_TOKEN` | secret | **Required.** Used to connect to the inference Space. Without it, every generation stops with an error | | |
| | `BACKEND_SPACE` | variable | Optional. Defaults to `whosouravsharma/diffusiondb-sd15-lora-inference` | | |
| | `LANGSMITH_API_KEY` | secret | Optional. Only needed with tracing | | |
| | `LANGSMITH_TRACING`, `LANGSMITH_PROJECT` | variable | Optional. Turn LangSmith tracing on and name the project | | |
| With tracing on, the UI and the inference Space appear in one trace, not | |
| two. The UI sends `langsmith-trace` and `baggage` headers with its request, | |
| so the backend's `sd15_lora_generate` run is recorded as a child of the | |
| UI's `ui_generate` run. | |
| ## Limitations | |
| - **The style is fixed:** saturation and contrast go up on every prompt, | |
| including ones that ask for muted colours. | |
| - **Some detailed scenes get simplified** or lose their composition. | |
| - Output is fixed at 512×512, the resolution the adapter was trained at. | |
| - The training images were themselves made by SD 1.x, so the adapter | |
| reproduces that model's artifacts. | |
| - **The safety checker is off** in the inference pipeline, so images are | |
| not screened. The training data was NSFW-filtered, but that filter is not | |
| a guarantee. | |
| - The first image after the inference Space has been idle is slow (see | |
| [Cold starts](#cold-starts)). | |
| ## License | |
| CreativeML OpenRAIL-M, the same license as the Stable Diffusion 1.5 base | |
| model. Training data comes from | |
| [DiffusionDB](https://huggingface.co/datasets/poloclub/diffusiondb) (CC0 1.0; | |
| Wang et al., 2022, [arXiv:2210.14896](https://arxiv.org/abs/2210.14896)). | |