Instructions to use madtune/pixeldit-diffusers with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use madtune/pixeldit-diffusers with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("nvidia/PixelDiT-1300M-1024px", dtype=torch.bfloat16, device_map="cuda") pipe.load_lora_weights("madtune/pixeldit-diffusers") prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k" image = pipe(prompt).images[0] - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- Draw Things
- DiffusionBee
Update README.md
Browse files
README.md
CHANGED
|
@@ -1,14 +1,13 @@
|
|
| 1 |
---
|
| 2 |
license: other
|
| 3 |
tags:
|
| 4 |
-
- text-to-image
|
| 5 |
-
- diffusion
|
| 6 |
-
- pixeldit
|
| 7 |
-
- nvidia
|
| 8 |
-
- pixel-space
|
| 9 |
-
- lora
|
| 10 |
base_model: nvidia/PixelDiT-1300M-1024px
|
| 11 |
-
library_name: diffusers
|
| 12 |
---
|
| 13 |
|
| 14 |

|
|
@@ -60,12 +59,6 @@ python generate.py --prompt "a viking warrior on a cliff at sunset, cinematic"
|
|
| 60 |
# Portrait mode
|
| 61 |
python generate.py --height 1280 --width 768 --steps 60 --cfg 8.5 --prompt "your prompt"
|
| 62 |
|
| 63 |
-
# Qwen encoder (creative/fantasy)
|
| 64 |
-
python generate.py --encoder qwen --proj qwen_proj.pt --prompt "A giant hamster emperor in a battle fortress"
|
| 65 |
-
|
| 66 |
-
# With LoRA
|
| 67 |
-
python generate.py --lora lora_yarn_out/best --prompt "a dark anime woman in a field, yarn art style"
|
| 68 |
-
|
| 69 |
# LCM fast mode (8 steps)
|
| 70 |
python generate.py --scheduler lcm --steps 8 --cfg 2.0 --prompt "your prompt"
|
| 71 |
```
|
|
@@ -76,22 +69,10 @@ python generate.py --scheduler lcm --steps 8 --cfg 2.0 --prompt "your prompt"
|
|
| 76 |
|
| 77 |
```python
|
| 78 |
import torch
|
| 79 |
-
from
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
tokenizer.padding_side = "right"
|
| 84 |
-
text_encoder = (
|
| 85 |
-
AutoModelForCausalLM.from_pretrained("Efficient-Large-Model/gemma-2-2b-it", dtype=torch.bfloat16)
|
| 86 |
-
.get_decoder().eval()
|
| 87 |
-
)
|
| 88 |
-
|
| 89 |
-
pipe = PixelDiTPipeline.from_pretrained(
|
| 90 |
-
"madtune/pixeldit-diffusers",
|
| 91 |
-
text_encoder=text_encoder,
|
| 92 |
-
tokenizer=tokenizer,
|
| 93 |
-
torch_dtype=torch.bfloat16,
|
| 94 |
-
)
|
| 95 |
pipe.enable_model_cpu_offload()
|
| 96 |
|
| 97 |
image = pipe(
|
|
@@ -101,49 +82,11 @@ image = pipe(
|
|
| 101 |
num_inference_steps=50,
|
| 102 |
guidance_scale=7.5,
|
| 103 |
).images[0]
|
| 104 |
-
image.save("out.jpg")
|
| 105 |
-
```
|
| 106 |
-
|
| 107 |
-
---
|
| 108 |
-
|
| 109 |
-
## LoRA
|
| 110 |
-
|
| 111 |
-
### Train a style LoRA
|
| 112 |
-
|
| 113 |
-
```bash
|
| 114 |
-
# 1. Download images (Pexels API key required)
|
| 115 |
-
python scripts/download_unsplash.py --query "yarn wool textile" --n 150 --out /data/lora_yarn
|
| 116 |
-
|
| 117 |
-
# 2. Precompute embeddings
|
| 118 |
-
python scripts/precompute_lora_data.py --images /data/lora_yarn --out /data/lora_yarn_cache --trigger "yarn art style" --recaption
|
| 119 |
|
| 120 |
-
|
| 121 |
-
python scripts/train_lora.py --data /data/lora_yarn_cache --out lora_yarn_out/ --epochs 50 --batch 2
|
| 122 |
-
```
|
| 123 |
-
|
| 124 |
-
### Load LoRA in pipeline
|
| 125 |
-
|
| 126 |
-
```python
|
| 127 |
-
pipe.load_lora_weights("lora_yarn_out/best")
|
| 128 |
-
pipe.set_adapters(["default"], adapter_weights=[1.0])
|
| 129 |
-
|
| 130 |
-
# merge multiple LoRAs
|
| 131 |
-
pipe.load_lora_weights("lora_style/best", adapter_name="style")
|
| 132 |
-
pipe.load_lora_weights("lora_char/best", adapter_name="char")
|
| 133 |
-
pipe.set_adapters(["style", "char"], adapter_weights=[1.0, 0.7])
|
| 134 |
-
|
| 135 |
-
# bake into weights
|
| 136 |
-
pipe.fuse_lora()
|
| 137 |
```
|
| 138 |
|
| 139 |
---
|
| 140 |
-
|
| 141 |
-
## Qwen Encoder
|
| 142 |
-
|
| 143 |
-
> **Coming soon.** Qwen3-2B integration (creative/fantasy prompts) is implemented in the pipeline but projection training scripts are not yet released. Watch this repo for updates.
|
| 144 |
-
|
| 145 |
-
---
|
| 146 |
-
|
| 147 |
## ComfyUI
|
| 148 |
|
| 149 |
```bash
|
|
@@ -163,13 +106,8 @@ Three nodes under **PixelDiT** category:
|
|
| 163 |
|---|---|
|
| 164 |
| `generate.py` | Main generation script |
|
| 165 |
| `scripts/upscale_images.py` | RealESRGAN 4× upscale before LoRA precompute |
|
| 166 |
-
| `scripts/precompute_lora_data.py` | Precompute image+caption pairs for LoRA training |
|
| 167 |
-
| `scripts/train_lora.py` | LoRA fine-tuning |
|
| 168 |
-
| `scripts/download_unsplash.py` | Download images from Pexels by search query |
|
| 169 |
| `scripts/setup_diffusers_pixeldit.py` | Install pipeline into active venv's diffusers |
|
| 170 |
|
| 171 |
-
See `howto_lora.md` for the full LoRA training walkthrough.
|
| 172 |
-
|
| 173 |
---
|
| 174 |
|
| 175 |
## Credits
|
|
|
|
| 1 |
---
|
| 2 |
license: other
|
| 3 |
tags:
|
| 4 |
+
- text-to-image
|
| 5 |
+
- diffusion
|
| 6 |
+
- pixeldit
|
| 7 |
+
- nvidia
|
| 8 |
+
- pixel-space
|
| 9 |
+
- lora
|
| 10 |
base_model: nvidia/PixelDiT-1300M-1024px
|
|
|
|
| 11 |
---
|
| 12 |
|
| 13 |

|
|
|
|
| 59 |
# Portrait mode
|
| 60 |
python generate.py --height 1280 --width 768 --steps 60 --cfg 8.5 --prompt "your prompt"
|
| 61 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 62 |
# LCM fast mode (8 steps)
|
| 63 |
python generate.py --scheduler lcm --steps 8 --cfg 2.0 --prompt "your prompt"
|
| 64 |
```
|
|
|
|
| 69 |
|
| 70 |
```python
|
| 71 |
import torch
|
| 72 |
+
from diffusers import PixelDiTPipeline
|
| 73 |
+
|
| 74 |
+
|
| 75 |
+
pipe = PixelDiTPipeline.from_pretrained("madtune/pixeldit-diffusers", torch_dtype=torch.bfloat16)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 76 |
pipe.enable_model_cpu_offload()
|
| 77 |
|
| 78 |
image = pipe(
|
|
|
|
| 82 |
num_inference_steps=50,
|
| 83 |
guidance_scale=7.5,
|
| 84 |
).images[0]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 85 |
|
| 86 |
+
image.save("out.jpg")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 87 |
```
|
| 88 |
|
| 89 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 90 |
## ComfyUI
|
| 91 |
|
| 92 |
```bash
|
|
|
|
| 106 |
|---|---|
|
| 107 |
| `generate.py` | Main generation script |
|
| 108 |
| `scripts/upscale_images.py` | RealESRGAN 4× upscale before LoRA precompute |
|
|
|
|
|
|
|
|
|
|
| 109 |
| `scripts/setup_diffusers_pixeldit.py` | Install pipeline into active venv's diffusers |
|
| 110 |
|
|
|
|
|
|
|
| 111 |
---
|
| 112 |
|
| 113 |
## Credits
|