Instructions to use torpedo2003/SAD with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use torpedo2003/SAD with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline from diffusers.utils import load_image # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("torpedo2003/SAD", dtype=torch.bfloat16, device_map="cuda") prompt = "Turn this cat into a dog" input_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png") image = pipe(image=input_image, prompt=prompt).images[0] - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -5,29 +5,36 @@ tags:
|
|
| 5 |
- diffusion
|
| 6 |
- vae
|
| 7 |
- decoder
|
| 8 |
-
library_name: diffusers
|
| 9 |
---
|
| 10 |
|
| 11 |
-
# Sample-Aligned Decoders
|
| 12 |
|
| 13 |
-
|
| 14 |
-
original autoencoder, fine-tuned with SAD on latents generated by one generator. The encoder, the latent space and the generator
|
| 15 |
-
are unchanged, and inference cost is identical.
|
| 16 |
|
| 17 |
-
|
| 18 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
|
| 20 |
## Usage
|
| 21 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
```python
|
| 23 |
from diffusers import DiTPipeline
|
| 24 |
-
from sad import load_sad_decoder
|
| 25 |
|
| 26 |
pipe = DiTPipeline.from_pretrained("facebook/DiT-XL-2-256").to("cuda")
|
| 27 |
-
load_sad_decoder(pipe.vae, "dit-xl2-256/sad-gan")
|
|
|
|
| 28 |
```
|
| 29 |
|
| 30 |
-
|
| 31 |
|
| 32 |
```python
|
| 33 |
from huggingface_hub import hf_hub_download
|
|
@@ -35,21 +42,28 @@ from safetensors.torch import load_file
|
|
| 35 |
pipe.vae.load_state_dict(load_file(hf_hub_download("torpedo2003/SAD", "dit-xl2-256/sad-gan.safetensors")), strict=False)
|
| 36 |
```
|
| 37 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
## Files
|
| 39 |
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
|
| 43 |
-
|
|
| 44 |
-
| `dit-xl2-256
|
| 45 |
-
| `dit-xl2-256
|
| 46 |
-
| `
|
| 47 |
-
| `
|
| 48 |
-
| `
|
| 49 |
-
| `
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
|
|
|
|
|
|
|
|
|
| 53 |
|
| 54 |
## Citation
|
| 55 |
|
|
|
|
| 5 |
- diffusion
|
| 6 |
- vae
|
| 7 |
- decoder
|
|
|
|
| 8 |
---
|
| 9 |
|
| 10 |
+
# Sample-Aligned Decoders for Latent Generative Models
|
| 11 |
|
| 12 |
+
[Yu Lei](https://torpedo2003.github.io/)<sup>1,2</sup>, Yi Li<sup>2</sup>, Hongcheng Gao<sup>3</sup>, Yipeng Wei<sup>2</sup>, Liang Li<sup>1</sup>, Li Su<sup>2</sup>, Qingming Huang<sup>2</sup>
|
|
|
|
|
|
|
| 13 |
|
| 14 |
+
<sup>1</sup>Institute of Computing Technology, Chinese Academy of Sciences <sup>2</sup>University of Chinese Academy of Sciences <sup>3</sup>Tsinghua University
|
| 15 |
+
|
| 16 |
+
[Project page](https://torpedo2003.github.io/SAD/) · [Code](https://github.com/torpedo2003/SAD)
|
| 17 |
+
|
| 18 |
+
These are drop-in decoders for latent generative models. Each file holds only the decoder (`decoder.*`, `post_quant_conv.*`) of
|
| 19 |
+
the original autoencoder. It was fine-tuned with SAD on latents from one generator. The encoder, the latent space and the
|
| 20 |
+
generator stay unchanged, and so does the inference cost.
|
| 21 |
|
| 22 |
## Usage
|
| 23 |
|
| 24 |
+
```bash
|
| 25 |
+
pip install git+https://github.com/torpedo2003/SAD
|
| 26 |
+
```
|
| 27 |
+
|
| 28 |
```python
|
| 29 |
from diffusers import DiTPipeline
|
| 30 |
+
from sad import load_sad_decoder
|
| 31 |
|
| 32 |
pipe = DiTPipeline.from_pretrained("facebook/DiT-XL-2-256").to("cuda")
|
| 33 |
+
load_sad_decoder(pipe.vae, "dit-xl2-256/sad-gan") # swaps the decoder in place
|
| 34 |
+
images = pipe(class_labels=[279, 388], num_inference_steps=250, guidance_scale=1.5).images
|
| 35 |
```
|
| 36 |
|
| 37 |
+
You can also load the weights without our package:
|
| 38 |
|
| 39 |
```python
|
| 40 |
from huggingface_hub import hf_hub_download
|
|
|
|
| 42 |
pipe.vae.load_state_dict(load_file(hf_hub_download("torpedo2003/SAD", "dit-xl2-256/sad-gan.safetensors")), strict=False)
|
| 43 |
```
|
| 44 |
|
| 45 |
+
The VA-VAE and E2E-VAE decoders are not diffusers models. Load them into the autoencoder classes of the
|
| 46 |
+
[LightningDiT](https://github.com/hustvl/LightningDiT) and [REPA-E](https://github.com/End2End-Diffusion/REPA-E) code bases,
|
| 47 |
+
using the same `load_sad_decoder` call.
|
| 48 |
+
|
| 49 |
## Files
|
| 50 |
|
| 51 |
+
Each decoder is trained on latents from its own generator and sampler, so use the one that matches your pipeline.
|
| 52 |
+
|
| 53 |
+
| File | Generator | Tokenizer | Sampler |
|
| 54 |
+
|---|---|---|---|
|
| 55 |
+
| `dit-xl2-256/sad-gan.safetensors` | DiT-XL/2 | SD-VAE (ft-EMA) | official 250-step DDPM, cfg 1.5 |
|
| 56 |
+
| `dit-xl2-256/sad-mmd.safetensors` | DiT-XL/2 | SD-VAE (ft-EMA) | official 250-step DDPM, cfg 1.5 |
|
| 57 |
+
| `dit-xl2-256-dpm50/sad-gan.safetensors` | DiT-XL/2 | SD-VAE (ft-EMA) | 50-step DPM-Solver++, cfg 1.5 |
|
| 58 |
+
| `dit-xl2-256-dpm50/sad-mmd.safetensors` | DiT-XL/2 | SD-VAE (ft-EMA) | 50-step DPM-Solver++, cfg 1.5 |
|
| 59 |
+
| `sit-xl2-repa-256/sad-gan.safetensors` | SiT-XL/2 + REPA | SD-VAE (ft-EMA) | SDE 250 steps |
|
| 60 |
+
| `lightningdit-xl-vavae-256/sad-gan.safetensors` | LightningDiT-XL | VA-VAE | official recipe |
|
| 61 |
+
| `repa-e-e2evae-256/sad-gan.safetensors` | REPA-E (SiT-XL/1) | E2E-VAE | SDE 250 steps |
|
| 62 |
+
| `pixart-alpha-256/sad-gan.safetensors` | PixArt-α 256px | SD-VAE (ft-EMA) | 20 steps, guidance 4.5 |
|
| 63 |
+
|
| 64 |
+
Every decoder was trained for 2,000 steps, about one GPU-hour. Training used the GAN discrepancy in DINOv2 space
|
| 65 |
+
(λ = 0.05 on ImageNet, 0.01 for PixArt-α) or MMD (λ = 1), plus the L1 encoder anchor.
|
| 66 |
+
To train a decoder for your own generator, see the [training guide](https://github.com/torpedo2003/SAD/tree/main/training).
|
| 67 |
|
| 68 |
## Citation
|
| 69 |
|