File size: 3,555 Bytes
2d23e77
 
e51dc43
 
2d23e77
 
 
 
 
 
 
6eb7beb
2d23e77
6eb7beb
2d23e77
be909b8
 
6eb7beb
3bae55e
6eb7beb
 
 
 
2d23e77
 
 
6eb7beb
 
 
 
2d23e77
 
6eb7beb
2d23e77
 
6eb7beb
 
2d23e77
 
6eb7beb
2d23e77
 
 
 
 
 
 
6eb7beb
 
 
 
2d23e77
 
6eb7beb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2d23e77
 
 
 
0106207
2d23e77
0106207
3bae55e
2d23e77
 
e51dc43
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
---
license: mit
library_name: diffusers
pipeline_tag: image-to-image
tags:
- image-generation
- diffusion
- vae
- decoder
---

# Sample-Aligned Decoders for Latent Generative Models

[Yu Lei](https://torpedo2003.github.io/)<sup>1,2</sup>, Yi Li<sup>2</sup>, Hongcheng Gao<sup>3</sup>, Yipeng Wei<sup>2</sup>, Liang Li<sup>1</sup>, Li Su<sup>2</sup>, Qingming Huang<sup>2</sup>

<sup>1</sup>Institute of Computing Technology, Chinese Academy of Sciences<br>
<sup>2</sup>University of Chinese Academy of Sciences &nbsp; <sup>3</sup>Tsinghua University

[Paper](https://arxiv.org/abs/2611.22458) 路 [Project page](https://torpedo2003.github.io/SAD/) 路 [Code](https://github.com/torpedo2003/SAD)

These are drop-in decoders for latent generative models. Each file holds only the decoder (`decoder.*`, `post_quant_conv.*`) of
the original autoencoder. It was fine-tuned with SAD on latents from one generator. The encoder, the latent space and the
generator stay unchanged, and so does the inference cost.

## Usage

```bash
pip install git+https://github.com/torpedo2003/SAD
```

```python
from diffusers import DiTPipeline
from sad import load_sad_decoder

pipe = DiTPipeline.from_pretrained("facebook/DiT-XL-2-256").to("cuda")
load_sad_decoder(pipe.vae, "dit-xl2-256/sad-gan")   # swaps the decoder in place
images = pipe(class_labels=[279, 388], num_inference_steps=250, guidance_scale=1.5).images
```

You can also load the weights without our package:

```python
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
pipe.vae.load_state_dict(load_file(hf_hub_download("torpedo2003/SAD", "dit-xl2-256/sad-gan.safetensors")), strict=False)
```

The VA-VAE and E2E-VAE decoders are not diffusers models. Load them into the autoencoder classes of the
[LightningDiT](https://github.com/hustvl/LightningDiT) and [REPA-E](https://github.com/End2End-Diffusion/REPA-E) code bases,
using the same `load_sad_decoder` call.

## Files

Each decoder is trained on latents from its own generator and sampler, so use the one that matches your pipeline.

| File | Generator | Tokenizer | Sampler |
|---|---|---|---|
| `dit-xl2-256/sad-gan.safetensors` | DiT-XL/2 | SD-VAE (ft-EMA) | official 250-step DDPM, cfg 1.5 |
| `dit-xl2-256/sad-mmd.safetensors` | DiT-XL/2 | SD-VAE (ft-EMA) | official 250-step DDPM, cfg 1.5 |
| `dit-xl2-256-dpm50/sad-gan.safetensors` | DiT-XL/2 | SD-VAE (ft-EMA) | 50-step DPM-Solver++, cfg 1.5 |
| `dit-xl2-256-dpm50/sad-mmd.safetensors` | DiT-XL/2 | SD-VAE (ft-EMA) | 50-step DPM-Solver++, cfg 1.5 |
| `sit-xl2-repa-256/sad-gan.safetensors` | SiT-XL/2 + REPA | SD-VAE (ft-EMA) | SDE 250 steps |
| `lightningdit-xl-vavae-256/sad-gan.safetensors` | LightningDiT-XL | VA-VAE | official recipe |
| `repa-e-e2evae-256/sad-gan.safetensors` | REPA-E (SiT-XL/1) | E2E-VAE | SDE 250 steps |
| `pixart-alpha-256/sad-gan.safetensors` | PixArt-伪 256px | SD-VAE (ft-EMA) | 20 steps, guidance 4.5 |

Every decoder was trained for 2,000 steps, about one GPU-hour. Training used the GAN discrepancy in DINOv2 space
(位 = 0.05 on ImageNet, 0.01 for PixArt-伪) or MMD (位 = 1), plus the L1 encoder anchor.
To train a decoder for your own generator, see the [training guide](https://github.com/torpedo2003/SAD/tree/main/training).

## Citation

```bibtex
@article{lei2026sad,
  title   = {Sample-Aligned Decoders for Latent Generative Models},
  author  = {Lei, Yu and Li, Yi and Gao, Hongcheng and Wei, Yipeng and Li, Liang and Su, Li and Huang, Qingming},
  journal = {arXiv preprint arXiv:2611.22458},
  year    = {2026}
}
```