torpedo2003 commited on
Commit
6eb7beb
·
verified ·
1 Parent(s): 0106207

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +37 -23
README.md CHANGED
@@ -5,29 +5,36 @@ tags:
5
  - diffusion
6
  - vae
7
  - decoder
8
- library_name: diffusers
9
  ---
10
 
11
- # Sample-Aligned Decoders (SAD)
12
 
13
- Drop-in decoders for latent generative models. Each file contains only the decoder (`decoder.*`, `post_quant_conv.*`) of the
14
- original autoencoder, fine-tuned with SAD on latents generated by one generator. The encoder, the latent space and the generator
15
- are unchanged, and inference cost is identical.
16
 
17
- - Project page: https://torpedo2003.github.io/SAD/
18
- - Code: https://github.com/torpedo2003/SAD
 
 
 
 
 
19
 
20
  ## Usage
21
 
 
 
 
 
22
  ```python
23
  from diffusers import DiTPipeline
24
- from sad import load_sad_decoder # pip install from https://github.com/torpedo2003/SAD
25
 
26
  pipe = DiTPipeline.from_pretrained("facebook/DiT-XL-2-256").to("cuda")
27
- load_sad_decoder(pipe.vae, "dit-xl2-256/sad-gan")
 
28
  ```
29
 
30
- Or without our package:
31
 
32
  ```python
33
  from huggingface_hub import hf_hub_download
@@ -35,21 +42,28 @@ from safetensors.torch import load_file
35
  pipe.vae.load_state_dict(load_file(hf_hub_download("torpedo2003/SAD", "dit-xl2-256/sad-gan.safetensors")), strict=False)
36
  ```
37
 
 
 
 
 
38
  ## Files
39
 
40
- | File | Generator | Tokenizer | Sampler used for the training latents | FID-50K (D₀ → SAD) |
41
- |---|---|---|---|---|
42
- | `dit-xl2-256/sad-gan.safetensors` | DiT-XL/2 | SD-VAE (ft-EMA) | official 250-step DDPM, cfg 1.5 | 2.30 → 2.13 |
43
- | `dit-xl2-256/sad-mmd.safetensors` | DiT-XL/2 | SD-VAE (ft-EMA) | official 250-step DDPM, cfg 1.5 | 2.30 → 2.17 |
44
- | `dit-xl2-256-dpm50/sad-gan.safetensors` | DiT-XL/2 | SD-VAE (ft-EMA) | 50-step DPM-Solver++, cfg 1.5 | ablation setting |
45
- | `dit-xl2-256-dpm50/sad-mmd.safetensors` | DiT-XL/2 | SD-VAE (ft-EMA) | 50-step DPM-Solver++, cfg 1.5 | ablation setting |
46
- | `sit-xl2-repa-256/sad-gan.safetensors` | SiT-XL/2 + REPA | SD-VAE (ft-EMA) | SDE 250 steps | 1.43 → 1.38 |
47
- | `lightningdit-xl-vavae-256/sad-gan.safetensors` | LightningDiT-XL | VA-VAE | official recipe | 1.38 → 1.23 |
48
- | `repa-e-e2evae-256/sad-gan.safetensors` | REPA-E (SiT-XL/1) | E2E-VAE | SDE 250 steps | 1.15 → 0.98 |
49
- | `pixart-alpha-256/sad-gan.safetensors` | PixArt-α 256px | SD-VAE (ft-EMA) | as in the paper | COCO FID 34.7 → 23.6 |
50
-
51
- All decoders were trained for 2,000 steps (about one GPU-hour) with the GAN discrepancy in DINOv2 space
52
- (λ = 0.05 on ImageNet, 0.01 on PixArt-α) or MMD (λ = 1), plus the L1 encoder anchor.
 
 
 
53
 
54
  ## Citation
55
 
 
5
  - diffusion
6
  - vae
7
  - decoder
 
8
  ---
9
 
10
+ # Sample-Aligned Decoders for Latent Generative Models
11
 
12
+ [Yu Lei](https://torpedo2003.github.io/)<sup>1,2</sup>, Yi Li<sup>2</sup>, Hongcheng Gao<sup>3</sup>, Yipeng Wei<sup>2</sup>, Liang Li<sup>1</sup>, Li Su<sup>2</sup>, Qingming Huang<sup>2</sup>
 
 
13
 
14
+ <sup>1</sup>Institute of Computing Technology, Chinese Academy of Sciences &nbsp; <sup>2</sup>University of Chinese Academy of Sciences &nbsp; <sup>3</sup>Tsinghua University
15
+
16
+ [Project page](https://torpedo2003.github.io/SAD/) · [Code](https://github.com/torpedo2003/SAD)
17
+
18
+ These are drop-in decoders for latent generative models. Each file holds only the decoder (`decoder.*`, `post_quant_conv.*`) of
19
+ the original autoencoder. It was fine-tuned with SAD on latents from one generator. The encoder, the latent space and the
20
+ generator stay unchanged, and so does the inference cost.
21
 
22
  ## Usage
23
 
24
+ ```bash
25
+ pip install git+https://github.com/torpedo2003/SAD
26
+ ```
27
+
28
  ```python
29
  from diffusers import DiTPipeline
30
+ from sad import load_sad_decoder
31
 
32
  pipe = DiTPipeline.from_pretrained("facebook/DiT-XL-2-256").to("cuda")
33
+ load_sad_decoder(pipe.vae, "dit-xl2-256/sad-gan") # swaps the decoder in place
34
+ images = pipe(class_labels=[279, 388], num_inference_steps=250, guidance_scale=1.5).images
35
  ```
36
 
37
+ You can also load the weights without our package:
38
 
39
  ```python
40
  from huggingface_hub import hf_hub_download
 
42
  pipe.vae.load_state_dict(load_file(hf_hub_download("torpedo2003/SAD", "dit-xl2-256/sad-gan.safetensors")), strict=False)
43
  ```
44
 
45
+ The VA-VAE and E2E-VAE decoders are not diffusers models. Load them into the autoencoder classes of the
46
+ [LightningDiT](https://github.com/hustvl/LightningDiT) and [REPA-E](https://github.com/End2End-Diffusion/REPA-E) code bases,
47
+ using the same `load_sad_decoder` call.
48
+
49
  ## Files
50
 
51
+ Each decoder is trained on latents from its own generator and sampler, so use the one that matches your pipeline.
52
+
53
+ | File | Generator | Tokenizer | Sampler |
54
+ |---|---|---|---|
55
+ | `dit-xl2-256/sad-gan.safetensors` | DiT-XL/2 | SD-VAE (ft-EMA) | official 250-step DDPM, cfg 1.5 |
56
+ | `dit-xl2-256/sad-mmd.safetensors` | DiT-XL/2 | SD-VAE (ft-EMA) | official 250-step DDPM, cfg 1.5 |
57
+ | `dit-xl2-256-dpm50/sad-gan.safetensors` | DiT-XL/2 | SD-VAE (ft-EMA) | 50-step DPM-Solver++, cfg 1.5 |
58
+ | `dit-xl2-256-dpm50/sad-mmd.safetensors` | DiT-XL/2 | SD-VAE (ft-EMA) | 50-step DPM-Solver++, cfg 1.5 |
59
+ | `sit-xl2-repa-256/sad-gan.safetensors` | SiT-XL/2 + REPA | SD-VAE (ft-EMA) | SDE 250 steps |
60
+ | `lightningdit-xl-vavae-256/sad-gan.safetensors` | LightningDiT-XL | VA-VAE | official recipe |
61
+ | `repa-e-e2evae-256/sad-gan.safetensors` | REPA-E (SiT-XL/1) | E2E-VAE | SDE 250 steps |
62
+ | `pixart-alpha-256/sad-gan.safetensors` | PixArt-α 256px | SD-VAE (ft-EMA) | 20 steps, guidance 4.5 |
63
+
64
+ Every decoder was trained for 2,000 steps, about one GPU-hour. Training used the GAN discrepancy in DINOv2 space
65
+ (λ = 0.05 on ImageNet, 0.01 for PixArt-α) or MMD (λ = 1), plus the L1 encoder anchor.
66
+ To train a decoder for your own generator, see the [training guide](https://github.com/torpedo2003/SAD/tree/main/training).
67
 
68
  ## Citation
69