| # Frechet-Distributional-Decoder-Alignment |
|
|
| Decoder weights of FDDA. FDDA post-trains the tokenizer decoder of a frozen latent generative model |
| with a Fréchet distance loss on generated samples. Code, installation and evaluation: |
| [github.com/sunset-clouds/Frechet-Distributional-Decoder-Alignment](https://github.com/sunset-clouds/Frechet-Distributional-Decoder-Alignment). |
|
|
| Each checkpoint stores the tokenizer state dict with the aligned decoder (`{"model", "epoch"}`) and |
| is used with the generator in `--generator_name`. ImageNet 256x256, 50,000 generated samples. |
| gFD<sub>r6</sub> is the mean over six representation spaces (Inception-v3, ConvNeXt-v2, DINOv2, MAE, |
| SigLIP2, CLIP) of FD divided by the FD of the ImageNet validation set. |
|
|
| ## Before generator-side FD post-training |
|
|
| `before_generator_side/<family>/`: decoders aligned to the pretrained generators. |
|
|
| Model | `--generator_name` | gFID | IS | gFD<sub>r6</sub> | checkpoint |
| --- | --- |:---:|:---:|:---:| --- |
| LlamaGen-B | `llamagen-B_256` | 2.74 | 209.3 | 10.88 | [`llamagen-B_256.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/before_generator_side/llamagen/llamagen-B_256.pt) |
| LlamaGen-L | `llamagen-L_256` | 1.87 | 303.5 | 5.60 | [`llamagen-L_256.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/before_generator_side/llamagen/llamagen-L_256.pt) |
| GigaTok-S-S | `gigatok-B_256` | 1.68 | 281.6 | 6.59 | [`gigatok-B_256.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/before_generator_side/gigatok/gigatok-B_256.pt) |
| TiTok-L-32 | `titok-L32` | 1.34 | 202.6 | 6.19 | [`titok-L32.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/before_generator_side/titok/titok-L32.pt) |
| TiTok-B-64 | `titok-B64` | 1.61 | 214.9 | 6.99 | [`titok-B64.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/before_generator_side/titok/titok-B64.pt) |
| VAR-d16 | `var-d16_256` | 1.62 | 280.1 | 7.03 | [`var-d16_256.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/before_generator_side/var/var-d16_256.pt) |
| VAR-d20 | `var-d20_256` | 1.32 | 296.9 | 5.34 | [`var-d20_256.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/before_generator_side/var/var-d20_256.pt) |
| VAR-d24 | `var-d24_256` | 1.24 | 304.4 | 4.26 | [`var-d24_256.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/before_generator_side/var/var-d24_256.pt) |
| iMF-B | `imf-B_256` | 1.65 | 268.1 | 8.54 | [`imf-B_256.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/before_generator_side/imf/imf-B_256.pt) |
| iMF-L | `imf-L_256` | 1.26 | 281.0 | 5.48 | [`imf-L_256.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/before_generator_side/imf/imf-L_256.pt) |
| iMF-XL | `imf-XL_256` | 1.14 | 288.3 | 5.03 | [`imf-XL_256.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/before_generator_side/imf/imf-XL_256.pt) |
|
|
| ## After generator-side FD post-training |
|
|
| `after_generator_side/<family>/`: decoders aligned to the FD post-trained generators (FDAR for |
| LlamaGen, GigaTok, TiTok and VAR; FD-SIM for iMF). |
|
|
| Model | `--generator_name` | gFID | IS | gFD<sub>r6</sub> | checkpoint |
| --- | --- |:---:|:---:|:---:| --- |
| LlamaGen-B | `llamagen-B_256-fdpt` | 2.23 | 275.9 | 5.27 | [`llamagen-B_256-fdpt.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/after_generator_side/llamagen/llamagen-B_256-fdpt.pt) |
| LlamaGen-L | `llamagen-L_256-fdpt` | 1.34 | 313.5 | 3.18 | [`llamagen-L_256-fdpt.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/after_generator_side/llamagen/llamagen-L_256-fdpt.pt) |
| GigaTok-S-S | `gigatok-B_256-fdpt` | 1.74 | 296.5 | 3.84 | [`gigatok-B_256-fdpt.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/after_generator_side/gigatok/gigatok-B_256-fdpt.pt) |
| TiTok-L-32 | `titok-L32-fdpt` | 1.35 | 215.1 | 5.49 | [`titok-L32-fdpt.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/after_generator_side/titok/titok-L32-fdpt.pt) |
| TiTok-B-64 | `titok-B64-fdpt` | 1.42 | 240.3 | 5.42 | [`titok-B64-fdpt.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/after_generator_side/titok/titok-B64-fdpt.pt) |
| VAR-d16 | `var-d16_256-fdpt` | 1.35 | 307.3 | 2.62 | [`var-d16_256-fdpt.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/after_generator_side/var/var-d16_256-fdpt.pt) |
| VAR-d20 | `var-d20_256-fdpt` | 1.08 | 308.1 | 1.98 | [`var-d20_256-fdpt.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/after_generator_side/var/var-d20_256-fdpt.pt) |
| VAR-d24 | `var-d24_256-fdpt` | 1.09 | 308.3 | 1.67 | [`var-d24_256-fdpt.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/after_generator_side/var/var-d24_256-fdpt.pt) |
| iMF-B | `imf-B_256-fdsim` | 0.91 | 304.0 | 4.54 | [`imf-B_256-fdsim.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/after_generator_side/imf/imf-B_256-fdsim.pt) |
| iMF-L | `imf-L_256-fdsim` | 0.80 | 299.1 | 2.43 | [`imf-L_256-fdsim.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/after_generator_side/imf/imf-L_256-fdsim.pt) |
| iMF-XL | `imf-XL_256-fdsim` | 0.78 | 303.7 | 2.21 | [`imf-XL_256-fdsim.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/after_generator_side/imf/imf-XL_256-fdsim.pt) |
|
|
| ## Reference statistics |
|
|
| `reference_stats/reconstruction/fid_stats/`: ImageNet-validation statistics used by the |
| reconstruction metrics. |
|
|
| ## Download |
|
|
| From the root of the code repository: |
|
|
| ```bash |
| hf download jiajunzhu/Frechet-Distributional-Decoder-Alignment \ |
| --include "*_generator_side/*" --local-dir checkpoints/fdda |
| hf download jiajunzhu/Frechet-Distributional-Decoder-Alignment \ |
| --include "reference_stats/*" --local-dir . |
| ``` |
|
|
| ## BibTeX |
|
|
| ```bibtex |
| ``` |
|
|
