Hugging Face's logo

jiajunzhu
/
Frechet-Distributional-Decoder-Alignment

Model card Files
xet
 Community
jiajunzhu commited on
Commit
f83f364
·
verified ·
1 Parent(s): 61a60ad

Add files using upload-large-folder tool

Browse files
Files changed (29) hide show
  1. README.md +72 -0
  2. after_generator_side/gigatok/gigatok-B_256-fdpt.pt +3 -0
  3. after_generator_side/imf/imf-B_256-fdsim.pt +3 -0
  4. after_generator_side/imf/imf-L_256-fdsim.pt +3 -0
  5. after_generator_side/imf/imf-XL_256-fdsim.pt +3 -0
  6. after_generator_side/llamagen/llamagen-B_256-fdpt.pt +3 -0
  7. after_generator_side/llamagen/llamagen-L_256-fdpt.pt +3 -0
  8. after_generator_side/titok/titok-B64-fdpt.pt +3 -0
  9. after_generator_side/titok/titok-L32-fdpt.pt +3 -0
  10. after_generator_side/var/var-d16_256-fdpt.pt +3 -0
  11. after_generator_side/var/var-d20_256-fdpt.pt +3 -0
  12. after_generator_side/var/var-d24_256-fdpt.pt +3 -0
  13. before_generator_side/gigatok/gigatok-B_256.pt +3 -0
  14. before_generator_side/imf/imf-B_256.pt +3 -0
  15. before_generator_side/imf/imf-L_256.pt +3 -0
  16. before_generator_side/imf/imf-XL_256.pt +3 -0
  17. before_generator_side/llamagen/llamagen-B_256.pt +3 -0
  18. before_generator_side/llamagen/llamagen-L_256.pt +3 -0
  19. before_generator_side/titok/titok-B64.pt +3 -0
  20. before_generator_side/titok/titok-L32.pt +3 -0
  21. before_generator_side/var/var-d16_256.pt +3 -0
  22. before_generator_side/var/var-d20_256.pt +3 -0
  23. before_generator_side/var/var-d24_256.pt +3 -0
  24. reference_stats/reconstruction/fid_stats/rec_convnext_in256_t224_stats.npz +3 -0
  25. reference_stats/reconstruction/fid_stats/rec_inception_stats.npz +3 -0
  26. reference_stats/reconstruction/fid_stats/rec_vit_large_patch14_clip_224_openai_in256_t256_stats.npz +3 -0
  27. reference_stats/reconstruction/fid_stats/rec_vit_large_patch14_dinov2_lvd142m_in256_t256_stats.npz +3 -0
  28. reference_stats/reconstruction/fid_stats/rec_vit_large_patch16_224_mae_in256_t224_stats.npz +3 -0
  29. reference_stats/reconstruction/fid_stats/rec_vit_so400m_patch16_siglip_256_v2_webli_in256_t224_stats.npz +3 -0
README.md ADDED
@@ -0,0 +1,72 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Frechet-Distributional-Decoder-Alignment
2
+
3
+ Decoder weights of FDDA. FDDA post-trains the tokenizer decoder of a frozen latent generative model
4
+ with a Fréchet distance loss on generated samples. Code, installation and evaluation:
5
+ [github.com/sunset-clouds/Frechet-Distributional-Decoder-Alignment](https://github.com/sunset-clouds/Frechet-Distributional-Decoder-Alignment).
6
+
7
+ Each checkpoint stores the tokenizer state dict with the adapted decoder (`{"model", "epoch"}`) and
8
+ is used with the generator in `--generator_name`. ImageNet 256x256, 50,000 generated samples.
9
+ gFD<sub>r6</sub> is the mean over six representation spaces (Inception-v3, ConvNeXt-v2, DINOv2, MAE,
10
+ SigLIP2, CLIP) of FD divided by the FD of the ImageNet validation set.
11
+
12
+ ## Before generator-side FD post-training
13
+
14
+ `before_generator_side/<family>/`: decoders adapted to the pretrained generators.
15
+
16
+ Model | `--generator_name` | gFID | IS | gFD<sub>r6</sub> | checkpoint
17
+ --- | --- |:---:|:---:|:---:| ---
18
+ LlamaGen-B | `llamagen-B_256` | 2.74 | 209.3 | 10.88 | [`llamagen-B_256.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/before_generator_side/llamagen/llamagen-B_256.pt)
19
+ LlamaGen-L | `llamagen-L_256` | 1.87 | 303.5 | 5.60 | [`llamagen-L_256.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/before_generator_side/llamagen/llamagen-L_256.pt)
20
+ GigaTok-S-S | `gigatok-B_256` | 1.68 | 281.6 | 6.59 | [`gigatok-B_256.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/before_generator_side/gigatok/gigatok-B_256.pt)
21
+ TiTok-L-32 | `titok-L32` | 1.34 | 202.6 | 6.19 | [`titok-L32.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/before_generator_side/titok/titok-L32.pt)
22
+ TiTok-B-64 | `titok-B64` | 1.61 | 214.9 | 6.99 | [`titok-B64.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/before_generator_side/titok/titok-B64.pt)
23
+ VAR-d16 | `var-d16_256` | 1.62 | 280.1 | 7.03 | [`var-d16_256.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/before_generator_side/var/var-d16_256.pt)
24
+ VAR-d20 | `var-d20_256` | 1.32 | 296.9 | 5.34 | [`var-d20_256.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/before_generator_side/var/var-d20_256.pt)
25
+ VAR-d24 | `var-d24_256` | 1.24 | 304.4 | 4.26 | [`var-d24_256.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/before_generator_side/var/var-d24_256.pt)
26
+ iMF-B | `imf-B_256` | 1.65 | 268.1 | 8.54 | [`imf-B_256.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/before_generator_side/imf/imf-B_256.pt)
27
+ iMF-L | `imf-L_256` | 1.26 | 281.0 | 5.48 | [`imf-L_256.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/before_generator_side/imf/imf-L_256.pt)
28
+ iMF-XL | `imf-XL_256` | 1.14 | 288.3 | 5.03 | [`imf-XL_256.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/before_generator_side/imf/imf-XL_256.pt)
29
+
30
+ ## After generator-side FD post-training
31
+
32
+ `after_generator_side/<family>/`: decoders adapted to the FD post-trained generators (FDAR for
33
+ LlamaGen, GigaTok, TiTok and VAR; FD-SIM for iMF).
34
+
35
+ Model | `--generator_name` | gFID | IS | gFD<sub>r6</sub> | checkpoint
36
+ --- | --- |:---:|:---:|:---:| ---
37
+ LlamaGen-B | `llamagen-B_256-fdpt` | 2.23 | 275.9 | 5.27 | [`llamagen-B_256-fdpt.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/after_generator_side/llamagen/llamagen-B_256-fdpt.pt)
38
+ LlamaGen-L | `llamagen-L_256-fdpt` | 1.34 | 313.5 | 3.18 | [`llamagen-L_256-fdpt.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/after_generator_side/llamagen/llamagen-L_256-fdpt.pt)
39
+ GigaTok-S-S | `gigatok-B_256-fdpt` | 1.74 | 296.5 | 3.84 | [`gigatok-B_256-fdpt.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/after_generator_side/gigatok/gigatok-B_256-fdpt.pt)
40
+ TiTok-L-32 | `titok-L32-fdpt` | 1.35 | 215.1 | 5.49 | [`titok-L32-fdpt.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/after_generator_side/titok/titok-L32-fdpt.pt)
41
+ TiTok-B-64 | `titok-B64-fdpt` | 1.42 | 240.3 | 5.42 | [`titok-B64-fdpt.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/after_generator_side/titok/titok-B64-fdpt.pt)
42
+ VAR-d16 | `var-d16_256-fdpt` | 1.35 | 307.3 | 2.62 | [`var-d16_256-fdpt.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/after_generator_side/var/var-d16_256-fdpt.pt)
43
+ VAR-d20 | `var-d20_256-fdpt` | 1.08 | 308.1 | 1.98 | [`var-d20_256-fdpt.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/after_generator_side/var/var-d20_256-fdpt.pt)
44
+ VAR-d24 | `var-d24_256-fdpt` | 1.09 | 308.3 | 1.67 | [`var-d24_256-fdpt.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/after_generator_side/var/var-d24_256-fdpt.pt)
45
+ iMF-B | `imf-B_256-fdsim` | 0.91 | 304.0 | 4.54 | [`imf-B_256-fdsim.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/after_generator_side/imf/imf-B_256-fdsim.pt)
46
+ iMF-L | `imf-L_256-fdsim` | 0.80 | 299.1 | 2.43 | [`imf-L_256-fdsim.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/after_generator_side/imf/imf-L_256-fdsim.pt)
47
+ iMF-XL | `imf-XL_256-fdsim` | 0.78 | 303.7 | 2.21 | [`imf-XL_256-fdsim.pt`](https://huggingface.co/jiajunzhu/Frechet-Distributional-Decoder-Alignment/blob/main/after_generator_side/imf/imf-XL_256-fdsim.pt)
48
+
49
+ ## Reference statistics
50
+
51
+ `reference_stats/reconstruction/fid_stats/`: ImageNet-validation statistics used by the
52
+ reconstruction metrics.
53
+
54
+ ## Download
55
+
56
+ From the root of the code repository, all decoders and the reference statistics:
57
+
58
+ ```bash
59
+ bash scripts/download_assets.sh fdda
60
+ ```
61
+
62
+ A single decoder:
63
+
64
+ ```bash
65
+ hf download jiajunzhu/Frechet-Distributional-Decoder-Alignment \
66
+ after_generator_side/llamagen/llamagen-B_256-fdpt.pt --local-dir checkpoints/fdda
67
+ ```
68
+
69
+ ## BibTeX
70
+
71
+ ```bibtex
72
+ ```
after_generator_side/gigatok/gigatok-B_256-fdpt.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:89716019abd7fa06f9308a8fb80a1566daf9daa58edc68e71420f9d83ff09226
3
+ size 544768392
after_generator_side/imf/imf-B_256-fdsim.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a7c30d083137a6f97762ec83919be664261e27d787ab763a6d1cd0994af4a93b
3
+ size 334717102
after_generator_side/imf/imf-L_256-fdsim.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0beea1f8966733334a7af0b3eb0a4051770ed7c1335e6cd5716a15efee087522
3
+ size 334717102
after_generator_side/imf/imf-XL_256-fdsim.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1642e41ed7750f13630799398dae8803ac92fb550d17b1aa3da7f9ec8cb8b2c4
3
+ size 334717356
after_generator_side/llamagen/llamagen-B_256-fdpt.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:65dbb5bd38be7b27f291a10e6422e00c1883081ae858a4199d6efd9003ab5d95
3
+ size 287668232
after_generator_side/llamagen/llamagen-L_256-fdpt.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4637241f40db545653b7bc56e50839af8fb370237210618a60f3aafea6264c2d
3
+ size 287668232
after_generator_side/titok/titok-B64-fdpt.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6621d5a56b46d8953b9d6e4110d6b2a5819acd847e3e41e6b5c85c1ab57b8a57
3
+ size 819411698
after_generator_side/titok/titok-L32-fdpt.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:577a6f7ade72ec6a0249045ee3722e9ce5df0396243b88c2ee627bda2066305e
3
+ size 2564484242
after_generator_side/var/var-d16_256-fdpt.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:259594ed2a24a409c1c6add96a48b8471e6dd988117925aad7ec767b721169e7
3
+ size 436083058
after_generator_side/var/var-d20_256-fdpt.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4ca15748715d57bc6ebc458d8423f45dec5f15d684132b31fa75bf8c4ebaeda7
3
+ size 436083058
after_generator_side/var/var-d24_256-fdpt.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2193bcd75666ec430c7135422ed92a9da86ead732cd4aec218a3ab0c4efa5abc
3
+ size 436083058
before_generator_side/gigatok/gigatok-B_256.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6dc9810663fcc1d1522f72ffcb24d977e21a75acf3d3ba48ffd1dd7a949e7d7f
3
+ size 544765618
before_generator_side/imf/imf-B_256.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4693338f6b38f27150ab07a7f1822c1a15febe6420f7fbe6164f87f06b06198f
3
+ size 334715578
before_generator_side/imf/imf-L_256.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c269c52197e2915e4d5f3ad9293550dfd397170deec842863c5f4213b2715cfa
3
+ size 334715578
before_generator_side/imf/imf-XL_256.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f0805d732808b38e5aee0a806810a3e16a55b197b9c269d8f88d07bf48482c4c
3
+ size 334715832
before_generator_side/llamagen/llamagen-B_256.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4bc8ba591cfc2801e1e075c9260e161bfc8a2d3946a8c728378f4423bdcbfb3f
3
+ size 287666497
before_generator_side/llamagen/llamagen-L_256.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:adb6db30e771ee0a81035455e2921825d9120c63f2efb29984cfe62be449ab17
3
+ size 287666497
before_generator_side/titok/titok-B64.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2e337d479712192c8bcb68abe7d5cee30578ba01cbea18715cabd72a96d8f726
3
+ size 819409658
before_generator_side/titok/titok-L32.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:de2fc54790569d8f9585066240b88b5745c73788baad9fdb28d7e440332ef9cb
3
+ size 2564480698
before_generator_side/var/var-d16_256.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:104163ed0c93c91e7bbf40500e69ffc5ac92d3c5532f9901e5a2abac57485702
3
+ size 436081418
before_generator_side/var/var-d20_256.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6c4c6920ee214c40cbf21c8a8e64c9f55de5a50723b004a2c2a83814988955ed
3
+ size 436081418
before_generator_side/var/var-d24_256.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a7eea1583380428a3fd5effe770104fbd9a1096eabafa545cfc54ca297d64abf
3
+ size 436081418
reference_stats/reconstruction/fid_stats/rec_convnext_in256_t224_stats.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f989bbc0a1a96f372120dbae84adc1d4629e718cf1531d6769937806cdf65b69
3
+ size 8397300
reference_stats/reconstruction/fid_stats/rec_inception_stats.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:758b856810c081e3686b01e5ebbae6c41983e731235f50f6921e697be73c95a7
3
+ size 33571316
reference_stats/reconstruction/fid_stats/rec_vit_large_patch14_clip_224_openai_in256_t256_stats.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e825c2ee98cc87b82529981623442c6c314133b7fe21d8c6f2eaf1ff2e7337c2
3
+ size 16794594
reference_stats/reconstruction/fid_stats/rec_vit_large_patch14_dinov2_lvd142m_in256_t256_stats.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dc5eceb142a05a110fc0d714b5e49536cf9a538e6e17875c0abbd182d78ddd3e
3
+ size 16794594
reference_stats/reconstruction/fid_stats/rec_vit_large_patch16_224_mae_in256_t224_stats.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c86c22c6531bfb1eda780cee73d73e95884965a9348ae8c8aeb83d1db1fb55dd
3
+ size 16794594
reference_stats/reconstruction/fid_stats/rec_vit_so400m_patch16_siglip_256_v2_webli_in256_t224_stats.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c65b821543c3aa7d19b6f8d847d0796b5c320a5f3ea4c66f3e3b9a830ade7794
3
+ size 21253090