nanoRAE: pretrained weights
Weights for nanoRAE, a small-scale reproduction of Diffusion Transformers with Representation Autoencoders (RAE) on CIFAR-10. The encoders were trained from scratch on CIFAR-10 with nanoDINO.
Encoders and shared files
| File | Content | Size |
|---|---|---|
encoders/encoder_dinov2_vit_tiny4.pt |
DINOv2 ViT-Tiny/4 encoder (EMA teacher) | 22 MB |
encoders/encoder_dinov2_vit_s4.pt |
DINOv2 ViT-S/4 encoder (experiment 04) | 85 MB |
stats/normalization_stats.pt |
per-channel, per-position latent mean / var of the Tiny encoder (train, with flips) | 0.1 MB |
stats/cifar10_train_fid_stats.npz |
FID reference: Inception mean / covariance of the 50k CIFAR-10 training images (torch-fidelity inception-v3-compat, 2048-d) |
34 MB |
stats/cifar10_test_fid_stats.npz |
the same for the 10k CIFAR-10 test images (not used here; for comparisons against test) | 34 MB |
encoders/disc_dino_vit_tiny4.pt |
DINO (v1) ViT-Tiny/4, frozen backbone of the GAN discriminator (only for training) | 22 MB |
Stage 1 decoders (EMA)
| File | Encoder | Decoder | GAN | Noise aug. | rFID | Experiment | Size |
|---|---|---|---|---|---|---|---|
stage1/decoder_vitb.pt |
Tiny | ViT-B | no | yes | 5.51 | 01 | 0.3 GB |
stage1/decoder_vitb_gan.pt |
Tiny | ViT-B | yes | yes | 6.57 | 01 | 0.3 GB |
stage1/decoder_vitl.pt |
Tiny | ViT-L | no | yes | 5.15 | 01 | 1.2 GB |
stage1/decoder_vitl_gan.pt |
Tiny | ViT-L | yes | yes | 5.80 | 01 | 1.2 GB |
stage1/decoder_vitxl.pt |
Tiny | ViT-XL | no | yes | 5.25 | 01 | 1.7 GB |
stage1/decoder_vitxl_gan.pt |
Tiny | ViT-XL | yes | yes | 5.71 | 01, used by Stage 2 | 1.7 GB |
stage1/decoder_vitxl_gan_nonoise.pt |
Tiny | ViT-XL | yes | no | 4.22 | 08 | 1.7 GB |
stage1/decoder_vitxl_s4.pt |
S | ViT-XL | no | yes | 3.97 | 04 | 1.7 GB |
stage1/decoder_vitxl_gan_s4.pt |
S | ViT-XL | yes | yes | 3.71 | 04 | 1.7 GB |
Stage 2 models (EMA, 2000 epochs, Tiny encoder; FID on 50k samples without CFG)
| File | Model | Config name | FID | Experiment | Size |
|---|---|---|---|---|---|
stage2/ditdh_s_h512_ema.pt |
DiTDH-S with a 512-wide head (default) | DiTDH-S-h512 |
7.94 | 05 | 0.2 GB |
stage2/lightningdit_s_ema.pt |
LightningDiT-S/1, no DDT head | LightningDiT-S |
8.70 | 06 | 0.1 GB |
stage2/ditdh_s_h512_noshift_ema.pt |
512-wide head, no time shift | DiTDH-S-h512-noshift |
9.01 | 07 | 0.2 GB |
stage2/ditdh_ti_ema.pt |
DiTDH-Ti (18.5M) | DiTDH-Ti |
9.64 | 05 | 0.1 GB |
stage2/ditdh_s_ema.pt |
DiTDH-S with the official 2048-wide head | DiTDH-S |
13.59 | 02 | 0.8 GB |
stage2/rjf_ditdh_s_h512_ema.pt |
512-wide head + RJF (R = 18) | DiTDH-S-h512-RJF |
6.81 | 12 | 0.2 GB |
stage2/rjf_lightningdit_s_ema.pt |
LightningDiT-S/1 + RJF (R = 18) | LightningDiT-S-RJF |
6.97 | 12 | 0.1 GB |
stage2/rjf_lightningdit_ti_ema.pt |
LightningDiT-Ti/1 + RJF (R = 18) | LightningDiT-Ti-RJF |
9.22 | 12 | 0.03 GB |
stage2/rfm_lightningdit_s_ema.pt |
LightningDiT-S/1 + RFM (R = 18) | LightningDiT-S-RFM |
7.09 | 13 | 0.1 GB |
stage2/rfm_lightningdit_ti_ema.pt |
LightningDiT-Ti/1 + RFM (R = 18) | LightningDiT-Ti-RFM |
9.41 | 13 | 0.03 GB |
stage2/xpred_ditdh_s_h512_ema.pt |
512-wide head, x-prediction | DiTDH-S-h512-xpred |
8.22 | 15 | 0.2 GB |
R is the per-token radius the RJF / RFM samples are scaled to before decoding (13.66 is the mean token norm of the normalized latent;
set transport.params.sphere_radius to change it).
AutoGuidance (the weak guides are short runs, 80 epochs; download the strong model as well)
| Guide file | Guides | Config name | w | FID | Experiment | Size |
|---|---|---|---|---|---|---|
stage2/guide_ditdh_ti_ema.pt |
DiTDH-S with a 512-wide head | DiTDH-S-h512-AG |
1.42 | 6.48 | 09 | 0.07 GB |
stage2/guide_ditdh_ti_rjf_ema.pt |
512-wide head + RJF (R = 13.66) | DiTDH-S-h512-RJF-AG |
1.5 | 6.31 | 14 | 0.07 GB |
stage2/guide_lightningdit_ti_rjf_ema.pt |
LightningDiT-S/1 + RJF (R = 13.66) | LightningDiT-S-RJF-AG |
1.5 | 6.07 | 14 | 0.03 GB |
stage2/guide_lightningdit_ti_rfm_ema.pt |
LightningDiT-S/1 + RFM (R = 13.66) | LightningDiT-S-RFM-AG |
1.5 | 6.11 | 14 | 0.03 GB |
REPA Guidance (internal guidance; the model has a REPA head after body block 4 and guides itself; FID at the best scale w)
| File | Model | Config name | w | FID | Experiment | Size |
|---|---|---|---|---|---|---|
stage2/xpred_ig_ditdh_s_h512_ema.pt |
512-wide head, x-prediction + REPA head, best | DiTDH-S-h512-xpred-IG |
3.0 | 4.70 | 16 | 0.2 GB |
stage2/vpred_ig_ditdh_s_h512_ema.pt |
512-wide head, velocity prediction + base head | DiTDH-S-h512-vpred-IG |
2.5 | 5.45 | 16 | 0.2 GB |
In the repository, the config DiTDH-S-h512-nonoise decodes the default model with stage1/decoder_vitxl_gan_nonoise.pt (experiment 08, FID 10.45).
All Stage 2 models use stage1/decoder_vitxl_gan.pt otherwise.
Details: docs.
Usage
git clone https://github.com/shumpei-takezaki/nanoRAE.git && cd nanoRAE
uv sync
uv run hf download shumpei777/nanoRAE --local-dir pretrained_models \
--include stage2/xpred_ig_ditdh_s_h512_ema.pt stage1/decoder_vitxl_gan.pt encoders/encoder_dinov2_vit_tiny4.pt stats/normalization_stats.pt
PYTHONPATH=src uv run python src/sample.py --config configs/stage2/sampling/CIFAR10/DiTDH-S-h512-xpred-IG_pretrained.yaml # -> sample.png
The encoders can also be loaded directly with nanoDINO: from nanodino.export import load_backbone.
License
MIT. Parts of the training code have their own licenses (the RAEv2-derived x-prediction / REPA head code is CC BY-NC 4.0): see THIRD_PARTY_NOTICES.md. The code builds on the official RAE implementation (MIT).