nanoRAE: pretrained weights

Weights for nanoRAE, a small-scale reproduction of Diffusion Transformers with Representation Autoencoders (RAE) on CIFAR-10. The encoders were trained from scratch on CIFAR-10 with nanoDINO.

Encoders and shared files

File Content Size
encoders/encoder_dinov2_vit_tiny4.pt DINOv2 ViT-Tiny/4 encoder (EMA teacher) 22 MB
encoders/encoder_dinov2_vit_s4.pt DINOv2 ViT-S/4 encoder (experiment 04) 85 MB
stats/normalization_stats.pt per-channel, per-position latent mean / var of the Tiny encoder (train, with flips) 0.1 MB
stats/cifar10_train_fid_stats.npz FID reference: Inception mean / covariance of the 50k CIFAR-10 training images (torch-fidelity inception-v3-compat, 2048-d) 34 MB
stats/cifar10_test_fid_stats.npz the same for the 10k CIFAR-10 test images (not used here; for comparisons against test) 34 MB
encoders/disc_dino_vit_tiny4.pt DINO (v1) ViT-Tiny/4, frozen backbone of the GAN discriminator (only for training) 22 MB

Stage 1 decoders (EMA)

File Encoder Decoder GAN Noise aug. rFID Experiment Size
stage1/decoder_vitb.pt Tiny ViT-B no yes 5.51 01 0.3 GB
stage1/decoder_vitb_gan.pt Tiny ViT-B yes yes 6.57 01 0.3 GB
stage1/decoder_vitl.pt Tiny ViT-L no yes 5.15 01 1.2 GB
stage1/decoder_vitl_gan.pt Tiny ViT-L yes yes 5.80 01 1.2 GB
stage1/decoder_vitxl.pt Tiny ViT-XL no yes 5.25 01 1.7 GB
stage1/decoder_vitxl_gan.pt Tiny ViT-XL yes yes 5.71 01, used by Stage 2 1.7 GB
stage1/decoder_vitxl_gan_nonoise.pt Tiny ViT-XL yes no 4.22 08 1.7 GB
stage1/decoder_vitxl_s4.pt S ViT-XL no yes 3.97 04 1.7 GB
stage1/decoder_vitxl_gan_s4.pt S ViT-XL yes yes 3.71 04 1.7 GB

Stage 2 models (EMA, 2000 epochs, Tiny encoder; FID on 50k samples without CFG)

File Model Config name FID Experiment Size
stage2/ditdh_s_h512_ema.pt DiTDH-S with a 512-wide head (default) DiTDH-S-h512 7.94 05 0.2 GB
stage2/lightningdit_s_ema.pt LightningDiT-S/1, no DDT head LightningDiT-S 8.70 06 0.1 GB
stage2/ditdh_s_h512_noshift_ema.pt 512-wide head, no time shift DiTDH-S-h512-noshift 9.01 07 0.2 GB
stage2/ditdh_ti_ema.pt DiTDH-Ti (18.5M) DiTDH-Ti 9.64 05 0.1 GB
stage2/ditdh_s_ema.pt DiTDH-S with the official 2048-wide head DiTDH-S 13.59 02 0.8 GB
stage2/rjf_ditdh_s_h512_ema.pt 512-wide head + RJF (R = 18) DiTDH-S-h512-RJF 6.81 12 0.2 GB
stage2/rjf_lightningdit_s_ema.pt LightningDiT-S/1 + RJF (R = 18) LightningDiT-S-RJF 6.97 12 0.1 GB
stage2/rjf_lightningdit_ti_ema.pt LightningDiT-Ti/1 + RJF (R = 18) LightningDiT-Ti-RJF 9.22 12 0.03 GB
stage2/rfm_lightningdit_s_ema.pt LightningDiT-S/1 + RFM (R = 18) LightningDiT-S-RFM 7.09 13 0.1 GB
stage2/rfm_lightningdit_ti_ema.pt LightningDiT-Ti/1 + RFM (R = 18) LightningDiT-Ti-RFM 9.41 13 0.03 GB
stage2/xpred_ditdh_s_h512_ema.pt 512-wide head, x-prediction DiTDH-S-h512-xpred 8.22 15 0.2 GB

R is the per-token radius the RJF / RFM samples are scaled to before decoding (13.66 is the mean token norm of the normalized latent; set transport.params.sphere_radius to change it).

AutoGuidance (the weak guides are short runs, 80 epochs; download the strong model as well)

Guide file Guides Config name w FID Experiment Size
stage2/guide_ditdh_ti_ema.pt DiTDH-S with a 512-wide head DiTDH-S-h512-AG 1.42 6.48 09 0.07 GB
stage2/guide_ditdh_ti_rjf_ema.pt 512-wide head + RJF (R = 13.66) DiTDH-S-h512-RJF-AG 1.5 6.31 14 0.07 GB
stage2/guide_lightningdit_ti_rjf_ema.pt LightningDiT-S/1 + RJF (R = 13.66) LightningDiT-S-RJF-AG 1.5 6.07 14 0.03 GB
stage2/guide_lightningdit_ti_rfm_ema.pt LightningDiT-S/1 + RFM (R = 13.66) LightningDiT-S-RFM-AG 1.5 6.11 14 0.03 GB

REPA Guidance (internal guidance; the model has a REPA head after body block 4 and guides itself; FID at the best scale w)

File Model Config name w FID Experiment Size
stage2/xpred_ig_ditdh_s_h512_ema.pt 512-wide head, x-prediction + REPA head, best DiTDH-S-h512-xpred-IG 3.0 4.70 16 0.2 GB
stage2/vpred_ig_ditdh_s_h512_ema.pt 512-wide head, velocity prediction + base head DiTDH-S-h512-vpred-IG 2.5 5.45 16 0.2 GB

In the repository, the config DiTDH-S-h512-nonoise decodes the default model with stage1/decoder_vitxl_gan_nonoise.pt (experiment 08, FID 10.45). All Stage 2 models use stage1/decoder_vitxl_gan.pt otherwise.

Details: docs.

Usage

git clone https://github.com/shumpei-takezaki/nanoRAE.git && cd nanoRAE
uv sync
uv run hf download shumpei777/nanoRAE --local-dir pretrained_models \
  --include stage2/xpred_ig_ditdh_s_h512_ema.pt stage1/decoder_vitxl_gan.pt encoders/encoder_dinov2_vit_tiny4.pt stats/normalization_stats.pt
PYTHONPATH=src uv run python src/sample.py --config configs/stage2/sampling/CIFAR10/DiTDH-S-h512-xpred-IG_pretrained.yaml   # -> sample.png

The encoders can also be loaded directly with nanoDINO: from nanodino.export import load_backbone.

License

MIT. Parts of the training code have their own licenses (the RAEv2-derived x-prediction / REPA head code is CC BY-NC 4.0): see THIRD_PARTY_NOTICES.md. The code builds on the official RAE implementation (MIT).

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train shumpei777/nanoRAE

Paper for shumpei777/nanoRAE