euswbnix/ddpm-cifar10

DDPM on CIFAR-10: a paper-faithful reproduction (FID 3.14)

A from-scratch, paper-faithful reproduction of DDPM (Ho et al., 2020) for unconditional 32×32 CIFAR-10, trained on a single RTX 5090. It reaches a legacy-TF FID of 3.139 (50k samples, 1000-step ancestral sampling, best of 3 seeds), matching the paper's 3.17.

These are the EMA weights (decay 0.99995) of the 700k-step snapshot, the best snapshot of the run. The model is the baseline entry of a single-GPU diffusion speedrun benchmark: https://github.com/Euswbnix/DDPM-Diffusion-Model.

64 uncurated samples: DDPM-1000 ancestral sampling, seed 0.

Results

FID is computed with clean-fid in legacy_tensorflow mode against CIFAR-10 train statistics, on 50k samples from DDPM-1000 ancestral sampling with EMA 0.99995. Each record uses exactly three distinct sampling seeds and reports the minimum. clean-mode FID is given as a disclosure.

Snapshot	Training time	FID (min of 3)	FID per seed (0 / 1 / 2)	clean-FID (min)
600k	8.24 h	3.1912	3.1912 / 3.2637 / 3.2757	3.8690
700k (this model)	9.61 h	3.1385	3.1385 / 3.2275 / 3.2149	3.8051
800k (paper budget)	10.98 h	3.1500	3.1500 / 3.1824 / 3.1643	3.8349

Training time is on-clock wall time from process start to the snapshot save. It includes torch.compile warmup, data loading and EMA updates, and excludes evaluation.

• The 700k snapshot beats the paper's full 800k-step budget on both FID and time: this recipe plateaus near FID 3.14 by 700k steps.
• The time to reach FID 3.17 falls between 8.24 h and 9.61 h. The 600k snapshot misses the target on all three seeds; the 700k snapshot clears it.
• FID gaps below about 0.04 are within seed-to-seed noise.

The raw evaluation files are in eval/.

Usage

You need torch, safetensors and huggingface_hub. The architecture and the samplers are the two Python files in this repository.

import json, sys
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

repo = "euswbnix/ddpm-cifar10"
for f in ("unet.py", "diffusion.py"):
    hf_hub_download(repo, f, local_dir=".")
sys.path.insert(0, ".")
from unet import UNet
from diffusion import GaussianDiffusion

cfg = json.load(open(hf_hub_download(repo, "config.json")))
model = UNet(
    ch=cfg["ch"], ch_mult=tuple(cfg["ch_mult"]), num_res_blocks=cfg["num_res_blocks"],
    attn_res=tuple(cfg["attn_res"]), dropout=cfg["dropout"], img_res=cfg["img_res"],
)
model.load_state_dict(load_file(hf_hub_download(repo, "model.safetensors")))

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device).eval()
diffusion = GaussianDiffusion(cfg["T"], cfg["beta_start"], cfg["beta_end"]).to(device)

gen = torch.Generator(device).manual_seed(0)
x = diffusion.p_sample_loop(model, (16, 3, 32, 32), device, generator=gen)  # in [-1, 1]
images = (x.clamp(-1, 1) + 1) / 2                                           # in [0, 1]

Use the ancestral sampler (p_sample_loop) to match the reported numbers. Few-step DDIM sampling hurts this ε-prediction, linear-β model badly: in 10k-sample checks, DDIM with 100 steps scored about 8 FID where DDPM-1000 scored about 5.

Training details

• Architecture: 35.7M-parameter U-Net as in the official implementation. Base width 128, channel multipliers (1, 2, 2, 2), two residual blocks per resolution, self-attention at 16×16, dropout 0.1, sinusoidal timestep embedding. Output layers are zero-initialized; all other weights use Xavier-uniform initialization.
• Diffusion: T = 1000, linear β from 1e-4 to 0.02, ε-prediction, simple MSE loss.
• Optimization: Adam, learning rate 2e-4 with a 5k-step linear warmup, gradient clipping at 1.0, batch size 128, random horizontal flips, seed 0.
• EMA: four decay rates tracked during training. 0.99995 gave the best FID and is the one released.
• Systems: bf16 autocast, torch.compile, PyTorch 2.11, one RTX 5090.
• Sampling: DDPM ancestral, 1000 steps, variance σ² = β, predicted x₀ clipped to [-1, 1].

Limitations

• Unconditional CIFAR-10 at 32×32 only. This is a research and benchmark artifact.
• Sampling takes 1000 network evaluations per image.

Citation

@inproceedings{ho2020denoising,
  title     = {Denoising Diffusion Probabilistic Models},
  author    = {Ho, Jonathan and Jain, Ajay and Abbeel, Pieter},
  booktitle = {Advances in Neural Information Processing Systems},
  volume    = {33},
  pages     = {6840--6851},
  year      = {2020}
}