Hugging Face's logo

rad-dit
/
ldit-radimagenet

Unconditional Image Generation
PyTorch
diffusion
medical-imaging
radimagenet
Model card Files
xet
 Community

You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

These models were trained on RadImageNet and are released for non-commercial research only (CC BY-NC 4.0). If you use the models or images generated with them, you must cite our paper and the papers in the Citation section. The models are not for clinical use.

Log in or Sign Up to review the conditions and access this model content.

rad-dit: LightningDiT-XL/1 on RadImageNet

A latent diffusion transformer (LightningDiT) with the VA-VAE tokeniser, initialised with ImageNet weights and trained on RadImageNet (256×256).

Available models

File Model gFID ↓
ldit-xl1-vavae.pt LightningDiT-XL/1 (675M) 16.43
latents_stats.pt ImageNet VA-VAE latent statistics, used in training and sampling (from hustvl/vavae-imagenet256-f16d32-dinov2, MIT) –

gFID: 256×256, cfg 1.0, Euler with 250 steps (see the paper). ldit-xl1-vavae.pt holds model, ema and config, as the official LightningDiT checkpoints. Sampling uses ema. The optimizer state is not included.

Architecture: inference.py builds the model from the inference config in the code repository (ldit/configs/inference_*.yaml, see the usage below). The checkpoint also holds a copy of the training config (config), for reference only. The inference config reads latents_stats.pt from the VA-VAE folder; the copy here is the same file.

Classes

Label Class
1001 CT abdomen
1002 CT lung
1003 MR hip
1004 MR knee
1005 MR shoulder
1006 No class (classifier-free guidance row)

The inference config sets num_classes: 1006. For example, class_probs: [0, 0, 0, 1, 0] gives MR knee only.

Usage

1. Get the code and log in

git clone https://github.com/CAIROLab-Bern/rad-dit.git && cd rad-dit
# Install the environment: see environment/README.md
hf auth login   # accept the terms on this page first

2. Download the model and the VA-VAE tokeniser

hf download rad-dit/ldit-radimagenet --local-dir ldit/checkpoints/radimagenet
# or in Python: from huggingface_hub import snapshot_download
#               snapshot_download("rad-dit/ldit-radimagenet", local_dir="ldit/checkpoints/radimagenet")
hf download hustvl/vavae-imagenet256-f16d32-dinov2 \
    --local-dir ldit/pretrained/vavae-imagenet256-f16d32-dinov2

3. Generate images (no dataset needed)

cd ldit
accelerate launch --num_processes=1 inference.py \
    --config configs/inference_radin_region5_pretrained.yaml \
    --ckpt checkpoints/radimagenet/ldit-xl1-vavae.pt \
    --output-dir samples --num-samples 256

Images are made in batches of per_proc_batch_size (256 in the config). Lower it in the config for fewer images or for a smaller GPU. The images are sampled uniformly over the classes. For selected classes, add class_probs to the data section of the config, with one value per class.

4. gFID: see ldit/README.md in the code repository.

Terms of use

  • Use the models for non-commercial research only (CC BY-NC 4.0).
  • Do not use the models or their images for diagnosis or other clinical decisions.
  • Generated images are synthetic. They are not real patient data.

Citation

If you use these models or images generated with them in your research, please cite our paper and consider giving the code repository a ⭐:

@inproceedings{georgiou2027natural,
  title     = {From Natural Images to Medical Scans: Transferability of Diffusion Models for Medical Image Generation},
  author    = {Georgiou, Efthymios and Kakogeorgiou, Ioannis and Br{\"u}ningk, Sarah},
  booktitle = {2027 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year      = {2027}
}

If you use our pretrained models, you must also cite the RadImageNet paper, as the data terms require:

  • Mei et al., RadImageNet: An Open Radiologic Deep Learning Research Dataset for Effective Transfer Learning, Radiology: Artificial Intelligence, 2022. Paper
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
Unconditional Image Generation
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Paper for rad-dit/ldit-radimagenet