--- pretty_name: "RLAD + SwinV2-Large (aka LUNetV2): Retinal Artery/Vein Segmentation" license: other license_name: apache-2.0-code-plus-noncommercial-weights library_name: transformers base_model: microsoft/swinv2-large-patch4-window12-192-22k tags: - retinal-fundus - artery-vein-segmentation - medical-imaging - ophthalmology - swinv2 datasets: - JonathanFhima/REYIA --- # Enhancing Retinal Vessel Segmentation Generalization via Layout-Aware Generative Modelling ### State-of-the-art AV segmentation in retinal fundus images
Jonathan Fhima  •  Jan Van Eijgen  •  Lennert Beeckmans  •  Thomas Jacob  •  Moti Freiman
Luis Filipe Nakayama  •  Ingeborg Stalmans  •  Chaim Baskin  •  Joachim A. Behar
[![arXiv](https://img.shields.io/badge/arXiv-2503.01190-red)](https://arxiv.org/abs/2503.01190) [![GitHub](https://img.shields.io/badge/github-RLAD-lightgrey)](https://github.com/aim-lab/RLAD) [![Dataset](https://img.shields.io/badge/%F0%9F%A4%97%20Dataset-REYIA-yellow)](https://huggingface.co/datasets/JonathanFhima/REYIA) [![RLAD + PVBM Tutorial](https://img.shields.io/badge/RLAD_%2B_PVBM_tutorial-Colab-orange)](https://colab.research.google.com/github/aim-lab/PVBM/blob/main/rlad_pvbm_tutorial.ipynb) The strongest open-weight model for retinal artery/vein segmentation we know of, trained with [RLAD](https://github.com/aim-lab/RLAD)-generated images and the [REYIA](https://huggingface.co/datasets/JonathanFhima/REYIA) dataset. Internally also known as LUNetV2. ![Example predictions across UZLF, LES-AV, INSPIRE, FIVES, DRIVE, TREND, MBRSET, and AV-WIDE](https://huggingface.co/JonathanFhima/RLAD-SwinV2-Large/resolve/main/benchmark_examples.png) ## Quickstart The model loads the standard Hugging Face way - `AutoModel` and `AutoImageProcessor` with `trust_remote_code=True` download the code, config, weights, and preprocessing directly, nothing to fetch by hand: ```python import numpy as np import torch from PIL import Image from transformers import AutoModel, AutoImageProcessor repo = "JonathanFhima/RLAD-SwinV2-Large" model = AutoModel.from_pretrained(repo, trust_remote_code=True).to("cuda").to(torch.bfloat16).eval() processor = AutoImageProcessor.from_pretrained(repo, trust_remote_code=True) image = Image.open("fundus.png").convert("RGB") inputs = processor(image, return_tensors="pt")["pixel_values"].to("cuda", torch.bfloat16) with torch.no_grad(): output = model(inputs).float() probs = torch.sigmoid(output)[0].cpu().numpy() mask = np.zeros((*probs.shape[1:], 3), dtype=np.uint8) mask[probs[0] > 0.5, 0] = 255 # arteries: red mask[(probs[1] > 0.5) & ~(probs[0] > 0.5), 2] = 255 # veins: blue ``` Or just run the CLI script, which does the same thing plus writes a blended overlay image: ```bash python3 -m venv lunetv2_env && source lunetv2_env/bin/activate pip install -r requirements.txt python infer.py fundus.png out/ ``` Writes `out/mask.png` and `out/overlay.png`. Runs in `bfloat16`, the precision the model was trained in. For a full pipeline from a raw fundus image to vasculature biomarkers, see the [RLAD + PVBM end-to-end tutorial](https://colab.research.google.com/github/aim-lab/PVBM/blob/main/rlad_pvbm_tutorial.ipynb), which feeds this model's output into [PVBM](https://github.com/aim-lab/PVBM). ## Example ![Input fundus image and its predicted artery/vein overlay](https://huggingface.co/JonathanFhima/RLAD-SwinV2-Large/resolve/main/example.png) ## Results If you plan to use this model or compare against it, we recommend scoring against our benchmark rather than re-running every baseline yourself. The scorer that produced these numbers lives in the [REYIA](https://huggingface.co/datasets/JonathanFhima/REYIA) repo (`scripts/eval/dice_score.py`). Artery/vein Dice score, averaged across the same local, external, and out-of-distribution (OOD) test sets; ∆ rows show the improvement RLAD adds on top of each backbone, and † marks results with data leakage during training — this release is the SwinV2-large row. ![Table 1: RLAD Results — quantitative comparison of RLAD-generated data integrated into DinoV2, RETFound, and SwinV2 across model sizes, evaluated on Local, External, and OOD benchmarks](https://raw.githubusercontent.com/aim-lab/RLAD/main/figures/table1_rlad_results.png) ## Ablations ![Table 3: Impact of increasing the number of training datasets — adding newly introduced or annotated datasets to the SwinV2-tiny training pipeline](https://huggingface.co/JonathanFhima/RLAD-SwinV2-Large/resolve/main/table3_dataset_ablation.png) ![Table 4: Pretraining Objective and Generation Method — impact of pretraining objectives and AV-conditioned generation on Local, External, and OOD performance](https://huggingface.co/JonathanFhima/RLAD-SwinV2-Large/resolve/main/table4_pretraining.png) ## License The code (`LUNetv2.py`, `infer.py`) is Apache 2.0. The weights (`model.safetensors`) are non-commercial, research use only — the training set mixes in subsets that are themselves non-commercial or restricted: | Source | Status | |---|---| | UZLF | [KU Leuven dataset](https://rdr.kuleuven.be/dataset.xhtml?persistentId=doi:10.48804/Z7SHGO) | | GRAPE | CC0 1.0 — see [REYIA](https://huggingface.co/datasets/JonathanFhima/REYIA) | | MAGRABIA | CC BY-NC 4.0 — see [REYIA](https://huggingface.co/datasets/JonathanFhima/REYIA) | | MESSIDOR | redistribution and commercial use explicitly prohibited — see [REYIA](https://huggingface.co/datasets/JonathanFhima/REYIA) | | PAPILA | figshare license field reads GPL 3.0+ — likely a depositor mis-click, not corrected as of this writing — see [REYIA](https://huggingface.co/datasets/JonathanFhima/REYIA) | | ENRICH | non-commercial use only, KU Leuven data-sharing protocol — see [REYIA](https://huggingface.co/datasets/JonathanFhima/REYIA) | ## Citation > Fhima, J., Van Eijgen, J., Beeckmans, L., Jacob, T., Freiman, M., > Nakayama, L.F., Stalmans, I., Baskin, C., Behar, J.A. *Enhancing Retinal > Vessel Segmentation Generalization via Layout-Aware Generative > Modelling.* arXiv:2503.01190, 2025. https://arxiv.org/abs/2503.01190