Drop the image-segmentation tag too - HF was inferring pipeline_tag back from it, which drove the misleading pipeline() snippet and task-specific Colab/Kaggle notebook templates (both assume single-label argmax output, which this multi-label model does not produce). artery-vein-segmentation tag already covers discoverability.

cf1f116 verified 12 days ago

preview code

Raw History Blame Contribute Delete

6.25 kB

---
pretty_name: "RLAD + SwinV2-Large (aka LUNetV2): Retinal Artery/Vein Segmentation"
license: other
license_name: apache-2.0-code-plus-noncommercial-weights
library_name: transformers
base_model: microsoft/swinv2-large-patch4-window12-192-22k
tags:
  - retinal-fundus
  - artery-vein-segmentation
  - medical-imaging
  - ophthalmology
  - swinv2
datasets:
  - JonathanFhima/REYIA
---
# Enhancing Retinal Vessel Segmentation Generalization via Layout-Aware Generative Modelling

### State-of-the-art AV segmentation in retinal fundus images

<div align="center">

Jonathan Fhima &nbsp;•&nbsp; Jan Van Eijgen &nbsp;•&nbsp; Lennert Beeckmans &nbsp;•&nbsp; Thomas Jacob &nbsp;•&nbsp; Moti Freiman
<br>
Luis Filipe Nakayama &nbsp;•&nbsp; Ingeborg Stalmans &nbsp;•&nbsp; Chaim Baskin &nbsp;•&nbsp; Joachim A. Behar

</div>

[![arXiv](https://img.shields.io/badge/arXiv-2503.01190-red)](https://arxiv.org/abs/2503.01190)
[![GitHub](https://img.shields.io/badge/github-RLAD-lightgrey)](https://github.com/aim-lab/RLAD)
[![Dataset](https://img.shields.io/badge/%F0%9F%A4%97%20Dataset-REYIA-yellow)](https://huggingface.co/datasets/JonathanFhima/REYIA)
[![RLAD + PVBM Tutorial](https://img.shields.io/badge/RLAD_%2B_PVBM_tutorial-Colab-orange)](https://colab.research.google.com/github/aim-lab/PVBM/blob/main/rlad_pvbm_tutorial.ipynb)

The strongest open-weight model for retinal artery/vein segmentation we know of, trained with [RLAD](https://github.com/aim-lab/RLAD)-generated images and the [REYIA](https://huggingface.co/datasets/JonathanFhima/REYIA) dataset. Internally also known as LUNetV2.

![Example predictions across UZLF, LES-AV, INSPIRE, FIVES, DRIVE, TREND, MBRSET, and AV-WIDE](https://huggingface.co/JonathanFhima/RLAD-SwinV2-Large/resolve/main/benchmark_examples.png)

## Quickstart

The model loads the standard Hugging Face way - `AutoModel` and
`AutoImageProcessor` with `trust_remote_code=True` download the code, config,
weights, and preprocessing directly, nothing to fetch by hand:

```python
import numpy as np
import torch
from PIL import Image
from transformers import AutoModel, AutoImageProcessor
repo = "JonathanFhima/RLAD-SwinV2-Large"
model = AutoModel.from_pretrained(repo, trust_remote_code=True).to("cuda").to(torch.bfloat16).eval()
processor = AutoImageProcessor.from_pretrained(repo, trust_remote_code=True)
image = Image.open("fundus.png").convert("RGB")
inputs = processor(image, return_tensors="pt")["pixel_values"].to("cuda", torch.bfloat16)
with torch.no_grad():
    output = model(inputs).float()
probs = torch.sigmoid(output)[0].cpu().numpy()
mask = np.zeros((*probs.shape[1:], 3), dtype=np.uint8)
mask[probs[0] > 0.5, 0] = 255                        # arteries: red
mask[(probs[1] > 0.5) & ~(probs[0] > 0.5), 2] = 255  # veins: blue
```

Or just run the CLI script, which does the same thing plus writes a blended
overlay image:

```bash
python3 -m venv lunetv2_env && source lunetv2_env/bin/activate
pip install -r requirements.txt
python infer.py fundus.png out/
```

Writes `out/mask.png` and `out/overlay.png`. Runs in `bfloat16`, the
precision the model was trained in.

For a full pipeline from a raw fundus image to vasculature biomarkers, see the
[RLAD + PVBM end-to-end tutorial](https://colab.research.google.com/github/aim-lab/PVBM/blob/main/rlad_pvbm_tutorial.ipynb),
which feeds this model's output into [PVBM](https://github.com/aim-lab/PVBM).

## Example

![Input fundus image and its predicted artery/vein overlay](https://huggingface.co/JonathanFhima/RLAD-SwinV2-Large/resolve/main/example.png)

## Results

If you plan to use this model or compare against it, we recommend
scoring against our benchmark rather than re-running every baseline
yourself. The scorer that produced these numbers lives in the
[REYIA](https://huggingface.co/datasets/JonathanFhima/REYIA) repo
(`scripts/eval/dice_score.py`).

Artery/vein Dice score, averaged across the same local, external, and out-of-distribution (OOD) test sets; ∆ rows show the improvement RLAD adds on top of each backbone, and † marks results with data leakage during training — this release is the SwinV2-large row.

![Table 1: RLAD Results — quantitative comparison of RLAD-generated data integrated into DinoV2, RETFound, and SwinV2 across model sizes, evaluated on Local, External, and OOD benchmarks](https://raw.githubusercontent.com/aim-lab/RLAD/main/figures/table1_rlad_results.png)

## Ablations

![Table 3: Impact of increasing the number of training datasets — adding newly introduced or annotated datasets to the SwinV2-tiny training pipeline](https://huggingface.co/JonathanFhima/RLAD-SwinV2-Large/resolve/main/table3_dataset_ablation.png)

![Table 4: Pretraining Objective and Generation Method — impact of pretraining objectives and AV-conditioned generation on Local, External, and OOD performance](https://huggingface.co/JonathanFhima/RLAD-SwinV2-Large/resolve/main/table4_pretraining.png)

## License

The code (`LUNetv2.py`, `infer.py`) is Apache 2.0. The weights
(`model.safetensors`) are non-commercial, research use only — the
training set mixes in subsets that are themselves non-commercial or
restricted:

| Source | Status |
|---|---|
| UZLF | [KU Leuven dataset](https://rdr.kuleuven.be/dataset.xhtml?persistentId=doi:10.48804/Z7SHGO) |
| GRAPE | CC0 1.0 — see [REYIA](https://huggingface.co/datasets/JonathanFhima/REYIA) |
| MAGRABIA | CC BY-NC 4.0 — see [REYIA](https://huggingface.co/datasets/JonathanFhima/REYIA) |
| MESSIDOR | redistribution and commercial use explicitly prohibited — see [REYIA](https://huggingface.co/datasets/JonathanFhima/REYIA) |
| PAPILA | figshare license field reads GPL 3.0+ — likely a depositor mis-click, not corrected as of this writing — see [REYIA](https://huggingface.co/datasets/JonathanFhima/REYIA) |
| ENRICH | non-commercial use only, KU Leuven data-sharing protocol — see [REYIA](https://huggingface.co/datasets/JonathanFhima/REYIA) |

## Citation

> Fhima, J., Van Eijgen, J., Beeckmans, L., Jacob, T., Freiman, M.,
> Nakayama, L.F., Stalmans, I., Baskin, C., Behar, J.A. *Enhancing Retinal
> Vessel Segmentation Generalization via Layout-Aware Generative
> Modelling.* arXiv:2503.01190, 2025. https://arxiv.org/abs/2503.01190

	---
	pretty_name: "RLAD + SwinV2-Large (aka LUNetV2): Retinal Artery/Vein Segmentation"
	license: other
	license_name: apache-2.0-code-plus-noncommercial-weights
	library_name: transformers
	base_model: microsoft/swinv2-large-patch4-window12-192-22k
	tags:
	- retinal-fundus
	- artery-vein-segmentation
	- medical-imaging
	- ophthalmology
	- swinv2
	datasets:
	- JonathanFhima/REYIA
	---

	# Enhancing Retinal Vessel Segmentation Generalization via Layout-Aware Generative Modelling

	### State-of-the-art AV segmentation in retinal fundus images

	<div align="center">

	Jonathan Fhima  •  Jan Van Eijgen  •  Lennert Beeckmans  •  Thomas Jacob  •  Moti Freiman
	<br>
	Luis Filipe Nakayama  •  Ingeborg Stalmans  •  Chaim Baskin  •  Joachim A. Behar

	</div>

	[![arXiv](https://img.shields.io/badge/arXiv-2503.01190-red)](https://arxiv.org/abs/2503.01190)
	[![GitHub](https://img.shields.io/badge/github-RLAD-lightgrey)](https://github.com/aim-lab/RLAD)
	[![Dataset](https://img.shields.io/badge/%F0%9F%A4%97%20Dataset-REYIA-yellow)](https://huggingface.co/datasets/JonathanFhima/REYIA)
	[![RLAD + PVBM Tutorial](https://img.shields.io/badge/RLAD_%2B_PVBM_tutorial-Colab-orange)](https://colab.research.google.com/github/aim-lab/PVBM/blob/main/rlad_pvbm_tutorial.ipynb)

	The strongest open-weight model for retinal artery/vein segmentation we know of, trained with [RLAD](https://github.com/aim-lab/RLAD)-generated images and the [REYIA](https://huggingface.co/datasets/JonathanFhima/REYIA) dataset. Internally also known as LUNetV2.

	![Example predictions across UZLF, LES-AV, INSPIRE, FIVES, DRIVE, TREND, MBRSET, and AV-WIDE](https://huggingface.co/JonathanFhima/RLAD-SwinV2-Large/resolve/main/benchmark_examples.png)

	## Quickstart

	The model loads the standard Hugging Face way - `AutoModel` and
	`AutoImageProcessor` with `trust_remote_code=True` download the code, config,
	weights, and preprocessing directly, nothing to fetch by hand:

	```python
	import numpy as np
	import torch
	from PIL import Image
	from transformers import AutoModel, AutoImageProcessor

	repo = "JonathanFhima/RLAD-SwinV2-Large"
	model = AutoModel.from_pretrained(repo, trust_remote_code=True).to("cuda").to(torch.bfloat16).eval()
	processor = AutoImageProcessor.from_pretrained(repo, trust_remote_code=True)

	image = Image.open("fundus.png").convert("RGB")
	inputs = processor(image, return_tensors="pt")["pixel_values"].to("cuda", torch.bfloat16)
	with torch.no_grad():
	output = model(inputs).float()

	probs = torch.sigmoid(output)[0].cpu().numpy()
	mask = np.zeros((*probs.shape[1:], 3), dtype=np.uint8)
	mask[probs[0] > 0.5, 0] = 255 # arteries: red
	mask[(probs[1] > 0.5) & ~(probs[0] > 0.5), 2] = 255 # veins: blue
	```

	Or just run the CLI script, which does the same thing plus writes a blended
	overlay image:

	```bash
	python3 -m venv lunetv2_env && source lunetv2_env/bin/activate
	pip install -r requirements.txt
	python infer.py fundus.png out/
	```

	Writes `out/mask.png` and `out/overlay.png`. Runs in `bfloat16`, the
	precision the model was trained in.

	For a full pipeline from a raw fundus image to vasculature biomarkers, see the
	[RLAD + PVBM end-to-end tutorial](https://colab.research.google.com/github/aim-lab/PVBM/blob/main/rlad_pvbm_tutorial.ipynb),
	which feeds this model's output into [PVBM](https://github.com/aim-lab/PVBM).

	## Example

	![Input fundus image and its predicted artery/vein overlay](https://huggingface.co/JonathanFhima/RLAD-SwinV2-Large/resolve/main/example.png)

	## Results

	If you plan to use this model or compare against it, we recommend
	scoring against our benchmark rather than re-running every baseline
	yourself. The scorer that produced these numbers lives in the
	[REYIA](https://huggingface.co/datasets/JonathanFhima/REYIA) repo
	(`scripts/eval/dice_score.py`).

	Artery/vein Dice score, averaged across the same local, external, and out-of-distribution (OOD) test sets; ∆ rows show the improvement RLAD adds on top of each backbone, and † marks results with data leakage during training — this release is the SwinV2-large row.

	![Table 1: RLAD Results — quantitative comparison of RLAD-generated data integrated into DinoV2, RETFound, and SwinV2 across model sizes, evaluated on Local, External, and OOD benchmarks](https://raw.githubusercontent.com/aim-lab/RLAD/main/figures/table1_rlad_results.png)

	## Ablations

	![Table 3: Impact of increasing the number of training datasets — adding newly introduced or annotated datasets to the SwinV2-tiny training pipeline](https://huggingface.co/JonathanFhima/RLAD-SwinV2-Large/resolve/main/table3_dataset_ablation.png)

	![Table 4: Pretraining Objective and Generation Method — impact of pretraining objectives and AV-conditioned generation on Local, External, and OOD performance](https://huggingface.co/JonathanFhima/RLAD-SwinV2-Large/resolve/main/table4_pretraining.png)

	## License

	The code (`LUNetv2.py`, `infer.py`) is Apache 2.0. The weights
	(`model.safetensors`) are non-commercial, research use only — the
	training set mixes in subsets that are themselves non-commercial or
	restricted:

	\| Source \| Status \|
	\|---\|---\|
	\| UZLF \| [KU Leuven dataset](https://rdr.kuleuven.be/dataset.xhtml?persistentId=doi:10.48804/Z7SHGO) \|
	\| GRAPE \| CC0 1.0 — see [REYIA](https://huggingface.co/datasets/JonathanFhima/REYIA) \|
	\| MAGRABIA \| CC BY-NC 4.0 — see [REYIA](https://huggingface.co/datasets/JonathanFhima/REYIA) \|
	\| MESSIDOR \| redistribution and commercial use explicitly prohibited — see [REYIA](https://huggingface.co/datasets/JonathanFhima/REYIA) \|
	\| PAPILA \| figshare license field reads GPL 3.0+ — likely a depositor mis-click, not corrected as of this writing — see [REYIA](https://huggingface.co/datasets/JonathanFhima/REYIA) \|
	\| ENRICH \| non-commercial use only, KU Leuven data-sharing protocol — see [REYIA](https://huggingface.co/datasets/JonathanFhima/REYIA) \|

	## Citation

	> Fhima, J., Van Eijgen, J., Beeckmans, L., Jacob, T., Freiman, M.,
	> Nakayama, L.F., Stalmans, I., Baskin, C., Behar, J.A. *Enhancing Retinal
	> Vessel Segmentation Generalization via Layout-Aware Generative
	> Modelling.* arXiv:2503.01190, 2025. https://arxiv.org/abs/2503.01190