Merlin-Cancer-Net

Merlin-Cancer-Net is an nnU-Net trained with tumor segmentation masks from the Merlin Plus dataset. It surpasses state-of-the-art public AI models and leading VLMs in the detection and segmentation of tumors in 9 organs: bladder, gallbladder, spleen, esophagus, stomach, duodenum, prostate, uterus, and adrenal glands.

Merlin Plus is a dataset with longitudinal metadata (patient IDs and scan dates) and per-voxel annotations for organs and 9 tumor types in the Merlin dataset (Stanford, 25,494 CT scans). Merlin Plus is part of a collaboration between the Merlin Project at Stanford and the R-Super Project at Johns Hopkins University. It is available at HuggingFace: https://huggingface.co/datasets/AbdomenAtlas/MerlinPlus. Merlin-Cancer-Net was trained with 423 masks from the Merlin Plus training dataset.

Papers

Scaling Artificial Intelligence for Multi-Tumor Early Detection with More Reports, Fewer Masks
Pedro R. A. S. Bassi, Xinze Zhou, Wenxuan Li, Szymon Płotka, Jieneng Chen, Qi Chen, Zheren Zhu, Jakub Prządo, Ibrahim E. Hamacı, Sezgin Er, Yuhan Wang, Ashwin Kumar, Bjoern Menze, Jarosław B. Ćwikła, Yuyin Zhou, Akshay S. Chaudhari, Curtis P. Langlotz, Sergio Decherchi, Andrea Cavalli, Kang Wang, Yang Yang, Alan L. Yuille, Zongwei Zhou*
Johns Hopkins University, Stanford University, University of Zurich, University of Warsaw, Italian Institute of Technology, UCSF
Preprint

Merlin Plus: A Large-Scale, Multi-cancer, Image-Mask-Report Dataset
Pedro R. A. S. Bassi†, Wenxuan Li†, Szymon Płotka†, Ruby Honjol, Jakub Prządo, Xinze Zhou, Kang Wang, Yang Yang, Malte Jensen, Akshay S. Chaudhari, Curtis P. Langlotz, Alan L. Yuille, and Zongwei Zhou.
MICCAI 2026, LNCS 16895. Springer Nature Switzerland.

Merlin: A Vision Language Foundation Model for 3D Computed Tomography
Louis Blankemeier, Joseph Paul Cohen, Ashwin Kumar, Dave Van Veen, Syed Jamal Safdar Gardezi, Magdalini Paschali, Zhihong Chen, Jean-Benoit Delbrouck, Eduardo Reis, Cesar Truyts, Christian Bluethgen, Malte Engmann Kjeldskov Jensen, Sophie Ostmeier, Maya Varma, Jeya Maria Jose Valanarasu, Zhongnan Fang, Zepeng Huo, Zaid Nabulsi, Diego Ardila, Wei-Hung Weng, Edson Amaro Junior, Neera Ahuja, Jason Fries, Nigam H. Shah, Andrew Johnston, Robert D. Boutin, Andrew Wentland, Curtis P. Langlotz, Jason Hom, Sergios Gatidis, Akshay S. Chaudhari
Stanford University
Nature, 2026.

nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation
Fabian Isensee, Paul F. Jaeger, Simon A. A. Kohl, Jens Petersen, Klaus H. Maier-Hein
German Cancer Research Center (DKFZ)
Nature Methods, 18(2), 203-211, 2021.

Instructions

Installation

conda create -n merlin_cancer_net python=3.12 -y
conda activate merlin_cancer_net
pip install torch torchvision torchaudio
pip install nnunetv2 pandas tqdm nibabel itk

Data format

Put every CT in one folder, as .nii.gz files. You can use any file name.

/path/to/dataset/
├── case_1.nii.gz
├── case_2.nii.gz
...

Inference

bash parallel_inference.sh \
    --pth /path/to/dataset/ \
    --outdir /path/to/joint/outputs/ \
    --checkpoint Dataset307_radiologist_annotations_Merlin_Plus/nnUNetTrainer__nnUNetPlannerResEncL_torchres_isotropic__3d_fullres/ \
    --gpus 0

Pass several GPUs as --gpus 0,1,2,3 to run in parallel.

Split the joint labels

The nnU-Net produces joint labels (one file for all segmentation classes). To split these labels:

python split_labels_multi_tumor.py \
    --input_dir /path/to/joint/outputs/ \
    --output_dir /path/to/outputs/

Outputs one folder per case, with one mask per tumor and organ:

/path/to/outputs/
├── BDMAP_00000001/
|    └── predictions/
|         ├── spleen_lesion.nii.gz
|         ├── stomach_lesion.nii.gz
|         ├── liver.nii.gz
|         └── ...

Citations

@article{bassi2025scaling,
  title={Scaling Artificial Intelligence for Multi-Tumor Early Detection with More Reports, Fewer Masks},
  author={Bassi, Pedro R. A. S. and Zhou, Xinze and Li, Wenxuan and P{\l}otka, Szymon and Chen, Jieneng and Chen, Qi and Zhu, Zheren and Prz{\k{a}}do, Jakub and Hamamci, Ibrahim E. and Er, Sezgin and others},
  journal={arXiv preprint arXiv:2510.14803},
  year={2025}
}


@InProceedings{BasPed_Merlin_MICCAI2026,
    author = { Bassi, Pedro R. A. S. AND Li, Wenxuan AND Płotka, Szymon AND Honjol, Ruby AND Prządo, Jakub AND Zhou, Xinze AND Wang, Kang AND Yang, Yang AND Jensen, Malte AND Chaudhari, Akshay S. AND Langlotz, Curtis P. AND Yuille, Alan L. AND Zhou, Zongwei},
    title = { { Merlin Plus: A Large-Scale, Multi-cancer, Image-Mask-Report Dataset } },
    booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
    year = {2026},
    publisher = {Springer Nature Switzerland},
    volume = {LNCS 16895},
    month = {September},
    page = {pending}
}


@article{blankemeier_kumar2026merlin,
  author = {Blankemeier, Louis and Kumar, Ashwin and Cohen, Joseph Paul and Liu, Jiaming and Liu, Longchao and Van Veen, Dave and Gardezi, Syed Jamal Safdar and Yu, Hongkun and Paschali, Magdalini and Chen, Zhihong and Delbrouck, Jean-Benoit and Reis, Eduardo and Holland, Robbie and Truyts, Cesar and Bluethgen, Christian and Wu, Yufu and Lian, Long and Jensen, Malte Engmann Kjeldskov and Ostmeier, Sophie and Varma, Maya and Valanarasu, Jeya Maria Jose and Fang, Zhongnan and Huo, Zepeng and Nabulsi, Zaid and Ardila, Diego and Weng, Wei-Hung and Amaro Junior, Edson and Ahuja, Neera and Fries, Jason and Shah, Nigam H. and Zaharchuk, Greg and Willis, Marc and Yala, Adam and Johnston, Andrew and Boutin, Robert D. and Wentland, Andrew and Langlotz, Curtis P. and Hom, Jason and Gatidis, Sergios and Chaudhari, Akshay S.},
  title   = {Merlin: a computed tomography vision-language foundation model and dataset},
  journal = {Nature},
  year    = {2026},
  doi     = {10.1038/s41586-026-10181-8},
  url     = {https://doi.org/10.1038/s41586-026-10181-8}
}

@article{isensee2021nnunet,
  title={nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation},
  author={Isensee, Fabian and Jaeger, Paul F. and Kohl, Simon A. A. and Petersen, Jens and Maier-Hein, Klaus H.},
  journal={Nature Methods},
  volume={18},
  number={2},
  pages={203--211},
  year={2021}
}

Acknowledgement

This work was supported by the Lustgarten Foundation for Pancreatic Cancer Research, the Patrick J. McGovern Foundation Award, and the National Institutes of Health (NIH) under Award Number R01EB037669. Paper content is covered by patents pending.

© The Johns Hopkins University. This work is openly licensed via CC BY-NC-ND. Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International Public License