| --- |
| language: |
| - en |
| library_name: CAPTOR |
| tags: |
| - medical |
| - radiology |
| - computed-tomography |
| - vision-language |
| - contrastive-learning |
| - 3d |
| - pytorch |
| pipeline_tag: image-feature-extraction |
| --- |
| |
| # CAPTOR |
|
|
| **CAPTOR** (Chest Abdomen Pelvis foundaTion model for Oncologic Radiology) is a CT vision-language model. It is trained CLIP-style to align 3D CT regions (thorax, abdomen, pelvis, ...) with the corresponding parts of radiology reports. |
|
|
| - **Image encoder:** 3D-inflated ResNet-152 (I3ResNet) |
| - **Text encoder:** Clinical-Longformer |
| - **Architecture:** based on [Merlin](https://github.com/StanfordMIMI/Merlin) |
| - **Code:** https://github.com/Guerbet-AI/CAPTOR |
| - **Paper:** [CAPTOR: a vision-language foundation model for Chest-Abdomen-Pelvis Oncologic CT](https://arxiv.org/abs/2610.14226) |
|
|
| ## Checkpoints |
|
|
| Each checkpoint is in its own subfolder. Only the one you request is downloaded. |
|
|
| `CAPTOR_30K` : ~30K CT studies from our proprietray dataset |
|
|
| `CAPTOR_100K` : ~100K CT studies from our proprietray dataset and public datasets : Merlin, CT-RATE, INSPECT, BIMCV-R |
|
|
| ## Quick start |
|
|
| ```bash |
| pip install git+https://github.com/Guerbet-AI/CAPTOR.git |
| ``` |
|
|
| ```python |
| from captor import MerlinArchitecture |
| |
| model = MerlinArchitecture.from_pretrained("Guerbet-AI/CAPTOR", subfolder="CAPTOR_30K") |
| print(model) |
| ``` |
|
|
| Weights, config and tokenizer are fetched automatically. No ResNet-152 or Clinical-Longformer initialization weights are needed. |
|
|
| The model is returned in eval mode. Call `model.train()` to fine-tune. |
|
|
| ### Image embeddings only |
|
|
| ```python |
| model = MerlinArchitecture.from_pretrained( |
| "Guerbet-AI/CAPTOR", subfolder="CAPTOR_30K", ImageEmbedding=True |
| ) |
| ``` |
|
|
| ### Offline use |
|
|
| On a machine with internet: |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| |
| snapshot_download("Guerbet-AI/CAPTOR", local_dir="./captor_weights", |
| allow_patterns=["tokenizer/*", "CAPTOR_30K/*"]) |
| ``` |
|
|
| Copy the folder to the offline machine, then: |
|
|
| ```python |
| model = MerlinArchitecture.from_pretrained("./captor_weights", subfolder="CAPTOR_30K") |
| ``` |
|
|
| ## Input preprocessing |
|
|
| The model expects CT volumes cropped around an anatomical region. Segmentation with TotalSegmentator, crop-center extraction and the dataloaders are part of the pipeline in the [GitHub repository](https://github.com/Guerbet-AI/CAPTOR), which also documents the expected data format. Loading the model alone does not preprocess raw NIfTI files. |
|
|
| ## License and acknowledgements |
|
|
| CAPTOR builds on Merlin, Clinical-Longformer, torchvision's ResNet-152 and TotalSegmentator. Please respect their licenses. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{jeanmactoux2026captorvisionlanguagefoundationmodel, |
| title={CAPTOR: a vision-language foundation model for Chest-Abdomen-Pelvis Oncologic CT}, |
| author={Séléna Jean-Mactoux and Sophie Loizillon and Ghina Jardali and Amandine Crombé and Nathalie Lassau and Alexandre Bône}, |
| year={2026}, |
| eprint={2610.14226}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CV}, |
| url={https://arxiv.org/abs/2610.14226}, |
| } |
| ``` |
