organic3d-encoder / README.md
zllhhhh's picture
Model card: arXiv paper link and citation
997d1f2 verified
|
Raw History Blame Contribute Delete
3.45 kB
---
license: cc-by-nc-sa-4.0
library_name: pytorch
pipeline_tag: zero-shot-classification
tags:
- 3D CT
- computed tomography
- vision-language
- medical imaging
- radiology
datasets:
- ibrahimhamamci/CT-RATE
---
# ORGANIC-3D encoder (`enc4_organic3d`)
A 3D CT image–text encoder that scores any sentence against a CT volume and follows the image evidence for each
finding: its score drops when the finding's organ is erased, stays when another organ is erased, and prefers the
sentence that names the correct side. Paper: [*ORGANIC-3D: Grounding 3D CT Vision Language Models in Organ-Level Image–Text Coupling*](https://arxiv.org/abs/2611.23529).
· Code: https://github.com/lzl199704/organic3d
· Benchmark: https://huggingface.co/datasets/zllhhhh/organic3d-benchmark
· VLM: https://huggingface.co/zllhhhh/organic3d-vlm
**Research use only. Not for clinical use.**
## Usage
```bash
pip install -e <organic3d repository> # the `organic` package and its requirements
python organic3d.py encoder --ct scan.nii.gz --region chest --totalseg \
--sentence "There is a nodule in the right upper lobe."
```
The script downloads this repository, resamples the CT to 1 mm RAS, centres a 384 mm crop with TotalSegmentator (or an
estimate without it) and prints the score of every finding of the region (Δf = S(V, "There is f.") − S(V, "No f."))
and of each `--sentence`. From Python: `from organic3d import Encoder`.
## Model
| | |
|---|---|
| Image tower | COLIPRI 3D ViT (Primus-M), 192³ input at 2 mm, fine-tuned |
| Anatomy tokens | 144 learned queries (16 organ groups × 8, left and right lung and kidney separate, + 16 global), 2 cross-attention layers |
| Score | each sentence is routed to the tokens of the organs it names and pools them with text-conditioned weights; cosine to the CXR-BERT sentence embedding / temperature |
| Files | `model.safetensors` (272 M parameters, fp32), `config.json`, `organic3d.py` |
## Training
Public data only: the CT-RATE and Merlin training splits (3 : 2). Contrastive loss, classification loss and the four
ORGANIC losses (organ-erasure with a control organ, mirror, matched-pair, anatomy-attention), 12,000 steps on 4 × H100.
Config: `configs/encoder/enc4_organic3d.yaml` in the code repository.
## Results (paper Table 1, %)
| | CT-RATE | RAD-ChestCT | Merlin |
|---|---|---|---|
| MLC-AUROC | 87.7 | 79.7 | 87.7 |
| Matched-pair AUROC (MP) | 82.5 | 78.2 | 86.5 |
| Organ-erasure ΔAUROC (OE) | 28.5 | 18.4 | 21.3 |
| Laterality accuracy (Side) | 76.9 | – | 63.8 |
Lung lobe erasure hit rate 74.7 (CT-RATE); lesion AUROC 79.5 and lesion-erasure hit rate 74.9 (ReXGroundingCT);
zero-shot tumour detection 75.0 and lesion-erasure hit rate 70.7 (AbdomenAtlas 3.0).
## Limitations
Trained on chest CT from one centre (CT-RATE) and abdominal CT from one centre (Merlin) with report-derived labels;
performance elsewhere is unknown. Laterality is far from perfect (Side 76.9 / 63.8). Licence terms of COLIPRI, CT-RATE
and Merlin apply.
## Citation
```bibtex
@article{liu2026organic3d,
title = {ORGANIC3D: Grounding 3D CT Vision Language Models in Organ Level Image Text Coupling},
author = {Liu, Zelong and Ahmed, Hamza and Lee, Wynn Wingyi and Torjman, Mickael and Dimitrakakis, Alexander and
Lu, Jiahua and Deyer, Timothy and Taouli, Bachir and Fayad, Zahi A. and Mei, Xueyan},
journal = {arXiv preprint arXiv:2611.23529},
year = {2026}
}
```