ORGANIC-3D encoder (enc4_organic3d)
Browse files- README.md +68 -0
- config.json +53 -0
- model.safetensors +3 -0
- organic3d.py +322 -0
README.md
ADDED
|
@@ -0,0 +1,68 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: cc-by-nc-sa-4.0
|
| 3 |
+
library_name: pytorch
|
| 4 |
+
pipeline_tag: zero-shot-classification
|
| 5 |
+
tags:
|
| 6 |
+
- 3D CT
|
| 7 |
+
- computed tomography
|
| 8 |
+
- vision-language
|
| 9 |
+
- medical imaging
|
| 10 |
+
- radiology
|
| 11 |
+
datasets:
|
| 12 |
+
- ibrahimhamamci/CT-RATE
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
+
# ORGANIC-3D encoder (`enc4_organic3d`)
|
| 16 |
+
|
| 17 |
+
A 3D CT image–text encoder that scores any sentence against a CT volume and follows the image evidence for each
|
| 18 |
+
finding: its score drops when the finding's organ is erased, stays when another organ is erased, and prefers the
|
| 19 |
+
sentence that names the correct side. Paper: *ORGANIC-3D: Grounding 3D CT Vision Language Models in Organ-Level
|
| 20 |
+
Image–Text Coupling*. Code: https://github.com/lzl199704/organic3d · Benchmark: https://huggingface.co/datasets/zllhhhh/organic3d-benchmark
|
| 21 |
+
· VLM: https://huggingface.co/zllhhhh/organic3d-vlm
|
| 22 |
+
|
| 23 |
+
**Research use only. Not for clinical use.**
|
| 24 |
+
|
| 25 |
+
## Usage
|
| 26 |
+
|
| 27 |
+
```bash
|
| 28 |
+
pip install -e <organic3d repository> # the `organic` package and its requirements
|
| 29 |
+
python organic3d.py encoder --ct scan.nii.gz --region chest --totalseg \
|
| 30 |
+
--sentence "There is a nodule in the right upper lobe."
|
| 31 |
+
```
|
| 32 |
+
|
| 33 |
+
The script downloads this repository, resamples the CT to 1 mm RAS, centres a 384 mm crop with TotalSegmentator (or an
|
| 34 |
+
estimate without it) and prints the score of every finding of the region (Δf = S(V, "There is f.") − S(V, "No f."))
|
| 35 |
+
and of each `--sentence`. From Python: `from organic3d import Encoder`.
|
| 36 |
+
|
| 37 |
+
## Model
|
| 38 |
+
|
| 39 |
+
| | |
|
| 40 |
+
|---|---|
|
| 41 |
+
| Image tower | COLIPRI 3D ViT (Primus-M), 192³ input at 2 mm, fine-tuned |
|
| 42 |
+
| Anatomy tokens | 144 learned queries (16 organ groups × 8, left and right lung and kidney separate, + 16 global), 2 cross-attention layers |
|
| 43 |
+
| Score | each sentence is routed to the tokens of the organs it names and pools them with text-conditioned weights; cosine to the CXR-BERT sentence embedding / temperature |
|
| 44 |
+
| Files | `model.safetensors` (272 M parameters, fp32), `config.json`, `organic3d.py` |
|
| 45 |
+
|
| 46 |
+
## Training
|
| 47 |
+
|
| 48 |
+
Public data only: the CT-RATE and Merlin training splits (3 : 2). Contrastive loss, classification loss and the four
|
| 49 |
+
ORGANIC losses (organ-erasure with a control organ, mirror, matched-pair, anatomy-attention), 12,000 steps on 4 × H100.
|
| 50 |
+
Config: `configs/encoder/enc4_organic3d.yaml` in the code repository.
|
| 51 |
+
|
| 52 |
+
## Results (paper Table 1, %)
|
| 53 |
+
|
| 54 |
+
| | CT-RATE | RAD-ChestCT | Merlin |
|
| 55 |
+
|---|---|---|---|
|
| 56 |
+
| MLC-AUROC | 87.7 | 79.7 | 87.7 |
|
| 57 |
+
| Matched-pair AUROC (MP) | 82.5 | 78.2 | 86.5 |
|
| 58 |
+
| Organ-erasure ΔAUROC (OE) | 28.5 | 18.4 | 21.3 |
|
| 59 |
+
| Laterality accuracy (Side) | 76.9 | – | 63.8 |
|
| 60 |
+
|
| 61 |
+
Lung lobe erasure hit rate 74.7 (CT-RATE); lesion AUROC 79.5 and lesion-erasure hit rate 74.9 (ReXGroundingCT);
|
| 62 |
+
zero-shot tumour detection 75.0 and lesion-erasure hit rate 70.7 (AbdomenAtlas 3.0).
|
| 63 |
+
|
| 64 |
+
## Limitations
|
| 65 |
+
|
| 66 |
+
Trained on chest CT from one centre (CT-RATE) and abdominal CT from one centre (Merlin) with report-derived labels;
|
| 67 |
+
performance elsewhere is unknown. Laterality is far from perfect (Side 76.9 / 63.8). Licence terms of COLIPRI, CT-RATE
|
| 68 |
+
and Merlin apply.
|
config.json
ADDED
|
@@ -0,0 +1,53 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model_type": "organic3d-encoder",
|
| 3 |
+
"architecture": "organic.model.Organic3D",
|
| 4 |
+
"model": {
|
| 5 |
+
"pretrained": false,
|
| 6 |
+
"backbone_dim": 864,
|
| 7 |
+
"dim": 768,
|
| 8 |
+
"queries": "anatomy",
|
| 9 |
+
"q_per_organ": 8,
|
| 10 |
+
"q_global": 16,
|
| 11 |
+
"query_layers": 2,
|
| 12 |
+
"heads": 12,
|
| 13 |
+
"token_pe": false,
|
| 14 |
+
"pool": "fixed",
|
| 15 |
+
"init_temperature": 0.07,
|
| 16 |
+
"grad_checkpointing": true,
|
| 17 |
+
"freeze_backbone_blocks": 0
|
| 18 |
+
},
|
| 19 |
+
"training_step": 12000,
|
| 20 |
+
"max_text_len": 96,
|
| 21 |
+
"input": {
|
| 22 |
+
"spacing_mm": 2.0,
|
| 23 |
+
"size": 192,
|
| 24 |
+
"field_of_view_mm": 384,
|
| 25 |
+
"hu_clip": [
|
| 26 |
+
-1000,
|
| 27 |
+
1000
|
| 28 |
+
],
|
| 29 |
+
"orientation": "RAS",
|
| 30 |
+
"resample_mm": 1.0
|
| 31 |
+
},
|
| 32 |
+
"crop_centre_organs": {
|
| 33 |
+
"chest": [
|
| 34 |
+
"lung_left",
|
| 35 |
+
"lung_right",
|
| 36 |
+
"heart"
|
| 37 |
+
],
|
| 38 |
+
"abdomen": [
|
| 39 |
+
"liver",
|
| 40 |
+
"spleen",
|
| 41 |
+
"kidney_left",
|
| 42 |
+
"kidney_right",
|
| 43 |
+
"pancreas"
|
| 44 |
+
]
|
| 45 |
+
},
|
| 46 |
+
"training_data": [
|
| 47 |
+
"CT-RATE (train split)",
|
| 48 |
+
"Merlin (train split)"
|
| 49 |
+
],
|
| 50 |
+
"n_tensors": 624,
|
| 51 |
+
"n_params": 272447407,
|
| 52 |
+
"run_id": "enc4_organic3d"
|
| 53 |
+
}
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ca4a8b50f4cb1f863fe556b958afd3b45449ed10dc52d45d137620df55f0c1a8
|
| 3 |
+
size 1089859876
|
organic3d.py
ADDED
|
@@ -0,0 +1,322 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""ORGANIC-3D inference: the encoder (finding scores, sentence scores) and the VLM (yes/no answers, reports) for one CT.
|
| 2 |
+
|
| 3 |
+
Needs the ORGANIC-3D code (the `organic` package, https://github.com/lzl199704/organic3d; `pip install -e .` in the repository) and its
|
| 4 |
+
requirements (torch, colipri, transformers >= 5, peft, nibabel, scipy, safetensors). Optional: TotalSegmentator for the
|
| 5 |
+
crop centre.
|
| 6 |
+
|
| 7 |
+
Preprocessing (as in the paper's evaluation):
|
| 8 |
+
1. reorient to RAS (nibabel as_closest_canonical) and resample to 1 mm (HU trilinear, align_corners=False, rounded);
|
| 9 |
+
2. crop centre: from a TotalSegmentator v2 'total' mask on the 1 mm grid (chest: lungs + heart; abdomen: liver,
|
| 10 |
+
spleen, kidneys, pancreas, head-foot from the top of the liver to the pelvic organs; recommended), or, without a
|
| 11 |
+
mask, an estimate (chest: the air inside the body; abdomen: the body), off by ~3 mm (chest) and ~11 mm (abdomen)
|
| 12 |
+
per axis on average, so scores differ from the paper's;
|
| 13 |
+
3. 384 mm cube at 1 mm (padded with -1000 HU), trilinear to 192^3 (2 mm), HU clipped to [-1000, 1000] / 1000.
|
| 14 |
+
|
| 15 |
+
Encoder: S(V, s) is the sentence-conditioned similarity; a finding f is scored by Delta_f = S(V, "There is f.") -
|
| 16 |
+
S(V, "No f."), the score used for every result in the paper. Any sentence can be scored with S(V, s).
|
| 17 |
+
VLM: P(Yes) against No at the first answer token of "Is there f in this CT scan? Answer Yes or No."; reports are
|
| 18 |
+
written answer-first (the model's own Yes answers, P(Yes) >= 0.5, are listed, then the report), and the report is
|
| 19 |
+
started with the first item of the findings order ("Trachea" chest, "Lower thorax:" abdomen), as in the paper.
|
| 20 |
+
|
| 21 |
+
python organic3d.py encoder --ct scan.nii.gz --region chest [--seg ts_1mm.nii.gz | --totalseg] \
|
| 22 |
+
[--sentence "There is a nodule in the right upper lobe."] [--weights <encoder repo or dir>]
|
| 23 |
+
python organic3d.py vlm --ct scan.nii.gz --region chest [--seg ... | --totalseg] [--question "..."] [--report] \
|
| 24 |
+
[--weights <vlm repo or dir>] [--encoder <encoder repo or dir>]
|
| 25 |
+
|
| 26 |
+
Research use only; not for clinical use.
|
| 27 |
+
"""
|
| 28 |
+
import argparse
|
| 29 |
+
import json
|
| 30 |
+
import os
|
| 31 |
+
import sys
|
| 32 |
+
|
| 33 |
+
import nibabel as nib
|
| 34 |
+
import numpy as np
|
| 35 |
+
import torch
|
| 36 |
+
import torch.nn.functional as F
|
| 37 |
+
|
| 38 |
+
QWEN_ID, QWEN_REV = 'Qwen/Qwen3.5-4B', '851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a'
|
| 39 |
+
ENCODER_REPO = 'zllhhhh/organic3d-encoder'
|
| 40 |
+
VLM_REPO = 'zllhhhh/organic3d-vlm'
|
| 41 |
+
DATASET = {'chest': 'ctrate', 'abdomen': 'merlin'} # region -> the evaluation convention it follows
|
| 42 |
+
REPORT_HEAD = {'chest': 'Trachea', 'abdomen': 'Lower thorax:'}
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
def resolve(repo):
|
| 46 |
+
if os.path.isdir(repo):
|
| 47 |
+
return repo
|
| 48 |
+
from huggingface_hub import snapshot_download
|
| 49 |
+
return snapshot_download(repo)
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
# ------------------------------------------------------------------ preprocessing
|
| 53 |
+
def load_ct(path, resample=True):
|
| 54 |
+
"""NIfTI (HU) -> (1 mm RAS float32 array, affine). resample=False: the file is already 1 mm RAS."""
|
| 55 |
+
im = nib.as_closest_canonical(nib.load(path))
|
| 56 |
+
a = np.ascontiguousarray(np.asarray(im.dataobj, dtype=np.float32))
|
| 57 |
+
if not resample:
|
| 58 |
+
return a, im.affine
|
| 59 |
+
zoom = np.asarray(im.header.get_zooms()[:3], float)
|
| 60 |
+
n_old = np.asarray(a.shape)
|
| 61 |
+
size = tuple(int(x) for x in np.maximum(1, np.round(n_old * zoom)).astype(int))
|
| 62 |
+
hu = F.interpolate(torch.from_numpy(a)[None, None], size=size, mode='trilinear', align_corners=False)[0, 0].numpy()
|
| 63 |
+
hu = np.clip(np.rint(hu), -32768, 32767).astype(np.int16).astype(np.float32)
|
| 64 |
+
sc = n_old / np.asarray(size)
|
| 65 |
+
S = np.eye(4)
|
| 66 |
+
S[:3, :3] = np.diag(sc)
|
| 67 |
+
S[:3, 3] = 0.5 * sc - 0.5
|
| 68 |
+
return hu, im.affine @ S
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
def totalsegmentator(hu, affine, device='gpu'):
|
| 72 |
+
"""TotalSegmentator v2 'total' (fast, multilabel) on the 1 mm volume -> uint8 mask on the same grid. Uses the python
|
| 73 |
+
API when TotalSegmentator is installed in this environment, else the `TotalSegmentator` command ($TOTALSEG_BIN or
|
| 74 |
+
PATH), e.g. from a separate environment."""
|
| 75 |
+
import shutil
|
| 76 |
+
import subprocess
|
| 77 |
+
import tempfile
|
| 78 |
+
with tempfile.TemporaryDirectory() as d:
|
| 79 |
+
src, dst = os.path.join(d, 'ct.nii.gz'), os.path.join(d, 'seg.nii.gz')
|
| 80 |
+
nib.save(nib.Nifti1Image(hu.astype(np.int16), affine), src)
|
| 81 |
+
try:
|
| 82 |
+
from totalsegmentator.python_api import totalsegmentator as ts
|
| 83 |
+
ts(src, dst, ml=True, fast=True, device=device, quiet=True)
|
| 84 |
+
except ImportError:
|
| 85 |
+
exe = os.environ.get('TOTALSEG_BIN') or shutil.which('TotalSegmentator')
|
| 86 |
+
if not exe:
|
| 87 |
+
raise RuntimeError('TotalSegmentator not found: install it, set TOTALSEG_BIN, or pass --seg')
|
| 88 |
+
subprocess.run([exe, '-i', src, '-o', dst, '--ml', '--fast', '--device', device, '--quiet'], check=True)
|
| 89 |
+
return np.asarray(nib.as_closest_canonical(nib.load(dst)).dataobj).astype(np.uint8)
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
# mean offset (mm) from the mask-free estimate to the TotalSegmentator centre (30 CT-RATE, 20 Merlin scans; on 50 other
|
| 93 |
+
# scans the corrected estimate is off by 2.4 / 2.8 / 3.5 mm (chest) and 10 / 11 / 11 mm (abdomen) per axis on average)
|
| 94 |
+
FALLBACK_OFFSET = {'chest': np.array([-3.6, 3.1, -4.7]), 'abdomen': np.array([32.8, 15.8, 16.4])}
|
| 95 |
+
|
| 96 |
+
|
| 97 |
+
def crop_centre(hu, region, seg=None):
|
| 98 |
+
"""Crop centre (1 mm voxel coordinates). seg: TotalSegmentator v2 multilabel mask on the 1 mm grid, or None.
|
| 99 |
+
Without a mask (approximation): chest = centroid of the air inside the body (the lungs), abdomen = centroid of the
|
| 100 |
+
body (largest region above -500 HU, holes filled per slice), each plus FALLBACK_OFFSET."""
|
| 101 |
+
from organic.data import _LUT, crop_center
|
| 102 |
+
if seg is not None:
|
| 103 |
+
return crop_center(_LUT[seg], DATASET[region])
|
| 104 |
+
from scipy import ndimage
|
| 105 |
+
sub = hu[::4, ::4, ::4]
|
| 106 |
+
lab, n = ndimage.label(sub > -500)
|
| 107 |
+
if not n:
|
| 108 |
+
return np.asarray(hu.shape) / 2
|
| 109 |
+
body = lab == (np.argmax(np.bincount(lab.ravel())[1:]) + 1)
|
| 110 |
+
body = np.stack([ndimage.binary_fill_holes(body[:, :, z]) for z in range(body.shape[2])], 2)
|
| 111 |
+
m = body
|
| 112 |
+
if region == 'chest':
|
| 113 |
+
lab, n = ndimage.label((sub < -400) & body)
|
| 114 |
+
if n:
|
| 115 |
+
sz = np.bincount(lab.ravel())
|
| 116 |
+
sz[0] = 0
|
| 117 |
+
m = np.isin(lab, [k for k in np.argsort(sz)[::-1][:2] if sz[k] > 0.1 * sz.max()])
|
| 118 |
+
return np.argwhere(m).mean(0) * 4 + FALLBACK_OFFSET[region]
|
| 119 |
+
|
| 120 |
+
|
| 121 |
+
def model_input(hu, centre, device):
|
| 122 |
+
"""1 mm HU volume + crop centre -> (1, 1, 192, 192, 192) input (as the evaluation harness)."""
|
| 123 |
+
from organic.augment import normalise
|
| 124 |
+
from organic.data import SIZE, SPACING, _crop_bounds
|
| 125 |
+
bounds = _crop_bounds(centre, hu.shape, int(SIZE * SPACING))
|
| 126 |
+
t = torch.as_tensor(hu, device=device)
|
| 127 |
+
out = torch.full([b - a for a, b in bounds], -1000.0, device=device)
|
| 128 |
+
src = tuple(slice(max(a, 0), min(b, n)) for (a, b), n in zip(bounds, t.shape))
|
| 129 |
+
dst = tuple(slice(s.start - a, s.stop - a) for s, (a, _) in zip(src, bounds))
|
| 130 |
+
out[dst] = t[src]
|
| 131 |
+
v = F.interpolate(out[None, None], size=(SIZE, SIZE, SIZE), mode='trilinear', align_corners=False)
|
| 132 |
+
return normalise(v)
|
| 133 |
+
|
| 134 |
+
|
| 135 |
+
# ------------------------------------------------------------------ encoder
|
| 136 |
+
class Encoder:
|
| 137 |
+
def __init__(self, weights=ENCODER_REPO, device='cuda', region=None):
|
| 138 |
+
"""region ('chest' / 'abdomen'): embed that region's finding prompts at load, before any image, as the evaluation
|
| 139 |
+
harness does (the bf16 image tower's last digits depend on the GPU kernels chosen earlier in the process; in this
|
| 140 |
+
order the scores equal the harness bit for bit)."""
|
| 141 |
+
from colipri import get_model, get_processor
|
| 142 |
+
from omegaconf import OmegaConf
|
| 143 |
+
from safetensors.torch import load_file
|
| 144 |
+
from organic.model import Organic3D
|
| 145 |
+
d = resolve(weights)
|
| 146 |
+
self.cfg = json.load(open(os.path.join(d, 'config.json')))
|
| 147 |
+
self.dev = torch.device(device)
|
| 148 |
+
self.m = Organic3D(OmegaConf.create({'model': self.cfg['model']}), get_model(pretrained=False))
|
| 149 |
+
self.m.load_state_dict(load_file(os.path.join(d, 'model.safetensors')), strict=True)
|
| 150 |
+
self.m.to(self.dev).eval()
|
| 151 |
+
self.tok = get_processor()._text_tokenizer
|
| 152 |
+
self.max_len = int(self.cfg.get('max_text_len', 96))
|
| 153 |
+
self.prompt_emb = {}
|
| 154 |
+
if region:
|
| 155 |
+
self.prompts(region)
|
| 156 |
+
|
| 157 |
+
@torch.no_grad()
|
| 158 |
+
def encode_image(self, x):
|
| 159 |
+
"""(1, 1, 192, 192, 192) input -> slot tokens T (1, K, 768)."""
|
| 160 |
+
with torch.autocast('cuda', dtype=torch.bfloat16):
|
| 161 |
+
T, _, _ = self.m.encode_image(x)
|
| 162 |
+
return T.float()
|
| 163 |
+
|
| 164 |
+
@torch.no_grad()
|
| 165 |
+
def encode_text(self, sents):
|
| 166 |
+
enc = self.tok(list(sents), max_length=self.max_len, padding='longest', truncation=True, return_tensors='pt')
|
| 167 |
+
with torch.autocast('cuda', dtype=torch.bfloat16):
|
| 168 |
+
t = self.m.encode_text(enc['input_ids'].to(self.dev), enc['attention_mask'].to(self.dev))
|
| 169 |
+
t = self.m.attach_route(t, list(sents)) # organ route of each sentence (fixed organ routing)
|
| 170 |
+
return t.float()
|
| 171 |
+
|
| 172 |
+
def prompts(self, region):
|
| 173 |
+
"""(findings, positive and negative prompt embeddings) of the region: "There is {f}." / "No {f}."."""
|
| 174 |
+
if region not in self.prompt_emb:
|
| 175 |
+
from organic.vocab import FAMILY, FINDING_NAMES, canonical_prompts
|
| 176 |
+
fs = [f for f in FINDING_NAMES if f.startswith(FAMILY[DATASET[region]])]
|
| 177 |
+
pos, neg = zip(*[canonical_prompts(f) for f in fs])
|
| 178 |
+
self.prompt_emb[region] = (fs, self.encode_text(pos), self.encode_text(neg))
|
| 179 |
+
return self.prompt_emb[region]
|
| 180 |
+
|
| 181 |
+
@torch.no_grad()
|
| 182 |
+
def score(self, T, t):
|
| 183 |
+
with torch.autocast('cuda', dtype=torch.bfloat16):
|
| 184 |
+
return self.m.score(T, t)[0].float().cpu().numpy()
|
| 185 |
+
|
| 186 |
+
def similarity(self, T, sents):
|
| 187 |
+
"""S(V, s) for each sentence (numpy)."""
|
| 188 |
+
return self.score(T, self.encode_text(sents))
|
| 189 |
+
|
| 190 |
+
def finding_scores(self, T, region):
|
| 191 |
+
"""{finding: Delta_f} for the region's findings (chest: the 18 CT-RATE findings; abdomen: 21 Merlin findings)."""
|
| 192 |
+
fs, tp, tn = self.prompts(region)
|
| 193 |
+
return dict(zip(fs, (self.score(T, tp) - self.score(T, tn)).tolist()))
|
| 194 |
+
|
| 195 |
+
|
| 196 |
+
# ------------------------------------------------------------------ VLM
|
| 197 |
+
class VLM:
|
| 198 |
+
def __init__(self, weights=VLM_REPO, encoder=None, device='cuda'):
|
| 199 |
+
from huggingface_hub import snapshot_download
|
| 200 |
+
from omegaconf import OmegaConf
|
| 201 |
+
from safetensors.torch import load_file
|
| 202 |
+
from organic.vlm import Organic3DVLM
|
| 203 |
+
d = resolve(weights)
|
| 204 |
+
self.cfg = json.load(open(os.path.join(d, 'config.json')))
|
| 205 |
+
self.enc = encoder if isinstance(encoder, Encoder) else Encoder(encoder or ENCODER_REPO, device)
|
| 206 |
+
self.dev = self.enc.dev
|
| 207 |
+
v = dict(self.cfg['vlm'])
|
| 208 |
+
v['llm'] = snapshot_download(self.cfg['base_model'], revision=self.cfg['base_model_revision'])
|
| 209 |
+
cfg = OmegaConf.create({'model': self.enc.cfg['model'], 'vlm': v})
|
| 210 |
+
self.m = Organic3DVLM(cfg, self.enc.m)
|
| 211 |
+
lora = {'llm.' + k.replace('.lora_A.', '.lora_A.default.').replace('.lora_B.', '.lora_B.default.'): x
|
| 212 |
+
for k, x in load_file(os.path.join(d, 'adapter_model.safetensors')).items()}
|
| 213 |
+
proj = {'proj.' + k: x for k, x in load_file(os.path.join(d, 'projector.safetensors')).items()}
|
| 214 |
+
missing, unexpected = self.m.load_state_dict({**lora, **proj}, strict=False)
|
| 215 |
+
assert not unexpected, unexpected[:5]
|
| 216 |
+
expect = {k for k in self.m.state_dict() if '.lora_' in k or k.startswith('proj.')}
|
| 217 |
+
assert not expect & set(missing), sorted(expect & set(missing))[:5]
|
| 218 |
+
self.m.to(self.dev).eval()
|
| 219 |
+
|
| 220 |
+
@torch.no_grad()
|
| 221 |
+
def visual_tokens(self, x):
|
| 222 |
+
with torch.autocast('cuda', dtype=torch.bfloat16):
|
| 223 |
+
V, _ = self.m.visual_tokens(x)
|
| 224 |
+
return V
|
| 225 |
+
|
| 226 |
+
@staticmethod
|
| 227 |
+
def question(f_or_phrase):
|
| 228 |
+
from organic.train_stageB import question
|
| 229 |
+
from organic.vocab import FINDINGS
|
| 230 |
+
if f_or_phrase in FINDINGS:
|
| 231 |
+
return question(f_or_phrase)
|
| 232 |
+
return f'Is there {f_or_phrase} in this CT scan? Answer Yes or No.'
|
| 233 |
+
|
| 234 |
+
@torch.no_grad()
|
| 235 |
+
def p_yes(self, V, questions):
|
| 236 |
+
"""P(Yes) against No at the first answer position, one value per question."""
|
| 237 |
+
n = len(questions)
|
| 238 |
+
with torch.autocast('cuda', dtype=torch.bfloat16):
|
| 239 |
+
emb, mask, _, _ = self.m.build(V.expand(n, -1, -1), list(questions), [''] * n, self.dev)
|
| 240 |
+
h, _ = self.m.hidden(emb, mask)
|
| 241 |
+
last = mask.sum(1) - 1
|
| 242 |
+
w = self.m.head().weight[[self.m.yes_id, self.m.no_id]].float()
|
| 243 |
+
two = h[torch.arange(n, device=self.dev), last].float() @ w.T
|
| 244 |
+
return torch.sigmoid(two[:, 0] - two[:, 1]).cpu().numpy()
|
| 245 |
+
|
| 246 |
+
def finding_answers(self, V, region):
|
| 247 |
+
from organic.vocab import FAMILY, FINDING_NAMES
|
| 248 |
+
fs = [f for f in FINDING_NAMES if f.startswith(FAMILY[DATASET[region]])]
|
| 249 |
+
return dict(zip(fs, self.p_yes(V, [self.question(f) for f in fs]).tolist()))
|
| 250 |
+
|
| 251 |
+
@torch.no_grad()
|
| 252 |
+
def report(self, V, region, max_new_tokens=768, head=True):
|
| 253 |
+
"""Answer-first report: (finding list, report). The report starts with REPORT_HEAD[region] (head=True)."""
|
| 254 |
+
from organic.train_stageB import finding_list, report_prompt
|
| 255 |
+
ans = self.finding_answers(V, region)
|
| 256 |
+
pre = finding_list([f for f, p in ans.items() if p >= 0.5])
|
| 257 |
+
h = REPORT_HEAD[region] if head else ''
|
| 258 |
+
if h:
|
| 259 |
+
pre = f'{pre} {h}'
|
| 260 |
+
with torch.autocast('cuda', dtype=torch.bfloat16):
|
| 261 |
+
out = self.m.generate(V, report_prompt(DATASET[region], self.cfg['vlm']['report_mode']), self.dev,
|
| 262 |
+
max_new_tokens=max_new_tokens, prefix=pre)
|
| 263 |
+
body = out[0].strip()
|
| 264 |
+
text = h + ('' if body[:1] in ',.;:' else ' ') + body if h else body
|
| 265 |
+
return pre.split('\n')[0].strip(), text
|
| 266 |
+
|
| 267 |
+
|
| 268 |
+
# ------------------------------------------------------------------ CLI
|
| 269 |
+
def prepare(a, device):
|
| 270 |
+
hu, aff = load_ct(a.ct, resample=not a.is_1mm)
|
| 271 |
+
seg = None
|
| 272 |
+
if a.seg:
|
| 273 |
+
seg = np.asarray(nib.as_closest_canonical(nib.load(a.seg)).dataobj).astype(np.uint8)
|
| 274 |
+
assert seg.shape == hu.shape, f'mask {seg.shape} is not on the 1 mm grid {hu.shape}'
|
| 275 |
+
elif a.totalseg:
|
| 276 |
+
seg = totalsegmentator(hu, aff)
|
| 277 |
+
return model_input(hu, crop_centre(hu, a.region, seg), device), seg is not None
|
| 278 |
+
|
| 279 |
+
|
| 280 |
+
def main():
|
| 281 |
+
ap = argparse.ArgumentParser(description=__doc__.split('\n')[0])
|
| 282 |
+
ap.add_argument('mode', choices=['encoder', 'vlm'])
|
| 283 |
+
ap.add_argument('--ct', required=True, help='CT NIfTI in HU')
|
| 284 |
+
ap.add_argument('--is_1mm', action='store_true', help='the CT is already 1 mm RAS (skip resampling)')
|
| 285 |
+
ap.add_argument('--region', choices=['chest', 'abdomen'], required=True)
|
| 286 |
+
ap.add_argument('--seg', default=None, help='TotalSegmentator v2 total multilabel mask on the 1 mm grid')
|
| 287 |
+
ap.add_argument('--totalseg', action='store_true', help='run TotalSegmentator for the crop centre')
|
| 288 |
+
ap.add_argument('--weights', default=None)
|
| 289 |
+
ap.add_argument('--encoder', default=ENCODER_REPO, help='vlm mode: the encoder repo or directory')
|
| 290 |
+
ap.add_argument('--sentence', action='append', default=[], help='encoder: extra sentence to score')
|
| 291 |
+
ap.add_argument('--question', action='append', default=[], help='vlm: extra finding phrase or yes/no question')
|
| 292 |
+
ap.add_argument('--report', action='store_true', help='vlm: write a report')
|
| 293 |
+
ap.add_argument('--max_new_tokens', type=int, default=768)
|
| 294 |
+
ap.add_argument('--out', default=None, help='write the results as JSON')
|
| 295 |
+
a = ap.parse_args()
|
| 296 |
+
dev = 'cuda'
|
| 297 |
+
x, masked = prepare(a, dev)
|
| 298 |
+
res = {'ct': os.path.basename(a.ct), 'region': a.region,
|
| 299 |
+
'crop_centre': 'TotalSegmentator' if masked else 'estimate without a mask'}
|
| 300 |
+
if a.mode == 'encoder':
|
| 301 |
+
enc = Encoder(a.weights or ENCODER_REPO, dev, region=a.region)
|
| 302 |
+
T = enc.encode_image(x)
|
| 303 |
+
res['finding_scores'] = enc.finding_scores(T, a.region)
|
| 304 |
+
if a.sentence:
|
| 305 |
+
res['sentence_scores'] = dict(zip(a.sentence, enc.similarity(T, a.sentence).tolist()))
|
| 306 |
+
else:
|
| 307 |
+
vlm = VLM(a.weights or VLM_REPO, a.encoder, dev)
|
| 308 |
+
V = vlm.visual_tokens(x)
|
| 309 |
+
res['p_yes'] = vlm.finding_answers(V, a.region)
|
| 310 |
+
if a.question:
|
| 311 |
+
qs = [q if q.endswith('?') or 'Answer Yes or No' in q else vlm.question(q) for q in a.question]
|
| 312 |
+
res['questions'] = dict(zip(qs, vlm.p_yes(V, qs).tolist()))
|
| 313 |
+
if a.report:
|
| 314 |
+
res['finding_list'], res['report'] = vlm.report(V, a.region, a.max_new_tokens)
|
| 315 |
+
s = json.dumps(res, indent=1)
|
| 316 |
+
if a.out:
|
| 317 |
+
open(a.out, 'w').write(s)
|
| 318 |
+
print(s)
|
| 319 |
+
|
| 320 |
+
|
| 321 |
+
if __name__ == '__main__':
|
| 322 |
+
sys.exit(main())
|