zllhhhh commited on
Commit
4c2ce45
·
verified ·
1 Parent(s): bdd0ab2

ORGANIC-3D encoder (enc4_organic3d)

Browse files
Files changed (4) hide show
  1. README.md +68 -0
  2. config.json +53 -0
  3. model.safetensors +3 -0
  4. organic3d.py +322 -0
README.md ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: cc-by-nc-sa-4.0
3
+ library_name: pytorch
4
+ pipeline_tag: zero-shot-classification
5
+ tags:
6
+ - 3D CT
7
+ - computed tomography
8
+ - vision-language
9
+ - medical imaging
10
+ - radiology
11
+ datasets:
12
+ - ibrahimhamamci/CT-RATE
13
+ ---
14
+
15
+ # ORGANIC-3D encoder (`enc4_organic3d`)
16
+
17
+ A 3D CT image–text encoder that scores any sentence against a CT volume and follows the image evidence for each
18
+ finding: its score drops when the finding's organ is erased, stays when another organ is erased, and prefers the
19
+ sentence that names the correct side. Paper: *ORGANIC-3D: Grounding 3D CT Vision Language Models in Organ-Level
20
+ Image–Text Coupling*. Code: https://github.com/lzl199704/organic3d · Benchmark: https://huggingface.co/datasets/zllhhhh/organic3d-benchmark
21
+ · VLM: https://huggingface.co/zllhhhh/organic3d-vlm
22
+
23
+ **Research use only. Not for clinical use.**
24
+
25
+ ## Usage
26
+
27
+ ```bash
28
+ pip install -e <organic3d repository> # the `organic` package and its requirements
29
+ python organic3d.py encoder --ct scan.nii.gz --region chest --totalseg \
30
+ --sentence "There is a nodule in the right upper lobe."
31
+ ```
32
+
33
+ The script downloads this repository, resamples the CT to 1 mm RAS, centres a 384 mm crop with TotalSegmentator (or an
34
+ estimate without it) and prints the score of every finding of the region (Δf = S(V, "There is f.") − S(V, "No f."))
35
+ and of each `--sentence`. From Python: `from organic3d import Encoder`.
36
+
37
+ ## Model
38
+
39
+ | | |
40
+ |---|---|
41
+ | Image tower | COLIPRI 3D ViT (Primus-M), 192³ input at 2 mm, fine-tuned |
42
+ | Anatomy tokens | 144 learned queries (16 organ groups × 8, left and right lung and kidney separate, + 16 global), 2 cross-attention layers |
43
+ | Score | each sentence is routed to the tokens of the organs it names and pools them with text-conditioned weights; cosine to the CXR-BERT sentence embedding / temperature |
44
+ | Files | `model.safetensors` (272 M parameters, fp32), `config.json`, `organic3d.py` |
45
+
46
+ ## Training
47
+
48
+ Public data only: the CT-RATE and Merlin training splits (3 : 2). Contrastive loss, classification loss and the four
49
+ ORGANIC losses (organ-erasure with a control organ, mirror, matched-pair, anatomy-attention), 12,000 steps on 4 × H100.
50
+ Config: `configs/encoder/enc4_organic3d.yaml` in the code repository.
51
+
52
+ ## Results (paper Table 1, %)
53
+
54
+ | | CT-RATE | RAD-ChestCT | Merlin |
55
+ |---|---|---|---|
56
+ | MLC-AUROC | 87.7 | 79.7 | 87.7 |
57
+ | Matched-pair AUROC (MP) | 82.5 | 78.2 | 86.5 |
58
+ | Organ-erasure ΔAUROC (OE) | 28.5 | 18.4 | 21.3 |
59
+ | Laterality accuracy (Side) | 76.9 | – | 63.8 |
60
+
61
+ Lung lobe erasure hit rate 74.7 (CT-RATE); lesion AUROC 79.5 and lesion-erasure hit rate 74.9 (ReXGroundingCT);
62
+ zero-shot tumour detection 75.0 and lesion-erasure hit rate 70.7 (AbdomenAtlas 3.0).
63
+
64
+ ## Limitations
65
+
66
+ Trained on chest CT from one centre (CT-RATE) and abdominal CT from one centre (Merlin) with report-derived labels;
67
+ performance elsewhere is unknown. Laterality is far from perfect (Side 76.9 / 63.8). Licence terms of COLIPRI, CT-RATE
68
+ and Merlin apply.
config.json ADDED
@@ -0,0 +1,53 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_type": "organic3d-encoder",
3
+ "architecture": "organic.model.Organic3D",
4
+ "model": {
5
+ "pretrained": false,
6
+ "backbone_dim": 864,
7
+ "dim": 768,
8
+ "queries": "anatomy",
9
+ "q_per_organ": 8,
10
+ "q_global": 16,
11
+ "query_layers": 2,
12
+ "heads": 12,
13
+ "token_pe": false,
14
+ "pool": "fixed",
15
+ "init_temperature": 0.07,
16
+ "grad_checkpointing": true,
17
+ "freeze_backbone_blocks": 0
18
+ },
19
+ "training_step": 12000,
20
+ "max_text_len": 96,
21
+ "input": {
22
+ "spacing_mm": 2.0,
23
+ "size": 192,
24
+ "field_of_view_mm": 384,
25
+ "hu_clip": [
26
+ -1000,
27
+ 1000
28
+ ],
29
+ "orientation": "RAS",
30
+ "resample_mm": 1.0
31
+ },
32
+ "crop_centre_organs": {
33
+ "chest": [
34
+ "lung_left",
35
+ "lung_right",
36
+ "heart"
37
+ ],
38
+ "abdomen": [
39
+ "liver",
40
+ "spleen",
41
+ "kidney_left",
42
+ "kidney_right",
43
+ "pancreas"
44
+ ]
45
+ },
46
+ "training_data": [
47
+ "CT-RATE (train split)",
48
+ "Merlin (train split)"
49
+ ],
50
+ "n_tensors": 624,
51
+ "n_params": 272447407,
52
+ "run_id": "enc4_organic3d"
53
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ca4a8b50f4cb1f863fe556b958afd3b45449ed10dc52d45d137620df55f0c1a8
3
+ size 1089859876
organic3d.py ADDED
@@ -0,0 +1,322 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """ORGANIC-3D inference: the encoder (finding scores, sentence scores) and the VLM (yes/no answers, reports) for one CT.
2
+
3
+ Needs the ORGANIC-3D code (the `organic` package, https://github.com/lzl199704/organic3d; `pip install -e .` in the repository) and its
4
+ requirements (torch, colipri, transformers >= 5, peft, nibabel, scipy, safetensors). Optional: TotalSegmentator for the
5
+ crop centre.
6
+
7
+ Preprocessing (as in the paper's evaluation):
8
+ 1. reorient to RAS (nibabel as_closest_canonical) and resample to 1 mm (HU trilinear, align_corners=False, rounded);
9
+ 2. crop centre: from a TotalSegmentator v2 'total' mask on the 1 mm grid (chest: lungs + heart; abdomen: liver,
10
+ spleen, kidneys, pancreas, head-foot from the top of the liver to the pelvic organs; recommended), or, without a
11
+ mask, an estimate (chest: the air inside the body; abdomen: the body), off by ~3 mm (chest) and ~11 mm (abdomen)
12
+ per axis on average, so scores differ from the paper's;
13
+ 3. 384 mm cube at 1 mm (padded with -1000 HU), trilinear to 192^3 (2 mm), HU clipped to [-1000, 1000] / 1000.
14
+
15
+ Encoder: S(V, s) is the sentence-conditioned similarity; a finding f is scored by Delta_f = S(V, "There is f.") -
16
+ S(V, "No f."), the score used for every result in the paper. Any sentence can be scored with S(V, s).
17
+ VLM: P(Yes) against No at the first answer token of "Is there f in this CT scan? Answer Yes or No."; reports are
18
+ written answer-first (the model's own Yes answers, P(Yes) >= 0.5, are listed, then the report), and the report is
19
+ started with the first item of the findings order ("Trachea" chest, "Lower thorax:" abdomen), as in the paper.
20
+
21
+ python organic3d.py encoder --ct scan.nii.gz --region chest [--seg ts_1mm.nii.gz | --totalseg] \
22
+ [--sentence "There is a nodule in the right upper lobe."] [--weights <encoder repo or dir>]
23
+ python organic3d.py vlm --ct scan.nii.gz --region chest [--seg ... | --totalseg] [--question "..."] [--report] \
24
+ [--weights <vlm repo or dir>] [--encoder <encoder repo or dir>]
25
+
26
+ Research use only; not for clinical use.
27
+ """
28
+ import argparse
29
+ import json
30
+ import os
31
+ import sys
32
+
33
+ import nibabel as nib
34
+ import numpy as np
35
+ import torch
36
+ import torch.nn.functional as F
37
+
38
+ QWEN_ID, QWEN_REV = 'Qwen/Qwen3.5-4B', '851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a'
39
+ ENCODER_REPO = 'zllhhhh/organic3d-encoder'
40
+ VLM_REPO = 'zllhhhh/organic3d-vlm'
41
+ DATASET = {'chest': 'ctrate', 'abdomen': 'merlin'} # region -> the evaluation convention it follows
42
+ REPORT_HEAD = {'chest': 'Trachea', 'abdomen': 'Lower thorax:'}
43
+
44
+
45
+ def resolve(repo):
46
+ if os.path.isdir(repo):
47
+ return repo
48
+ from huggingface_hub import snapshot_download
49
+ return snapshot_download(repo)
50
+
51
+
52
+ # ------------------------------------------------------------------ preprocessing
53
+ def load_ct(path, resample=True):
54
+ """NIfTI (HU) -> (1 mm RAS float32 array, affine). resample=False: the file is already 1 mm RAS."""
55
+ im = nib.as_closest_canonical(nib.load(path))
56
+ a = np.ascontiguousarray(np.asarray(im.dataobj, dtype=np.float32))
57
+ if not resample:
58
+ return a, im.affine
59
+ zoom = np.asarray(im.header.get_zooms()[:3], float)
60
+ n_old = np.asarray(a.shape)
61
+ size = tuple(int(x) for x in np.maximum(1, np.round(n_old * zoom)).astype(int))
62
+ hu = F.interpolate(torch.from_numpy(a)[None, None], size=size, mode='trilinear', align_corners=False)[0, 0].numpy()
63
+ hu = np.clip(np.rint(hu), -32768, 32767).astype(np.int16).astype(np.float32)
64
+ sc = n_old / np.asarray(size)
65
+ S = np.eye(4)
66
+ S[:3, :3] = np.diag(sc)
67
+ S[:3, 3] = 0.5 * sc - 0.5
68
+ return hu, im.affine @ S
69
+
70
+
71
+ def totalsegmentator(hu, affine, device='gpu'):
72
+ """TotalSegmentator v2 'total' (fast, multilabel) on the 1 mm volume -> uint8 mask on the same grid. Uses the python
73
+ API when TotalSegmentator is installed in this environment, else the `TotalSegmentator` command ($TOTALSEG_BIN or
74
+ PATH), e.g. from a separate environment."""
75
+ import shutil
76
+ import subprocess
77
+ import tempfile
78
+ with tempfile.TemporaryDirectory() as d:
79
+ src, dst = os.path.join(d, 'ct.nii.gz'), os.path.join(d, 'seg.nii.gz')
80
+ nib.save(nib.Nifti1Image(hu.astype(np.int16), affine), src)
81
+ try:
82
+ from totalsegmentator.python_api import totalsegmentator as ts
83
+ ts(src, dst, ml=True, fast=True, device=device, quiet=True)
84
+ except ImportError:
85
+ exe = os.environ.get('TOTALSEG_BIN') or shutil.which('TotalSegmentator')
86
+ if not exe:
87
+ raise RuntimeError('TotalSegmentator not found: install it, set TOTALSEG_BIN, or pass --seg')
88
+ subprocess.run([exe, '-i', src, '-o', dst, '--ml', '--fast', '--device', device, '--quiet'], check=True)
89
+ return np.asarray(nib.as_closest_canonical(nib.load(dst)).dataobj).astype(np.uint8)
90
+
91
+
92
+ # mean offset (mm) from the mask-free estimate to the TotalSegmentator centre (30 CT-RATE, 20 Merlin scans; on 50 other
93
+ # scans the corrected estimate is off by 2.4 / 2.8 / 3.5 mm (chest) and 10 / 11 / 11 mm (abdomen) per axis on average)
94
+ FALLBACK_OFFSET = {'chest': np.array([-3.6, 3.1, -4.7]), 'abdomen': np.array([32.8, 15.8, 16.4])}
95
+
96
+
97
+ def crop_centre(hu, region, seg=None):
98
+ """Crop centre (1 mm voxel coordinates). seg: TotalSegmentator v2 multilabel mask on the 1 mm grid, or None.
99
+ Without a mask (approximation): chest = centroid of the air inside the body (the lungs), abdomen = centroid of the
100
+ body (largest region above -500 HU, holes filled per slice), each plus FALLBACK_OFFSET."""
101
+ from organic.data import _LUT, crop_center
102
+ if seg is not None:
103
+ return crop_center(_LUT[seg], DATASET[region])
104
+ from scipy import ndimage
105
+ sub = hu[::4, ::4, ::4]
106
+ lab, n = ndimage.label(sub > -500)
107
+ if not n:
108
+ return np.asarray(hu.shape) / 2
109
+ body = lab == (np.argmax(np.bincount(lab.ravel())[1:]) + 1)
110
+ body = np.stack([ndimage.binary_fill_holes(body[:, :, z]) for z in range(body.shape[2])], 2)
111
+ m = body
112
+ if region == 'chest':
113
+ lab, n = ndimage.label((sub < -400) & body)
114
+ if n:
115
+ sz = np.bincount(lab.ravel())
116
+ sz[0] = 0
117
+ m = np.isin(lab, [k for k in np.argsort(sz)[::-1][:2] if sz[k] > 0.1 * sz.max()])
118
+ return np.argwhere(m).mean(0) * 4 + FALLBACK_OFFSET[region]
119
+
120
+
121
+ def model_input(hu, centre, device):
122
+ """1 mm HU volume + crop centre -> (1, 1, 192, 192, 192) input (as the evaluation harness)."""
123
+ from organic.augment import normalise
124
+ from organic.data import SIZE, SPACING, _crop_bounds
125
+ bounds = _crop_bounds(centre, hu.shape, int(SIZE * SPACING))
126
+ t = torch.as_tensor(hu, device=device)
127
+ out = torch.full([b - a for a, b in bounds], -1000.0, device=device)
128
+ src = tuple(slice(max(a, 0), min(b, n)) for (a, b), n in zip(bounds, t.shape))
129
+ dst = tuple(slice(s.start - a, s.stop - a) for s, (a, _) in zip(src, bounds))
130
+ out[dst] = t[src]
131
+ v = F.interpolate(out[None, None], size=(SIZE, SIZE, SIZE), mode='trilinear', align_corners=False)
132
+ return normalise(v)
133
+
134
+
135
+ # ------------------------------------------------------------------ encoder
136
+ class Encoder:
137
+ def __init__(self, weights=ENCODER_REPO, device='cuda', region=None):
138
+ """region ('chest' / 'abdomen'): embed that region's finding prompts at load, before any image, as the evaluation
139
+ harness does (the bf16 image tower's last digits depend on the GPU kernels chosen earlier in the process; in this
140
+ order the scores equal the harness bit for bit)."""
141
+ from colipri import get_model, get_processor
142
+ from omegaconf import OmegaConf
143
+ from safetensors.torch import load_file
144
+ from organic.model import Organic3D
145
+ d = resolve(weights)
146
+ self.cfg = json.load(open(os.path.join(d, 'config.json')))
147
+ self.dev = torch.device(device)
148
+ self.m = Organic3D(OmegaConf.create({'model': self.cfg['model']}), get_model(pretrained=False))
149
+ self.m.load_state_dict(load_file(os.path.join(d, 'model.safetensors')), strict=True)
150
+ self.m.to(self.dev).eval()
151
+ self.tok = get_processor()._text_tokenizer
152
+ self.max_len = int(self.cfg.get('max_text_len', 96))
153
+ self.prompt_emb = {}
154
+ if region:
155
+ self.prompts(region)
156
+
157
+ @torch.no_grad()
158
+ def encode_image(self, x):
159
+ """(1, 1, 192, 192, 192) input -> slot tokens T (1, K, 768)."""
160
+ with torch.autocast('cuda', dtype=torch.bfloat16):
161
+ T, _, _ = self.m.encode_image(x)
162
+ return T.float()
163
+
164
+ @torch.no_grad()
165
+ def encode_text(self, sents):
166
+ enc = self.tok(list(sents), max_length=self.max_len, padding='longest', truncation=True, return_tensors='pt')
167
+ with torch.autocast('cuda', dtype=torch.bfloat16):
168
+ t = self.m.encode_text(enc['input_ids'].to(self.dev), enc['attention_mask'].to(self.dev))
169
+ t = self.m.attach_route(t, list(sents)) # organ route of each sentence (fixed organ routing)
170
+ return t.float()
171
+
172
+ def prompts(self, region):
173
+ """(findings, positive and negative prompt embeddings) of the region: "There is {f}." / "No {f}."."""
174
+ if region not in self.prompt_emb:
175
+ from organic.vocab import FAMILY, FINDING_NAMES, canonical_prompts
176
+ fs = [f for f in FINDING_NAMES if f.startswith(FAMILY[DATASET[region]])]
177
+ pos, neg = zip(*[canonical_prompts(f) for f in fs])
178
+ self.prompt_emb[region] = (fs, self.encode_text(pos), self.encode_text(neg))
179
+ return self.prompt_emb[region]
180
+
181
+ @torch.no_grad()
182
+ def score(self, T, t):
183
+ with torch.autocast('cuda', dtype=torch.bfloat16):
184
+ return self.m.score(T, t)[0].float().cpu().numpy()
185
+
186
+ def similarity(self, T, sents):
187
+ """S(V, s) for each sentence (numpy)."""
188
+ return self.score(T, self.encode_text(sents))
189
+
190
+ def finding_scores(self, T, region):
191
+ """{finding: Delta_f} for the region's findings (chest: the 18 CT-RATE findings; abdomen: 21 Merlin findings)."""
192
+ fs, tp, tn = self.prompts(region)
193
+ return dict(zip(fs, (self.score(T, tp) - self.score(T, tn)).tolist()))
194
+
195
+
196
+ # ------------------------------------------------------------------ VLM
197
+ class VLM:
198
+ def __init__(self, weights=VLM_REPO, encoder=None, device='cuda'):
199
+ from huggingface_hub import snapshot_download
200
+ from omegaconf import OmegaConf
201
+ from safetensors.torch import load_file
202
+ from organic.vlm import Organic3DVLM
203
+ d = resolve(weights)
204
+ self.cfg = json.load(open(os.path.join(d, 'config.json')))
205
+ self.enc = encoder if isinstance(encoder, Encoder) else Encoder(encoder or ENCODER_REPO, device)
206
+ self.dev = self.enc.dev
207
+ v = dict(self.cfg['vlm'])
208
+ v['llm'] = snapshot_download(self.cfg['base_model'], revision=self.cfg['base_model_revision'])
209
+ cfg = OmegaConf.create({'model': self.enc.cfg['model'], 'vlm': v})
210
+ self.m = Organic3DVLM(cfg, self.enc.m)
211
+ lora = {'llm.' + k.replace('.lora_A.', '.lora_A.default.').replace('.lora_B.', '.lora_B.default.'): x
212
+ for k, x in load_file(os.path.join(d, 'adapter_model.safetensors')).items()}
213
+ proj = {'proj.' + k: x for k, x in load_file(os.path.join(d, 'projector.safetensors')).items()}
214
+ missing, unexpected = self.m.load_state_dict({**lora, **proj}, strict=False)
215
+ assert not unexpected, unexpected[:5]
216
+ expect = {k for k in self.m.state_dict() if '.lora_' in k or k.startswith('proj.')}
217
+ assert not expect & set(missing), sorted(expect & set(missing))[:5]
218
+ self.m.to(self.dev).eval()
219
+
220
+ @torch.no_grad()
221
+ def visual_tokens(self, x):
222
+ with torch.autocast('cuda', dtype=torch.bfloat16):
223
+ V, _ = self.m.visual_tokens(x)
224
+ return V
225
+
226
+ @staticmethod
227
+ def question(f_or_phrase):
228
+ from organic.train_stageB import question
229
+ from organic.vocab import FINDINGS
230
+ if f_or_phrase in FINDINGS:
231
+ return question(f_or_phrase)
232
+ return f'Is there {f_or_phrase} in this CT scan? Answer Yes or No.'
233
+
234
+ @torch.no_grad()
235
+ def p_yes(self, V, questions):
236
+ """P(Yes) against No at the first answer position, one value per question."""
237
+ n = len(questions)
238
+ with torch.autocast('cuda', dtype=torch.bfloat16):
239
+ emb, mask, _, _ = self.m.build(V.expand(n, -1, -1), list(questions), [''] * n, self.dev)
240
+ h, _ = self.m.hidden(emb, mask)
241
+ last = mask.sum(1) - 1
242
+ w = self.m.head().weight[[self.m.yes_id, self.m.no_id]].float()
243
+ two = h[torch.arange(n, device=self.dev), last].float() @ w.T
244
+ return torch.sigmoid(two[:, 0] - two[:, 1]).cpu().numpy()
245
+
246
+ def finding_answers(self, V, region):
247
+ from organic.vocab import FAMILY, FINDING_NAMES
248
+ fs = [f for f in FINDING_NAMES if f.startswith(FAMILY[DATASET[region]])]
249
+ return dict(zip(fs, self.p_yes(V, [self.question(f) for f in fs]).tolist()))
250
+
251
+ @torch.no_grad()
252
+ def report(self, V, region, max_new_tokens=768, head=True):
253
+ """Answer-first report: (finding list, report). The report starts with REPORT_HEAD[region] (head=True)."""
254
+ from organic.train_stageB import finding_list, report_prompt
255
+ ans = self.finding_answers(V, region)
256
+ pre = finding_list([f for f, p in ans.items() if p >= 0.5])
257
+ h = REPORT_HEAD[region] if head else ''
258
+ if h:
259
+ pre = f'{pre} {h}'
260
+ with torch.autocast('cuda', dtype=torch.bfloat16):
261
+ out = self.m.generate(V, report_prompt(DATASET[region], self.cfg['vlm']['report_mode']), self.dev,
262
+ max_new_tokens=max_new_tokens, prefix=pre)
263
+ body = out[0].strip()
264
+ text = h + ('' if body[:1] in ',.;:' else ' ') + body if h else body
265
+ return pre.split('\n')[0].strip(), text
266
+
267
+
268
+ # ------------------------------------------------------------------ CLI
269
+ def prepare(a, device):
270
+ hu, aff = load_ct(a.ct, resample=not a.is_1mm)
271
+ seg = None
272
+ if a.seg:
273
+ seg = np.asarray(nib.as_closest_canonical(nib.load(a.seg)).dataobj).astype(np.uint8)
274
+ assert seg.shape == hu.shape, f'mask {seg.shape} is not on the 1 mm grid {hu.shape}'
275
+ elif a.totalseg:
276
+ seg = totalsegmentator(hu, aff)
277
+ return model_input(hu, crop_centre(hu, a.region, seg), device), seg is not None
278
+
279
+
280
+ def main():
281
+ ap = argparse.ArgumentParser(description=__doc__.split('\n')[0])
282
+ ap.add_argument('mode', choices=['encoder', 'vlm'])
283
+ ap.add_argument('--ct', required=True, help='CT NIfTI in HU')
284
+ ap.add_argument('--is_1mm', action='store_true', help='the CT is already 1 mm RAS (skip resampling)')
285
+ ap.add_argument('--region', choices=['chest', 'abdomen'], required=True)
286
+ ap.add_argument('--seg', default=None, help='TotalSegmentator v2 total multilabel mask on the 1 mm grid')
287
+ ap.add_argument('--totalseg', action='store_true', help='run TotalSegmentator for the crop centre')
288
+ ap.add_argument('--weights', default=None)
289
+ ap.add_argument('--encoder', default=ENCODER_REPO, help='vlm mode: the encoder repo or directory')
290
+ ap.add_argument('--sentence', action='append', default=[], help='encoder: extra sentence to score')
291
+ ap.add_argument('--question', action='append', default=[], help='vlm: extra finding phrase or yes/no question')
292
+ ap.add_argument('--report', action='store_true', help='vlm: write a report')
293
+ ap.add_argument('--max_new_tokens', type=int, default=768)
294
+ ap.add_argument('--out', default=None, help='write the results as JSON')
295
+ a = ap.parse_args()
296
+ dev = 'cuda'
297
+ x, masked = prepare(a, dev)
298
+ res = {'ct': os.path.basename(a.ct), 'region': a.region,
299
+ 'crop_centre': 'TotalSegmentator' if masked else 'estimate without a mask'}
300
+ if a.mode == 'encoder':
301
+ enc = Encoder(a.weights or ENCODER_REPO, dev, region=a.region)
302
+ T = enc.encode_image(x)
303
+ res['finding_scores'] = enc.finding_scores(T, a.region)
304
+ if a.sentence:
305
+ res['sentence_scores'] = dict(zip(a.sentence, enc.similarity(T, a.sentence).tolist()))
306
+ else:
307
+ vlm = VLM(a.weights or VLM_REPO, a.encoder, dev)
308
+ V = vlm.visual_tokens(x)
309
+ res['p_yes'] = vlm.finding_answers(V, a.region)
310
+ if a.question:
311
+ qs = [q if q.endswith('?') or 'Answer Yes or No' in q else vlm.question(q) for q in a.question]
312
+ res['questions'] = dict(zip(qs, vlm.p_yes(V, qs).tolist()))
313
+ if a.report:
314
+ res['finding_list'], res['report'] = vlm.report(V, a.region, a.max_new_tokens)
315
+ s = json.dumps(res, indent=1)
316
+ if a.out:
317
+ open(a.out, 'w').write(s)
318
+ print(s)
319
+
320
+
321
+ if __name__ == '__main__':
322
+ sys.exit(main())