medical
ct
yipengsun commited on
Commit
c6c96c7
·
0 Parent(s):

LightAlign CT-RATE weights

Browse files
.gitattributes ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,62 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: cc-by-nc-sa-4.0
3
+ datasets:
4
+ - ibrahimhamamci/CT-RATE
5
+ base_model:
6
+ - rmaguado/VoxelFM
7
+ - Qwen/Qwen3.5-9B
8
+ tags:
9
+ - medical
10
+ - ct
11
+ ---
12
+
13
+ # LightAlign CT-RATE weights
14
+
15
+ Linear map and report-side finding classifiers for 18 CT-RATE findings, from the paper "What Matters for Cross-Modal Alignment".
16
+
17
+ Code: [github.com/sypsyp97/lightalign](https://github.com/sypsyp97/lightalign) · Project page: [sypsyp97.github.io/lightalign](https://sypsyp97.github.io/lightalign/)
18
+
19
+ The image encoder is frozen VoxelFM ViT-B 3D (`rmaguado/VoxelFM`, revision `7a58beab42c0ba213f40b9dbcaae43d3488416dd`; code commit `fba41a34c3e4dcfebbe902be5e2b2ce26db27d52`). The feature is the CLS token after block 11, after the final layer norm.
20
+
21
+ The text encoder is frozen Qwen3.5-9B (`Qwen/Qwen3.5-9B`, revision `c202236235762e1c871ad0ccb60c8ee5ba337b9a`). The feature is the mean over tokens of the middle hidden state (`len(hidden_states) // 2`). The vision tower is not used. The public commit is pinned here. The command that produced the original text arrays did not pass a revision.
22
+
23
+ The map is ridge regression from image features to text features, fit on all 5,994 paired training patients (seed 0). Each modality is L2-normalized, centered by the mean of those unit vectors on the full training pool, and L2-normalized again. Ridge uses `RidgeCV` with alphas `logspace(-3, 3, 13)` and `fit_intercept=False`. There is no intercept in this file.
24
+
25
+ Each finding then has a `StandardScaler` and a logistic regression fit on the centered report embeddings and the report-derived label. A new scan is mapped into that space. `TC` is the decision function. `P` is the cosine with the positive prompt embedding minus the cosine with the negative prompt embedding. The prompt strings are `IMPRESSION:\n{label} is present.` and `IMPRESSION:\nNo {label.lower()}.`.
26
+
27
+ Files: `lightalign-ctrate-n5994-seed0.npz` and `lightalign-ctrate-n5994-seed0.json`.
28
+
29
+ ## Intended use
30
+
31
+ Research on chest CT representation and zero-shot finding readout.
32
+
33
+ Not for clinical use. Not a medical device. Do not use the scores to diagnose, triage, or treat a patient.
34
+
35
+ ## Training data
36
+
37
+ CT-RATE (`ibrahimhamamci/CT-RATE`), gated, license CC BY-NC-SA 4.0. Training rows are the hashed 30% of `train_fixed` reconstructions `*_a_1` after the dataset non-chest list is removed (5,994 patients: 2,024 `sel`, 1,945 `train`, 2,025 `test`). The map uses every row. Labels are the 18 report-derived abnormality columns shipped with CT-RATE. They are not a new manual annotation.
38
+
39
+ These weights inherit CC BY-NC-SA 4.0.
40
+
41
+ ## How to use
42
+
43
+ Score one NIfTI volume with the `lightalign` package:
44
+
45
+ ```bash
46
+ lightalign-predict --ct volume.nii.gz --weights yipengsun/lightalign-ctrate --voxelfm-code /path/to/VoxelFM
47
+ ```
48
+
49
+ Prediction needs a CUDA GPU, the VoxelFM code checkout (`--voxelfm-code` or `VOXELFM_CODE`), the VoxelFM weights (downloaded from `rmaguado/VoxelFM` by default), and this file. It does not need Qwen3.5-9B. The same package rebuilds this file from CT-RATE embeddings. See the package README for the case rules, the pair counts, and the chest-unit table.
50
+
51
+ ## Limitations
52
+
53
+ - Research only. The scores are not calibrated probabilities. This file does not contain an operating point. Weighted F1 and accuracy from the reproduction script use a threshold chosen on the evaluated volumes.
54
+ - Eighteen findings, taken from report text in one public dataset. A finding the report does not name is a negative label in that table. That is not a radiologist saying the finding is absent on the scan.
55
+ - The reader expects a 3D chest CT in NIfTI. Voxels are read from `dataobj` with no rescale slope or intercept. The CT-RATE fixed release already stores corrected Hounsfield units. A file that still stores unscaled values in `dataobj` will be wrong.
56
+ - The map is linear. It was fit on 5,994 pairs from CT-RATE. Other hospitals, scanners, and reconstructions were not part of the fit.
57
+ - A different Qwen3.5-9B snapshot would not match these coefficients.
58
+ - Attention uses fused scaled dot-product attention. Each process checks it once against the reference attention on a reduced volume, and stops if the cosine is below 0.9999.
59
+
60
+ ## License
61
+
62
+ cc-by-nc-sa-4.0
lightalign-ctrate-n5994-seed0.json ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "format": "lightalign-ctrate-n5994-seed0",
3
+ "n_pairs": 5994,
4
+ "seed": 0,
5
+ "findings": [
6
+ "Medical material",
7
+ "Arterial wall calcification",
8
+ "Cardiomegaly",
9
+ "Pericardial effusion",
10
+ "Coronary artery wall calcification",
11
+ "Hiatal hernia",
12
+ "Lymphadenopathy",
13
+ "Emphysema",
14
+ "Atelectasis",
15
+ "Lung nodule",
16
+ "Lung opacity",
17
+ "Pulmonary fibrotic sequela",
18
+ "Pleural effusion",
19
+ "Mosaic attenuation pattern",
20
+ "Peribronchial thickening",
21
+ "Consolidation",
22
+ "Bronchiectasis",
23
+ "Interlobular septal thickening"
24
+ ],
25
+ "image": {
26
+ "repo": "rmaguado/VoxelFM",
27
+ "revision": "7a58beab42c0ba213f40b9dbcaae43d3488416dd",
28
+ "code": "https://github.com/rmaguado/VoxelFM",
29
+ "code_commit": "fba41a34c3e4dcfebbe902be5e2b2ce26db27d52",
30
+ "files": [
31
+ "vitb_3d/config.yaml",
32
+ "vitb_3d/checkpoints/99999.pth"
33
+ ],
34
+ "feature": "normed CLS token after block 11",
35
+ "attention": "fused scaled_dot_product_attention",
36
+ "dtype": "bfloat16"
37
+ },
38
+ "text": {
39
+ "repo": "Qwen/Qwen3.5-9B",
40
+ "revision": "c202236235762e1c871ad0ccb60c8ee5ba337b9a",
41
+ "pooling": "mean over tokens of hidden_states[len(hidden_states)//2]",
42
+ "max_length": 1024,
43
+ "dtype": "bfloat16",
44
+ "padding_side": "right",
45
+ "loader": "AutoModelForImageTextToText; vision tower deleted"
46
+ },
47
+ "normalization": "L2-normalize, subtract the mean of the training-pool unit vectors, L2-normalize again",
48
+ "ridge": {
49
+ "alphas": "logspace(-3, 3, 13)",
50
+ "fit_intercept": false,
51
+ "alpha": 3.1622776601683795
52
+ },
53
+ "logistic_regression": {
54
+ "C": 1.0,
55
+ "fit_intercept": true,
56
+ "solver": "lbfgs",
57
+ "max_iter": 100,
58
+ "tol": 0.0001,
59
+ "scaler": "StandardScaler",
60
+ "note": "explicit scikit-learn 1.6.1 defaults"
61
+ },
62
+ "prompt_embeddings": "centered and L2-normalized; positive then negative per finding",
63
+ "fit_libraries": {
64
+ "numpy": "2.4.6",
65
+ "scikit-learn": "1.9.1",
66
+ "scipy": "1.17.1"
67
+ }
68
+ }
lightalign-ctrate-n5994-seed0.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d35d979cb5323bd9e740447a5ae3d4dbb61d37489a69aeba58193b5a7a5de03a
3
+ size 31305542