bolkv commited on
Commit
6f8a674
·
verified ·
1 Parent(s): a2f1c9e

EviJEPA challenge weights: abnormality classifier (ep19) and report generator (stage2 ep9)

Browse files
README.md ADDED
@@ -0,0 +1,119 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ license_name: research-only
4
+ tags:
5
+ - medical
6
+ - computed-tomography
7
+ - chest-ct
8
+ - radiology-report-generation
9
+ - multi-label-classification
10
+ - vision-language
11
+ language:
12
+ - en
13
+ pipeline_tag: image-classification
14
+ ---
15
+
16
+ # EviJEPA — Challenge Weights (CT Abnormality Classification & CT Report Generation)
17
+
18
+ Trained weights for **EviJEPA**, an evidence-aware CT–report representation learning framework.
19
+ Both checkpoints placed **3rd** in their respective VLM3D Challenge tracks.
20
+
21
+ | Track | Rank | Metrics |
22
+ |---|---|---|
23
+ | CT Abnormality Classification | **3rd** | CRG 0.5703 · AUROC 0.8646 · F1 0.6270 · Acc 0.8071 |
24
+ | CT Report Generation | **3rd** | CRG 0.5555 · F1 0.6109 · BLEU 0.2576 · ROUGE-L 0.2615 · METEOR 0.2333 |
25
+
26
+ **Code, training recipes and full documentation:** see the EviJEPA GitHub repository.
27
+
28
+ ---
29
+
30
+ ## Files
31
+
32
+ | File | Size | Contents |
33
+ |---|---|---|
34
+ | `classifier_epoch19.pt` | 68 MB | Abnormality classifier: label-attention head (18 findings) + adapted vision encoder (LoRA + patch embedding). Self-contained. |
35
+ | `reportgen_vision_adaptation.pt` | 40 MB | Vision encoder adaptation for report generation (LoRA + patch embedding), kept frozen during generation. |
36
+ | `reportgen_stage2_epoch9.pt` | 312 MB | Report generator: 256-query resampler + MedGemma LoRA adapter (stage 2, epoch 9). |
37
+ | `calib_bias.json` | 811 B | Per-label logit bias for the classifier (balanced F1/CRG operating point). Monotonic, so AUROC is unchanged. |
38
+
39
+ Report generation needs **both** `reportgen_vision_adaptation.pt` (input side) and
40
+ `reportgen_stage2_epoch9.pt` (generator side).
41
+
42
+ ## Base models required (not included)
43
+
44
+ These checkpoints are **adapters and task heads only**. Download the base models separately under
45
+ their own licenses:
46
+
47
+ | Base model | Needed by |
48
+ |---|---|
49
+ | V-JEPA 2.1 ViT-L (`vjepa2_1_vitl_dist_vitG_384.pt`) | both |
50
+ | `google/medgemma-1.5-4b-it` | report generation |
51
+
52
+ No Llama, Gemma or MedGemma base weights are redistributed here — only LoRA deltas and newly
53
+ trained modules.
54
+
55
+ ## Download
56
+
57
+ ```bash
58
+ huggingface-cli download bolkv/EviJEPA --local-dir weights
59
+ ```
60
+
61
+ ## Usage
62
+
63
+ ```bash
64
+ git clone <EviJEPA-github-repo> && cd EviJEPA
65
+ pip install -r requirements.txt
66
+ export VJEPA_REPO=/path/to/vjepa2
67
+
68
+ python infer_classifier.py \
69
+ --config configs/abnormality_classification.yaml \
70
+ --ckpt weights/classifier_epoch19.pt \
71
+ --input /path/to/input --output /path/to/output \
72
+ --calibration weights/calib_bias.json
73
+
74
+ python infer_reportgen.py \
75
+ --config configs/report_generation.yaml \
76
+ --ckpt weights/reportgen_stage2_epoch9.pt \
77
+ --input /path/to/input --output /path/to/output \
78
+ --classifier_ckpt weights/classifier_epoch19.pt \
79
+ --calibration weights/calib_bias.json
80
+ ```
81
+
82
+ Set `vision_ckpt: weights/reportgen_vision_adaptation.pt` in `configs/report_generation.yaml`.
83
+
84
+ **Input**: 3D chest CT in NIfTI (`*.nii.gz`), Hounsfield units, any spacing. Resampling to
85
+ 1 x 1 x 2.5 mm and 384 x 384 x 128 is performed internally.
86
+ **Output**: `predictions.json` with 18 per-label probabilities, or generated reports.
87
+
88
+ ## Method
89
+
90
+ Each volume is resampled to 1 x 1 x 2.5 mm, expanded into three HU windows (lung, soft tissue,
91
+ wide), tokenized with z-axis pixel unshuffle, and encoded by a LoRA-adapted V-JEPA 2.1 ViT-L.
92
+ EviJEPA aligns CT and report through a global branch (learnable attention pooling + contrastive
93
+ learning) and a local evidence branch (expert-choice routing matched by Sinkhorn optimal transport,
94
+ with route and evidence diversity regularization). The pretrained encoder is then adapted with
95
+ label-wise queries for classification and a two-stage projector plus MedGemma fine-tuning for
96
+ report generation.
97
+
98
+ ## Team
99
+
100
+ Hyunjun Park, Woojin Na, Jinah Park — Korea Advanced Institute of Science and Technology (KAIST)
101
+
102
+ ## Limitations
103
+
104
+ - Trained and evaluated on **CT-RATE only**; robustness to other scanners, protocols and
105
+ populations is unverified.
106
+ - Labels are automatically extracted from reports, not radiologist-verified.
107
+ - Rare and focal findings (lung nodule, pulmonary fibrotic sequela) are the weakest labels.
108
+ - Generated reports can over-call findings on normal studies.
109
+ - **Research use only. Not a medical device. Not for diagnosis or clinical decision-making.**
110
+
111
+ ## License and terms
112
+
113
+ Research use only, provided "as is" without warranty. CT-RATE and all base models remain under
114
+ their own licenses and acceptable-use policies; users must obtain them from their original sources
115
+ and comply with those terms.
116
+
117
+ ## Citation
118
+
119
+ To be added.
calib_bias.json ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "Medical material": -0.9047009912802484,
3
+ "Arterial wall calcification": -0.6858179453317352,
4
+ "Cardiomegaly": -1.670602611947551,
5
+ "Pericardial effusion": -2.460788794070786,
6
+ "Coronary artery wall calcification": -0.5969757405377963,
7
+ "Hiatal hernia": -0.8519606851255502,
8
+ "Lymphadenopathy": -0.4715268077887384,
9
+ "Emphysema": 0.120654947953491,
10
+ "Atelectasis": 0.19182830449403848,
11
+ "Lung nodule": 0.7373416780748351,
12
+ "Lung opacity": 0.6699719471420577,
13
+ "Pulmonary fibrotic sequela": -0.12687102212306492,
14
+ "Pleural effusion": -1.6293811338619186,
15
+ "Mosaic attenuation pattern": -1.1960276902479205,
16
+ "Peribronchial thickening": -1.3180886966615555,
17
+ "Consolidation": -0.9792865004271506,
18
+ "Bronchiectasis": -0.879089215397013,
19
+ "Interlobular septal thickening": -0.8515947370936057
20
+ }
classifier_epoch19.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0d897e105bc59fe9d38ddf49df0d54c3196b5b5d45f60d0027aee03422922c61
3
+ size 70528944
reportgen_stage2_epoch9.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:56a05224b747122bd10b5bba8e0417e6339023a96655fc31bf2853bb252c797c
3
+ size 326775325
reportgen_vision_adaptation.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:42225958e38fbc1f234d2f447fa6f4dbb93cd324be022cf756355f4606524cfd
3
+ size 40971995