EviJEPA challenge weights: abnormality classifier (ep19) and report generator (stage2 ep9)
Browse files- README.md +119 -0
- calib_bias.json +20 -0
- classifier_epoch19.pt +3 -0
- reportgen_stage2_epoch9.pt +3 -0
- reportgen_vision_adaptation.pt +3 -0
README.md
ADDED
|
@@ -0,0 +1,119 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: other
|
| 3 |
+
license_name: research-only
|
| 4 |
+
tags:
|
| 5 |
+
- medical
|
| 6 |
+
- computed-tomography
|
| 7 |
+
- chest-ct
|
| 8 |
+
- radiology-report-generation
|
| 9 |
+
- multi-label-classification
|
| 10 |
+
- vision-language
|
| 11 |
+
language:
|
| 12 |
+
- en
|
| 13 |
+
pipeline_tag: image-classification
|
| 14 |
+
---
|
| 15 |
+
|
| 16 |
+
# EviJEPA — Challenge Weights (CT Abnormality Classification & CT Report Generation)
|
| 17 |
+
|
| 18 |
+
Trained weights for **EviJEPA**, an evidence-aware CT–report representation learning framework.
|
| 19 |
+
Both checkpoints placed **3rd** in their respective VLM3D Challenge tracks.
|
| 20 |
+
|
| 21 |
+
| Track | Rank | Metrics |
|
| 22 |
+
|---|---|---|
|
| 23 |
+
| CT Abnormality Classification | **3rd** | CRG 0.5703 · AUROC 0.8646 · F1 0.6270 · Acc 0.8071 |
|
| 24 |
+
| CT Report Generation | **3rd** | CRG 0.5555 · F1 0.6109 · BLEU 0.2576 · ROUGE-L 0.2615 · METEOR 0.2333 |
|
| 25 |
+
|
| 26 |
+
**Code, training recipes and full documentation:** see the EviJEPA GitHub repository.
|
| 27 |
+
|
| 28 |
+
---
|
| 29 |
+
|
| 30 |
+
## Files
|
| 31 |
+
|
| 32 |
+
| File | Size | Contents |
|
| 33 |
+
|---|---|---|
|
| 34 |
+
| `classifier_epoch19.pt` | 68 MB | Abnormality classifier: label-attention head (18 findings) + adapted vision encoder (LoRA + patch embedding). Self-contained. |
|
| 35 |
+
| `reportgen_vision_adaptation.pt` | 40 MB | Vision encoder adaptation for report generation (LoRA + patch embedding), kept frozen during generation. |
|
| 36 |
+
| `reportgen_stage2_epoch9.pt` | 312 MB | Report generator: 256-query resampler + MedGemma LoRA adapter (stage 2, epoch 9). |
|
| 37 |
+
| `calib_bias.json` | 811 B | Per-label logit bias for the classifier (balanced F1/CRG operating point). Monotonic, so AUROC is unchanged. |
|
| 38 |
+
|
| 39 |
+
Report generation needs **both** `reportgen_vision_adaptation.pt` (input side) and
|
| 40 |
+
`reportgen_stage2_epoch9.pt` (generator side).
|
| 41 |
+
|
| 42 |
+
## Base models required (not included)
|
| 43 |
+
|
| 44 |
+
These checkpoints are **adapters and task heads only**. Download the base models separately under
|
| 45 |
+
their own licenses:
|
| 46 |
+
|
| 47 |
+
| Base model | Needed by |
|
| 48 |
+
|---|---|
|
| 49 |
+
| V-JEPA 2.1 ViT-L (`vjepa2_1_vitl_dist_vitG_384.pt`) | both |
|
| 50 |
+
| `google/medgemma-1.5-4b-it` | report generation |
|
| 51 |
+
|
| 52 |
+
No Llama, Gemma or MedGemma base weights are redistributed here — only LoRA deltas and newly
|
| 53 |
+
trained modules.
|
| 54 |
+
|
| 55 |
+
## Download
|
| 56 |
+
|
| 57 |
+
```bash
|
| 58 |
+
huggingface-cli download bolkv/EviJEPA --local-dir weights
|
| 59 |
+
```
|
| 60 |
+
|
| 61 |
+
## Usage
|
| 62 |
+
|
| 63 |
+
```bash
|
| 64 |
+
git clone <EviJEPA-github-repo> && cd EviJEPA
|
| 65 |
+
pip install -r requirements.txt
|
| 66 |
+
export VJEPA_REPO=/path/to/vjepa2
|
| 67 |
+
|
| 68 |
+
python infer_classifier.py \
|
| 69 |
+
--config configs/abnormality_classification.yaml \
|
| 70 |
+
--ckpt weights/classifier_epoch19.pt \
|
| 71 |
+
--input /path/to/input --output /path/to/output \
|
| 72 |
+
--calibration weights/calib_bias.json
|
| 73 |
+
|
| 74 |
+
python infer_reportgen.py \
|
| 75 |
+
--config configs/report_generation.yaml \
|
| 76 |
+
--ckpt weights/reportgen_stage2_epoch9.pt \
|
| 77 |
+
--input /path/to/input --output /path/to/output \
|
| 78 |
+
--classifier_ckpt weights/classifier_epoch19.pt \
|
| 79 |
+
--calibration weights/calib_bias.json
|
| 80 |
+
```
|
| 81 |
+
|
| 82 |
+
Set `vision_ckpt: weights/reportgen_vision_adaptation.pt` in `configs/report_generation.yaml`.
|
| 83 |
+
|
| 84 |
+
**Input**: 3D chest CT in NIfTI (`*.nii.gz`), Hounsfield units, any spacing. Resampling to
|
| 85 |
+
1 x 1 x 2.5 mm and 384 x 384 x 128 is performed internally.
|
| 86 |
+
**Output**: `predictions.json` with 18 per-label probabilities, or generated reports.
|
| 87 |
+
|
| 88 |
+
## Method
|
| 89 |
+
|
| 90 |
+
Each volume is resampled to 1 x 1 x 2.5 mm, expanded into three HU windows (lung, soft tissue,
|
| 91 |
+
wide), tokenized with z-axis pixel unshuffle, and encoded by a LoRA-adapted V-JEPA 2.1 ViT-L.
|
| 92 |
+
EviJEPA aligns CT and report through a global branch (learnable attention pooling + contrastive
|
| 93 |
+
learning) and a local evidence branch (expert-choice routing matched by Sinkhorn optimal transport,
|
| 94 |
+
with route and evidence diversity regularization). The pretrained encoder is then adapted with
|
| 95 |
+
label-wise queries for classification and a two-stage projector plus MedGemma fine-tuning for
|
| 96 |
+
report generation.
|
| 97 |
+
|
| 98 |
+
## Team
|
| 99 |
+
|
| 100 |
+
Hyunjun Park, Woojin Na, Jinah Park — Korea Advanced Institute of Science and Technology (KAIST)
|
| 101 |
+
|
| 102 |
+
## Limitations
|
| 103 |
+
|
| 104 |
+
- Trained and evaluated on **CT-RATE only**; robustness to other scanners, protocols and
|
| 105 |
+
populations is unverified.
|
| 106 |
+
- Labels are automatically extracted from reports, not radiologist-verified.
|
| 107 |
+
- Rare and focal findings (lung nodule, pulmonary fibrotic sequela) are the weakest labels.
|
| 108 |
+
- Generated reports can over-call findings on normal studies.
|
| 109 |
+
- **Research use only. Not a medical device. Not for diagnosis or clinical decision-making.**
|
| 110 |
+
|
| 111 |
+
## License and terms
|
| 112 |
+
|
| 113 |
+
Research use only, provided "as is" without warranty. CT-RATE and all base models remain under
|
| 114 |
+
their own licenses and acceptable-use policies; users must obtain them from their original sources
|
| 115 |
+
and comply with those terms.
|
| 116 |
+
|
| 117 |
+
## Citation
|
| 118 |
+
|
| 119 |
+
To be added.
|
calib_bias.json
ADDED
|
@@ -0,0 +1,20 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"Medical material": -0.9047009912802484,
|
| 3 |
+
"Arterial wall calcification": -0.6858179453317352,
|
| 4 |
+
"Cardiomegaly": -1.670602611947551,
|
| 5 |
+
"Pericardial effusion": -2.460788794070786,
|
| 6 |
+
"Coronary artery wall calcification": -0.5969757405377963,
|
| 7 |
+
"Hiatal hernia": -0.8519606851255502,
|
| 8 |
+
"Lymphadenopathy": -0.4715268077887384,
|
| 9 |
+
"Emphysema": 0.120654947953491,
|
| 10 |
+
"Atelectasis": 0.19182830449403848,
|
| 11 |
+
"Lung nodule": 0.7373416780748351,
|
| 12 |
+
"Lung opacity": 0.6699719471420577,
|
| 13 |
+
"Pulmonary fibrotic sequela": -0.12687102212306492,
|
| 14 |
+
"Pleural effusion": -1.6293811338619186,
|
| 15 |
+
"Mosaic attenuation pattern": -1.1960276902479205,
|
| 16 |
+
"Peribronchial thickening": -1.3180886966615555,
|
| 17 |
+
"Consolidation": -0.9792865004271506,
|
| 18 |
+
"Bronchiectasis": -0.879089215397013,
|
| 19 |
+
"Interlobular septal thickening": -0.8515947370936057
|
| 20 |
+
}
|
classifier_epoch19.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0d897e105bc59fe9d38ddf49df0d54c3196b5b5d45f60d0027aee03422922c61
|
| 3 |
+
size 70528944
|
reportgen_stage2_epoch9.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:56a05224b747122bd10b5bba8e0417e6339023a96655fc31bf2853bb252c797c
|
| 3 |
+
size 326775325
|
reportgen_vision_adaptation.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:42225958e38fbc1f234d2f447fa6f4dbb93cd324be022cf756355f4606524cfd
|
| 3 |
+
size 40971995
|