EviJEPA — Challenge Weights (CT Abnormality Classification & CT Report Generation)
Trained weights for EviJEPA, an evidence-aware CT–report representation learning framework. Both checkpoints placed 3rd in their respective VLM3D Challenge tracks.
| Track | Rank | Metrics |
|---|---|---|
| CT Abnormality Classification | 3rd | CRG 0.5703 · AUROC 0.8646 · F1 0.6270 · Acc 0.8071 |
| CT Report Generation | 3rd | CRG 0.5555 · F1 0.6109 · BLEU 0.2576 · ROUGE-L 0.2615 · METEOR 0.2333 |
Code, training recipes and full documentation: see the EviJEPA GitHub repository.
Files
| File | Size | Contents |
|---|---|---|
classifier_epoch19.pt |
68 MB | Abnormality classifier: label-attention head (18 findings) + adapted vision encoder (LoRA + patch embedding). Self-contained. |
reportgen_vision_adaptation.pt |
40 MB | Vision encoder adaptation for report generation (LoRA + patch embedding), kept frozen during generation. |
reportgen_stage2_epoch9.pt |
312 MB | Report generator: 256-query resampler + MedGemma LoRA adapter (stage 2, epoch 9). |
calib_bias.json |
811 B | Per-label logit bias for the classifier (balanced F1/CRG operating point). Monotonic, so AUROC is unchanged. |
Report generation needs both reportgen_vision_adaptation.pt (input side) and
reportgen_stage2_epoch9.pt (generator side).
Base models required (not included)
These checkpoints are adapters and task heads only. Download the base models separately under their own licenses:
| Base model | Needed by |
|---|---|
V-JEPA 2.1 ViT-L (vjepa2_1_vitl_dist_vitG_384.pt) |
both |
google/medgemma-1.5-4b-it |
report generation |
No Llama, Gemma or MedGemma base weights are redistributed here — only LoRA deltas and newly trained modules.
Download
huggingface-cli download bolkv/EviJEPA --local-dir weights
Usage
git clone <EviJEPA-github-repo> && cd EviJEPA
pip install -r requirements.txt
export VJEPA_REPO=/path/to/vjepa2
python infer_classifier.py \
--config configs/abnormality_classification.yaml \
--ckpt weights/classifier_epoch19.pt \
--input /path/to/input --output /path/to/output \
--calibration weights/calib_bias.json
python infer_reportgen.py \
--config configs/report_generation.yaml \
--ckpt weights/reportgen_stage2_epoch9.pt \
--input /path/to/input --output /path/to/output \
--classifier_ckpt weights/classifier_epoch19.pt \
--calibration weights/calib_bias.json
Set vision_ckpt: weights/reportgen_vision_adaptation.pt in configs/report_generation.yaml.
Input: 3D chest CT in NIfTI (*.nii.gz), Hounsfield units, any spacing. Resampling to
1 x 1 x 2.5 mm and 384 x 384 x 128 is performed internally.
Output: predictions.json with 18 per-label probabilities, or generated reports.
Method
Each volume is resampled to 1 x 1 x 2.5 mm, expanded into three HU windows (lung, soft tissue, wide), tokenized with z-axis pixel unshuffle, and encoded by a LoRA-adapted V-JEPA 2.1 ViT-L. EviJEPA aligns CT and report through a global branch (learnable attention pooling + contrastive learning) and a local evidence branch (expert-choice routing matched by Sinkhorn optimal transport, with route and evidence diversity regularization). The pretrained encoder is then adapted with label-wise queries for classification and a two-stage projector plus MedGemma fine-tuning for report generation.
Team
Hyunjun Park, Woojin Na, Jinah Park — Korea Advanced Institute of Science and Technology (KAIST)
Limitations
- Trained and evaluated on CT-RATE only; robustness to other scanners, protocols and populations is unverified.
- Labels are automatically extracted from reports, not radiologist-verified.
- Rare and focal findings (lung nodule, pulmonary fibrotic sequela) are the weakest labels.
- Generated reports can over-call findings on normal studies.
- Research use only. Not a medical device. Not for diagnosis or clinical decision-making.
License and terms
Released under CC BY-NC 4.0 (attribution, non-commercial) — see the LICENSE file in the
EviJEPA GitHub repository for the full terms, medical disclaimer and third-party notices.
These checkpoints contain only LoRA adapters, newly trained modules (label-attention head, query resampler) and the fine-tuned patch-embedding layer; no transformer-block weights of Llama, Gemma, MedGemma or V-JEPA are redistributed. CT-RATE and every base model remain under their own licenses and acceptable-use policies and must be obtained from their original sources.
Research use only. Not a medical device. Not for diagnosis or clinical decision-making.
Citation
To be added.