EviJEPA — Challenge Weights (CT Abnormality Classification & CT Report Generation)

Trained weights for EviJEPA, an evidence-aware CT–report representation learning framework. Both checkpoints placed 3rd in their respective VLM3D Challenge tracks.

Track Rank Metrics
CT Abnormality Classification 3rd CRG 0.5703 · AUROC 0.8646 · F1 0.6270 · Acc 0.8071
CT Report Generation 3rd CRG 0.5555 · F1 0.6109 · BLEU 0.2576 · ROUGE-L 0.2615 · METEOR 0.2333

Code, training recipes and full documentation: see the EviJEPA GitHub repository.


Files

File Size Contents
classifier_epoch19.pt 68 MB Abnormality classifier: label-attention head (18 findings) + adapted vision encoder (LoRA + patch embedding). Self-contained.
reportgen_vision_adaptation.pt 40 MB Vision encoder adaptation for report generation (LoRA + patch embedding), kept frozen during generation.
reportgen_stage2_epoch9.pt 312 MB Report generator: 256-query resampler + MedGemma LoRA adapter (stage 2, epoch 9).
calib_bias.json 811 B Per-label logit bias for the classifier (balanced F1/CRG operating point). Monotonic, so AUROC is unchanged.

Report generation needs both reportgen_vision_adaptation.pt (input side) and reportgen_stage2_epoch9.pt (generator side).

Base models required (not included)

These checkpoints are adapters and task heads only. Download the base models separately under their own licenses:

Base model Needed by
V-JEPA 2.1 ViT-L (vjepa2_1_vitl_dist_vitG_384.pt) both
google/medgemma-1.5-4b-it report generation

No Llama, Gemma or MedGemma base weights are redistributed here — only LoRA deltas and newly trained modules.

Download

huggingface-cli download bolkv/EviJEPA --local-dir weights

Usage

git clone <EviJEPA-github-repo> && cd EviJEPA
pip install -r requirements.txt
export VJEPA_REPO=/path/to/vjepa2

python infer_classifier.py \
  --config configs/abnormality_classification.yaml \
  --ckpt   weights/classifier_epoch19.pt \
  --input  /path/to/input --output /path/to/output \
  --calibration weights/calib_bias.json

python infer_reportgen.py \
  --config configs/report_generation.yaml \
  --ckpt   weights/reportgen_stage2_epoch9.pt \
  --input  /path/to/input --output /path/to/output \
  --classifier_ckpt weights/classifier_epoch19.pt \
  --calibration     weights/calib_bias.json

Set vision_ckpt: weights/reportgen_vision_adaptation.pt in configs/report_generation.yaml.

Input: 3D chest CT in NIfTI (*.nii.gz), Hounsfield units, any spacing. Resampling to 1 x 1 x 2.5 mm and 384 x 384 x 128 is performed internally. Output: predictions.json with 18 per-label probabilities, or generated reports.

Method

Each volume is resampled to 1 x 1 x 2.5 mm, expanded into three HU windows (lung, soft tissue, wide), tokenized with z-axis pixel unshuffle, and encoded by a LoRA-adapted V-JEPA 2.1 ViT-L. EviJEPA aligns CT and report through a global branch (learnable attention pooling + contrastive learning) and a local evidence branch (expert-choice routing matched by Sinkhorn optimal transport, with route and evidence diversity regularization). The pretrained encoder is then adapted with label-wise queries for classification and a two-stage projector plus MedGemma fine-tuning for report generation.

Team

Hyunjun Park, Woojin Na, Jinah Park — Korea Advanced Institute of Science and Technology (KAIST)

Limitations

  • Trained and evaluated on CT-RATE only; robustness to other scanners, protocols and populations is unverified.
  • Labels are automatically extracted from reports, not radiologist-verified.
  • Rare and focal findings (lung nodule, pulmonary fibrotic sequela) are the weakest labels.
  • Generated reports can over-call findings on normal studies.
  • Research use only. Not a medical device. Not for diagnosis or clinical decision-making.

License and terms

Released under CC BY-NC 4.0 (attribution, non-commercial) — see the LICENSE file in the EviJEPA GitHub repository for the full terms, medical disclaimer and third-party notices.

These checkpoints contain only LoRA adapters, newly trained modules (label-attention head, query resampler) and the fine-tuned patch-embedding layer; no transformer-block weights of Llama, Gemma, MedGemma or V-JEPA are redistributed. CT-RATE and every base model remain under their own licenses and acceptable-use policies and must be obtained from their original sources.

Research use only. Not a medical device. Not for diagnosis or clinical decision-making.

Citation

To be added.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support