Image-to-Text
uvlm
medical-imaging
ct
report-generation
3d
vision-language

U-VLM: Hierarchical Vision Language Modeling for Report Generation

Paper GitHub HuggingFace

Released U-VLM checkpoints for 3D CT radiology report generation on CT-RATE and AbdomenAtlas 3.0. U-VLM pairs a residual U-Net encoder pretrained through segmentation and classification with a 0.1B language decoder trained from scratch. Encoder features are injected into multiple decoder layers.

Released Checkpoints

Dataset F1 BLEU-mean
CT-RATE 0.4497 0.3404
AbdomenAtlas 3.0 (structured) 0.5718 0.4342

Both entries use checkpoint_final.pth. CT-RATE F1 is RadBERT macro F1; AbdomenAtlas F1 uses DeepSeek label extraction. BLEU-mean is computed with pycocoevalcap.

Retraining Note

The released checkpoints are final-epoch models retrained after the paper. Their recomputed metrics differ from the paper table:

Dataset Paper F1 Released F1 Paper BLEU-mean Released BLEU-mean
CT-RATE 0.414 0.4497 0.349 0.3404
AbdomenAtlas 3.0 0.624 0.5718 0.437 0.4342

The CT-RATE checkpoint is a narrative-report model. The AbdomenAtlas checkpoint is a structured-report model (report_structured).

Layout

.
β”œβ”€β”€ config.json
β”œβ”€β”€ Dataset1131_CT-RATE/
β”‚   └── nnUNetTrainer_UVLM__CTRATE_report_192x256x256_clsseg_bal_aug_ds_fr_narr__3d_fullres/
β”‚       β”œβ”€β”€ dataset.json
β”‚       β”œβ”€β”€ plans.json
β”‚       β”œβ”€β”€ dataset_fingerprint.json
β”‚       β”œβ”€β”€ metrics.json
β”‚       └── fold_0/checkpoint_final.pth
β”œβ”€β”€ Dataset1141_AbdomenAtlas3.0Mini/
β”‚   └── nnUNetTrainer_UVLM__ABDOMEN_report_192x256x256_clsseg_seg_coarse_lesion_bal_aug_ds_fr_struct__3d_fullres/
β”‚       β”œβ”€β”€ dataset.json
β”‚       β”œβ”€β”€ plans.json
β”‚       β”œβ”€β”€ dataset_fingerprint.json
β”‚       β”œβ”€β”€ metrics.json
β”‚       └── fold_0/checkpoint_final.pth
└── tokenizer_qwen3_4b/

Only inference weights, JSON configs, and metrics are included. Training logs, optimizer states, images, reports, and patient identifiers are not included.

Method

U-VLM progressively trains a shared residual U-Net encoder:

  1. Segmentation pretraining β€” dense per-voxel supervision.
  2. Classification pretraining β€” 18-class chest or 3-class abdominal labels.
  3. Report generation β€” image-report pairs.

For report generation, encoder features from six stages are routed to six decoder layers. The decoder has 512 hidden dimensions, 8 heads, and is trained from scratch. The released models use the Qwen3-4B tokenizer; the Qwen3-4B language-model weights are not used.

Requirements

Use the code repository's environment:

git clone https://github.com/yinghemedical/U-VLM.git
cd U-VLM
bash uvlm/scripts/setup_env.sh

Python 3.10, PyTorch, nnU-Net v2, blosc2, and transformers are required. See the code repository for the full installation instructions.

Inference

Download the weights:

hf download yinghemedical/U-VLM --repo-type model --local-dir /path/to/uvlm_weights

The input CSV must contain series_id and either a preprocessed Blosc2 path (blosc2_path) or a NIfTI path (image_path / nifti_path). NIfTI volumes are normalized and resized on the fly. Run report generation with the code repository:

# CT-RATE, narrative report
uvlm_inference report \
    --csv-path /path/to/ct_rate_test.csv \
    --model-dir /path/to/uvlm_weights/Dataset1131_CT-RATE/nnUNetTrainer_UVLM__CTRATE_report_192x256x256_clsseg_bal_aug_ds_fr_narr__3d_fullres \
    --output-dir /path/to/output/ct_rate \
    --gpu-config "0:1"

# AbdomenAtlas 3.0, structured report
uvlm_inference report \
    --csv-path /path/to/abdomen_test.csv \
    --model-dir /path/to/uvlm_weights/Dataset1141_AbdomenAtlas3.0Mini/nnUNetTrainer_UVLM__ABDOMEN_report_192x256x256_clsseg_seg_coarse_lesion_bal_aug_ds_fr_struct__3d_fullres \
    --output-dir /path/to/output/abdomen_atlas \
    --gpu-config "0:1"

Output is output/.../results.csv with a generated_report column.

The tokenizer is included in this repository and in the code repository. Set UVLM_TOKENIZER_PATH to override the default:

export UVLM_TOKENIZER_PATH=/path/to/uvlm_weights/tokenizer_qwen3_4b

Evaluation Details

See the code repository's docs/RELEASED_CHECKPOINTS.md for the full metric breakdown and protocol.

Dataset F1 BLEU-1 BLEU-2 BLEU-3 BLEU-4 BLEU-mean
CT-RATE 0.4497 0.4819 0.3595 0.2845 0.2359 0.3404
AbdomenAtlas 3.0 (structured) 0.5718 0.5569 0.4635 0.3921 0.3244 0.4342

These are local evaluation results, not official challenge scores. Metrics are reported on the CT-RATE validation split and the AbdomenAtlas test split, respectively.

Data and License

CT-RATE and AbdomenAtlas 3.0 images and reports are not redistributed. Users must obtain the datasets under their upstream terms.

  • Code: Apache-2.0.
  • Model weights in this repository: CC BY-NC 4.0 (non-commercial use only).
  • Qwen3-4B tokenizer and upstream components retain their own licenses.

Citation

@article{shi2026u,
  title={U-VLM: Hierarchical Vision Language Modeling for Report Generation},
  author={Shi, Pengcheng and Zhang, Minghui and Song, Kehan and Liu, Jiaqi and Gu, Yun and Zhang, Xinglin},
  journal={arXiv preprint arXiv:2603.00479},
  year={2026}
}

Acknowledgements

U-VLM builds on nnU-Net and the Qwen3 tokenizer. We thank the CT-RATE and AbdomenAtlas teams for their datasets. U-VLM is developed and maintained by Medical Image Insights and Shanghai Jiao Tong University.

Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Datasets used to train yinghemedical/U-VLM

Paper for yinghemedical/U-VLM