U-VLM: Hierarchical Vision Language Modeling for Report Generation
Released U-VLM checkpoints for 3D CT radiology report generation on CT-RATE and AbdomenAtlas 3.0. U-VLM pairs a residual U-Net encoder pretrained through segmentation and classification with a 0.1B language decoder trained from scratch. Encoder features are injected into multiple decoder layers.
- Code: github.com/yinghemedical/U-VLM
- Paper: arXiv:2603.00479
Released Checkpoints
| Dataset | F1 | BLEU-mean |
|---|---|---|
| CT-RATE | 0.4497 | 0.3404 |
| AbdomenAtlas 3.0 (structured) | 0.5718 | 0.4342 |
Both entries use checkpoint_final.pth. CT-RATE F1 is RadBERT macro F1;
AbdomenAtlas F1 uses DeepSeek label extraction. BLEU-mean is computed with
pycocoevalcap.
Retraining Note
The released checkpoints are final-epoch models retrained after the paper. Their recomputed metrics differ from the paper table:
| Dataset | Paper F1 | Released F1 | Paper BLEU-mean | Released BLEU-mean |
|---|---|---|---|---|
| CT-RATE | 0.414 | 0.4497 | 0.349 | 0.3404 |
| AbdomenAtlas 3.0 | 0.624 | 0.5718 | 0.437 | 0.4342 |
The CT-RATE checkpoint is a narrative-report model. The AbdomenAtlas
checkpoint is a structured-report model (report_structured).
Layout
.
βββ config.json
βββ Dataset1131_CT-RATE/
β βββ nnUNetTrainer_UVLM__CTRATE_report_192x256x256_clsseg_bal_aug_ds_fr_narr__3d_fullres/
β βββ dataset.json
β βββ plans.json
β βββ dataset_fingerprint.json
β βββ metrics.json
β βββ fold_0/checkpoint_final.pth
βββ Dataset1141_AbdomenAtlas3.0Mini/
β βββ nnUNetTrainer_UVLM__ABDOMEN_report_192x256x256_clsseg_seg_coarse_lesion_bal_aug_ds_fr_struct__3d_fullres/
β βββ dataset.json
β βββ plans.json
β βββ dataset_fingerprint.json
β βββ metrics.json
β βββ fold_0/checkpoint_final.pth
βββ tokenizer_qwen3_4b/
Only inference weights, JSON configs, and metrics are included. Training logs, optimizer states, images, reports, and patient identifiers are not included.
Method
U-VLM progressively trains a shared residual U-Net encoder:
- Segmentation pretraining β dense per-voxel supervision.
- Classification pretraining β 18-class chest or 3-class abdominal labels.
- Report generation β image-report pairs.
For report generation, encoder features from six stages are routed to six decoder layers. The decoder has 512 hidden dimensions, 8 heads, and is trained from scratch. The released models use the Qwen3-4B tokenizer; the Qwen3-4B language-model weights are not used.
Requirements
Use the code repository's environment:
git clone https://github.com/yinghemedical/U-VLM.git
cd U-VLM
bash uvlm/scripts/setup_env.sh
Python 3.10, PyTorch, nnU-Net v2, blosc2, and transformers are required.
See the code repository for the full installation instructions.
Inference
Download the weights:
hf download yinghemedical/U-VLM --repo-type model --local-dir /path/to/uvlm_weights
The input CSV must contain series_id and either a preprocessed Blosc2 path
(blosc2_path) or a NIfTI path (image_path / nifti_path). NIfTI volumes are
normalized and resized on the fly. Run report generation with the code
repository:
# CT-RATE, narrative report
uvlm_inference report \
--csv-path /path/to/ct_rate_test.csv \
--model-dir /path/to/uvlm_weights/Dataset1131_CT-RATE/nnUNetTrainer_UVLM__CTRATE_report_192x256x256_clsseg_bal_aug_ds_fr_narr__3d_fullres \
--output-dir /path/to/output/ct_rate \
--gpu-config "0:1"
# AbdomenAtlas 3.0, structured report
uvlm_inference report \
--csv-path /path/to/abdomen_test.csv \
--model-dir /path/to/uvlm_weights/Dataset1141_AbdomenAtlas3.0Mini/nnUNetTrainer_UVLM__ABDOMEN_report_192x256x256_clsseg_seg_coarse_lesion_bal_aug_ds_fr_struct__3d_fullres \
--output-dir /path/to/output/abdomen_atlas \
--gpu-config "0:1"
Output is output/.../results.csv with a generated_report column.
The tokenizer is included in this repository and in the code repository. Set
UVLM_TOKENIZER_PATH to override the default:
export UVLM_TOKENIZER_PATH=/path/to/uvlm_weights/tokenizer_qwen3_4b
Evaluation Details
See the code repository's
docs/RELEASED_CHECKPOINTS.md
for the full metric breakdown and protocol.
| Dataset | F1 | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | BLEU-mean |
|---|---|---|---|---|---|---|
| CT-RATE | 0.4497 | 0.4819 | 0.3595 | 0.2845 | 0.2359 | 0.3404 |
| AbdomenAtlas 3.0 (structured) | 0.5718 | 0.5569 | 0.4635 | 0.3921 | 0.3244 | 0.4342 |
These are local evaluation results, not official challenge scores. Metrics are reported on the CT-RATE validation split and the AbdomenAtlas test split, respectively.
Data and License
CT-RATE and AbdomenAtlas 3.0 images and reports are not redistributed. Users must obtain the datasets under their upstream terms.
- Code: Apache-2.0.
- Model weights in this repository: CC BY-NC 4.0 (non-commercial use only).
- Qwen3-4B tokenizer and upstream components retain their own licenses.
Citation
@article{shi2026u,
title={U-VLM: Hierarchical Vision Language Modeling for Report Generation},
author={Shi, Pengcheng and Zhang, Minghui and Song, Kehan and Liu, Jiaqi and Gu, Yun and Zhang, Xinglin},
journal={arXiv preprint arXiv:2603.00479},
year={2026}
}
Acknowledgements
U-VLM builds on nnU-Net and the Qwen3 tokenizer. We thank the CT-RATE and AbdomenAtlas teams for their datasets. U-VLM is developed and maintained by Medical Image Insights and Shanghai Jiao Tong University.
- Downloads last month
- -