| --- |
| license: apache-2.0 |
| language: |
| - en |
| pipeline_tag: image-to-text |
| tags: |
| - medical |
| --- |
| |
| # Model Card for PathBLIP-2 |
|
|
| A vision-language model built upon the BLIP-2 framework using BioGPT and HIPT for pathology report generation and cross-modal retrieval of melanocytic lesions. |
|
|
| ## Model Details |
|
|
| This repository contains multiple checkpoints for the model which was used for the experiments in the paper. |
| The model was trained and evaluated on a dataset of 19,636 melanocytic lesion cases, consisting of one or more whole slide images (WSIs) and a pathology report, using different training configurations. |
| The supporting code is available from the corresponding GitHub repository. |
| We refer to the paper for more information regarding the dataset, training, evaluation, and limitations. |
|
|
| - **Paper: *["On the Importance of Text Preprocessing for Multimodal Representation Learning and Pathology Report Generation"](https://openreview.net/forum?id=5fQchwJgQr&referrer=%5Bthe%20profile%20of%20Ruben%20Lucassen%5D(%2Fprofile%3Fid%3D~Ruben_Lucassen1))*** |
| - **Repository:** [GitHub](https://github.com/nuldertien/PathBLIP-2) |
| - **Framework:** [BLIP-2](https://github.com/salesforce/LAVIS/tree/main/projects/blip2) |
| - **Base language model:** [BioGPT](https://huggingface.co/microsoft/biogpt) |
| - **WSI feature extractor:** [HIPT](https://github.com/mahmoodlab/HIPT) |
| - **License:** Apache-2.0 |
