ViDRiP-LLaVA-SigLIP

ViDRiP-LLaVA with the SigLIP image encoder (ViDRiP-LLaVA_SigLIP in the paper), from

ViDRiP-LLaVA: Learning diagnostic reasoning from narrated pathology videos for images, videos and whole slides

Code, training scripts and evaluation: https://github.com/trinhvg/ViDRiP-LLaVA

Best for

Open-ended diagnostic descriptions: given a pathology video clip or image, describe the morphology and give a diagnosis. For multiple-choice questions, use ViDRiP-LLaVA-SigLIP-wo-ClipDesc.

Model

• Image encoder: SigLIP (google/siglip-so400m-patch14-384), projector, and a 7B-parameter Qwen2.5 language decoder (LLaVA-OneVision architecture, llava_qwen).
• Input: a single image or a sequence of video frames (32 frames in our video evaluation).

Training stages

This is the full training path: stages 0 and 1, the ClipDesc stage, then ViDRiP-Instruct.

Stage	Training	Data	This model
0	Image–caption alignment (projector only)	949,575 image–caption pairs (Quilt-1M, PathAsst, bladder image–text)	yes
1	Image instruction tuning	140,363 image conversations (Quilt-LLaVA, PathAsst)	yes
2	Image instructions + ClipDesc-Instruct	stage-1 conversations + 139,941 ClipDesc-Instruct clips	yes
3	ViDRiP-Instruct tuning (decoder LoRA, merged)	4,036 ViDRiP-Instruct videos	yes

Each stage ran for one epoch. Details are in the paper's Methods and Supplementary Information.

Usage

Load with the LLaVA-NeXT code in the GitHub repository:

from llava.model.builder import load_pretrained_model

tokenizer, model, image_processor, max_length = load_pretrained_model(
    "trinhvg/ViDRiP-LLaVA-SigLIP", None, "llava_qwen", device_map="auto", attn_implementation="sdpa"
)

A full image/video example is in docs/VidDiag_LLaVA_trial.py in the repository.

Released checkpoints

Checkpoint	Training	Best for
ViDRiP-LLaVA-SigLIP (this repo)	stages 0, 1, ClipDesc, ViDRiP-Instruct	open-ended diagnostic descriptions
ViDRiP-LLaVA-SigLIP-wo-ClipDesc	stages 0, 1, ViDRiP-Instruct	multiple-choice questions

The CONCH- and MUSK-based ViDRiP-LLaVA models contain fine-tuned copies of the CONCH v1.5 and MUSK encoders, whose licenses (CC BY-NC-ND 4.0) do not permit redistribution of derivatives, so these weights are not released. They can be reproduced with the released training code and instruction data by researchers who obtain the encoders from their licensors.

Training data notice

The models were trained on an internal version of the video data that included full-frame-rate clips and content prior to OCR filtering. The 502-video benchmark used in the paper is held out and available on request.

License and intended use

• Released under CC BY-NC 3.0 (Attribution–NonCommercial), for non-commercial research only.
• Not for clinical use or medical decision-making.
• Components such as Qwen2.5, SigLIP, LLaVA-NeXT, Quilt-1M and PathAsst are subject to their own licenses and terms of use.

Citation

Coming soon.

This repository was previously named trinhvg/ViDRiP_LLaVA_video; the weights are unchanged.