license: apache-2.0
base_model: Qwen/Qwen3-VL-8B-Instruct
library_name: transformers
pipeline_tag: image-text-to-text
language:
- en
tags:
- medical
- ophthalmology
- retinal-oct
- visual-grounding
- vision-language
RetGrounder
RetGrounder is an 8B multimodal large language model fine-tuned for visual grounding of biomarkers in retinal optical coherence tomography (OCT), introduced with the RetinaGroundBench benchmark. It is a LoRA fine-tune of Qwen/Qwen3-VL-8B-Instruct, trained with an OCT-specific strategy that integrates grounding and textual supervision, then merged into the base weights.
- Base model:
Qwen/Qwen3-VL-8B-Instruct - Tasks: biomarker visual grounding (bounding boxes) and presence VQA on retinal OCT B-scans
- Benchmark & evaluation code: https://github.com/ronnief1/RetinaGroundBench
Results
On the RetinaGroundBench visual-grounding benchmark (13 biomarker classes, AggIoU with t₁=0.1, t₂=0.3), RetGrounder reaches a macro F1 of 33.9% (AggIoU 18.5%, precision 48.0%, recall 27.5%) — a 43% relative improvement in F1 over the best general-purpose model evaluated (InternVL3.5-30B-A3B, F1 23.7%), despite its smaller size.
Usage
from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
from PIL import Image
model = Qwen3VLForConditionalGeneration.from_pretrained(
"ronnief1/RetGrounder", dtype="auto", device_map="auto")
processor = AutoProcessor.from_pretrained("ronnief1/RetGrounder")
image = Image.open("oct_bscan.jpg").convert("RGB")
prompt = ('Locate every instance that belongs to the following categories: '
'"intraretinal fluid, drusen". Report bbox coordinates in JSON format.')
messages = [{"role": "user", "content": [
{"type": "image", "image": image},
{"type": "text", "text": prompt}]}]
inputs = processor.apply_chat_template(
messages, tokenize=True, add_generation_prompt=True,
return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=1024)
print(processor.batch_decode(
[o[len(i):] for i, o in zip(inputs.input_ids, out)],
skip_special_tokens=True)[0])
Predicted boxes are in a 0–1000 normalized coordinate frame (scale back to pixels with the image width/height).
License
The RetGrounder weights are released under Apache-2.0, inherited from the base model Qwen/Qwen3-VL-8B-Instruct (also Apache-2.0). The RetinaGroundBench annotations are released separately under CC BY 4.0, and the underlying B-scans derive from the public OCTDL and OLIVES datasets — please also observe the terms of those original datasets.
Citation
If you use RetGrounder or RetinaGroundBench, please cite the paper:
@misc{fecso2026visualgroundingbenchmarkretinal,
title={A Visual Grounding Benchmark for Retinal OCT with Multimodal Large Language Models},
author={Ronald Fecso and José Morano and Botond Fazekas and Paul Widmann-Sedlnitzky and Dorota Himel and Laetita Hinterhuber and Virginia Mares and Andreas Pollreisz and Hrvoje Bogunović},
year={2026},
eprint={2610.31205},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2610.31205},
}