RetGrounder

RetGrounder is an 8B multimodal large language model fine-tuned for visual grounding of biomarkers in retinal optical coherence tomography (OCT), introduced with the RetinaGroundBench benchmark. It is a LoRA fine-tune of Qwen/Qwen3-VL-8B-Instruct, trained with an OCT-specific strategy that integrates grounding and textual supervision, then merged into the base weights.

• Base model: Qwen/Qwen3-VL-8B-Instruct
• Tasks: biomarker visual grounding (bounding boxes) and presence VQA on retinal OCT B-scans
• Benchmark & evaluation code: https://github.com/ronnief1/RetinaGroundBench

Results

On the RetinaGroundBench visual-grounding benchmark (13 biomarker classes, AggIoU with t₁=0.1, t₂=0.3), RetGrounder reaches a macro F1 of 33.9% (AggIoU 18.5%, precision 48.0%, recall 27.5%) — a 43% relative improvement in F1 over the best general-purpose model evaluated (InternVL3.5-30B-A3B, F1 23.7%), despite its smaller size.

Usage

from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
from PIL import Image

model = Qwen3VLForConditionalGeneration.from_pretrained(
    "ronnief1/RetGrounder", dtype="auto", device_map="auto")
processor = AutoProcessor.from_pretrained("ronnief1/RetGrounder")

image = Image.open("oct_bscan.jpg").convert("RGB")
prompt = ('Locate every instance that belongs to the following categories: '
          '"intraretinal fluid, drusen". Report bbox coordinates in JSON format.')
messages = [{"role": "user", "content": [
    {"type": "image", "image": image},
    {"type": "text", "text": prompt}]}]
inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=1024)
print(processor.batch_decode(
    [o[len(i):] for i, o in zip(inputs.input_ids, out)],
    skip_special_tokens=True)[0])

Predicted boxes are in a 0–1000 normalized coordinate frame (scale back to pixels with the image width/height).

License

The RetGrounder weights are released under Apache-2.0, inherited from the base model Qwen/Qwen3-VL-8B-Instruct (also Apache-2.0). The RetinaGroundBench annotations are released separately under CC BY 4.0, and the underlying B-scans derive from the public OCTDL and OLIVES datasets — please also observe the terms of those original datasets.

Citation

If you use RetGrounder or RetinaGroundBench, please cite the paper:

@misc{fecso2026visualgroundingbenchmarkretinal,
      title={A Visual Grounding Benchmark for Retinal OCT with Multimodal Large Language Models}, 
      author={Ronald Fecso and José Morano and Botond Fazekas and Paul Widmann-Sedlnitzky and Dorota Himel and Laetita Hinterhuber and Virginia Mares and Andreas Pollreisz and Hrvoje Bogunović},
      year={2026},
      eprint={2610.31205},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2610.31205}, 
}