---
license: apache-2.0
base_model: Qwen/Qwen3-VL-8B-Instruct
library_name: transformers
pipeline_tag: image-text-to-text
language:
- en
tags:
- medical
- ophthalmology
- retinal-oct
- visual-grounding
- vision-language
---

# RetGrounder

**RetGrounder** is an 8B multimodal large language model fine-tuned for **visual grounding of biomarkers in retinal optical coherence tomography (OCT)**, introduced with the **RetinaGroundBench** benchmark. It is a LoRA fine-tune of [`Qwen/Qwen3-VL-8B-Instruct`](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct), trained with an OCT-specific strategy that integrates grounding and textual supervision, then merged into the base weights.

- **Base model:** `Qwen/Qwen3-VL-8B-Instruct`
- **Tasks:** biomarker visual grounding (bounding boxes) and presence VQA on retinal OCT B-scans
- **Benchmark & evaluation code:** https://github.com/ronnief1/RetinaGroundBench

## Results

On the RetinaGroundBench visual-grounding benchmark (13 biomarker classes, AggIoU with t₁=0.1, t₂=0.3), RetGrounder reaches a **macro F1 of 33.9%** (AggIoU 18.5%, precision 48.0%, recall 27.5%) — a **43% relative improvement** in F1 over the best general-purpose model evaluated (InternVL3.5-30B-A3B, F1 23.7%), despite its smaller size.

## Usage

```python
from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
from PIL import Image

model = Qwen3VLForConditionalGeneration.from_pretrained(
    "ronnief1/RetGrounder", dtype="auto", device_map="auto")
processor = AutoProcessor.from_pretrained("ronnief1/RetGrounder")

image = Image.open("oct_bscan.jpg").convert("RGB")
prompt = ('Locate every instance that belongs to the following categories: '
          '"intraretinal fluid, drusen". Report bbox coordinates in JSON format.')
messages = [{"role": "user", "content": [
    {"type": "image", "image": image},
    {"type": "text", "text": prompt}]}]
inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=1024)
print(processor.batch_decode(
    [o[len(i):] for i, o in zip(inputs.input_ids, out)],
    skip_special_tokens=True)[0])
```

Predicted boxes are in a **0–1000 normalized coordinate frame** (scale back to pixels with the image width/height).

## License

The RetGrounder weights are released under **Apache-2.0**, inherited from the base model [`Qwen/Qwen3-VL-8B-Instruct`](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct) (also Apache-2.0). The **RetinaGroundBench annotations** are released separately under **CC BY 4.0**, and the underlying B-scans derive from the public **OCTDL** and **OLIVES** datasets — please also observe the terms of those original datasets.

## Citation

If you use RetGrounder or RetinaGroundBench, please cite the paper:

```bibtex
@misc{fecso2026visualgroundingbenchmarkretinal,
      title={A Visual Grounding Benchmark for Retinal OCT with Multimodal Large Language Models}, 
      author={Ronald Fecso and José Morano and Botond Fazekas and Paul Widmann-Sedlnitzky and Dorota Himel and Laetita Hinterhuber and Virginia Mares and Andreas Pollreisz and Hrvoje Bogunović},
      year={2026},
      eprint={2610.31205},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2610.31205}, 
}
```