| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3-VL-8B-Instruct |
| library_name: transformers |
| pipeline_tag: image-text-to-text |
| language: |
| - en |
| tags: |
| - medical |
| - ophthalmology |
| - retinal-oct |
| - visual-grounding |
| - vision-language |
| --- |
| |
| # RetGrounder |
|
|
| **RetGrounder** is an 8B multimodal large language model fine-tuned for **visual grounding of biomarkers in retinal optical coherence tomography (OCT)**, introduced with the **RetinaGroundBench** benchmark. It is a LoRA fine-tune of [`Qwen/Qwen3-VL-8B-Instruct`](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct), trained with an OCT-specific strategy that integrates grounding and textual supervision, then merged into the base weights. |
|
|
| - **Base model:** `Qwen/Qwen3-VL-8B-Instruct` |
| - **Tasks:** biomarker visual grounding (bounding boxes) and presence VQA on retinal OCT B-scans |
| - **Benchmark & evaluation code:** https://github.com/ronnief1/RetinaGroundBench |
|
|
| ## Results |
|
|
| On the RetinaGroundBench visual-grounding benchmark (13 biomarker classes, AggIoU with t₁=0.1, t₂=0.3), RetGrounder reaches a **macro F1 of 33.9%** (AggIoU 18.5%, precision 48.0%, recall 27.5%) — a **43% relative improvement** in F1 over the best general-purpose model evaluated (InternVL3.5-30B-A3B, F1 23.7%), despite its smaller size. |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import Qwen3VLForConditionalGeneration, AutoProcessor |
| from PIL import Image |
| |
| model = Qwen3VLForConditionalGeneration.from_pretrained( |
| "ronnief1/RetGrounder", dtype="auto", device_map="auto") |
| processor = AutoProcessor.from_pretrained("ronnief1/RetGrounder") |
| |
| image = Image.open("oct_bscan.jpg").convert("RGB") |
| prompt = ('Locate every instance that belongs to the following categories: ' |
| '"intraretinal fluid, drusen". Report bbox coordinates in JSON format.') |
| messages = [{"role": "user", "content": [ |
| {"type": "image", "image": image}, |
| {"type": "text", "text": prompt}]}] |
| inputs = processor.apply_chat_template( |
| messages, tokenize=True, add_generation_prompt=True, |
| return_dict=True, return_tensors="pt").to(model.device) |
| out = model.generate(**inputs, max_new_tokens=1024) |
| print(processor.batch_decode( |
| [o[len(i):] for i, o in zip(inputs.input_ids, out)], |
| skip_special_tokens=True)[0]) |
| ``` |
|
|
| Predicted boxes are in a **0–1000 normalized coordinate frame** (scale back to pixels with the image width/height). |
|
|
| ## License |
|
|
| The RetGrounder weights are released under **Apache-2.0**, inherited from the base model [`Qwen/Qwen3-VL-8B-Instruct`](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct) (also Apache-2.0). The **RetinaGroundBench annotations** are released separately under **CC BY 4.0**, and the underlying B-scans derive from the public **OCTDL** and **OLIVES** datasets — please also observe the terms of those original datasets. |
|
|
| ## Citation |
|
|
| If you use RetGrounder or RetinaGroundBench, please cite the paper: |
|
|
| ```bibtex |
| @misc{fecso2026visualgroundingbenchmarkretinal, |
| title={A Visual Grounding Benchmark for Retinal OCT with Multimodal Large Language Models}, |
| author={Ronald Fecso and José Morano and Botond Fazekas and Paul Widmann-Sedlnitzky and Dorota Himel and Laetita Hinterhuber and Virginia Mares and Andreas Pollreisz and Hrvoje Bogunović}, |
| year={2026}, |
| eprint={2610.31205}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CV}, |
| url={https://arxiv.org/abs/2610.31205}, |
| } |
| ``` |
|
|
