---
license: apache-2.0
library_name: transformers
pipeline_tag: image-text-to-text
base_model: Qwen/Qwen3.5-9B
tags:
- medical
- multimodal
- grounding
- visual-grounding
- multi-image
- reasoning
- chain-of-thought
- radiology
- mri
- ct
- arxiv:2611.00041
---
# MedChain-R1-9B: Think Before You Locate: Clinical Chain-of-Thought Supervision for Sequential Medical Image Grounding
[](https://arxiv.org/abs/2611.00041)
[](https://github.com/adinathdukre/MedChain-R1)
[](https://huggingface.co/adidukrembzuai/MedChain-R1-4B)
[](https://huggingface.co/adidukrembzuai/MedChain-R1-9B)
[](https://opensource.org/licenses/Apache-2.0)
**MedChain-R1-9B** is a 9B vision-language model for **sequential medical image grounding**: given a short sequence of
medical images and a clinical question, it first writes an image-grounded rationale in `…` and then
returns one bounding box `(x1,y1),(x2,y2)` in pixel coordinates. It is built on
[Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) with **deliberative chain-of-thought supervision
(dCoT-SFT)** on the **MedChain** rationale corpus, followed by **GRPO** with an IoU + format reward.
**MedChain** contains 182,573 filtered rationales over the 8 MedSG-Bench tasks and 10 imaging modalities. A medical VLM
(FlemingVL-38B) drafts a caption-like rationale; a multimodal refiner (Qwen3.5-397B-A17B) then re-examines the images,
question and annotated box and writes a box-conditioned explanation with explicit cross-image reasoning.
---
## 🔑 Key findings (from the paper)
1. **Reasoning supervision sets a new SOTA.** dCoT-SFT outperforms MedSeq-Grounder at every scale, reaching 74.96, 76.60
and 77.24 mIoU at 4B, 9B and 27B; the 27B model improves over the strongest prior model by 4.69 mIoU and 5.49 Acc@0.5.
2. **The gains depend on the task.** Against an answer-only 4B model, dCoT-SFT improves mIoU by 3.56 (95% CI 3.13–4.00):
+14.7–16.1 on difference grounding and +6.5 on object tracking, but −2.2 to −3.2 on multi-view, patch and referring.
3. **Predictions depend on the reasoning context.** Forcing an immediate answer lowers mIoU by 2.9; conditioning on a
rationale from another task lowers it by 12.2.
4. **GRPO adds little after dCoT-SFT.** +0.37 mIoU on the benchmark (4B), with task-dependent effects.
---
## 🏆 Benchmark Performance
### MedSG-Bench (Table 1 of the paper)
Each cell is **mIoU / Acc@0.5** in percent on all 9,630 test questions. IDG: registered (RDG) and non-registered (NRDG)
difference grounding. ICG: multi-view (MV), object tracking (OT), concept (VCG), patch (VPG), cross-modal (CMG) and
referring (RG) grounding. **Bold**: our results that exceed MedSeq-Grounder, the strongest prior model, on the same metric.
| Group | Model | Size | RDG | NRDG | MV | OT | VCG | VPG | CMG | RG | **Avg.** |
|---|---|---|---|---|---|---|---|---|---|---|---|
| General | Qwen2.5-VL | 72B | 4.37 / 2.60 | 3.46 / 0.80 | 7.22 / 2.78 | 13.11 / 7.70 | 10.33 / 3.50 | 26.45 / 6.30 | 16.32 / 7.00 | 20.19 / 14.10 | 13.35 / 6.12 |
| General | InternVL3 | 78B | 0.20 / 0.00 | 0.53 / 0.00 | 6.35 / 2.43 | 13.03 / 8.00 | 3.57 / 0.90 | 11.81 / 2.50 | 3.34 / 0.85 | 12.76 / 8.10 | 6.44 / 2.90 |
| General | LLaVA-OneVision | 72B | 2.58 / 0.80 | 2.87 / 0.90 | 11.74 / 1.39 | 9.61 / 2.30 | 10.95 / 3.30 | 32.38 / 20.30 | 16.24 / 5.40 | 15.43 / 6.68 | 13.21 / 5.18 |
| General | Migician | 7B | 15.26 / 7.80 | 14.49 / 6.10 | 18.16 / 7.84 | 21.38 / 14.90 | 14.23 / 7.20 | 28.87 / 13.70 | 21.41 / 12.15 | 25.30 / 18.02 | 20.29 / 11.39 |
| General | Qwen3.5 (zero-shot) | 27B | 0.00 / 0.00 | 0.00 / 0.00 | 8.48 / 4.88 | 28.39 / 24.50 | 0.25 / 0.00 | 2.50 / 0.20 | 0.25 / 0.00 | 0.82 / 0.00 | 4.42 / 3.15 |
| Medical | HuatuoGPT-Vision | 34B | 1.44 / 0.00 | 2.15 / 0.00 | 9.41 / 1.65 | 13.25 / 8.30 | 6.43 / 0.70 | 14.53 / 1.40 | 10.60 / 2.60 | 8.60 / 1.75 | 8.57 / 2.09 |
| Medical | MedGemma | 27B | 0.01 / 0.00 | 0.06 / 0.00 | 5.29 / 2.26 | 22.26 / 16.10 | 2.27 / 0.00 | 3.11 / 0.30 | 2.92 / 0.35 | 5.35 / 0.61 | 4.94 / 2.14 |
| Medical | MedGRPO-7B-RL | 7B | 0.09 / 0.00 | 0.06 / 0.00 | 3.68 / 1.48 | 17.63 / 15.00 | 4.57 / 0.40 | 18.44 / 1.20 | 0.78 / 0.15 | 9.31 / 3.37 | 6.27 / 2.45 |
| Medical | MedSeq-Grounder | 7B | 83.29 / 93.20 | 83.72 / 94.10 | 55.03 / 60.19 | 62.10 / 67.20 | 74.11 / 82.60 | 85.25 / 98.80 | 78.77 / 82.75 | 60.43 / 65.59 | 72.55 / 79.71 |
| Ours | MedChain-R1 (dCoT-SFT) | 4B | 82.86 / **93.30** | 81.30 / 91.20 | 54.01 / 58.97 | **64.77** / **71.00** | 70.64 / 78.80 | **90.21** / **99.70** | **83.39** / **87.00** | **69.67** / **79.28** | **74.96** / **82.37** |
| Ours | MedChain-R1 (+GRPO) | 4B | **83.82** / **94.50** | 83.21 / 92.70 | 54.80 / 59.93 | **67.90** / **76.20** | 66.09 / 72.10 | **89.43** / **99.80** | **83.59** / **87.55** | **70.72** / **80.30** | **75.32** / **82.89** |
| Ours | MedChain-R1 (dCoT-SFT) | 9B | **84.00** / 93.10 | 83.02 / 91.60 | **55.87** / **60.98** | **67.46** / **75.00** | **74.64** / **83.20** | **91.22** / **99.90** | **84.16** / **87.40** | **70.77** / **78.95** | **76.60** / **83.55** |
| Ours | MedChain-R1 (+GRPO) | 9B | **84.16** / 93.20 | 83.12 / 91.40 | **56.98** / **62.02** | **67.65** / **74.90** | **74.80** / **82.80** | **91.22** / **99.80** | **84.82** / **88.25** | **70.53** / **79.22** | **76.90** / **83.82** |
| Ours | MedChain-R1 (dCoT-SFT) | 27B | 81.87 / 93.00 | 80.97 / 91.60 | **60.14** / **66.72** | **69.53** / **78.30** | **74.64** / **83.30** | **91.87** / **99.90** | **84.24** / **88.15** | **72.51** / **81.98** | **77.24** / **85.20** |
> Baselines are evaluated zero-shot with one unified prompt (MedSeq-Grounder is the supervised SOTA). Full results for all
> baselines are in Table 10 of the paper. Benchmarked on AMD MI210 GPUs.
### This checkpoint
The paper's 9B GRPO row (76.90 / 83.82) comes from a different GRPO checkpoint. **This repository
holds:** dCoT-SFT checkpoint-2500 → GRPO (IoU + format reward) on the held-out RL30K pool, a second pass (seed 45) continuing the seed-42 GRPO model, step 2500. Evaluated with the same protocol (all 9,630 questions, greedy decoding, batch size 1, bf16,
1,200 new tokens), it scores:
| RDG | NRDG | MV | OT | VCG | VPG | CMG | RG | **mIoU / Acc@0.5** |
|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| 84.41 | 83.53 | 58.44 | 68.52 | 75.49 | 91.34 | 84.79 | 72.01 | **77.54 / 84.87** |
Per-task cells are mIoU. The per-example predictions are in the [code repository](https://github.com/adinathdukre/MedChain-R1).
---
## 🚀 Quick Start
### Installation
```bash
pip install "transformers==5.2.0" torch qwen-vl-utils==0.0.14
```
### Basic Usage
```python
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from qwen_vl_utils import process_vision_info
repo = "adidukrembzuai/MedChain-R1-9B"
model = AutoModelForImageTextToText.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto").eval()
processor = AutoProcessor.from_pretrained(repo)
question = "Compare these two images carefully and give me the coordinates of their difference."
messages = [{
"role": "user",
"content": [
{"type": "image", "image": "scan_1.png"},
{"type": "image", "image": "scan_2.png"},
{"type": "text", "text": question + "\nFirst output the thinking process in tags "
"and then output the bounding box in (x1,y1),(x2,y2) tags."},
],
}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, videos=video_inputs, return_tensors="pt").to(model.device)
with torch.inference_mode():
out = model.generate(**inputs, max_new_tokens=1200, do_sample=False)
print(processor.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
# ... (x1,y1),(x2,y2)
```
**Notes**
* Append the instruction above to every question; it is the prompt the model was trained and evaluated with.
* Generate one example at a time: batched generation produced degenerate outputs for some checkpoints.
* Use the processor bundled in this repository (the dCoT-SFT processor). The paper (Section 5.3) shows that an exported
processor with a lower image-resolution limit costs 3–5 points.
* `eval/eval_bench.py` in the [code repository](https://github.com/adinathdukre/MedChain-R1) is the exact evaluation script.
---
## 📦 Model Family
| Model | Base | This checkpoint, MedSG-Bench mIoU / Acc@0.5 | Best for |
|---|---|---:|---|
| [MedChain-R1-9B](https://huggingface.co/adidukrembzuai/MedChain-R1-9B) | Qwen3.5-9B | 77.54 / 84.87 | highest accuracy — **recommended** |
| [MedChain-R1-4B](https://huggingface.co/adidukrembzuai/MedChain-R1-4B) | Qwen3.5-4B | 75.97 / 83.67 | resource-constrained environments |
---
## ⚠️ Limitations
Research artefact, **not validated for autonomous diagnosis or clinical deployment**. The MedChain rationales are
conditioned on the annotated box, so they explain a known answer and may be post-hoc; they should not be read as a
faithful record of the model's internal computation. The rationales can inherit errors and biases from their teacher
models and were reviewed in a single-expert audit of 2,000 examples (98.25% accepted). Evaluation is limited to MedSG-Bench.
---
## 📄 Citation
If you use MedChain-R1 in your research, please cite our paper:
```bibtex
@article{dukre2026think,
title = {Think Before You Locate: Clinical Chain-of-Thought Supervision for Sequential Medical Image Grounding},
author = {Dukre, Adinath Madhavrao and Kolli, Govinda and Lu, Yifan and Zou, Ziyun and Wasi, Azmine Toushik and
Atito, Sara and Bozorgtabar, Behzad and Mahapatra, Dwarikanath and Awais, Muhammad and Razzak, Imran},
journal = {arXiv preprint arXiv:2611.00041},
year = {2026}
}
```
---
## 📜 License
Released under the **Apache License 2.0**. Built on Qwen3.5 (Apache-2.0). MedSG images keep their original licences and
are not included.