adidukrembzuai commited on
Commit
c57337c
Β·
verified Β·
1 Parent(s): fbb0b69

Add model card

Browse files
Files changed (3) hide show
  1. .gitattributes +1 -0
  2. README.md +146 -0
  3. overview.png +3 -0
.gitattributes CHANGED
@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  tokenizer.json filter=lfs diff=lfs merge=lfs -text
 
 
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  tokenizer.json filter=lfs diff=lfs merge=lfs -text
37
+ overview.png filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,146 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: transformers
4
+ pipeline_tag: image-text-to-text
5
+ base_model: Qwen/Qwen3.5-9B
6
+ tags:
7
+ - medical
8
+ - multimodal
9
+ - grounding
10
+ - visual-grounding
11
+ - multi-image
12
+ - reasoning
13
+ - radiology
14
+ - mri
15
+ - ct
16
+ ---
17
+
18
+ # MedChain-R1-9B: Reasoning-Driven Sequential Grounding for Medical Images
19
+
20
+ [![Code](https://img.shields.io/badge/GitHub-Code-black)](https://github.com/adinathdukre/MedChain-R1)
21
+ [![Model](https://img.shields.io/badge/πŸ€—-MedChain--R1--4B-blue)](https://huggingface.co/adidukrembzuai/MedChain-R1-4B)
22
+ [![Model](https://img.shields.io/badge/πŸ€—-MedChain--R1--9B-blue)](https://huggingface.co/adidukrembzuai/MedChain-R1-9B)
23
+ [![License](https://img.shields.io/badge/License-Apache%202.0-green.svg)](https://opensource.org/licenses/Apache-2.0)
24
+
25
+ <p align="center">
26
+ <img src="overview.png" alt="MedChain-R1 overview" width="900"/>
27
+ </p>
28
+
29
+ **MedChain-R1-9B** is a 9B vision-language model for **sequential medical image grounding**: given a short sequence of
30
+ medical images and a question, it writes a short rationale in `<think>…</think>` and then returns one bounding box
31
+ `<answer>(x1,y1),(x2,y2)</answer>` in pixel coordinates of the target image. It is built on
32
+ [Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) in two stages:
33
+
34
+ 1. **CoT-SFT** on the MedChain rationale corpus: 152,396 MedSG training sequences, each with a box-conditioned rationale
35
+ (FlemingVL-38B draft, refined by Qwen3.5-397B-A17B with the annotated box as privileged input).
36
+ 2. **GRPO** with an IoU reward on a held-out 30,000-prompt pool (3,750 per task), 8 rollouts per prompt, frozen vision tower.
37
+
38
+ Training stage of this checkpoint: CoT-SFT (checkpoint-2500) β†’ GRPO on RL30K, two epochs (step 2500 of the second epoch, seed 45).
39
+
40
+ On MedSG-Bench (8 tasks, 10 modalities, 9,630 test examples), MedChain-R1-9B reaches **77.54 mIoU / 84.87 Acc@0.5**.
41
+
42
+ ---
43
+
44
+ ## πŸ† Benchmark Performance
45
+
46
+ ### MedSG-Bench (IoU per task, average IoU / Acc@0.5)
47
+
48
+ | Model | Size | RDG | NRDG | MV | OT | Concept | Patch | Cross-modal | Referring | **Avg IoU / Acc@0.5** |
49
+ |---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
50
+ | Qwen2.5-VL | 72B | 4.37 | 3.46 | 7.22 | 13.11 | 10.33 | 26.45 | 16.32 | 20.19 | 13.35 / 6.12 |
51
+ | InternVL3 | 78B | 0.20 | 0.53 | 6.35 | 13.03 | 3.57 | 11.81 | 3.34 | 12.76 | 6.44 / 2.90 |
52
+ | Migician | 7B | 15.26 | 14.49 | 18.16 | 21.38 | 14.23 | 28.87 | 21.41 | 25.30 | 20.29 / 11.39 |
53
+ | HuatuoGPT-Vision | 34B | 1.44 | 2.15 | 9.41 | 13.25 | 6.43 | 14.53 | 10.60 | 8.60 | 8.57 / 2.09 |
54
+ | MedSeq-Grounder | 7B | 83.29 | 83.72 | 55.03 | 62.10 | 74.11 | 85.25 | 78.77 | 60.43 | 72.55 / 79.71 |
55
+ | Qwen3.5 (zero-shot) | 4B / 9B / 27B | 0.0 | 0.0 | 8.3–8.6 | 26.7–28.4 | 0.3–1.5 | 1.6–2.7 | 0.3 | 0.8–1.7 | 4.4–4.5 / 3.0–3.2 |
56
+ | MedChain-R1-9B (CoT-SFT only) | 9B | 84.0 | 83.0 | 55.9 | 67.5 | 74.8 | 91.2 | 84.3 | 70.5 | 76.60 / 83.42 |
57
+ | MedChain-R1-4B | 4B | 83.5 | 82.0 | 56.6 | 66.7 | 71.8 | 90.3 | 84.0 | 70.2 | 75.97 / 83.67 |
58
+ | **MedChain-R1-9B (this model)** | 9B | 84.4 | 83.5 | 58.4 | 68.5 | 75.5 | 91.3 | 84.8 | 72.0 | 77.54 / 84.87 |
59
+
60
+ > RDG / NRDG = registered / non-registered difference grounding, MV = multi-view, OT = object tracking.
61
+ > Baselines are the numbers reported for MedSG-Bench (zero-shot general and medical MLLMs; MedSeq-Grounder is trained on MedSG-188K).
62
+ > MedChain-R1 rows are recomputed from per-example predictions: greedy decoding, batch size 1, bf16, 1,200 new tokens.
63
+ > Benchmarked on AMD MI210 GPUs.
64
+
65
+ ### Supported tasks
66
+
67
+ | Task | What the model grounds |
68
+ |---|---|
69
+ | Registered / non-registered difference | the region that changed between two scans |
70
+ | Multi-view | the same structure seen from another view |
71
+ | Object tracking | a lesion or structure across slices or time points |
72
+ | Concept | the region matching a described concept across images |
73
+ | Patch | where a given patch lies in the target image |
74
+ | Cross-modal | the same finding in another modality (e.g. FLAIR β†’ DWI) |
75
+ | Referring | the region a text description refers to |
76
+
77
+ ---
78
+
79
+ ## πŸš€ Quick Start
80
+
81
+ ### Installation
82
+
83
+ ```bash
84
+ pip install "transformers==5.2.0" torch qwen-vl-utils==0.0.14
85
+ ```
86
+
87
+ ### Basic Usage
88
+
89
+ ```python
90
+ import torch
91
+ from transformers import AutoModelForImageTextToText, AutoProcessor
92
+ from qwen_vl_utils import process_vision_info
93
+
94
+ repo = "adidukrembzuai/MedChain-R1-9B"
95
+ model = AutoModelForImageTextToText.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto").eval()
96
+ processor = AutoProcessor.from_pretrained(repo)
97
+
98
+ question = "Locate the region that differs between the first and the second image."
99
+ messages = [{
100
+ "role": "user",
101
+ "content": [
102
+ {"type": "image", "image": "scan_t0.png"},
103
+ {"type": "image", "image": "scan_t1.png"},
104
+ {"type": "text", "text": question + "\nFirst output the thinking process in <think> </think> tags "
105
+ "and then output the bounding box in <answer>(x1,y1),(x2,y2)</answer> tags."},
106
+ ],
107
+ }]
108
+
109
+ text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
110
+ image_inputs, video_inputs = process_vision_info(messages)
111
+ inputs = processor(text=[text], images=image_inputs, videos=video_inputs, return_tensors="pt").to(model.device)
112
+
113
+ with torch.inference_mode():
114
+ out = model.generate(**inputs, max_new_tokens=1200, do_sample=False)
115
+ print(processor.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
116
+ # <think> ... </think><answer>(x1,y1),(x2,y2)</answer>
117
+ ```
118
+
119
+ **Notes**
120
+
121
+ * Append the instruction above to every question; it is the prompt the model was trained and evaluated with.
122
+ * Generate one example at a time: batched generation produced degenerate outputs for some checkpoints.
123
+ * `eval/eval_bench.py` in the [code repository](https://github.com/adinathdukre/MedChain-R1) is the exact evaluation script.
124
+
125
+ ---
126
+
127
+ ## πŸ“¦ Model Family
128
+
129
+ | Model | Base | MedSG-Bench mIoU / Acc@0.5 | Best for |
130
+ |---|---|---:|---|
131
+ | [MedChain-R1-9B](https://huggingface.co/adidukrembzuai/MedChain-R1-9B) | Qwen3.5-9B | 77.54 / 84.87 | highest accuracy β€” **recommended** |
132
+ | [MedChain-R1-4B](https://huggingface.co/adidukrembzuai/MedChain-R1-4B) | Qwen3.5-4B | 75.97 / 83.67 | resource-constrained environments |
133
+
134
+ ---
135
+
136
+ ## ⚠️ Limitations
137
+
138
+ Research artefact. Trained and evaluated on MedSG data only; **not validated for diagnosis or any clinical use**. The
139
+ rationales imitate teacher-written, box-conditioned explanations and are not a faithful account of how the box was chosen.
140
+
141
+ ---
142
+
143
+ ## πŸ“œ License
144
+
145
+ Released under the **Apache License 2.0**. Built on Qwen3.5 (Apache-2.0); the GRPO code derives from open-r1-multimodal and
146
+ UniVG-R1 (Apache-2.0). MedSG images keep their original licences and are not included.
overview.png ADDED

Git LFS Details

  • SHA256: dde135fc315ad4513badbcca32e75b16f65e65c5ed03f6ecf6e4acc7dc3668a0
  • Pointer size: 131 Bytes
  • Size of remote file: 798 kB