yuchenwu73 commited on
Commit
a7ffa7e
·
1 Parent(s): 774e461

Card: English first with Chinese section; exact training prompts

Browse files

The usage snippet dropped the spaces inside the coordinate lists
([[x1,y1],...] instead of [[x1, y1], ...]). Training and evaluation use the
spaced form, and the difference changes tokenization, so anyone copying the
snippet was not prompting the model the way it was trained.

Files changed (1) hide show
  1. README.md +100 -36
README.md CHANGED
@@ -13,37 +13,41 @@ tags:
13
  - qwen3-vl
14
  ---
15
 
 
 
16
  # GeoBox-R1
17
 
18
- 统一边界框级遥感视觉定位模型,同时输出水平框(HBB)与旋转框(OBB)。
 
19
 
20
  > **GeoBox-R1: Curriculum-Guided SFT and Geometric RL for Unified Box-Level Remote Sensing Visual Grounding**
21
  > Chenxi Lan\*, Yuchen Wu\*, Minghang Zhou, Tianyu Li, Zhihao Qiu, Guoqing Wang
22
- > AAAI 2027 投稿中。(\* 共同一作)
23
  >
24
- > [项目主页](https://yuchenwu73.github.io/GeoBox-R1/) ·
25
- > [代码](https://github.com/yuchenwu73/GeoBox-R1) ·
26
- > [训练数据](https://huggingface.co/datasets/yuchenwu73/GeoBox-R1-Data) ·
27
- > [Stage-1 SFT 检查点](https://huggingface.co/yuchenwu73/GeoBox-R1-SFT)
28
 
29
- 本仓库是**最终模型**(Stage-1 SFT + Stage-2 GDPO 后的合并权重)。
30
- 只需要第一阶段结果请用 [`GeoBox-R1-SFT`](https://huggingface.co/yuchenwu73/GeoBox-R1-SFT)。
31
 
32
- ## 训练流程
33
 
34
- 基座 **Qwen3-VL-4B-Instruct**,两阶段:
35
 
36
- 1. **课程式 SFT** — 训练数据按 HBB → OBB → HBB-to-OBB CoT 由易到难排列。
37
- LoRA(rank 16,alpha 32),冻结视觉编码器与 merger,lr `1e-4`,1 epoch,2× RTX 4090。
38
- 2. **几何强化学习(GDPO)** — 在 SFT 检查点上用两个基于规则的几何奖励细化 OBB:
39
- Rotated IoU 与自适应 Wasserstein 距离(λ 各 0.5)。
40
- G=8 rollouts,β=0.02,τ_c=8,lr `5e-6`,1 epoch,3× A100 40G(1 个 vLLM rollout 服务 + 2 个 GDPO worker)。
 
41
 
42
- ## 结果
43
 
44
- 7 个 HBB 与 3 个 OBB 评测集,三项指标的宏平均均取得最佳。
45
 
46
- ### HBB(7 个评测集)
47
 
48
  | Model | Params | DIOR-Test | DIOR-Val | RSVG-Test | RSVG-Val | GeoChat* | VRSBench* | AVVG | **Avg.** |
49
  | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
@@ -53,28 +57,30 @@ tags:
53
  | GeoBox-R1 (SFT) | 4B | 74.91 | 74.21 | 48.57 | 46.29 | 61.39 | 64.45 | 30.33 | 57.17 |
54
  | **GeoBox-R1** | **4B** | 76.61 | 75.11 | **51.26** | **48.38** | 61.13 | **66.84** | **32.14** | **58.78** |
55
 
56
- (Acc@0.5。Acc@0.7 与 mIoU 的宏平均分别为 **42.22** 与 **50.39**,同样最佳。)
57
 
58
- ### OBB(3 个评测集)
59
 
60
- | Model | Params | GeoChat* @0.5 / @0.7 / mRIoU | VRSBench* | AVVG | **Avg.** |
61
  | --- | --- | --- | --- | --- | --- |
62
  | InternVL3 (SFT) | 8B | 49.79 / 23.10 / 41.66 | 42.01 / 20.65 / 40.68 | 15.43 / 7.40 / 15.39 | 35.74 / 17.05 / 32.58 |
63
  | GeoGround | 7B | 58.72 / 25.49 / 46.89 | 53.26 / 29.82 / 48.35 | 13.89 / 4.10 / 15.64 | 41.96 / 19.81 / 36.96 |
64
  | GeoBox-R1 (SFT) | 4B | 55.96 / 30.45 / 45.45 | 51.14 / 27.69 / 45.91 | 22.23 / 15.07 / 19.20 | 43.11 / 24.40 / 36.85 |
65
  | **GeoBox-R1** | **4B** | **60.56 / 35.19 / 48.92** | **56.61 / 30.55 / 49.43** | **24.79 / 16.89 / 21.18** | **47.32 / 27.55 / 39.85** |
66
 
67
- 以 4B 参数超过 7B–8B 的现有最优模型:HBB 领先 **3.68/4.89/2.61** 点,OBB 领先 **5.36/7.74/2.89** 点,
68
- 在更严格的 Acc@0.7 上增益最大。
69
 
70
- **OBB-only RL 不牺牲 HBB**:GDPO 只用 OBB 样本训练,HBB 宏平均反而从 57.17/40.33/48.82 升到
71
- 58.78/42.22/50.39 —— Acc@0.5 在 7 个集合中的 6 个上升,Acc@0.7 与 mIoU 全部 7 个上升。
 
72
 
73
- ## 使用
74
 
75
- 模型使用 HBB / OBB 两种提示词,输出 JSON 格式的框坐标。
 
76
 
77
- ```python
78
  from transformers import AutoModelForImageTextToText, AutoProcessor
79
  from PIL import Image
80
 
@@ -85,23 +91,38 @@ processor = AutoProcessor.from_pretrained(model_id)
85
  image = Image.open("scene.png")
86
  expression = "the brown suv on the right"
87
 
88
- # 旋转框(OBB)
89
- prompt = (f"Locate the instance that matches the description: [{expression}]. "
90
- "Report oriented bbox coordinates in following JSON format:\n"
91
- "```json\n[\n\t{\"oriented_bbox\": [[x1,y1],[x2,y2],[x3,y3],[x4,y4]]}\n]\n```")
 
 
 
92
 
93
  messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": prompt}]}]
94
  text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
95
  inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)
96
  out = model.generate(**inputs, max_new_tokens=256)
97
  print(processor.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
98
- ```
99
 
100
- 水平框把提示词换成 `Report horizontal bbox coordinates ...` 与 `{"horizontal_bbox": [x1,y1,x2,y2]}` 即可。
101
 
102
- 也可用 [ms-swift](https://github.com/modelscope/ms-swift)(训练所用框架)推理与部署。
 
 
 
 
 
 
 
103
 
104
- ## 引用
 
 
 
 
 
105
 
106
  ```bibtex
107
  @inproceedings{geoboxr1,
@@ -114,3 +135,46 @@ print(processor.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=T
114
  note = {Under review}
115
  }
116
  ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
13
  - qwen3-vl
14
  ---
15
 
16
+ **English** | [简体中文](#geobox-r1-中文)
17
+
18
  # GeoBox-R1
19
 
20
+ Unified box-level remote sensing visual grounding — one model producing both horizontal (HBB)
21
+ and oriented (OBB) bounding boxes.
22
 
23
  > **GeoBox-R1: Curriculum-Guided SFT and Geometric RL for Unified Box-Level Remote Sensing Visual Grounding**
24
  > Chenxi Lan\*, Yuchen Wu\*, Minghang Zhou, Tianyu Li, Zhihao Qiu, Guoqing Wang
25
+ > Under review at AAAI 2027. (\* equal contribution)
26
  >
27
+ > [Project page](https://yuchenwu73.github.io/GeoBox-R1/) ·
28
+ > [Code](https://github.com/yuchenwu73/GeoBox-R1) ·
29
+ > [Training data](https://huggingface.co/datasets/yuchenwu73/GeoBox-R1-Data) ·
30
+ > [Stage-1 SFT checkpoint](https://huggingface.co/yuchenwu73/GeoBox-R1-SFT)
31
 
32
+ This repository holds the **final model** — merged weights after Stage-1 SFT and Stage-2 GDPO.
33
+ For the first stage alone, use [`GeoBox-R1-SFT`](https://huggingface.co/yuchenwu73/GeoBox-R1-SFT).
34
 
35
+ ## Training
36
 
37
+ Built on **Qwen3-VL-4B-Instruct** in two stages:
38
 
39
+ 1. **Curriculum-guided SFT** — training data ordered easy-to-hard: HBB → OBB → HBB-to-OBB CoT.
40
+ LoRA (rank 16, alpha 32), vision encoder and merger frozen, lr `1e-4`, 1 epoch, 2× RTX 4090.
41
+ 2. **Geometric RL (GDPO)** — refines OBB prediction on top of the SFT checkpoint with two
42
+ rule-based geometric rewards: Rotated IoU and an adaptive Wasserstein distance (λ = 0.5 each).
43
+ G = 8 rollouts, β = 0.02, τ_c = 8, lr `5e-6`, 1 epoch, 3× A100 40G (one vLLM rollout server
44
+ and two GDPO workers).
45
 
46
+ ## Results
47
 
48
+ Best macro averages on all three metrics, across 7 HBB and 3 OBB evaluation sets.
49
 
50
+ ### HBB (7 evaluation sets, Acc@0.5)
51
 
52
  | Model | Params | DIOR-Test | DIOR-Val | RSVG-Test | RSVG-Val | GeoChat* | VRSBench* | AVVG | **Avg.** |
53
  | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
 
57
  | GeoBox-R1 (SFT) | 4B | 74.91 | 74.21 | 48.57 | 46.29 | 61.39 | 64.45 | 30.33 | 57.17 |
58
  | **GeoBox-R1** | **4B** | 76.61 | 75.11 | **51.26** | **48.38** | 61.13 | **66.84** | **32.14** | **58.78** |
59
 
60
+ Acc@0.7 and mIoU macro averages are **42.22** and **50.39**, also the best.
61
 
62
+ ### OBB (3 evaluation sets, Acc@0.5 / Acc@0.7 / mRIoU)
63
 
64
+ | Model | Params | GeoChat* | VRSBench* | AVVG | **Avg.** |
65
  | --- | --- | --- | --- | --- | --- |
66
  | InternVL3 (SFT) | 8B | 49.79 / 23.10 / 41.66 | 42.01 / 20.65 / 40.68 | 15.43 / 7.40 / 15.39 | 35.74 / 17.05 / 32.58 |
67
  | GeoGround | 7B | 58.72 / 25.49 / 46.89 | 53.26 / 29.82 / 48.35 | 13.89 / 4.10 / 15.64 | 41.96 / 19.81 / 36.96 |
68
  | GeoBox-R1 (SFT) | 4B | 55.96 / 30.45 / 45.45 | 51.14 / 27.69 / 45.91 | 22.23 / 15.07 / 19.20 | 43.11 / 24.40 / 36.85 |
69
  | **GeoBox-R1** | **4B** | **60.56 / 35.19 / 48.92** | **56.61 / 30.55 / 49.43** | **24.79 / 16.89 / 21.18** | **47.32 / 27.55 / 39.85** |
70
 
71
+ At 4B parameters this beats the 7B–8B state of the art by **3.68/4.89/2.61** points on HBB and
72
+ **5.36/7.74/2.89** on OBB, with the largest gains at the stricter Acc@0.7 threshold.
73
 
74
+ **OBB-only RL does not cost HBB accuracy.** GDPO trains on OBB samples alone, yet the HBB macro
75
+ average rises from 57.17/40.33/48.82 to 58.78/42.22/50.39 — Acc@0.5 improves on 6 of 7 sets,
76
+ and Acc@0.7 and mIoU improve on all 7.
77
 
78
+ ## Usage
79
 
80
+ The prompts below are **byte-for-byte identical to the ones used in training and evaluation**,
81
+ including the spaces inside the coordinate lists. Changing the spacing changes tokenization.
82
 
83
+ ````python
84
  from transformers import AutoModelForImageTextToText, AutoProcessor
85
  from PIL import Image
86
 
 
91
  image = Image.open("scene.png")
92
  expression = "the brown suv on the right"
93
 
94
+ # Oriented box (OBB)
95
+ prompt = f"""Locate the instance that matches the description: [{expression}]. Report oriented bbox coordinates in following JSON format:
96
+ ```json
97
+ [
98
+ \t{{"oriented_bbox": [[x1, y1], [x2, y2], [x3, y3], [x4, y4]]}}
99
+ ]
100
+ ```"""
101
 
102
  messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": prompt}]}]
103
  text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
104
  inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)
105
  out = model.generate(**inputs, max_new_tokens=256)
106
  print(processor.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
107
+ ````
108
 
109
+ For horizontal boxes, use the same call with:
110
 
111
+ ````python
112
+ prompt = f"""Locate the instance that matches the description: [{expression}]. Report horizontal bbox coordinates in following JSON format:
113
+ ```json
114
+ [
115
+ \t{{"horizontal_bbox": [x1, y1, x2, y2]}}
116
+ ]
117
+ ```"""
118
+ ````
119
 
120
+ Coordinates are quantized to `[0, 1000]`; scale by image width and height to recover pixels.
121
+
122
+ The model can also be served with [ms-swift](https://github.com/modelscope/ms-swift), the
123
+ framework used for training.
124
+
125
+ ## Citation
126
 
127
  ```bibtex
128
  @inproceedings{geoboxr1,
 
135
  note = {Under review}
136
  }
137
  ```
138
+
139
+ ---
140
+
141
+ # GeoBox-R1 中文
142
+
143
+ [English](#geobox-r1) | **简体中文**
144
+
145
+ 统一边界框级遥感视觉定位模型,同时输出水平框(HBB)与旋转框(OBB)。
146
+
147
+ 本仓库是**最终模型**(Stage-1 SFT + Stage-2 GDPO 后的合并权重)。
148
+ 只需要第一阶段结果请用 [`GeoBox-R1-SFT`](https://huggingface.co/yuchenwu73/GeoBox-R1-SFT)。
149
+
150
+ ## 训练流程
151
+
152
+ 基座 **Qwen3-VL-4B-Instruct**,两阶段:
153
+
154
+ 1. **课程式 SFT** — 训练数据按 HBB → OBB → HBB-to-OBB CoT 由易到难排列。
155
+ LoRA(rank 16,alpha 32),冻结视觉编码器与 merger,lr `1e-4`,1 epoch,2× RTX 4090。
156
+ 2. **几何强化学习(GDPO)** — 在 SFT 检查点上用两个基于规则的几何奖励细化 OBB:
157
+ Rotated IoU 与自适应 Wasserstein 距离(λ 各 0.5)。
158
+ G=8 rollouts,β=0.02,τ_c=8,lr `5e-6`,1 epoch,3× A100 40G(1 个 vLLM rollout 服务 + 2 个 GDPO worker)。
159
+
160
+ ## 结果
161
+
162
+ 7 个 HBB 与 3 个 OBB 评测集,三项指标的宏平均均取得最佳:
163
+
164
+ | 任务 | Acc@0.5 | Acc@0.7 | mIoU / mRIoU |
165
+ | --- | --- | --- | --- |
166
+ | HBB(7 个集) | **58.78** | **42.22** | **50.39** |
167
+ | OBB(3 个集) | **47.32** | **27.55** | **39.85** |
168
+
169
+ 以 4B 参数超过 7B–8B 的现有最优模型:HBB 领先 **3.68/4.89/2.61** 点,
170
+ OBB 领先 **5.36/7.74/2.89** 点,在更严格的 Acc@0.7 上增益最大。
171
+ 逐数据集的完整结果见上方英文表格或[项目主页](https://yuchenwu73.github.io/GeoBox-R1/)。
172
+
173
+ **OBB-only RL 不牺牲 HBB**:GDPO 只用 OBB 样本训练,HBB 宏平均反而从 57.17/40.33/48.82
174
+ 升到 58.78/42.22/50.39 —— Acc@0.5 在 7 个集合中的 6 个上升,Acc@0.7 与 mIoU 全部 7 个上升。
175
+
176
+ ## 使用
177
+
178
+ 代码见上方英文 [Usage](#usage) 一节。提示词与训练、评测时**逐字节一致**,
179
+ 包括坐标列表里的空格 —— 改动空格会改变分词结果。坐标量化到 `[0, 1000]`,
180
+ 按图像宽高缩放即可还原到像素。