Hugging Face's logo

osunlp
/
UGround

Image-Text-to-Text
Safetensors
llava_llama
Model card Files
xet
 Community
4
BoyuNLP commited on
Commit
c61a302
·
verified ·
1 Parent(s): 1d01d95

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +37 -24
README.md CHANGED
@@ -13,49 +13,62 @@ UGround is a storng GUI visual grounding model trained with a simple recipe. Che
13
  - **Demo:** https://huggingface.co/spaces/orby-osu/UGround
14
  - **Point of Contact:** [Boyu Gou](mailto:gou.43@osu.edu)
15
 
16
-
17
  - [x] Model Weights
 
18
  - [ ] Code
19
  - [ ] Inference Code of UGround
20
  - [x] Offline Experiments
21
  - [x] Screenspot (along with referring expressions generated by GPT-4/4o)
22
  - [x] Multimodal-Mind2Web
23
  - [x] OmniAct
 
24
  - [ ] Online Experiments
25
- - [ ] Mind2Web-Live
26
- - [ ] AndroidWorld
27
- - [ ] Data
28
  - [ ] Data Examples
29
  - [ ] Data Construction Scripts
30
- - [ ] Guidance of Open-source Data
 
31
  - [x] Online Demo (HF Spaces)
32
 
 
33
 
 
 
 
 
 
 
 
34
 
35
 
36
  ## Main Results
37
 
38
  ### ScreenSpot (GUI Visual Grounding)
39
 
40
- | ScreenSpot (Standard) | Arch | SFT data | Mobile-Text | Mobile-Icon | Desktop-Text | Desktop-Icon | Web-Text | Web-Icon | Avg |
41
- | ------------------------ | ---------------- | ---------------- | ----------- | ----------- | ------------ | ------------ | -------- | -------- | -------- |
42
- | Qwen-VL | Qwen-VL | | 9.5 | 4.8 | 5.7 | 5.0 | 3.5 | 2.4 | 5.2 |
43
- | CogAgent | CogAgent | CogAgent | 67 | 24 | 74.2 | 20 | 70.4 | 28.6 | 47.4 |
44
- | SeeClick | Qwen-VL | SeeClick | 78.0 | 52.0 | 72.2 | 30.0 | 55.7 | 32.5 | 53.4 |
45
- | Qwen-GUI | Qwen-VL | GUICourse | 52.4 | 10.9 | 45.9 | 5.7 | 43.0 | 13.6 | 28.6 |
46
- | UGround-V1 (Qwen-VL) | Qwen-VL | Web-Hybrid | 68.5 | 28.4 | 69.6 | 34.3 | 63.5 | 39.3 | 50.6 |
47
- | UGround-V1 | LLaVA-UGround-V1 | UGround-V1 | 82.8 | 60.3 | 82.5 | 63.6 | 80.4 | 70.4 | 73.3 |
48
- | Qwen2-VL | Qwen2-VL | | 61.3 | 39.3 | 52.0 | 45.0 | 33.0 | 21.8 | 42.1 |
49
- | Auguvis-G-7B | Qwen2-VL | Aguvis-Stage-1 | 88.3 | 78.2 | 88.1 | 70.7 | 85.7 | 74.8 | 81.0 |
50
- | Auguvis-7B | Qwen2-VL | Aguvis-Stage-1&2 | **95.6** | 77.7 | **93.8** | 67.1 | 88.3 | 75.2 | 83.0 |
51
- | OS-Atlas-Base-4B | InternVL | OS-Atlas | 85.7 | 58.5 | 72.2 | 45.7 | 82.6 | 63.1 | 68.0 |
52
- | OS-Atlas-Base-7B | Qwen2-VL | OS-Atlas | 93.0 | 72.9 | 91.8 | 62.9 | **90.9** | 74.3 | 81.0 |
53
- | ShowUI-G | ShowUI | ShowUI | 91.6 | 69.0 | 81.8 | 59.0 | 83.0 | 65.5 | 75.0 |
54
- | ShowUI | ShowUI | ShowUI | 92.3 | 75.5 | 76.3 | 61.1 | 81.7 | 63.6 | 75.1 |
55
- | Iris | Iris | SeeClick | 85.3 | 64.2 | 86.7 | 57.5 | 82.6 | 71.2 | 74.6 |
56
- | Aria-UI | Aria | Aria-UI | 92.3 | 73.8 | 93.3 | 64.3 | 86.5 | 76.2 | 81.1 |
57
- | UGround-V1-2B (Qwen2-VL) | Qwen2-VL | UGround-V1 | 89.4 | 72.0 | 88.7 | 65.7 | 81.3 | 68.9 | 77.7 |
58
- | UGround-V1-7B (Qwen2-VL) | Qwen2-VL | UGround-V1 | 93.0 | **79.9** | **93.8** | **76.4** | **90.9** | **84.0** | **86.3** |
 
 
 
59
 
60
 
61
 
 
13
  - **Demo:** https://huggingface.co/spaces/orby-osu/UGround
14
  - **Point of Contact:** [Boyu Gou](mailto:gou.43@osu.edu)
15
 
 
16
  - [x] Model Weights
17
+ - [x] Qwen2-VL-based V1
18
  - [ ] Code
19
  - [ ] Inference Code of UGround
20
  - [x] Offline Experiments
21
  - [x] Screenspot (along with referring expressions generated by GPT-4/4o)
22
  - [x] Multimodal-Mind2Web
23
  - [x] OmniAct
24
+ - [ ] Android Control
25
  - [ ] Online Experiments
26
+ - [ ] Mind2Web-Live-SeeAct-V
27
+ - [ ] AndroidWorld-SeeAct-V
28
+ - [ ] Data-V1
29
  - [ ] Data Examples
30
  - [ ] Data Construction Scripts
31
+ - [ ] Guidance of Open-source Data
32
+ - [ ] Data-V1.1
33
  - [x] Online Demo (HF Spaces)
34
 
35
+ ## Models
36
 
37
+ - Initial UGround-V1: https://huggingface.co/osunlp/UGround
38
+ - UGround-V1-2B (Qwen2-VL): https://huggingface.co/osunlp/UGround-V1-2B
39
+ - UGround-V1-7B (Qwen2-VL): https://huggingface.co/osunlp/UGround-V1-7B
40
+ - UGround-V1-72B (Qwen2-VL): Coming Soon
41
+ - UGround-V1.1-2B (Qwen2-VL): Coming Soon
42
+ - UGround-V1.1-7B (Qwen2-VL): Coming Soon
43
+ - UGround-V1.1-72B (Qwen2-VL): Coming Soon
44
 
45
 
46
  ## Main Results
47
 
48
  ### ScreenSpot (GUI Visual Grounding)
49
 
50
+ | ScreenSpot (Standard) | Arch | SFT data | Mobile-Text | Mobile-Icon | Desktop-Text | Desktop-Icon | Web-Text | Web-Icon | Avg |
51
+ | ---------------------------- | ---------------- | ---------------- | ----------- | ----------- | ------------ | ------------ | -------- | -------- | -------- |
52
+ | GPT-4 | | | 22.6 | 24.5 | 20.2 | 11.8 | 9.2 | 8.8 | 16.2 |
53
+ | GPT-4o | | | 20.2 | 24.9 | 21.1 | 23.6 | 12.2 | 7.8 | 18.3 |
54
+ | MiniGPT-v2 | MiniGPT-v2 | | 8.4 | 6.6 | 6.2 | 2.9 | 6.5 | 3.4 | 5.7 |
55
+ | Groma | Groma | | 10.3 | 2.6 | 4.6 | 4.3 | 5.7 | 3.4 | 5.2 |
56
+ | Fuyu | Fuyu | | 41.0 | 1.3 | 33.0 | 3.6 | 33.9 | 4.4 | 19.5 |
57
+ | Qwen-VL | Qwen-VL | | 9.5 | 4.8 | 5.7 | 5.0 | 3.5 | 2.4 | 5.2 |
58
+ | SeeClick | Qwen-VL | SeeClick | 78.0 | 52.0 | 72.2 | 30.0 | 55.7 | 32.5 | 53.4 |
59
+ | Qwen-GUI | Qwen-VL | GUICourse | 52.4 | 10.9 | 45.9 | 5.7 | 43.0 | 13.6 | 28.6 |
60
+ | **UGround-V1** | LLaVA-UGround-V1 | UGround-V1 | 82.8 | 60.3 | 82.5 | 63.6 | 80.4 | 70.4 | 73.3 |
61
+ | Qwen2-VL | Qwen2-VL | | 61.3 | 39.3 | 52.0 | 45.0 | 33.0 | 21.8 | 42.1 |
62
+ | Auguvis-G-7B | Qwen2-VL | Aguvis-Stage-1 | 88.3 | 78.2 | 88.1 | 70.7 | 85.7 | 74.8 | 81.0 |
63
+ | Auguvis-7B | Qwen2-VL | Aguvis-Stage-1&2 | **95.6** | 77.7 | **93.8** | 67.1 | 88.3 | 75.2 | 83.0 |
64
+ | OS-Atlas-Base-4B | InternVL | OS-Atlas | 85.7 | 58.5 | 72.2 | 45.7 | 82.6 | 63.1 | 68.0 |
65
+ | OS-Atlas-Base-7B | Qwen2-VL | OS-Atlas | 93.0 | 72.9 | 91.8 | 62.9 | **90.9** | 74.3 | 81.0 |
66
+ | ShowUI-G | ShowUI | ShowUI | 91.6 | 69.0 | 81.8 | 59.0 | 83.0 | 65.5 | 75.0 |
67
+ | ShowUI | ShowUI | ShowUI | 92.3 | 75.5 | 76.3 | 61.1 | 81.7 | 63.6 | 75.1 |
68
+ | Iris | Iris | SeeClick | 85.3 | 64.2 | 86.7 | 57.5 | 82.6 | 71.2 | 74.6 |
69
+ | Aria-UI | Aria | Aria-UI | 92.3 | 73.8 | 93.3 | 64.3 | 86.5 | 76.2 | 81.1 |
70
+ | **UGround-V1-2B (Qwen2-VL)** | Qwen2-VL | UGround-V1 | 89.4 | 72.0 | 88.7 | 65.7 | 81.3 | 68.9 | 77.7 |
71
+ | **UGround-V1-7B (Qwen2-VL)** | Qwen2-VL | UGround-V1 | 93.0 | **79.9** | **93.8** | **76.4** | **90.9** | **84.0** | **86.3** |
72
 
73
 
74