HelloKKMe commited on
Commit
610cd6f
·
verified ·
1 Parent(s): 665ae8c

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +86 -3
README.md CHANGED
@@ -18,7 +18,7 @@ library_name: transformers
18
 
19
  Reinforcement learning (RL) (e.g., GRPO) helps with grounding because of its inherent objective alignment—rewarding successful clicks—rather than encouraging long textual Chain-of-Thought (CoT) reasoning. Unlike approaches that rely heavily on verbose CoT reasoning, GRPO directly incentivizes actionable and grounded responses. Based on findings from our [blog](https://huggingface.co/blog/HelloKKMe/grounding-r1), we share state-of-the-art GUI grounding models trained using GRPO.
20
 
21
- # Performance
22
 
23
  We follow the standard evaluation protocol and benchmark our model on three challenging datasets. Our method consistently achieves the best results among all open-source model families. Below are the comparative results:
24
 
@@ -39,6 +39,58 @@ We follow the standard evaluation protocol and benchmark our model on three chal
39
  | GTA1-2507 (Ours) | 7B | ✅ | 92.4 <sub>*(∆ +2.7)*</sub> | 50.1<sub>*(∆ +8.1)*</sub> | 55.1 <sub>*(∆ +2.3)*</sub> | 67.7 <sub>*(∆ +3.5)*</sub> |
40
  | GTA1 (Ours) | 32B | ✅ | 95.2 <sub>*(∆ +1.8)*</sub> | 63.6<sub>*(∆ +8.3)*</sub> | 65.2 <sub>*(∆ +5.6)*</sub> | 72.2<sub>*(∆ +2.0)*</sub> |
41
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
42
 
43
  > **Note:**
44
  > - Model size is indicated in billions (B) of parameters.
@@ -688,5 +740,36 @@ def main():
688
  if __name__ == "__main__":
689
  main()
690
  ```
691
-
692
- Refer to our [code](https://github.com/Yan98/GTA1) for more details.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
18
 
19
  Reinforcement learning (RL) (e.g., GRPO) helps with grounding because of its inherent objective alignment—rewarding successful clicks—rather than encouraging long textual Chain-of-Thought (CoT) reasoning. Unlike approaches that rely heavily on verbose CoT reasoning, GRPO directly incentivizes actionable and grounded responses. Based on findings from our [blog](https://huggingface.co/blog/HelloKKMe/grounding-r1), we share state-of-the-art GUI grounding models trained using GRPO.
20
 
21
+ # Grounding Performance
22
 
23
  We follow the standard evaluation protocol and benchmark our model on three challenging datasets. Our method consistently achieves the best results among all open-source model families. Below are the comparative results:
24
 
 
39
  | GTA1-2507 (Ours) | 7B | ✅ | 92.4 <sub>*(∆ +2.7)*</sub> | 50.1<sub>*(∆ +8.1)*</sub> | 55.1 <sub>*(∆ +2.3)*</sub> | 67.7 <sub>*(∆ +3.5)*</sub> |
40
  | GTA1 (Ours) | 32B | ✅ | 95.2 <sub>*(∆ +1.8)*</sub> | 63.6<sub>*(∆ +8.3)*</sub> | 65.2 <sub>*(∆ +5.6)*</sub> | 72.2<sub>*(∆ +2.0)*</sub> |
41
 
42
+ # Agent Performance
43
+
44
+ ## OSWorld and OSWorld-Verified Benchmarks
45
+
46
+ We evaluate our models on the OSWorld and OSWorld-Verified benchmarks following the standard evaluation protocol. The results demonstrate strong performance across both datasets.
47
+
48
+ | **Agent Model** | **Step** | **OSWorld** | **OSWorld-Verified** |
49
+ |-----------------|:--------:|:-----------:|:-------------------:|
50
+ | **Proprietary Models** |
51
+ | Claude 3.7 Sonnet | 100 | 28.0 | — |
52
+ | OpenAI CUA 4o | 200 | 38.1 | — |
53
+ | UI-TARS-1.5 | 100 | 42.5 | 41.8 |
54
+ | OpenAI CUA o3 | 200 | 42.9 | — |
55
+ | **Open-Source Models** |
56
+ | Aria-UI w/ GPT-4o | 15 | 15.2 | — |
57
+ | Aguvis-72B w/ GPT-4o | 15 | 17.0 | — |
58
+ | UI-TARS-72B-SFT | 50 | 18.8 | — |
59
+ | Agent S w/ Claude-3.5-Sonnet | 15 | 20.5 | — |
60
+ | Agent S w/ GPT-4o | 15 | 20.6 | — |
61
+ | UI-TARS-72B-DPO | 15 | 22.7 | — |
62
+ | UI-TARS-72B-DPO | 50 | 24.6 | — |
63
+ | UI-TARS-1.5-7B | 100 | 26.9 | 27.4 |
64
+ | Jedi-7B w/ o3 | 100 | — | 51.0 |
65
+ | Jedi-7B w/ GPT-4o | 100 | 27.0 | — |
66
+ | Agent S2 w/ Claude-3.7-Sonnet | 50 | 34.5 | — |
67
+ | Agent S2 w/ Gemini-2.5-Pro | 50 | 41.4 | 45.8 |
68
+ | Agent S2.5 w/ o3 | 100 | — | 56.0 |
69
+ | Agent S2.5 w/ GPT-5 | 100 | — | 58.4 |
70
+ | CoAct-1 w/o3 & o4mini & OpenAI CUA 4o | 150 | — | 60.8 |
71
+ | GTA1-7B w/ o3 | 100 | 45.2 | 53.1 |
72
+ | GTA1-7B w/ GPT-5 | 100 | — | 61.0 |
73
+ | GTA1-32B w/ o3 | 100 | — | 55.4 |
74
+ | GTA1-32B w/ GPT-5 | 100 | — | 62.0 |
75
+
76
+ > **Note:** A dash (—) indicates unavailable results.
77
+
78
+ ## WindowsAgentArena Benchmark
79
+
80
+ We also evaluate our models on the WindowsAgentArena benchmark, demonstrating strong performance in Windows-specific GUI automation tasks.
81
+
82
+ | **Agent Model** | **Step** | **Success Rate** |
83
+ |-----------------|:--------:|:---------------:|
84
+ | Kimi-VL | 15 | 10.4 |
85
+ | WAA | — | 19.5 |
86
+ | Jedi w/ GPT-4o | 100 | 33.7 |
87
+ | GTA1-7B w/ o3 | 100 | 47.9 |
88
+ | GTA1-7B w/ GPT-5 | 100 | 49.2 |
89
+ | GTA1-32B w/ o3 | 100 | 51.2 |
90
+ | GTA1-32B w/ GPT-5 | 100 | 50.6 |
91
+
92
+ > **Note:** A dash (—) indicates unavailable results.
93
+
94
 
95
  > **Note:**
96
  > - Model size is indicated in billions (B) of parameters.
 
740
  if __name__ == "__main__":
741
  main()
742
  ```
743
+ ## Ethical Considerations
744
+
745
+ This model is released for research and educational purposes. While our model demonstrates strong performance on GUI benchmarks, users should carefully evaluate its suitability for their specific use cases.
746
+
747
+ **Important Considerations:**
748
+ - **Accuracy Limitations:** Like all AI systems, this model may produce incorrect outputs or fail to accurately identify GUI elements in certain scenarios.
749
+ - **Safety and Security:** Exercise caution when deploying GUI automation agents, especially in production environments where incorrect actions could affect system integrity or data security.
750
+ - **Human Oversight:** We recommend maintaining appropriate human supervision when using this model for automated GUI interactions.
751
+ - **Compliance:** Users are responsible for ensuring their use of this model complies with applicable laws, regulations, and organizational policies.
752
+
753
+ **Recommended Best Practices:**
754
+ - Thoroughly test the model in controlled environments before production deployment
755
+ - Implement safeguards and error handling mechanisms
756
+ - Consider the potential impact of automated actions on user systems and data
757
+ - Regularly monitor and validate model performance in your specific domain
758
+
759
+ For further guidance on use cases, refer to our AUP and AI AUP.
760
+
761
+ ## Citation
762
+
763
+ If you're using any GTA model or find it helpful in your research, please cite it as follows:
764
+
765
+ ```markdown
766
+ @article{yang2025gta1guitesttimescaling,
767
+ title={GTA1: GUI Test-time Scaling Agent},
768
+ author={Yan Yang and Dongxu Li and Yutong Dai and Yuhao Yang and Ziyang Luo and Zirui Zhao and Zhiyuan Hu and Junzhe Huang and Amrita Saha and Zeyuan Chen and Ran Xu and Liyuan Pan and Caiming Xiong and Junnan Li},
769
+ year={2025},
770
+ eprint={2507.05791},
771
+ archivePrefix={arXiv},
772
+ primaryClass={cs.AI},
773
+ url={https://arxiv.org/abs/2507.05791},
774
+ }
775
+ ```