Image-Text-to-Text
Transformers
Safetensors
English
qwen2_5_vl
VLM
Computer-Use-Agent
OS-Agent
GUI
Grounding
conversational
Eval Results
text-generation-inference
Instructions to use Salesforce/GTA1-32B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Salesforce/GTA1-32B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="Salesforce/GTA1-32B") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("Salesforce/GTA1-32B") model = AutoModelForMultimodalLM.from_pretrained("Salesforce/GTA1-32B", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Salesforce/GTA1-32B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Salesforce/GTA1-32B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Salesforce/GTA1-32B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/Salesforce/GTA1-32B
- SGLang
How to use Salesforce/GTA1-32B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Salesforce/GTA1-32B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Salesforce/GTA1-32B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Salesforce/GTA1-32B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Salesforce/GTA1-32B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use Salesforce/GTA1-32B with Docker Model Runner:
docker model run hf.co/Salesforce/GTA1-32B
Update README.md
Browse files
README.md
CHANGED
|
@@ -18,7 +18,7 @@ library_name: transformers
|
|
| 18 |
|
| 19 |
Reinforcement learning (RL) (e.g., GRPO) helps with grounding because of its inherent objective alignment—rewarding successful clicks—rather than encouraging long textual Chain-of-Thought (CoT) reasoning. Unlike approaches that rely heavily on verbose CoT reasoning, GRPO directly incentivizes actionable and grounded responses. Based on findings from our [blog](https://huggingface.co/blog/HelloKKMe/grounding-r1), we share state-of-the-art GUI grounding models trained using GRPO.
|
| 20 |
|
| 21 |
-
# Performance
|
| 22 |
|
| 23 |
We follow the standard evaluation protocol and benchmark our model on three challenging datasets. Our method consistently achieves the best results among all open-source model families. Below are the comparative results:
|
| 24 |
|
|
@@ -39,6 +39,58 @@ We follow the standard evaluation protocol and benchmark our model on three chal
|
|
| 39 |
| GTA1-2507 (Ours) | 7B | ✅ | 92.4 <sub>*(∆ +2.7)*</sub> | 50.1<sub>*(∆ +8.1)*</sub> | 55.1 <sub>*(∆ +2.3)*</sub> | 67.7 <sub>*(∆ +3.5)*</sub> |
|
| 40 |
| GTA1 (Ours) | 32B | ✅ | 95.2 <sub>*(∆ +1.8)*</sub> | 63.6<sub>*(∆ +8.3)*</sub> | 65.2 <sub>*(∆ +5.6)*</sub> | 72.2<sub>*(∆ +2.0)*</sub> |
|
| 41 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 42 |
|
| 43 |
> **Note:**
|
| 44 |
> - Model size is indicated in billions (B) of parameters.
|
|
@@ -688,5 +740,36 @@ def main():
|
|
| 688 |
if __name__ == "__main__":
|
| 689 |
main()
|
| 690 |
```
|
| 691 |
-
|
| 692 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
|
| 19 |
Reinforcement learning (RL) (e.g., GRPO) helps with grounding because of its inherent objective alignment—rewarding successful clicks—rather than encouraging long textual Chain-of-Thought (CoT) reasoning. Unlike approaches that rely heavily on verbose CoT reasoning, GRPO directly incentivizes actionable and grounded responses. Based on findings from our [blog](https://huggingface.co/blog/HelloKKMe/grounding-r1), we share state-of-the-art GUI grounding models trained using GRPO.
|
| 20 |
|
| 21 |
+
# Grounding Performance
|
| 22 |
|
| 23 |
We follow the standard evaluation protocol and benchmark our model on three challenging datasets. Our method consistently achieves the best results among all open-source model families. Below are the comparative results:
|
| 24 |
|
|
|
|
| 39 |
| GTA1-2507 (Ours) | 7B | ✅ | 92.4 <sub>*(∆ +2.7)*</sub> | 50.1<sub>*(∆ +8.1)*</sub> | 55.1 <sub>*(∆ +2.3)*</sub> | 67.7 <sub>*(∆ +3.5)*</sub> |
|
| 40 |
| GTA1 (Ours) | 32B | ✅ | 95.2 <sub>*(∆ +1.8)*</sub> | 63.6<sub>*(∆ +8.3)*</sub> | 65.2 <sub>*(∆ +5.6)*</sub> | 72.2<sub>*(∆ +2.0)*</sub> |
|
| 41 |
|
| 42 |
+
# Agent Performance
|
| 43 |
+
|
| 44 |
+
## OSWorld and OSWorld-Verified Benchmarks
|
| 45 |
+
|
| 46 |
+
We evaluate our models on the OSWorld and OSWorld-Verified benchmarks following the standard evaluation protocol. The results demonstrate strong performance across both datasets.
|
| 47 |
+
|
| 48 |
+
| **Agent Model** | **Step** | **OSWorld** | **OSWorld-Verified** |
|
| 49 |
+
|-----------------|:--------:|:-----------:|:-------------------:|
|
| 50 |
+
| **Proprietary Models** |
|
| 51 |
+
| Claude 3.7 Sonnet | 100 | 28.0 | — |
|
| 52 |
+
| OpenAI CUA 4o | 200 | 38.1 | — |
|
| 53 |
+
| UI-TARS-1.5 | 100 | 42.5 | 41.8 |
|
| 54 |
+
| OpenAI CUA o3 | 200 | 42.9 | — |
|
| 55 |
+
| **Open-Source Models** |
|
| 56 |
+
| Aria-UI w/ GPT-4o | 15 | 15.2 | — |
|
| 57 |
+
| Aguvis-72B w/ GPT-4o | 15 | 17.0 | — |
|
| 58 |
+
| UI-TARS-72B-SFT | 50 | 18.8 | — |
|
| 59 |
+
| Agent S w/ Claude-3.5-Sonnet | 15 | 20.5 | — |
|
| 60 |
+
| Agent S w/ GPT-4o | 15 | 20.6 | — |
|
| 61 |
+
| UI-TARS-72B-DPO | 15 | 22.7 | — |
|
| 62 |
+
| UI-TARS-72B-DPO | 50 | 24.6 | — |
|
| 63 |
+
| UI-TARS-1.5-7B | 100 | 26.9 | 27.4 |
|
| 64 |
+
| Jedi-7B w/ o3 | 100 | — | 51.0 |
|
| 65 |
+
| Jedi-7B w/ GPT-4o | 100 | 27.0 | — |
|
| 66 |
+
| Agent S2 w/ Claude-3.7-Sonnet | 50 | 34.5 | — |
|
| 67 |
+
| Agent S2 w/ Gemini-2.5-Pro | 50 | 41.4 | 45.8 |
|
| 68 |
+
| Agent S2.5 w/ o3 | 100 | — | 56.0 |
|
| 69 |
+
| Agent S2.5 w/ GPT-5 | 100 | — | 58.4 |
|
| 70 |
+
| CoAct-1 w/o3 & o4mini & OpenAI CUA 4o | 150 | — | 60.8 |
|
| 71 |
+
| GTA1-7B w/ o3 | 100 | 45.2 | 53.1 |
|
| 72 |
+
| GTA1-7B w/ GPT-5 | 100 | — | 61.0 |
|
| 73 |
+
| GTA1-32B w/ o3 | 100 | — | 55.4 |
|
| 74 |
+
| GTA1-32B w/ GPT-5 | 100 | — | 62.0 |
|
| 75 |
+
|
| 76 |
+
> **Note:** A dash (—) indicates unavailable results.
|
| 77 |
+
|
| 78 |
+
## WindowsAgentArena Benchmark
|
| 79 |
+
|
| 80 |
+
We also evaluate our models on the WindowsAgentArena benchmark, demonstrating strong performance in Windows-specific GUI automation tasks.
|
| 81 |
+
|
| 82 |
+
| **Agent Model** | **Step** | **Success Rate** |
|
| 83 |
+
|-----------------|:--------:|:---------------:|
|
| 84 |
+
| Kimi-VL | 15 | 10.4 |
|
| 85 |
+
| WAA | — | 19.5 |
|
| 86 |
+
| Jedi w/ GPT-4o | 100 | 33.7 |
|
| 87 |
+
| GTA1-7B w/ o3 | 100 | 47.9 |
|
| 88 |
+
| GTA1-7B w/ GPT-5 | 100 | 49.2 |
|
| 89 |
+
| GTA1-32B w/ o3 | 100 | 51.2 |
|
| 90 |
+
| GTA1-32B w/ GPT-5 | 100 | 50.6 |
|
| 91 |
+
|
| 92 |
+
> **Note:** A dash (—) indicates unavailable results.
|
| 93 |
+
|
| 94 |
|
| 95 |
> **Note:**
|
| 96 |
> - Model size is indicated in billions (B) of parameters.
|
|
|
|
| 740 |
if __name__ == "__main__":
|
| 741 |
main()
|
| 742 |
```
|
| 743 |
+
## Ethical Considerations
|
| 744 |
+
|
| 745 |
+
This model is released for research and educational purposes. While our model demonstrates strong performance on GUI benchmarks, users should carefully evaluate its suitability for their specific use cases.
|
| 746 |
+
|
| 747 |
+
**Important Considerations:**
|
| 748 |
+
- **Accuracy Limitations:** Like all AI systems, this model may produce incorrect outputs or fail to accurately identify GUI elements in certain scenarios.
|
| 749 |
+
- **Safety and Security:** Exercise caution when deploying GUI automation agents, especially in production environments where incorrect actions could affect system integrity or data security.
|
| 750 |
+
- **Human Oversight:** We recommend maintaining appropriate human supervision when using this model for automated GUI interactions.
|
| 751 |
+
- **Compliance:** Users are responsible for ensuring their use of this model complies with applicable laws, regulations, and organizational policies.
|
| 752 |
+
|
| 753 |
+
**Recommended Best Practices:**
|
| 754 |
+
- Thoroughly test the model in controlled environments before production deployment
|
| 755 |
+
- Implement safeguards and error handling mechanisms
|
| 756 |
+
- Consider the potential impact of automated actions on user systems and data
|
| 757 |
+
- Regularly monitor and validate model performance in your specific domain
|
| 758 |
+
|
| 759 |
+
For further guidance on use cases, refer to our AUP and AI AUP.
|
| 760 |
+
|
| 761 |
+
## Citation
|
| 762 |
+
|
| 763 |
+
If you're using any GTA model or find it helpful in your research, please cite it as follows:
|
| 764 |
+
|
| 765 |
+
```markdown
|
| 766 |
+
@article{yang2025gta1guitesttimescaling,
|
| 767 |
+
title={GTA1: GUI Test-time Scaling Agent},
|
| 768 |
+
author={Yan Yang and Dongxu Li and Yutong Dai and Yuhao Yang and Ziyang Luo and Zirui Zhao and Zhiyuan Hu and Junzhe Huang and Amrita Saha and Zeyuan Chen and Ran Xu and Liyuan Pan and Caiming Xiong and Junnan Li},
|
| 769 |
+
year={2025},
|
| 770 |
+
eprint={2507.05791},
|
| 771 |
+
archivePrefix={arXiv},
|
| 772 |
+
primaryClass={cs.AI},
|
| 773 |
+
url={https://arxiv.org/abs/2507.05791},
|
| 774 |
+
}
|
| 775 |
+
```
|