README.md

---
license: apache-2.0
base_model: Qwen/Qwen3-VL-8B-Instruct
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- gui-agent
- gui-grounding
- qwen3-vl
datasets:
- PrentisAI/ScreenRef
---

# GroundGUI-8B

A GUI grounding model built on [Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct).
Given a screenshot and a natural-language instruction, it outputs a single click action as a
`<tool_call>` with a coordinate on a 0–1000 grid.

- **Paper:** [arXiv:2610.19100](https://arxiv.org/abs/2610.19100)
- **Code:** [PrentisAI/GroundGUI](https://github.com/PrentisAI/GroundGUI)
- **Data:** [PrentisAI/ScreenRef](https://huggingface.co/datasets/PrentisAI/ScreenRef)

## Training

Fine-tuned from Qwen3-VL-8B-Instruct on [ScreenRef](https://huggingface.co/datasets/PrentisAI/ScreenRef).
Details are described in the [paper](https://arxiv.org/abs/2610.19100).

## Results

Accuracy (%) is the fraction of predicted click points that land inside the ground-truth box.
Coordinates are mapped back to pixel space and checked against the box exactly, with no rounding
tolerance.

| ScreenSpot-v2 | ScreenSpot-Pro | MMBench-GUI L2 | UI-Vision | OSWorld-G |
|:---:|:---:|:---:|:---:|:---:|
| 95.2 (1211/1272) | 63.9 (1011/1581) | 86.8 (3121/3594) | 38.3 (2101/5479) | 65.4 (369/564) |

- Inference used vLLM with greedy decoding (temperature 0) and `max_tokens` 1024.
- OSWorld-G is the original 564-question split.

## Usage

The model expects a specific system prompt, and the chat template does not add it for you. Use the
evaluation code in [GroundGUI](https://github.com/PrentisAI/GroundGUI) (`scripts/eval.sh`) to
reproduce the numbers above; it sets the system prompt (`scalecua_toolcall`) and the scoring for
each benchmark.

The output looks like:

```
<tool_call>
{"name": "computer_use", "arguments": {"action": "left_click", "coordinate": [x, y]}}
</tool_call>
```

To get pixel coordinates from the output: `x_px = x / 1000 * image_width` and
`y_px = y / 1000 * image_height`.

## Notes

- The tool name depends on the platform: `computer_use` for desktop, `browser_use` for web,
  `mobile_use` for mobile.
- `generation_config.json` is inherited from the base model and enables sampling. Pass
  `do_sample=False` to reproduce the numbers above.
- The weights were saved with transformers 5.12.

## License

[Apache License 2.0](LICENSE), the same as the base model. This model is a modified version of
[Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct): Prentis AI fine-tuned the
original weights.

The training data, [ScreenRef](https://huggingface.co/datasets/PrentisAI/ScreenRef), is distributed
under its own terms; this license covers the model only and grants no rights in the dataset or in
the screenshots it contains.

## Citation

GroundGUI-8B is introduced in [GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning](https://arxiv.org/abs/2610.19100). If you use this model, please cite:

```bibtex
@misc{xie2026groundguiguigroundinghighquality,
      title={GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning},
      author={Qingsong Xie and Jiahao Tian and Jiacheng Tang and Zhifeng Li and Madalina Ciobanu and Navan Preet Singh and Qingqing Mao and Ritankar Das},
      year={2026},
      eprint={2610.19100},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2610.19100},
}
```