| ---
|
| license: mit
|
| library_name: ultralytics
|
| tags:
|
| - object-detection
|
| - yolo
|
| - gui
|
| - ui-detection
|
| - omniparser
|
| pipeline_tag: object-detection
|
| ---
|
|
|
| # GPA-GUI-Detector
|
|
|
| A YOLO-based GUI element detection model for detecting interactive UI elements (icons, buttons, etc.) on screen for GUI Process Automation. This model is finetuned from the [OmniParser](https://github.com/microsoft/OmniParser) ecosystem.
|
|
|
| ## Model
|
|
|
| The model weight file is `model.pt`. It is a YOLO model trained with the [Ultralytics](https://github.com/ultralytics/ultralytics) framework.
|
|
|
| ## Installation
|
|
|
| ```bash
|
| pip install ultralytics
|
| ```
|
|
|
| ## Usage
|
|
|
| ### Basic Inference
|
|
|
| ```python
|
| from ultralytics import YOLO
|
|
|
| model = YOLO("model.pt")
|
| results = model("screenshot.png")
|
| ```
|
|
|
| ### Detection with Custom Parameters
|
|
|
| ```python
|
| from ultralytics import YOLO
|
| from PIL import Image
|
|
|
| # Load the model
|
| model = YOLO("model.pt")
|
|
|
| # Run inference with custom confidence and image size
|
| results = model.predict(
|
| source="screenshot.png",
|
| conf=0.05, # confidence threshold
|
| imgsz=640, # input image size
|
| iou=0.7, # NMS IoU threshold
|
| )
|
|
|
| # Parse results
|
| boxes = results[0].boxes.xyxy.cpu().numpy() # bounding boxes in [x1, y1, x2, y2]
|
| scores = results[0].boxes.conf.cpu().numpy() # confidence scores
|
|
|
| # Draw results on image
|
| img = Image.open("screenshot.png")
|
| for box, score in zip(boxes, scores):
|
| x1, y1, x2, y2 = box
|
| print(f"Detected UI element at [{x1:.0f}, {y1:.0f}, {x2:.0f}, {y2:.0f}] (conf: {score:.2f})")
|
|
|
| # Or save the annotated image directly
|
| results[0].save("result.png")
|
| ```
|
|
|
| ### Integration with OmniParser
|
|
|
| ```python
|
| import sys
|
| sys.path.append("/path/to/OmniParser")
|
|
|
| from util.utils import get_yolo_model, predict_yolo
|
| from PIL import Image
|
|
|
| model = get_yolo_model("model.pt")
|
| image = Image.open("screenshot.png")
|
|
|
| boxes, confidences, phrases = predict_yolo(
|
| model=model,
|
| image=image,
|
| box_threshold=0.05,
|
| imgsz=640,
|
| scale_img=False,
|
| iou_threshold=0.7,
|
| )
|
|
|
| for i, (box, conf) in enumerate(zip(boxes, confidences)):
|
| print(f"Element {i}: box={box.tolist()}, confidence={conf:.2f}")
|
| ```
|
|
|
| ## Example
|
|
|
| Detection results on a sample screenshot (1920x1080) from the [ScreenSpot-Pro](https://github.com/likaixin2000/ScreenSpot-Pro-GUI-Grounding) benchmark (`conf=0.05`, `iou=0.1`, `imgsz=1280`).
|
|
|
| **Input Screenshot**
|
|
|
| <p align="center">
|
| <img src="images/example_input.png" width="80%" alt="Input Screenshot"/>
|
| </p>
|
|
|
| <table>
|
| <tr>
|
| <th align="center">OmniParser V2</th>
|
| <th align="center">GPA-GUI-Detector</th>
|
| </tr>
|
| <tr>
|
| <td align="center"><img src="images/example_omniparser.png" width="92%" alt="OmniParser V2"/></td>
|
| <td align="center"><img src="images/example_gpa.png" width="99%" alt="GPA-GUI-Detector"/></td>
|
| </tr>
|
| </table>
|
|
|
| ## License
|
|
|
| This model is released under the [MIT License](https://opensource.org/licenses/MIT).
|
|
|
