| --- |
| license: mit |
| tags: |
| - object-detection |
| - yolo11 |
| - ui-elements |
| - windows |
| - ultralytics |
| datasets: |
| - ui_synth_v2 |
| pipeline_tag: object-detection |
| --- |
| |
| # Windows UI Element Detector — YOLO11s for Windows UI Elements |
|
|
| ## Model Summary |
|
|
| A YOLO11s (small) model fine-tuned on 3 000 synthetic Windows-style UI screenshots to detect interactive UI elements. Designed as a lightweight computer-vision fallback for Windows UI automation agents when native UI Automation APIs fail. |
|
|
| ## Classes |
|
|
| | ID | Class | |
| |----|------------| |
| | 0 | button | |
| | 1 | textbox | |
| | 2 | checkbox | |
| | 3 | dropdown | |
| | 4 | icon | |
| | 5 | tab | |
| | 6 | menu_item | |
| |
| ## Training Data |
| |
| Trained on `ui_synth_v2`, a synthetic dataset of 3 000 Windows-style UI screenshots generated via HTML/CSS templates rendered with Playwright. Includes domain randomization (themes, fonts, scaling, noise). |
| |
| ## Metrics |
| |
| | Metric | Value | |
| |--------------|--------| |
| | mAP50 | 0.9886 | |
| | mAP50-95 | 0.9543 | |
| | Precision | 0.9959 | |
| | Recall | 0.9730 | |
| |
| ### Per-Class AP@50 |
| |
| | Class | AP@50 | |
| |------------|--------| |
| | button | 0.9919 | |
| | textbox | 0.9771 | |
| | checkbox | 0.9864 | |
| | dropdown | 0.9829 | |
| | icon | 0.9950 | |
| | tab | 0.9950 | |
| | menu_item | 0.9915 | |
|
|
| ## Usage |
|
|
| ```python |
| from local_ui_locator import detect_elements, find_by_text, safe_click_point |
| |
| # Detect all UI elements in a screenshot |
| detections = detect_elements("screenshot.png", conf=0.3) |
| for det in detections: |
| print(f"{det.type}: {det.bbox} score={det.score:.2f}") |
| |
| # Find element by text |
| match = find_by_text("screenshot.png", query="Submit") |
| if match: |
| x, y = safe_click_point(match.bbox) |
| print(f"Click at ({x}, {y})") |
| ``` |
|
|
| ### Direct Ultralytics usage |
|
|
| ```python |
| from ultralytics import YOLO |
| |
| model = YOLO("best.pt") |
| results = model.predict("screenshot.png", conf=0.3) |
| ``` |
|
|
| ## Architecture |
|
|
| - **Base model:** YOLO11s (Ultralytics) |
| - **Input size:** 640px |
| - **Parameters:** ~9.4M |
| - **GFLOPs:** ~21.3 |
| - **Inference speed:** ~44-80ms on CPU (M2 Pro), ~2-5ms on GPU (RTX 5060) |
|
|
| ## Training |
|
|
| - **GPU:** NVIDIA RTX 5060 8GB (Blackwell) |
| - **Dataset:** 3 000 synthetic images (2 400 train / 300 val / 300 test) |
| - **Epochs:** 120 (early stopping with patience=25) |
| - **Batch size:** 16 |
| - **Image size:** 640px |
| - **Optimizer:** SGD with cosine LR scheduler |
|
|
| ## Limitations |
|
|
| - Trained on synthetic data only — real-world Windows UI may show domain gap |
| - Best on standard Windows 10/11 UI; custom-styled applications may perform worse |
| - Does not detect text content (use OCR for that) |
| - 7 classes only; complex widget types are not supported |
|
|
| ## License |
|
|
| MIT |
|
|
