Instructions to use microsoft/OmniParser-v2.0 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use microsoft/OmniParser-v2.0 with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("microsoft/OmniParser-v2.0", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Use 0.05 confidence threshold in icon_detect_v3 example to match icon_detect default
Browse files
README.md
CHANGED
|
@@ -35,7 +35,7 @@ from torchvision.ops import nms
|
|
| 35 |
|
| 36 |
IMGSZ = 1280
|
| 37 |
STRIDES = (8, 16, 32)
|
| 38 |
-
CONF_THRESHOLD = 0.
|
| 39 |
IOU_THRESHOLD = 0.45
|
| 40 |
|
| 41 |
model = torch.jit.load("icon_detect_v3/model.pt", map_location="cpu").eval()
|
|
@@ -89,7 +89,7 @@ boxes[:, 1::2] = boxes[:, 1::2].clamp(0, height)
|
|
| 89 |
|
| 90 |
Notes:
|
| 91 |
- The model is single class (interactable region), so `cls` has one channel per scale.
|
| 92 |
-
- `CONF_THRESHOLD`
|
| 93 |
- The letterbox padding above is top-left anchored, which keeps the coordinate mapping to a single `scale` division. If you center the padding instead, subtract the pad offsets before dividing.
|
| 94 |
|
| 95 |
# Responsible AI Considerations
|
|
|
|
| 35 |
|
| 36 |
IMGSZ = 1280
|
| 37 |
STRIDES = (8, 16, 32)
|
| 38 |
+
CONF_THRESHOLD = 0.05
|
| 39 |
IOU_THRESHOLD = 0.45
|
| 40 |
|
| 41 |
model = torch.jit.load("icon_detect_v3/model.pt", map_location="cpu").eval()
|
|
|
|
| 89 |
|
| 90 |
Notes:
|
| 91 |
- The model is single class (interactable region), so `cls` has one channel per scale.
|
| 92 |
+
- `CONF_THRESHOLD` defaults to `0.05` to match the `box_threshold` that OmniParser uses with `icon_detect`. The detector is deliberately low confidence on small UI elements, so raising this much above `0.1` starts dropping real elements (icons, toolbar buttons, footer links) rather than just filtering noise. Tune `CONF_THRESHOLD` / `IOU_THRESHOLD` for your screenshots the same way you would tune `box_threshold` / `iou_threshold` for `icon_detect`.
|
| 93 |
- The letterbox padding above is top-left anchored, which keeps the coordinate mapping to a single `scale` division. If you center the padding instead, subtract the pad offsets before dividing.
|
| 94 |
|
| 95 |
# Responsible AI Considerations
|