Instructions to use microsoft/OmniParser-v2.0 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use microsoft/OmniParser-v2.0 with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("microsoft/OmniParser-v2.0", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
`handler.py` checks for `"RBGA"` (typo) so RGBA screenshots never convert — plus Hub `intersection_area` still wrong
Hi — while wiring OmniParser into a computer-use agent harness on CPU I hit two handler.py issues that are easy to miss and matter for real screenshot pipelines. (Not model-quality complaints.)
1. Typo: "RBGA" instead of "RGBA" (two call sites)
In both check_ocr_bboxes and get_som_labeled_img:
if image.mode == "RBGA":
image = image.convert("RGB")
"RBGA" is never a real PIL mode, so the branch never runs. RGBA captures (common from some OS / browser screenshot APIs) stay 4-channel.
Repro:
from PIL import Image
import numpy as np
def handler_convert(image):
# same check as handler.py today
if image.mode == "RBGA":
image = image.convert("RGB")
return image
img = Image.new("RGBA", (64, 64), (255, 0, 0, 128))
out = handler_convert(img)
print(img.mode, "->", out.mode, np.array(out).shape)
# RGBA -> RGBA (64, 64, 4) # conversion skipped
Fix: "RGBA" (and ideally also handle "LA" / other non-RGB modes, or always image.convert("RGB")).
2. Hub intersection_area still uses min for the top-left edge
Open PR #26 already flags this against the GitHub util; the Hub handler.py copy is still the broken version:
def intersection_area(bbox_left, bbox_right) -> int:
return max(0, min(bbox_left[2], bbox_right[2]) - min(bbox_left[0], bbox_right[0])) * max(
0, min(bbox_left[3], bbox_right[3]) - min(bbox_left[1], bbox_right[1])
)
Correct IoU intersection uses max for the top-left and min for the bottom-right. Measured on the Hub function:
| case | Hub area | correct area |
|---|---|---|
| partial overlap | 10000 | 2500 |
| contained | 5625 | 2500 |
| no overlap | 100 | 0 |
| identical | 1600 | 1600 |
Non-overlapping boxes getting a non-zero intersection is the scary one for agents: overlap suppression / OCR-vs-icon fusion can drop or keep the wrong region, so the policy model clicks the wrong control.
Happy to open a tiny PR that only flips RBGA → RGBA (and leave the IoU fix to #26) if that's easier to merge.
Side note for people hitting empty config.json / Colab AutoModel (#35, #2)
Root config.json is still an empty file, so AutoModel.from_pretrained("microsoft/OmniParser-v2.0") fails. The card's MIT icon_detect_v3 TorchScript path works on CPU without ultralytics. I ran it on a synthetic 800×500 UI (torch 2.15.0+cpu, pure-torch NMS): 13 boxes, top conf ~0.41, ~17s for one forward pass on this box. Happy to paste the exact script if useful.
— Sam (building / red-teaming tool-using and computer-use agents)