`handler.py` checks for `"RBGA"` (typo) so RGBA screenshots never convert — plus Hub `intersection_area` still wrong

#38
by SamJoshua - opened

Hi — while wiring OmniParser into a computer-use agent harness on CPU I hit two handler.py issues that are easy to miss and matter for real screenshot pipelines. (Not model-quality complaints.)

1. Typo: "RBGA" instead of "RGBA" (two call sites)

In both check_ocr_bboxes and get_som_labeled_img:

if image.mode == "RBGA":
    image = image.convert("RGB")

"RBGA" is never a real PIL mode, so the branch never runs. RGBA captures (common from some OS / browser screenshot APIs) stay 4-channel.

Repro:

from PIL import Image
import numpy as np

def handler_convert(image):
    # same check as handler.py today
    if image.mode == "RBGA":
        image = image.convert("RGB")
    return image

img = Image.new("RGBA", (64, 64), (255, 0, 0, 128))
out = handler_convert(img)
print(img.mode, "->", out.mode, np.array(out).shape)
# RGBA -> RGBA (64, 64, 4)   # conversion skipped

Fix: "RGBA" (and ideally also handle "LA" / other non-RGB modes, or always image.convert("RGB")).

2. Hub intersection_area still uses min for the top-left edge

Open PR #26 already flags this against the GitHub util; the Hub handler.py copy is still the broken version:

def intersection_area(bbox_left, bbox_right) -> int:
    return max(0, min(bbox_left[2], bbox_right[2]) - min(bbox_left[0], bbox_right[0])) * max(
        0, min(bbox_left[3], bbox_right[3]) - min(bbox_left[1], bbox_right[1])
    )

Correct IoU intersection uses max for the top-left and min for the bottom-right. Measured on the Hub function:

case Hub area correct area
partial overlap 10000 2500
contained 5625 2500
no overlap 100 0
identical 1600 1600

Non-overlapping boxes getting a non-zero intersection is the scary one for agents: overlap suppression / OCR-vs-icon fusion can drop or keep the wrong region, so the policy model clicks the wrong control.

Happy to open a tiny PR that only flips RBGA → RGBA (and leave the IoU fix to #26) if that's easier to merge.

Side note for people hitting empty config.json / Colab AutoModel (#35, #2)

Root config.json is still an empty file, so AutoModel.from_pretrained("microsoft/OmniParser-v2.0") fails. The card's MIT icon_detect_v3 TorchScript path works on CPU without ultralytics. I ran it on a synthetic 800×500 UI (torch 2.15.0+cpu, pure-torch NMS): 13 boxes, top conf ~0.41, ~17s for one forward pass on this box. Happy to paste the exact script if useful.

— Sam (building / red-teaming tool-using and computer-use agents)

Sign up or log in to comment