macos-gui-agent-qwen2.5-vl-3b

LoRA fine-tune of Qwen/Qwen2.5-VL-3B-Instruct for macOS GUI automation from screenshots.

This model is intended to look at a macOS/app screenshot plus a natural-language task and emit the next GUI action as a JSON object. It is an experimental screenshot-to-action adapter, not a complete desktop automation product.

Repository:

• Model: https://huggingface.co/GumbiiDigital/macos-gui-agent-qwen2.5-vl-3b
• Training dataset: https://huggingface.co/datasets/GumbiiDigital/macos-gui-agent-train
• Base model: https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct
• Trackio run: https://GumbiiDigital-mlintern-macgui.hf.space?project=macos-gui-agent&runs=macos-gui-agent-v2&sidebar=collapsed

What this is for

Given:

1. A screenshot.
2. A task such as click the UI element Create new... or Open the Settings tab.

The adapter should respond with JSON similar to:

{
  "action": "click",
  "position": [0.1575, 0.134],
  "value": null
}

position is expected to use normalized screen coordinates in [x, y], where both values should be between 0 and 1.

The target action schema used during training was:

{
  "action": "click|type|scroll|drag|hotkey",
  "position": [0.0, 0.0],
  "value": "text to type, if applicable"
}

Some training examples also include metadata keys such as app, category, or target_description.

Important: this is a LoRA adapter

This repository contains a PEFT/LoRA adapter, not a merged full model. Load it on top of the base model Qwen/Qwen2.5-VL-3B-Instruct.

Main adapter file:

• adapter_model.safetensors — 594,546,704 bytes

LoRA config:

• rank r: 64
• alpha: 16
• dropout: 0.05
• target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• PEFT version: 0.18.1

Quick start

Install dependencies:

pip install torch transformers peft qwen-vl-utils pillow accelerate

Run inference:

import torch
from PIL import Image
from peft import PeftModel
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration

BASE_MODEL = "Qwen/Qwen2.5-VL-3B-Instruct"
ADAPTER = "GumbiiDigital/macos-gui-agent-qwen2.5-vl-3b"

processor = AutoProcessor.from_pretrained(
    BASE_MODEL,
    min_pixels=256 * 28 * 28,
    max_pixels=1344 * 28 * 28,
)

base_model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, ADAPTER)
model.eval()

image = Image.open("screenshot.png").convert("RGB")

task = "click the UI element Create new..."
messages = [
    {
        "role": "system",
        "content": [
            {
                "type": "text",
                "text": (
                    "You are a macOS automation assistant that controls the computer "
                    "via the Accessibility API. Given a screenshot and a task, output "
                    "the next action as a JSON object with keys: 'action' "
                    "(click|type|scroll|drag|hotkey), 'position' [x_norm, y_norm] "
                    "normalized 0-1, and 'value' for text if applicable."
                ),
            }
        ],
    },
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": f"Task: {task}"},
        ],
    },
]

text = processor.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = processor(
    text=[text],
    images=[image],
    padding=True,
    return_tensors="pt",
).to(model.device)

with torch.no_grad():
    generated_ids = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=False,
    )

trimmed = [
    output_ids[len(input_ids):]
    for input_ids, output_ids in zip(inputs.input_ids, generated_ids)
]

output = processor.batch_decode(
    trimmed,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False,
)[0]
print(output)

Expected style:

{"action": "click", "position": [0.1575, 0.134], "value": null}

Training data

Training dataset: GumbiiDigital/macos-gui-agent-train

The dataset contains 6,000 screenshot/action examples built from:

• 2,000 examples from ritzzai/GUI-R1
• 2,000 examples from macpaw-research/GUIrilla-Task
• 2,000 examples from macpaw-research/Screen2AX-Task

Dataset shape:

{
    "messages": [
        {"role": "system", "content": [{"type": "text", "text": "..."}]},
        {"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "Task: ..."}]},
        {"role": "assistant", "content": [{"type": "text", "text": "{...action json...}"}]},
    ],
    "images": [PIL image]
}

Dataset audit at training time:

• total rows: 6,000
• action distribution:
  • click: 5,934 examples
  • type: 66 examples
• non-null value fields: 0
• target positions normalized to 0-1: yes, 5,934/5,934 examples with positions were in range
• metadata keys:
  • app: 2,000 examples
  • category: 2,000 examples
  • target_description: 2,000 examples

This means the adapter is primarily a click-location model. It should not be considered trained for reliable typing, scrolling, dragging, or hotkeys.

Training procedure

Training command used TRL SFTTrainer with PEFT LoRA.

Core settings:

• base model: Qwen/Qwen2.5-VL-3B-Instruct
• epochs: 1
• dataset rows: 6,000
• optimizer steps: 750
• per-device train batch size: 1
• gradient accumulation: 8
• learning rate: 2e-5
• scheduler: cosine
• warmup ratio: 0.03
• max length: 2048
• precision: bf16
• gradient checkpointing: enabled
• save strategy: epoch

Framework versions:

• TRL: 1.3.0
• Transformers: 4.57.6
• PyTorch: 2.11.0+cu130
• Datasets: 4.8.4
• PEFT: 0.18.1
• Tokenizers: 0.22.2

Final training state:

• global step: 750
• epoch: 1.0
• final loss: 7.015
• final mean token accuracy: 0.0914150146767497
• final tokens seen: 8,578,259
• adapter size: 594,546,704 bytes

Training loss summary from trainer_state.json:

• first logged loss: 16.9928
• final logged loss: 7.015
• minimum logged loss: 6.9813
• mean first 10 logged losses: 13.01959
• mean last 10 logged losses: 7.01031
• mean first 10 token accuracy: 0.04403
• mean last 10 token accuracy: 0.09214

Evaluation status

No proper held-out benchmark has been run yet. The numbers below are smoke-test checks, not a real validation score.

A 6-example in-training smoke sample gave:

• JSON-valid outputs: 6/6
• action label match: 6/6
• median normalized position L2 error: 0.1147

The same 6 examples using the unfine-tuned base model gave:

• JSON-valid outputs: 6/6
• action label match: 1/6
• outputs often used left_click, raw pixel coordinates, or extra markdown fences

Observed adapter behavior from smoke testing:

• Strong improvement in following the requested JSON style compared with the base model.
• Strong improvement in using click instead of non-schema actions like left_click.
• Usually emits normalized coordinates, but at least one smoke output still produced raw pixel coordinates ([763, 842]).
• Coordinate accuracy is inconsistent. Some examples are close; others are far from target.

Practical verdict:

• Good enough as a first experimental adapter proving the pipeline works.
• Not yet good enough to trust for autonomous macOS control without guardrails.
• Needs a held-out evaluation set and better action diversity before being called production-ready.

Limitations

Major limitations:

1. The dataset is almost entirely click actions.

   • 5,934/6,000 examples are click.
   • Only 66 examples are type.
   • No meaningful training coverage for scroll, drag, hotkey, or text entry values.

2. The model may still output invalid coordinate scale.

   • Target format is normalized 0-1.
   • Smoke testing found at least one adapter output using raw pixel coordinates.

3. No held-out validation score yet.

   • The current evaluation is an in-training smoke test.
   • It proves formatting improved, not real generalization.

4. The model does not execute actions.

   • It only predicts JSON.
   • A separate macOS Accessibility API executor would be required.
   • Any executor should validate JSON, clamp coordinates, require safety checks, and avoid destructive actions.

5. It is not a general web/desktop agent.

   • It was trained on GUI screenshots and action labels.
   • It does not have planning, memory, browsing, or tool execution built in.

Recommended safety wrapper

Before sending actions to a real computer, wrap outputs with checks:

• parse strict JSON only
• reject markdown/fenced code
• require action in allowed set
• clamp position to [0, 1]
• reject raw pixel coordinates unless explicitly converted
• require confirmation for destructive actions
• add a screenshot-before/screenshot-after loop
• keep a human abort hotkey

Example validation rule:

def validate_action(action):
    allowed = {"click", "type", "scroll", "drag", "hotkey"}
    if action.get("action") not in allowed:
        raise ValueError("unsupported action")
    pos = action.get("position")
    if action.get("action") in {"click", "drag"}:
        if not isinstance(pos, list) or len(pos) != 2:
            raise ValueError("missing normalized position")
        x, y = pos
        if not (0 <= x <= 1 and 0 <= y <= 1):
            raise ValueError("position must be normalized 0-1")
    return action

Intended users

This adapter is intended for researchers and builders experimenting with:

• GUI grounding
• macOS computer-use agents
• screenshot-to-action models
• Accessibility API automation
• VLM fine-tuning pipelines

It is not intended for unsupervised control of a personal computer.

How to improve next

Recommended next steps:

1. Build a held-out test split and report exact metrics.
2. Add action-balanced data for type, scroll, drag, and hotkey.
3. Include non-null value examples for text entry.
4. Enforce normalized-coordinate output during decoding or post-processing.
5. Evaluate coordinate error by dataset source and task category.
6. Add real macOS screenshot rollouts with a human approval gate.
7. Consider a second training pass after fixing the output schema and balancing examples.

Citation

@software{vonwerra2020trl,
  title   = {{TRL: Transformers Reinforcement Learning}},
  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
  license = {Apache-2.0},
  url     = {https://github.com/huggingface/trl},
  year    = {2020}
}