You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

Mage-VL-Cowork-Grounding for Apple Silicon (MLX, 8-bit)

Mage-VL-Cowork-Grounding v0.1 vs original

A fine-tune of microsoft/Mage-VL-Cowork (revision 6fbe7f9), Microsoft's 4B hybrid computer-use model, that makes it click the right element on a screenshot. Mage-VL-Cowork drives a computer through computer_use (mouse and keyboard) and bash; this version keeps that contract and adds precise grounding: on ScreenSpot-v2 it clicks the target 72% of the time, against 13% for the original, and it chooses the shell for shell tasks far more often. This build quantises the language model to 8-bit (affine, group 64) and keeps the vision tower in bf16. Size: 5.3 GB.

Version 0.1, published by NetProjects (Internet Projects Ltd), 18 November 2026. Not affiliated with Microsoft.

Results

Test (offline, same harness, screenshots โ‰ค 1M px) Mage-VL-Cowork This fine-tune
ScreenSpot-v2, 300 single-click samples 13.0% 72.0%
ยท desktop / mobile / web 10.1 / 16.1 / 11.7% 51.9 / 81.4 / 76.7%
Agent loop: Microsoft's hybrid agent, up to 5 steps, 100 ScreenSpot-v2 tasks 13% 68%
AndroidControl, 70 test episodes, 333 recorded steps: step success 11.7% 47.1%
ยท click within 14% of the target 11.2% 48.0%
ยท correct action type 67.9% 73.3%
Picks bash for 30 held-out shell tasks 3 / 30 19 / 30
Valid tool calls on 16 agent prompts 15 / 16 16 / 16

All numbers come from offline evaluations on a MacBook Pro M5 Max (MLX, greedy decoding): each answer is scored, nothing is executed, and a screenshot step is answered with the same screen. ScreenSpot-v2 and the agent loop use our frozen 300-sample set (79 desktop, 118 mobile, 103 web) and Microsoft's unmodified agent code; AndroidControl uses Microsoft's GUI-agent prompt with the episode goal and the recorded history, a click counting as correct within 14% of the target (the usual threshold). Desktop is the weakest platform; mobile and web gain the most. Microsoft's own serving setup reports a higher ScreenSpot score for the original (about 43%, in a comment in their SGLang serving code) than our 13% at 1M pixels, so the gap against their best setup is smaller than this table shows. Both models here use the same harness, prompts and image size.

More results

Results by platform

Phone tasks by action type

One screenshot, two models: the original misses, ours lands on the target

The example is a ScreenSpot-v2 web screenshot (Apache-2.0) with the instruction "search my projects": the original clicks in the project list, the fine-tune clicks the search box (outlined).

How it was trained

  • Method. LoRA (rank 16, alpha 32) on the Qwen3-4B language model's attention and MLP layers, merged into the weights; the Mage-ViT vision tower is unchanged. Loss on the assistant answer only. Every training prompt is built with Microsoft's own agent code, so it matches what the agent sends at run time.
  • Data (Apache-2.0). Single-click grounding examples from OS-Atlas-data (macOS, Windows, Linux, Android, web) and Aria-UI_Data (desktop, web, mobile); multi-step phone steps from the AndroidControl train split; the base model's own agent turns (including bash routing), so its existing agent behaviour is kept. Screenshots are resized to at most 1,000,000 pixels.
  • Clean test sets. Every training screenshot was checked against all ScreenSpot-v2 images and the AndroidControl test episodes we evaluate on (exact and perceptual hashes); matches were removed before training.
  • Released weights (v0.1). The checkpoint measured above, merged into the weights. The released builds pass our quick agent checks (valid tool calls, the click lands on the target).

Intended use

Made for computer-use agents driven by Microsoft's agent system prompts (see the original model card). Like the original Mage-VL-Cowork, it is not a general chat model: without an agent prompt it tends to answer with tool calls. Plain-chat support is planned for a later version.

Run it

pip install -U "mlx-vlm>=0.7.11"
mlx_vlm.server --model netprojects/Mage-VL-Cowork-Grounding-MLX-8bit --port 8100

Use Microsoft's system prompts and turn format from the original model card and its agent code. computer_use coordinates are in [0, 999] on both axes: x_px = round(x / 999 * (width - 1)), the same for y. The processor in this repo resizes screenshots to at most 1,000,000 pixels, the size the model was trained and tested at.

Use it in a sandbox

This is an autonomous agent that can run shell commands and GUI actions. Follow Microsoft's guidance: run it in an isolated container or VM, confirm irreversible actions with a human, keep credentials out of its workspace, and restrict network access when it reads untrusted content.

License

Apache-2.0, like Mage-VL-Cowork, Qwen3-4B and the training datasets. LICENSE is the Apache-2.0 text; NOTICE describes the changes and credits Microsoft, Alibaba Cloud (Qwen), OS-Atlas, Aria-UI and AndroidControl. Please also cite Microsoft's paper (see the original model card).

More builds from NetProjects: huggingface.co/netprojects ยท netprojects.com

Downloads last month
20
Safetensors
Model size
5B params
Tensor type
U32
ยท
BF16
ยท
MLX
Hardware compatibility
Log In to add your hardware

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for netprojects/Mage-VL-Cowork-Grounding-MLX-8bit

Finetuned
(7)
this model

Datasets used to train netprojects/Mage-VL-Cowork-Grounding-MLX-8bit