Instructions to use netprojects/Mage-VL-Cowork-Grounding-MLX-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use netprojects/Mage-VL-Cowork-Grounding-MLX-4bit with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("netprojects/Mage-VL-Cowork-Grounding-MLX-4bit") config = load_config("netprojects/Mage-VL-Cowork-Grounding-MLX-4bit") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use netprojects/Mage-VL-Cowork-Grounding-MLX-4bit with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "netprojects/Mage-VL-Cowork-Grounding-MLX-4bit"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "netprojects/Mage-VL-Cowork-Grounding-MLX-4bit" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent
How to use netprojects/Mage-VL-Cowork-Grounding-MLX-4bit with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "netprojects/Mage-VL-Cowork-Grounding-MLX-4bit"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default netprojects/Mage-VL-Cowork-Grounding-MLX-4bit
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use netprojects/Mage-VL-Cowork-Grounding-MLX-4bit with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "netprojects/Mage-VL-Cowork-Grounding-MLX-4bit"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "netprojects/Mage-VL-Cowork-Grounding-MLX-4bit" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Mage-VL-Cowork-Grounding for Apple Silicon (MLX, 4-bit)
A fine-tune of microsoft/Mage-VL-Cowork (revision 6fbe7f9), Microsoft's 4B hybrid computer-use model, that
makes it click the right element on a screenshot. Mage-VL-Cowork drives a computer through computer_use (mouse and
keyboard) and bash; this version keeps that contract and adds precise grounding: on ScreenSpot-v2 it clicks the
target 72% of the time, against 13% for the original, and it chooses the shell for shell tasks far more often.
This build quantises the language model to 4-bit (affine, group 64) and keeps the vision tower in bf16. Size: 3.1 GB.
Version 0.1, published by NetProjects (Internet Projects Ltd), 18 November 2026. Not affiliated with Microsoft.
| Build | Size |
|---|---|
| netprojects/Mage-VL-Cowork-Grounding-MLX-6bit (recommended) | 4.2 GB |
| netprojects/Mage-VL-Cowork-Grounding-MLX-8bit | 5.3 GB |
| netprojects/Mage-VL-Cowork-Grounding-MLX-4bit | 3.1 GB |
| netprojects/Mage-VL-Cowork-Grounding-MLX-bf16 | 9.5 GB |
Results
| Test (offline, same harness, screenshots โค 1M px) | Mage-VL-Cowork | This fine-tune |
|---|---|---|
| ScreenSpot-v2, 300 single-click samples | 13.0% | 72.0% |
| ยท desktop / mobile / web | 10.1 / 16.1 / 11.7% | 51.9 / 81.4 / 76.7% |
| Agent loop: Microsoft's hybrid agent, up to 5 steps, 100 ScreenSpot-v2 tasks | 13% | 68% |
| AndroidControl, 70 test episodes, 333 recorded steps: step success | 11.7% | 47.1% |
| ยท click within 14% of the target | 11.2% | 48.0% |
| ยท correct action type | 67.9% | 73.3% |
Picks bash for 30 held-out shell tasks |
3 / 30 | 19 / 30 |
| Valid tool calls on 16 agent prompts | 15 / 16 | 16 / 16 |
All numbers come from offline evaluations on a MacBook Pro M5 Max (MLX, greedy decoding): each answer is scored, nothing is executed, and a screenshot step is answered with the same screen. ScreenSpot-v2 and the agent loop use our frozen 300-sample set (79 desktop, 118 mobile, 103 web) and Microsoft's unmodified agent code; AndroidControl uses Microsoft's GUI-agent prompt with the episode goal and the recorded history, a click counting as correct within 14% of the target (the usual threshold). Desktop is the weakest platform; mobile and web gain the most. Microsoft's own serving setup reports a higher ScreenSpot score for the original (about 43%, in a comment in their SGLang serving code) than our 13% at 1M pixels, so the gap against their best setup is smaller than this table shows. Both models here use the same harness, prompts and image size.
More results
The example is a ScreenSpot-v2 web screenshot (Apache-2.0) with the instruction "search my projects": the original clicks in the project list, the fine-tune clicks the search box (outlined).
How it was trained
- Method. LoRA (rank 16, alpha 32) on the Qwen3-4B language model's attention and MLP layers, merged into the weights; the Mage-ViT vision tower is unchanged. Loss on the assistant answer only. Every training prompt is built with Microsoft's own agent code, so it matches what the agent sends at run time.
- Data (Apache-2.0). Single-click grounding examples from OS-Atlas-data
(macOS, Windows, Linux, Android, web) and Aria-UI_Data
(desktop, web, mobile); multi-step phone steps from the AndroidControl
train split; the base model's own agent turns (including
bashrouting), so its existing agent behaviour is kept. Screenshots are resized to at most 1,000,000 pixels. - Clean test sets. Every training screenshot was checked against all ScreenSpot-v2 images and the AndroidControl test episodes we evaluate on (exact and perceptual hashes); matches were removed before training.
- Released weights (v0.1). The checkpoint measured above, merged into the weights. The released builds pass our quick agent checks (valid tool calls, the click lands on the target).
Intended use
Made for computer-use agents driven by Microsoft's agent system prompts (see the original model card). Like the original Mage-VL-Cowork, it is not a general chat model: without an agent prompt it tends to answer with tool calls. Plain-chat support is planned for a later version.
Run it
pip install -U "mlx-vlm>=0.7.11"
mlx_vlm.server --model netprojects/Mage-VL-Cowork-Grounding-MLX-4bit --port 8100
Use Microsoft's system prompts and turn format from the original model card and its
agent code. computer_use coordinates are in
[0, 999] on both axes: x_px = round(x / 999 * (width - 1)), the same for y. The processor in this repo
resizes screenshots to at most 1,000,000 pixels, the size the model was trained and tested at.
Use it in a sandbox
This is an autonomous agent that can run shell commands and GUI actions. Follow Microsoft's guidance: run it in an isolated container or VM, confirm irreversible actions with a human, keep credentials out of its workspace, and restrict network access when it reads untrusted content.
License
Apache-2.0, like Mage-VL-Cowork, Qwen3-4B and the training datasets. LICENSE is the Apache-2.0 text; NOTICE
describes the changes and credits Microsoft, Alibaba Cloud (Qwen), OS-Atlas, Aria-UI and AndroidControl. Please also
cite Microsoft's paper (see the original model card).
More builds from NetProjects: huggingface.co/netprojects ยท netprojects.com
- Downloads last month
- 18
4-bit



