Mage-VL-Cowork for Apple Core AI

An experimental conversion of Microsoft's Mage-VL-Cowork for Apple's Core AI framework on Apple silicon.

Mage-VL-Cowork is a vision-language model for computer use: it works with screen images and instructions to propose actions. This project explores running that model with Core AI. Microsoft trained the original model; Taylor Arndt's work here is the conversion and testing.

This download is for developers and researchers who want to inspect the converted assets, run a small included test, or build on the conversion. It includes Core AI model assets, tokenizer and processor files, and a Swift runner with a Python smoke-test script.

Current status: the included smoke test runs, and selected tests show matching action proposals. Other tests show numerical differences, different actions, or missed targets. This is an experimental release, not a validated desktop agent. It does not control your computer or provide a ready-to-use screenshot-to-action command.

Run the included smoke test

The smoke test is a small check that the packaged decoder can load and reproduce its first two saved output tokens. It uses prepared synthetic input, not a new screenshot.

What you need

An Apple silicon Mac with the Core AI runtime and Xcode command-line tools. The recorded test setup is listed below; compatibility with other setups has not been established.
Python and the psutil package.
The downloaded repository, including its model assets and test fixture. Keep its folder structure intact.

In a terminal, open the downloaded repository folder. Install the script's Python dependency in your Python environment:

python -m pip install psutil

Then run:

python smoke.py --output /tmp/mage-smoke-new-run

Choose an output directory that does not already exist. The script compiles Greedy.swift using xcrun swiftc, then runs the test with time, memory, and disk guards. Compilation has a 300-second timeout; the runtime has a 600-second limit. On completion, read smoke-result.json in your output folder for the pass/fail result. If execution fails, inspect runtime.log.

The test was run on a Mac with 128 GiB of memory. The script requires 25 GiB of currently available memory and 35 GiB of free disk at startup. These checks are safeguards, not a promise that a machine with those resources is supported. It stops below 20 GiB available memory or 25 GiB free disk and limits its process tree to 32 GiB RSS. It does not close unrelated apps.

A passing smoke test confirms only this small packaged example. It does not establish general answer quality, screenshot understanding, or successful computer tasks. See PACKAGE-CHECKS.json for the completed checks.

Using your own inputs

There is no supported command in this release that takes an arbitrary screenshot and returns actions. The Swift runner takes a prepared-run folder and the package folder. Fresh vision inference requires prepared FP32 patches, cos, and sin arrays through the vision asset's main entry point. Full image preprocessing and prompt construction require additional integration work.

This is a Core AI asset package, not a Transformers-compatible checkpoint. Core AI and Core ML are different frameworks.

Test results

These are selected offline diagnostics, not a benchmark score. No proposed actions were executed during testing.

The checks answer different questions:

Numerical agreement: how closely the converted model's numbers match the reference when given the same preceding tokens.
Token agreement: whether the models generate the same sequence.
Action agreement: whether their parsed action lists match.
Target hit: whether proposed coordinates land inside the expected target.
Task success: whether an executed action completes a real task. This was not tested.

A match on one check does not imply a pass on the others.

Earlier twelve-case CPU tests

The completed greedy, teacher-forced suite compared 2,973 decisions:

Five of twelve cases passed the numerical checks.
Nine numerical positions failed across seven cases.
There were 32 disagreements about the highest-scoring next token.
Two of twelve cases had fully matching parsed action lists.
The lowest PSNR was 33.8336 dB on Calculator, below the 40 dB threshold used in these tests. PSNR measures numerical similarity; higher is better.

These tests used an older nine-group Python route. Earlier failures with the original BF16 conversion also remain unresolved. They should not be treated as fixed by the newer mixed-precision candidate.

Newer Swift tests

The newer split Swift route has encouraging individual results, but its larger planned test suite is incomplete.

ScreenSpot-v2 row018, with fresh native vision: generation reached the end-of-sequence token, the action list matched the source, and the proposed coordinates hit the target. Generated tokens still diverged at decision 111. Numerical checks on the shared prefix do not cover the later divergent outputs.
ScreenSpot-v2 row005, replaying source features: generation reached the end-of-sequence token with exactly matching source tokens and action proposals. A numerical check still failed at decision 192: 31.5773 dB against the 40 dB threshold. Both the source and converted model missed the target.
Other selected Swift cases include matching actions, differing actions, and missed targets.

The source comparison helper uses eager text attention. Microsoft's SGLang/FlashInfer/Triton execution and full transport behavior were not reproduced. These results are not a reproduction of Microsoft's published OSWorld result.

Limitations and safe use

Do not connect the model's proposals directly to a live desktop. No action executor is included in this release. Any future agent integration needs a disposable, isolated environment, limited permissions, human review, restricted network access, and no exposed credentials. Also follow the original model's Responsible AI guidance.

The following have not been validated:

Batching, multiple images, video, or long contexts.
iOS deployment or broad input-shape support.
Installation on a fresh machine.
Performance or actual GPU/Neural Engine placement.
General computer-use reliability or end-to-end task completion.

Tested images include selected 320 × 192 and 1024 × 512 synthetic inputs, plus 960 × 540 ScreenSpot-v2 inputs. Declared dynamic shapes do not establish support for every shape. Overlap between evaluation data and original training data has not been verified.

Observed warnings included CoreAnalytics Context leak detected and Python Failed to import MPS module. The two limited relocation regression checks still produced bit-exact outputs, but the warning causes and compute placement were not established.

Conversion details

Model assets and precision

The package contains 75 Core AI assets:

One FP32 vision asset.
36 BF16 decoder pre-attention assets.
36 BF16 decoder post-attention assets.
Shared attention and an output head.

It also includes the original BF16 embedding table as raw little-endian data with shape [151936, 2560].

Vision weights retain their original BF16 values, represented exactly in FP32, and vision computation uses FP32. Processor pixels are rounded to BF16 before being represented in FP32. The decoder keeps BF16 weights and computation. This changes vision arithmetic; it is not quantization or unchanged original-BF16 execution.

The decoder allocates 2,048 cache positions. That allocation is not a validated 2,048-token context-window claim. The original checkpoint and historical failed candidates are preserved separately.

Packaging checks

Assets were loaded with public AIModelAsset APIs, stripped using coreai_torch.debugging.debug_info.strip_debug_info, and saved as new assets. Original files were preserved.

All 75 packaged assets passed validity checks and binary scans for local home-directory prefixes. Asset paths resolve relative to the package root. After relocation, vision reproduced a saved synthetic native output bit-for-bit, and the decoder reproduced the first two tokens and BF16 logits bit-for-bit. These packaging checks do not resolve the accuracy failures above.

The package excludes upstream model Python source, the MLX reference implementation, GUI agent harnesses, dependencies, compiled binaries, and conversion caches. Install required dependencies separately from their official sources.

Recorded test environment

These versions describe the test machine, not minimum requirements or a clean-install guarantee:

Apple silicon Mac17,6, 128 GiB memory.
macOS 27.2 (26B123); Xcode 27.2 (27B52).
Python 3.12.16; torch 2.9.0; torchvision 0.24.0.
transformers 5.12.1; numpy 2.4.6.
coreai-torch 0.4.3; coreai-core 1.0.0b3.

Credits and license

Microsoft created and trained Mage-VL-Cowork. Its model card identifies Mage-ViT and the Qwen3-4B-Instruct-2507 text backbone. Apple provides Core AI and its toolchain. This is an independent conversion project by Taylor Arndt; no Microsoft or Apple endorsement is claimed.

The source checkpoint declares Apache-2.0. Preserve the full license, citation, applicable notices, and modification attribution when redistributing. See NOTICE.md and the licenses folder for the included attribution and license texts. The upstream file listing contains a LICENSE file and no NOTICE file.

MLX-VLM was consulted as an architectural reference, not used as the inference backend. Its reference revision is 4693cd0eaf022ae7ef61c6fdb55e17e6399879a7, under the MIT license, copyright 2025 Prince Canuma. The license is preserved in licenses/MLX-VLM-MIT.txt.

Cite the original model

@article{zhang2026magecowork,
  title={Mage-VL-Cowork: A Full Training Recipe to Learn Hybrid Computer Use from Evolved Harnesses},
  author={Zhang, Xiaoyi and Wang, Haoqing and Zhang, Xinjie and Ming, Xiang and Wang, Xing and Yang, Senqiao and Zhang, Kaichen and Jia, Zhaoyang and Tan, Andong and Wang, Xinjiang and Guo, Jinghao and Shen, Yifei and Li, Xiao and Zhang, Peng and Guo, Xun and Li, Jiahao and Zheng, Shicheng and Wang, Jinglu and Guo, Zongyu and Xie, Wenxuan and Li, Bin and Lu, Yan},
  journal={arXiv preprint},
  year={2026}
}

Downloads last month: -; Downloads are not tracked for this model. How to track

Model tree for tayarndt/Mage-VL-Cowork-CoreAI

Base model

microsoft/Mage-VL

Finetuned

microsoft/Mage-VL-Cowork

Finetuned

(2)

this model