dongjintao/MAI-Swift-2B

Single-Pass Mobile GUI Action Prediction
Examples · Results · Quick start · 简体中文

MAI-Swift-2B maps screenshots, instructions and executed-action history to one Android action through a single native causal forward over 105 fixed value slots. It is the mobile member of the GUI-Swift family, built on MAI-UI-2B.

Release: AndroidControl SFT · step4658   /   Adapter: 433.68M parameters · 1.73 GB

This is an independent research release, not an official MAI-team model. Single-pass describes the mechanism; no measured speedup or live task-success advantage is claimed.

Results at a glance

690 saved validation steps. Tolerance match requires the same action and fields, each coordinate axis within ±5%, text edit similarity ≥90%, and all other parameters exact. Thresholds were chosen after inspecting validation. This is a next-action metric, not a completed-task success rate.

Offline benchmark results

Evaluated on 2026-11-08 with the released step4658 adapter and unchanged native Mobile105 inference. These are offline predictions, not live task completion rates.

Benchmark	Metric	Correct / total	Result
ScreenSpot-v2 mobile	Tap point inside target box	310 / 501	61.88%
AndroidControl test	Tolerance action match	5,902 / 8,423	70.07%
AndroidControl test	Action type accuracy (diagnostic)	8,035 / 8,423	95.39%

• ScreenSpot-v2: all 501 mobile rows (238 ios, 211 android, 52 shop, preserving source labels). A prediction must be a tap inside the inclusive target box. All 61 non-tap predictions count as misses. Dataset revision 5efbb1f1b5463a575f2eb7bc30fe29e49c15f93c; inclusive hit-test reused from SeeClick revision 0ef37ac4d7aaf37ba7b990e3e3c3ca77e1fb8f93. Source annotations were not corrected for this evaluation.
• AndroidControl: all 8,423 local test steps with recorded demonstration history and low-level instructions, rather than self-generated rollout history. Match requires identical action/field sets, each coordinate axis within ±50/1000, text normalized Levenshtein similarity ≥90%, and other parameters exact. Text is case- and whitespace-sensitive. Thresholds were selected on validation before this test; this is a project-defined metric, not an official leaderboard score.
• Limitations: zero invalid decoded actions in either run, but parameter errors remain: AndroidControl tap 3,636/5,070 (71.72%), text entry 128/632 (20.25%), open app 277/600 (46.17%). Local AndroidControl trajectory splits are disjoint; base-model pretraining exposure is unknown. No identical-protocol base-model comparison or isolated speed benchmark is reported. AndroidWorld results are not yet available.

Machine-readable results, subgroup counts and prediction hashes. Raw trajectories and screenshots are not redistributed in this update. Sources: ScreenSpot-v2, SeeClick, AndroidControl.

Real validation examples

Actual saved checkpoint predictions, selected after evaluation. The screenshots show the current input; history was also supplied where recorded. This gallery includes a text failure and does not estimate performance. Original screenshots and predictions are linked with their hashes; no new inference was run to select improved outputs.

Locate a product WITHIN ±5% PER AXIS Instruction Click on the Modern Graceful Men Sports Shoes Prediction {"action": "tap", "coordinate": [252, 458]} Reference {"action": "tap", "coordinate": [219, 469]} Inspect saved record android-control:3851:0 · Historical observations: 0 JSON + SHA-256	Enter a search term TEXT MATCHES REFERENCE Instruction type the text running shoes Prediction {"action": "type_text", "text": "running shoes"} Reference {"action": "type_text", "text": "running shoes"} Inspect saved record android-control:10026:5 · Historical observations: 5 JSON + SHA-256
Scroll for details SCROLL DIRECTION MATCHES Instruction Swipe up for details Prediction {"action": "scroll_down"} Reference {"action": "scroll_down"} Inspect saved record android-control:3851:1 · Historical observations: 1 JSON + SHA-256	A limitation · Repeated text TEXT MISMATCH Instruction Search for Black day Prediction {"action": "type_text", "text": "Black day day"} Reference {"action": "type_text", "text": "Black day"} Inspect saved record android-control:3911:1 · Historical observations: 1 JSON + SHA-256

Quick start

Python 3.10 was used for verification. Install requirements.txt in an isolated environment with the official CUDA-enabled Torch 2.9.1 wheel appropriate to your machine. Use a CUDA GPU with sufficient memory (verification uses a 24 GB RTX 4090).

python prepare_dependencies.py --destination /path/on/data-disk/mai-swift-deps
python infer.py --base-model /path/on/data-disk/mai-swift-deps/MAI-UI-2B \
  --bundle /path/on/data-disk/mai-swift-deps/runtime-bundle \
  --request request.json

A request uses screenshot paths relative to the request file:

{"turns":[{"instruction":"Tap Settings.","image":"screen.png","receipt":null}]}

For subsequent observations, append a turn with its current instruction, screenshot, and a receipt containing the actually executed previous action and observed outcome:

{"instruction":"Open Wi-Fi settings.","image":"screen2.png","receipt":{"executed_action":{"action":"tap","coordinate":[500,300]},"outcome":"unverified"}}

Coordinates in receipts and decoded actions are normalized to 0–1000. The runner prints an action; it does not operate a device or verify task success. Preserve real outcomes; a requested action is not evidence it was executed. Historical assistant MASK tables are retained as scaffolds, and oldest complete turns are removed only when the 16,384-token window is exceeded.

Output protocol

One native causal forward reads logits at each slot's preceding position. The 105 slots are action (1), two coordinate pairs (4), and text (100). Action codes A–P represent tap, long_press, swipe, scroll_up/down/left/right, type_text, open_app, back, home, enter, menu, wait, done, failed. Coordinates reuse the pinned 256-bin GUI-Decisions codebook and local2 readout. Text is limited to 99 payload tokens plus EOS. Inactive fields are ignored by the decoder.

The adapter covers 301 linear modules in the vision tower, projections, language model and output head. Convolutional patch projection, embeddings and normalization are excluded. Frozen tied input/output weights remain shared; the output head has a separate LoRA delta. Do not blindly merge into tied embeddings.

Training and observed validation

AndroidControl only: 74,525 admitted training targets, 690 validation targets, 8,423 held-out test targets (not evaluated). One epoch, 4,658 optimizer updates, two RTX 4090s, global batch 16. Official LLaMA-Factory training loop with a local fixed-slot data/loss adapter; official PEFT rank 256, alpha 16, dropout 0.05, learning rate 1e-5. This checkpoint uses the Desktop131 objective: L_action + 2 * mean(active parameter-field losses), then a sample mean. Coordinate soft targets use sigma 0.005 with full-vocabulary normalization. This is not RL training.

Offline validation measure	Result
Action type accuracy	665/690 (96.38%)
Tolerance action match	494/690 (71.59%)
Invalid decoded actions	0/690
Mean validation loss	4.246688

Tolerance match requires the same action and field set; each coordinate axis within ±50/1000; text normalized Levenshtein similarity ≥90%; remaining parameters exact. These thresholds were chosen after inspecting validation and are not an established benchmark metric. They do not establish UI hit-testing, text semantic equivalence, or complete task success. Raw historical metrics remain in evaluation.json for provenance.

No AndroidWorld result, live mobile task completion rate, or measured latency advantage is claimed. One forward describes the implementation, not an independently benchmarked speedup. Validation was observed during development; held-out test results are reported separately above.

Reproduction and attribution

Base revision: 503050934809558c8dfd2ddedaf9621fa74ac2de. Final adapter SHA-256: d9ae8b33a987d515554bb16234d38506f663c881e6b776aa7eb0fb2f3eee8849.

release-validation.json records comparison against frozen inference; provenance.json, mai_swift_config.json and SHA256SUMS.json pin the release. Four curated AndroidControl validation screenshots and saved step predictions are included with provenance. Full conversations, training data, optimizer state and credentials are not included.

See THIRD_PARTY.md for dependencies and licensing. GUI-Decisions is external and hash-pinned because its selected public snapshot has no LICENSE file; it is not redistributed or relicensed here.

task_type is omitted from the adapter JSON: PEFT retains its default None and the generic adapter wrapper. This avoids the Hub parser rejecting a JSON null; it does not change the inference task or weights.