GUI-Swift-4B / docs /technical-details.md
dongjintao's picture
Report position tolerance at 50/1000 and clarify future RL direction
8da55cc verified
|
Raw History Blame Contribute Delete
10.6 kB

GUI-Swift-4B 路 Technical details

Single-Pass GUI Action Prediction

GUI-Swift-4B is a research adapter for GUI-Owl-1.5-4B-Instruct. It predicts a desktop GUI action, including its applicable parameters and text, through one native causal model forward over 131 fixed value slots. The frozen backbone retains its original vocabulary head. There is no separate autoregressive text writer in this release.

This first release corresponds to Desktop131 SFT step3368. It is a desktop-only checkpoint, not Unified133, not a mobile-trained model, and not an RL checkpoint. The name describes the intended efficient execution approach; this release does not establish a matched task-success/latency advantage over the base model.

Attribution and relationship to GUI-Decisions

Our principal methodological inspiration from GUI-Decisions is its E-arm 256-bin coordinate representation and readout. Each x/y axis is represented by one label from a 256-token codebook. Historical coordinates use exactly the same codebook as predicted coordinates.

Single-forward complete-action prediction was already being explored in this project before adopting that coordinate scheme. We distinguish that design history from implementation reuse: the released runtime calls the original GUI-Decisions slotread.train.read and uses its label/numerical utilities from commit 4afc7f24694bf41d84b58e761b218c694c43f13b. We gratefully acknowledge this contribution. This is an independent project, not an official GUI-Decisions release or a full reproduction of its training method.

The pinned public tree omits slotread/data, although the training module imports it. The inference adapter therefore loads only the SHA-256-verified read function from that file, preserving its body, rather than importing unrelated training-data code. Label and numeric utilities are imported normally. This small compatibility adapter is checked against the original complete private runtime snapshot.

The selected upstream snapshot has no license file. Its source is not bundled or relicensed here; users obtain the pinned dependency from the author's repository. Our Apache-2.0 license applies to the project-owned release code and adapter, not that external dependency. See THIRD_PARTY.md.

We also acknowledge GUI-Owl / Mobile-Agent for the pretrained backbone, and PEFT/Transformers for model and adapter support.

Quick start

Use Python 3.10+ and an NVIDIA GPU. The release uses a custom inference entry point; ordinary generate() or loading the adapter into the whole VLM with a generic PeftModel.from_pretrained call does not reproduce this protocol. The adapter is attached to the language-model submodule, preserving the original loader.

python -m pip install -r requirements.txt
python prepare_dependencies.py --destination /path/on/data-disk/gui-swift-deps
python infer.py \
  --base-model /path/on/data-disk/gui-swift-deps/GUI-Owl-1.5-4B-Instruct \
  --gui-decisions /path/on/data-disk/gui-swift-deps/GUI-Decisions \
  --request examples/request.json

Dependency preparation downloads the pinned base model (approximately 8.9 GB) and the external GUI-Decisions code. Existing matching local directories can instead be passed directly to infer.py. Every required base and upstream source file is checked against recorded hashes. The inference script prints an action and does not control the mouse or keyboard.

A request contains:

{
  "instruction": "Click the Save button.",
  "image": "screen.png",
  "history": []
}

Image paths are relative to the request file. For history, add records containing image and actions (a list of actually executed canonical actions). All supplied historical actions and the last two historical screenshots are retained. Do not add ground-truth next actions to the input. The included example is a synthetic UI smoke input, not a benchmark result or real-application task success.

Action protocol

Field Slots
Action 3
x, y 1 each
Keys 16
Scroll amount 4
Time 5
Status 1
Text 100
Total 131

Coordinates returned by the decoder are normalized integers in 0..1000; coordinate distributions use the original local2 readout. Text is limited to 99 native tokens plus EOS. Drag has one destination coordinate pair and starts at the current pointer; this version does not encode both endpoints. Scroll amount retains the historical wheel-count convention despite the field name pixels. The runtime preserves catalogue-constrained action/key decoding and the original invalid-output behavior. It has no arbitrary shell action.

Training

The checkpoint combines a 64-update pilot over 512 examples with a continuation of 3,368 updates over the remaining 26,944 examples, visiting the 27,456-example training split once in total. step3368 is the continuation step number, not the number of updates including the pilot. The saved validation split has 1,967 rows and was used repeatedly during development.

  • LoRA: rank 16, alpha 32, dropout 0.05; 252 language attention/MLP linear modules.
  • Adapted projections: q, k, v, o, gate, up and down; 33,030,144 FP32 parameters.
  • Frozen: vision encoder/mergers, input embeddings, original tied output head and BF16 base parameters.
  • Historical learning rate: 2e-4; effective global batch: 8.
  • Objective: action-field CE plus twice the mean of active parameter-field losses, with full-vocabulary normalization; coordinates use fixed sigma 0.005 targets.
  • This is not the later planned all-linear rank-256 configuration, token-mean Unified133 objective, or box-adaptive/equal-field-mean training plan.

The existing data audit describes a mixture of AgentNet (15,754 training steps), HybridCUA (11,303), and OS-Atlas (399). All 1,967 validation rows are from AgentNet. The raw corpus is not bundled. Five selected validation examples are shared in examples/validation/ under the original AgentNet MIT terms. The release does not establish absence of semantic overlap with public benchmarks.

Existing evaluation

These are saved historical measurements, not a fresh benchmark run for release.

Evaluation Result Scope
OSWorld-derived selected suite 58 / 356 = 16.29% full success 361 selected of 369 official tasks; 5 unscored
Mean environment reward 0.1712077 356 scored tasks only
Validation action-type accuracy 75.60% 1,967 development rows
Validation canonical exact match 24.05% Same 1,967 rows; strict full-action matching
Validation text exact match 120 / 399 = 30.08% Ground-truth type rows

The five unscored tasks are not silently converted to zero rewards. The OSWorld number is not a complete official leaderboard score. This checkpoint was selected for release after observing historical results. Protocol differences prevent a causal comparison with other historical models or the upstream model-card scores. Terra-guided demonstrations, including any direct teacher actions, are not included as standalone model benchmark results. See evaluation.json for the saved counts.

Coordinate tolerance reporting update

On 2026-11-06 the user selected a 50/1000 per-axis tolerance for the model-card headline after inspecting the original results. This is a post-hoc recount of the same saved predictions; no inference or environment evaluation was rerun. The original evaluation.json remains unchanged.

Position criterion, with matching action type Matches / reference-coordinate rows Rate
Both axis errors <= 20/1000 (original) 456 / 1,104 41.30%
Both axis errors <= 50/1000 (updated headline) 548 / 1,104 49.64%

All 1,104 reference actions with a primary coordinate field stay in the denominator. Missing or invalid predictions are non-matches. Each coordinate is normalized to 0-1000; 50 allows 5% of screen width horizontally and 5% of screen height vertically. This metric checks the action type and primary point only; widget boundaries, drag endpoints, scroll amounts and other parameters are not checked. It is not a task-success metric. Direct coordinate differences were cross-checked against the saved per-axis errors, reproducing the original 20/1000 result. Recount definition, counts and source hash.

Strict canonical full-action matching remains 24.05% (473/1,967). It requires literal decoded parameter equality, so even a prediction in the correct coordinate bins can fail this metric. The GIMP gallery example has both bins correct but decoded [155,243] versus reference [153,243].

Future reinforcement learning

This SFT checkpoint is a starting point for future reinforcement learning using environment-verifier feedback, with the goal of improving end-to-end task completion. The reported 16.29% remains the measured SFT result; RL improvement is a future objective, not an established gain for this release.

Limitations

This research model can misclick, repeat actions, emit invalid text, and stop prematurely. Text and dragging remain weak. A single forward does not imply single-step task completion or a guaranteed latency advantage. There is no verified mobile compatibility in this release. Integrating with an executor requires correct coordinate conversion, scroll conventions and environment-level outcome checks.

Release identity

The original checkpoint SHA-256 is bd82d3c508a7b591b7c1578614231b142689982c842c6864c80baaa213e5dde5. The released adapter_model.safetensors contains only the adapter tensors; no optimizer state, credentials or local research datasets are included. Frozen runtime sources and extracted compatibility components are recorded in provenance.json; SHA256SUMS.json binds the published files.

Release validation on one RTX 4090 compared the original runtime and this package in separate processes. On two saved inputs (current screenshot only, and full history), input tensors, logits, and decoded actions matched exactly, with one forward per input. All 504 exported adapter tensors also matched exactly. The synthetic example's actual output is in examples/output.json. Peak allocated GPU memory was about 9.1 GiB in this bounded check; this is not a maximum-memory claim for every image/context length. See release-validation.json for the environment.