# GUI-Swift-4B ยท Technical details **Single-Pass GUI Action Prediction** GUI-Swift-4B is a research adapter for GUI-Owl-1.5-4B-Instruct. It predicts a desktop GUI action, including its applicable parameters and text, through **one native causal model forward over 131 fixed value slots**. The frozen backbone retains its original vocabulary head. There is no separate autoregressive text writer in this release. This first release corresponds to **Desktop131 SFT step3368**. It is a desktop-only checkpoint, not Unified133, not a mobile-trained model, and not an RL checkpoint. The name describes the intended efficient execution approach; this release does not establish a matched task-success/latency advantage over the base model. ## Attribution and relationship to GUI-Decisions Our principal methodological inspiration from [GUI-Decisions](https://github.com/infinitylogesh/GUI-Decisions) is its **E-arm 256-bin coordinate representation and readout**. Each x/y axis is represented by one label from a 256-token codebook. Historical coordinates use exactly the same codebook as predicted coordinates. Single-forward complete-action prediction was already being explored in this project before adopting that coordinate scheme. We distinguish that design history from implementation reuse: the released runtime calls the original GUI-Decisions `slotread.train.read` and uses its label/numerical utilities from commit `4afc7f24694bf41d84b58e761b218c694c43f13b`. We gratefully acknowledge this contribution. This is an independent project, not an official GUI-Decisions release or a full reproduction of its training method. The pinned public tree omits `slotread/data`, although the training module imports it. The inference adapter therefore loads only the SHA-256-verified `read` function from that file, preserving its body, rather than importing unrelated training-data code. Label and numeric utilities are imported normally. This small compatibility adapter is checked against the original complete private runtime snapshot. The selected upstream snapshot has no license file. Its source is **not bundled or relicensed here**; users obtain the pinned dependency from the author's repository. Our Apache-2.0 license applies to the project-owned release code and adapter, not that external dependency. See [THIRD_PARTY.md](../THIRD_PARTY.md). We also acknowledge [GUI-Owl / Mobile-Agent](https://github.com/X-PLUG/MobileAgent) for the pretrained backbone, and PEFT/Transformers for model and adapter support. ## Quick start Use Python 3.10+ and an NVIDIA GPU. The release uses a custom inference entry point; ordinary `generate()` or loading the adapter into the whole VLM with a generic `PeftModel.from_pretrained` call does not reproduce this protocol. The adapter is attached to the **language-model submodule**, preserving the original loader. ```bash python -m pip install -r requirements.txt python prepare_dependencies.py --destination /path/on/data-disk/gui-swift-deps python infer.py \ --base-model /path/on/data-disk/gui-swift-deps/GUI-Owl-1.5-4B-Instruct \ --gui-decisions /path/on/data-disk/gui-swift-deps/GUI-Decisions \ --request examples/request.json ``` Dependency preparation downloads the pinned base model (approximately 8.9 GB) and the external GUI-Decisions code. Existing matching local directories can instead be passed directly to `infer.py`. Every required base and upstream source file is checked against recorded hashes. The inference script prints an action and **does not control the mouse or keyboard**. A request contains: ```json { "instruction": "Click the Save button.", "image": "screen.png", "history": [] } ``` Image paths are relative to the request file. For history, add records containing `image` and `actions` (a list of actually executed canonical actions). All supplied historical actions and the last two historical screenshots are retained. Do not add ground-truth next actions to the input. The included example is a synthetic UI smoke input, not a benchmark result or real-application task success. ## Action protocol | Field | Slots | |---|---:| | Action | 3 | | x, y | 1 each | | Keys | 16 | | Scroll amount | 4 | | Time | 5 | | Status | 1 | | Text | 100 | | **Total** | **131** | Coordinates returned by the decoder are normalized integers in `0..1000`; coordinate distributions use the original `local2` readout. Text is limited to 99 native tokens plus EOS. Drag has **one destination coordinate pair** and starts at the current pointer; this version does not encode both endpoints. Scroll amount retains the historical wheel-count convention despite the field name `pixels`. The runtime preserves catalogue-constrained action/key decoding and the original invalid-output behavior. It has no arbitrary shell action. ## Training The checkpoint combines a 64-update pilot over 512 examples with a continuation of 3,368 updates over the remaining 26,944 examples, visiting the 27,456-example training split once in total. `step3368` is the continuation step number, not the number of updates including the pilot. The saved validation split has 1,967 rows and was used repeatedly during development. - LoRA: rank 16, alpha 32, dropout 0.05; 252 language attention/MLP linear modules. - Adapted projections: q, k, v, o, gate, up and down; 33,030,144 FP32 parameters. - Frozen: vision encoder/mergers, input embeddings, original tied output head and BF16 base parameters. - Historical learning rate: 2e-4; effective global batch: 8. - Objective: action-field CE plus twice the mean of active parameter-field losses, with full-vocabulary normalization; coordinates use fixed sigma 0.005 targets. - This is **not** the later planned all-linear rank-256 configuration, token-mean Unified133 objective, or box-adaptive/equal-field-mean training plan. The existing data audit describes a mixture of AgentNet (15,754 training steps), HybridCUA (11,303), and OS-Atlas (399). All 1,967 validation rows are from AgentNet. The raw corpus is not bundled. Five selected validation examples are shared in `examples/validation/` under the original AgentNet MIT terms. The release does not establish absence of semantic overlap with public benchmarks. ## Existing evaluation These are saved historical measurements, not a fresh benchmark run for release. | Evaluation | Result | Scope | |---|---:|---| | OSWorld-derived selected suite | 58 / 356 = **16.29%** full success | 361 selected of 369 official tasks; 5 unscored | | Mean environment reward | **0.1712077** | 356 scored tasks only | | Validation action-type accuracy | **75.60%** | 1,967 development rows | | Validation canonical exact match | **24.05%** | Same 1,967 rows; strict full-action matching | | Validation text exact match | 120 / 399 = **30.08%** | Ground-truth `type` rows | The five unscored tasks are not silently converted to zero rewards. The OSWorld number is not a complete official leaderboard score. This checkpoint was selected for release after observing historical results. Protocol differences prevent a causal comparison with other historical models or the upstream model-card scores. Terra-guided demonstrations, including any direct teacher actions, are not included as standalone model benchmark results. See `evaluation.json` for the saved counts. ## Coordinate tolerance reporting update On 2026-11-06 the user selected a 50/1000 per-axis tolerance for the model-card headline after inspecting the original results. This is a post-hoc recount of the same saved predictions; no inference or environment evaluation was rerun. The original `evaluation.json` remains unchanged. | Position criterion, with matching action type | Matches / reference-coordinate rows | Rate | |---|---:|---:| | Both axis errors <= 20/1000 (original) | 456 / 1,104 | 41.30% | | Both axis errors <= 50/1000 (updated headline) | 548 / 1,104 | 49.64% | All 1,104 reference actions with a primary `coordinate` field stay in the denominator. Missing or invalid predictions are non-matches. Each coordinate is normalized to 0-1000; 50 allows 5% of screen width horizontally and 5% of screen height vertically. This metric checks the action type and primary point only; widget boundaries, drag endpoints, scroll amounts and other parameters are not checked. It is not a task-success metric. Direct coordinate differences were cross-checked against the saved per-axis errors, reproducing the original 20/1000 result. [Recount definition, counts and source hash](coordinate-tolerance.json). Strict canonical full-action matching remains 24.05% (473/1,967). It requires literal decoded parameter equality, so even a prediction in the correct coordinate bins can fail this metric. The GIMP gallery example has both bins correct but decoded `[155,243]` versus reference `[153,243]`. ## Future reinforcement learning This SFT checkpoint is a starting point for future reinforcement learning using environment-verifier feedback, with the goal of improving end-to-end task completion. The reported 16.29% remains the measured SFT result; RL improvement is a future objective, not an established gain for this release. ## Limitations This research model can misclick, repeat actions, emit invalid text, and stop prematurely. Text and dragging remain weak. A single forward does not imply single-step task completion or a guaranteed latency advantage. There is no verified mobile compatibility in this release. Integrating with an executor requires correct coordinate conversion, scroll conventions and environment-level outcome checks. ## Release identity The original checkpoint SHA-256 is `bd82d3c508a7b591b7c1578614231b142689982c842c6864c80baaa213e5dde5`. The released `adapter_model.safetensors` contains only the adapter tensors; no optimizer state, credentials or local research datasets are included. Frozen runtime sources and extracted compatibility components are recorded in `provenance.json`; `SHA256SUMS.json` binds the published files. Release validation on one RTX 4090 compared the original runtime and this package in separate processes. On two saved inputs (current screenshot only, and full history), input tensors, logits, and decoded actions matched exactly, with one forward per input. All 504 exported adapter tensors also matched exactly. The synthetic example's actual output is in `examples/output.json`. Peak allocated GPU memory was about 9.1 GiB in this bounded check; this is not a maximum-memory claim for every image/context length. See `release-validation.json` for the environment.