|
Download docs/technical-details.md from dongjintao/GUI-Swift-4B: direct link, hf CLI and curl.
- Browser
- Download file 10.6 kB
-
https://huggingface.co/dongjintao/GUI-Swift-4B/resolve/main/docs/technical-details.md
- Command line
-
hf download hf://dongjintao/GUI-Swift-4B/docs/technical-details.md
-
curl -L -o technical-details.md https://huggingface.co/dongjintao/GUI-Swift-4B/resolve/main/docs/technical-details.md
10.6 kB
| # GUI-Swift-4B · Technical details | |
| **Single-Pass GUI Action Prediction** | |
| GUI-Swift-4B is a research adapter for GUI-Owl-1.5-4B-Instruct. It predicts a | |
| desktop GUI action, including its applicable parameters and text, through **one | |
| native causal model forward over 131 fixed value slots**. The frozen backbone | |
| retains its original vocabulary head. There is no separate autoregressive text | |
| writer in this release. | |
| This first release corresponds to **Desktop131 SFT step3368**. It is a desktop-only | |
| checkpoint, not Unified133, not a mobile-trained model, and not an RL checkpoint. | |
| The name describes the intended efficient execution approach; this release does | |
| not establish a matched task-success/latency advantage over the base model. | |
| ## Attribution and relationship to GUI-Decisions | |
| Our principal methodological inspiration from | |
| [GUI-Decisions](https://github.com/infinitylogesh/GUI-Decisions) is its **E-arm | |
| 256-bin coordinate representation and readout**. Each x/y axis is represented by | |
| one label from a 256-token codebook. Historical coordinates use exactly the same | |
| codebook as predicted coordinates. | |
| Single-forward complete-action prediction was already being explored in this | |
| project before adopting that coordinate scheme. We distinguish that design | |
| history from implementation reuse: the released runtime calls the original | |
| GUI-Decisions `slotread.train.read` and uses its label/numerical utilities from | |
| commit `4afc7f24694bf41d84b58e761b218c694c43f13b`. We gratefully acknowledge this | |
| contribution. This is an independent project, not an official GUI-Decisions release | |
| or a full reproduction of its training method. | |
| The pinned public tree omits `slotread/data`, although the training module imports | |
| it. The inference adapter therefore loads only the SHA-256-verified `read` function | |
| from that file, preserving its body, rather than importing unrelated training-data | |
| code. Label and numeric utilities are imported normally. This small compatibility | |
| adapter is checked against the original complete private runtime snapshot. | |
| The selected upstream snapshot has no license file. Its source is **not bundled | |
| or relicensed here**; users obtain the pinned dependency from the author's | |
| repository. Our Apache-2.0 license applies to the project-owned release code and | |
| adapter, not that external dependency. See [THIRD_PARTY.md](../THIRD_PARTY.md). | |
| We also acknowledge [GUI-Owl / Mobile-Agent](https://github.com/X-PLUG/MobileAgent) | |
| for the pretrained backbone, and PEFT/Transformers for model and adapter support. | |
| ## Quick start | |
| Use Python 3.10+ and an NVIDIA GPU. The release uses a custom inference entry point; | |
| ordinary `generate()` or loading the adapter into the whole VLM with a generic | |
| `PeftModel.from_pretrained` call does not reproduce this protocol. The adapter is | |
| attached to the **language-model submodule**, preserving the original loader. | |
| ```bash | |
| python -m pip install -r requirements.txt | |
| python prepare_dependencies.py --destination /path/on/data-disk/gui-swift-deps | |
| python infer.py \ | |
| --base-model /path/on/data-disk/gui-swift-deps/GUI-Owl-1.5-4B-Instruct \ | |
| --gui-decisions /path/on/data-disk/gui-swift-deps/GUI-Decisions \ | |
| --request examples/request.json | |
| ``` | |
| Dependency preparation downloads the pinned base model (approximately 8.9 GB) and | |
| the external GUI-Decisions code. Existing matching local directories can instead | |
| be passed directly to `infer.py`. Every required base and upstream source file is | |
| checked against recorded hashes. The inference script prints an action and **does | |
| not control the mouse or keyboard**. | |
| A request contains: | |
| ```json | |
| { | |
| "instruction": "Click the Save button.", | |
| "image": "screen.png", | |
| "history": [] | |
| } | |
| ``` | |
| Image paths are relative to the request file. For history, add records containing | |
| `image` and `actions` (a list of actually executed canonical actions). All supplied | |
| historical actions and the last two historical screenshots are retained. Do not | |
| add ground-truth next actions to the input. The included example is a synthetic | |
| UI smoke input, not a benchmark result or real-application task success. | |
| ## Action protocol | |
| | Field | Slots | | |
| |---|---:| | |
| | Action | 3 | | |
| | x, y | 1 each | | |
| | Keys | 16 | | |
| | Scroll amount | 4 | | |
| | Time | 5 | | |
| | Status | 1 | | |
| | Text | 100 | | |
| | **Total** | **131** | | |
| Coordinates returned by the decoder are normalized integers in `0..1000`; | |
| coordinate distributions use the original `local2` readout. Text is limited to | |
| 99 native tokens plus EOS. Drag has **one destination coordinate pair** and starts | |
| at the current pointer; this version does not encode both endpoints. Scroll amount | |
| retains the historical wheel-count convention despite the field name `pixels`. | |
| The runtime preserves catalogue-constrained action/key decoding and the original | |
| invalid-output behavior. It has no arbitrary shell action. | |
| ## Training | |
| The checkpoint combines a 64-update pilot over 512 examples with a continuation | |
| of 3,368 updates over the remaining 26,944 examples, visiting the 27,456-example | |
| training split once in total. `step3368` is the continuation step number, not the | |
| number of updates including the pilot. The saved validation split has 1,967 rows | |
| and was used repeatedly during development. | |
| - LoRA: rank 16, alpha 32, dropout 0.05; 252 language attention/MLP linear modules. | |
| - Adapted projections: q, k, v, o, gate, up and down; 33,030,144 FP32 parameters. | |
| - Frozen: vision encoder/mergers, input embeddings, original tied output head and | |
| BF16 base parameters. | |
| - Historical learning rate: 2e-4; effective global batch: 8. | |
| - Objective: action-field CE plus twice the mean of active parameter-field losses, | |
| with full-vocabulary normalization; coordinates use fixed sigma 0.005 targets. | |
| - This is **not** the later planned all-linear rank-256 configuration, token-mean | |
| Unified133 objective, or box-adaptive/equal-field-mean training plan. | |
| The existing data audit describes a mixture of AgentNet (15,754 training steps), | |
| HybridCUA (11,303), and OS-Atlas (399). All 1,967 validation rows are from AgentNet. | |
| The raw corpus is not bundled. Five selected validation examples are shared in `examples/validation/` under the original AgentNet MIT terms. | |
| The release does not establish absence of semantic overlap with public benchmarks. | |
| ## Existing evaluation | |
| These are saved historical measurements, not a fresh benchmark run for release. | |
| | Evaluation | Result | Scope | | |
| |---|---:|---| | |
| | OSWorld-derived selected suite | 58 / 356 = **16.29%** full success | 361 selected of 369 official tasks; 5 unscored | | |
| | Mean environment reward | **0.1712077** | 356 scored tasks only | | |
| | Validation action-type accuracy | **75.60%** | 1,967 development rows | | |
| | Validation canonical exact match | **24.05%** | Same 1,967 rows; strict full-action matching | | |
| | Validation text exact match | 120 / 399 = **30.08%** | Ground-truth `type` rows | | |
| The five unscored tasks are not silently converted to zero rewards. The OSWorld | |
| number is not a complete official leaderboard score. This checkpoint was selected | |
| for release after observing historical results. Protocol differences prevent a | |
| causal comparison with other historical models or the upstream model-card scores. | |
| Terra-guided demonstrations, including any direct teacher actions, are not included | |
| as standalone model benchmark results. See `evaluation.json` for the saved counts. | |
| ## Coordinate tolerance reporting update | |
| On 2026-11-06 the user selected a 50/1000 per-axis tolerance for the model-card | |
| headline after inspecting the original results. This is a post-hoc recount of | |
| the same saved predictions; no inference or environment evaluation was rerun. | |
| The original `evaluation.json` remains unchanged. | |
| | Position criterion, with matching action type | Matches / reference-coordinate rows | Rate | | |
| |---|---:|---:| | |
| | Both axis errors <= 20/1000 (original) | 456 / 1,104 | 41.30% | | |
| | Both axis errors <= 50/1000 (updated headline) | 548 / 1,104 | 49.64% | | |
| All 1,104 reference actions with a primary `coordinate` field stay in the | |
| denominator. Missing or invalid predictions are non-matches. Each coordinate is | |
| normalized to 0-1000; 50 allows 5% of screen width horizontally and 5% of screen | |
| height vertically. This metric checks the action type and primary point only; | |
| widget boundaries, drag endpoints, scroll amounts and other parameters are not | |
| checked. It is not a task-success metric. Direct coordinate differences were | |
| cross-checked against the saved per-axis errors, reproducing the original | |
| 20/1000 result. [Recount definition, counts and source hash](coordinate-tolerance.json). | |
| Strict canonical full-action matching remains 24.05% (473/1,967). It requires | |
| literal decoded parameter equality, so even a prediction in the correct | |
| coordinate bins can fail this metric. The GIMP gallery example has both bins | |
| correct but decoded `[155,243]` versus reference `[153,243]`. | |
| ## Future reinforcement learning | |
| This SFT checkpoint is a starting point for future reinforcement learning using | |
| environment-verifier feedback, with the goal of improving end-to-end task | |
| completion. The reported 16.29% remains the measured SFT result; RL improvement | |
| is a future objective, not an established gain for this release. | |
| ## Limitations | |
| This research model can misclick, repeat actions, emit invalid text, and stop | |
| prematurely. Text and dragging remain weak. A single forward does not imply | |
| single-step task completion or a guaranteed latency advantage. There is no verified | |
| mobile compatibility in this release. Integrating with an executor requires correct | |
| coordinate conversion, scroll conventions and environment-level outcome checks. | |
| ## Release identity | |
| The original checkpoint SHA-256 is | |
| `bd82d3c508a7b591b7c1578614231b142689982c842c6864c80baaa213e5dde5`. | |
| The released `adapter_model.safetensors` contains only the adapter tensors; no | |
| optimizer state, credentials or local research datasets are included. Frozen | |
| runtime sources and extracted compatibility components are recorded in | |
| `provenance.json`; `SHA256SUMS.json` binds the published files. | |
| Release validation on one RTX 4090 compared the original runtime and this package | |
| in separate processes. On two saved inputs (current screenshot only, and full | |
| history), input tensors, logits, and decoded actions matched exactly, with one | |
| forward per input. All 504 exported adapter tensors also matched exactly. The | |
| synthetic example's actual output is in `examples/output.json`. Peak allocated GPU | |
| memory was about 9.1 GiB in this bounded check; this is not a maximum-memory claim | |
| for every image/context length. See `release-validation.json` for the environment. | |