Download docs/technical-details.md from dongjintao/GUI-Swift-4B: direct link, hf CLI and curl.
- Browser
- Download file 10.6 kB
-
https://huggingface.co/dongjintao/GUI-Swift-4B/resolve/main/docs/technical-details.md
- Command line
-
hf download hf://dongjintao/GUI-Swift-4B/docs/technical-details.md
-
curl -L -o technical-details.md https://huggingface.co/dongjintao/GUI-Swift-4B/resolve/main/docs/technical-details.md
GUI-Swift-4B 路 Technical details
Single-Pass GUI Action Prediction
GUI-Swift-4B is a research adapter for GUI-Owl-1.5-4B-Instruct. It predicts a desktop GUI action, including its applicable parameters and text, through one native causal model forward over 131 fixed value slots. The frozen backbone retains its original vocabulary head. There is no separate autoregressive text writer in this release.
This first release corresponds to Desktop131 SFT step3368. It is a desktop-only checkpoint, not Unified133, not a mobile-trained model, and not an RL checkpoint. The name describes the intended efficient execution approach; this release does not establish a matched task-success/latency advantage over the base model.
Attribution and relationship to GUI-Decisions
Our principal methodological inspiration from GUI-Decisions is its E-arm 256-bin coordinate representation and readout. Each x/y axis is represented by one label from a 256-token codebook. Historical coordinates use exactly the same codebook as predicted coordinates.
Single-forward complete-action prediction was already being explored in this
project before adopting that coordinate scheme. We distinguish that design
history from implementation reuse: the released runtime calls the original
GUI-Decisions slotread.train.read and uses its label/numerical utilities from
commit 4afc7f24694bf41d84b58e761b218c694c43f13b. We gratefully acknowledge this
contribution. This is an independent project, not an official GUI-Decisions release
or a full reproduction of its training method.
The pinned public tree omits slotread/data, although the training module imports
it. The inference adapter therefore loads only the SHA-256-verified read function
from that file, preserving its body, rather than importing unrelated training-data
code. Label and numeric utilities are imported normally. This small compatibility
adapter is checked against the original complete private runtime snapshot.
The selected upstream snapshot has no license file. Its source is not bundled or relicensed here; users obtain the pinned dependency from the author's repository. Our Apache-2.0 license applies to the project-owned release code and adapter, not that external dependency. See THIRD_PARTY.md.
We also acknowledge GUI-Owl / Mobile-Agent for the pretrained backbone, and PEFT/Transformers for model and adapter support.
Quick start
Use Python 3.10+ and an NVIDIA GPU. The release uses a custom inference entry point;
ordinary generate() or loading the adapter into the whole VLM with a generic
PeftModel.from_pretrained call does not reproduce this protocol. The adapter is
attached to the language-model submodule, preserving the original loader.
python -m pip install -r requirements.txt
python prepare_dependencies.py --destination /path/on/data-disk/gui-swift-deps
python infer.py \
--base-model /path/on/data-disk/gui-swift-deps/GUI-Owl-1.5-4B-Instruct \
--gui-decisions /path/on/data-disk/gui-swift-deps/GUI-Decisions \
--request examples/request.json
Dependency preparation downloads the pinned base model (approximately 8.9 GB) and
the external GUI-Decisions code. Existing matching local directories can instead
be passed directly to infer.py. Every required base and upstream source file is
checked against recorded hashes. The inference script prints an action and does
not control the mouse or keyboard.
A request contains:
{
"instruction": "Click the Save button.",
"image": "screen.png",
"history": []
}
Image paths are relative to the request file. For history, add records containing
image and actions (a list of actually executed canonical actions). All supplied
historical actions and the last two historical screenshots are retained. Do not
add ground-truth next actions to the input. The included example is a synthetic
UI smoke input, not a benchmark result or real-application task success.
Action protocol
| Field | Slots |
|---|---|
| Action | 3 |
| x, y | 1 each |
| Keys | 16 |
| Scroll amount | 4 |
| Time | 5 |
| Status | 1 |
| Text | 100 |
| Total | 131 |
Coordinates returned by the decoder are normalized integers in 0..1000;
coordinate distributions use the original local2 readout. Text is limited to
99 native tokens plus EOS. Drag has one destination coordinate pair and starts
at the current pointer; this version does not encode both endpoints. Scroll amount
retains the historical wheel-count convention despite the field name pixels.
The runtime preserves catalogue-constrained action/key decoding and the original
invalid-output behavior. It has no arbitrary shell action.
Training
The checkpoint combines a 64-update pilot over 512 examples with a continuation
of 3,368 updates over the remaining 26,944 examples, visiting the 27,456-example
training split once in total. step3368 is the continuation step number, not the
number of updates including the pilot. The saved validation split has 1,967 rows
and was used repeatedly during development.
- LoRA: rank 16, alpha 32, dropout 0.05; 252 language attention/MLP linear modules.
- Adapted projections: q, k, v, o, gate, up and down; 33,030,144 FP32 parameters.
- Frozen: vision encoder/mergers, input embeddings, original tied output head and BF16 base parameters.
- Historical learning rate: 2e-4; effective global batch: 8.
- Objective: action-field CE plus twice the mean of active parameter-field losses, with full-vocabulary normalization; coordinates use fixed sigma 0.005 targets.
- This is not the later planned all-linear rank-256 configuration, token-mean Unified133 objective, or box-adaptive/equal-field-mean training plan.
The existing data audit describes a mixture of AgentNet (15,754 training steps),
HybridCUA (11,303), and OS-Atlas (399). All 1,967 validation rows are from AgentNet.
The raw corpus is not bundled. Five selected validation examples are shared in examples/validation/ under the original AgentNet MIT terms.
The release does not establish absence of semantic overlap with public benchmarks.
Existing evaluation
These are saved historical measurements, not a fresh benchmark run for release.
| Evaluation | Result | Scope |
|---|---|---|
| OSWorld-derived selected suite | 58 / 356 = 16.29% full success | 361 selected of 369 official tasks; 5 unscored |
| Mean environment reward | 0.1712077 | 356 scored tasks only |
| Validation action-type accuracy | 75.60% | 1,967 development rows |
| Validation canonical exact match | 24.05% | Same 1,967 rows; strict full-action matching |
| Validation text exact match | 120 / 399 = 30.08% | Ground-truth type rows |
The five unscored tasks are not silently converted to zero rewards. The OSWorld
number is not a complete official leaderboard score. This checkpoint was selected
for release after observing historical results. Protocol differences prevent a
causal comparison with other historical models or the upstream model-card scores.
Terra-guided demonstrations, including any direct teacher actions, are not included
as standalone model benchmark results. See evaluation.json for the saved counts.
Coordinate tolerance reporting update
On 2026-11-06 the user selected a 50/1000 per-axis tolerance for the model-card
headline after inspecting the original results. This is a post-hoc recount of
the same saved predictions; no inference or environment evaluation was rerun.
The original evaluation.json remains unchanged.
| Position criterion, with matching action type | Matches / reference-coordinate rows | Rate |
|---|---|---|
| Both axis errors <= 20/1000 (original) | 456 / 1,104 | 41.30% |
| Both axis errors <= 50/1000 (updated headline) | 548 / 1,104 | 49.64% |
All 1,104 reference actions with a primary coordinate field stay in the
denominator. Missing or invalid predictions are non-matches. Each coordinate is
normalized to 0-1000; 50 allows 5% of screen width horizontally and 5% of screen
height vertically. This metric checks the action type and primary point only;
widget boundaries, drag endpoints, scroll amounts and other parameters are not
checked. It is not a task-success metric. Direct coordinate differences were
cross-checked against the saved per-axis errors, reproducing the original
20/1000 result. Recount definition, counts and source hash.
Strict canonical full-action matching remains 24.05% (473/1,967). It requires
literal decoded parameter equality, so even a prediction in the correct
coordinate bins can fail this metric. The GIMP gallery example has both bins
correct but decoded [155,243] versus reference [153,243].
Future reinforcement learning
This SFT checkpoint is a starting point for future reinforcement learning using environment-verifier feedback, with the goal of improving end-to-end task completion. The reported 16.29% remains the measured SFT result; RL improvement is a future objective, not an established gain for this release.
Limitations
This research model can misclick, repeat actions, emit invalid text, and stop prematurely. Text and dragging remain weak. A single forward does not imply single-step task completion or a guaranteed latency advantage. There is no verified mobile compatibility in this release. Integrating with an executor requires correct coordinate conversion, scroll conventions and environment-level outcome checks.
Release identity
The original checkpoint SHA-256 is
bd82d3c508a7b591b7c1578614231b142689982c842c6864c80baaa213e5dde5.
The released adapter_model.safetensors contains only the adapter tensors; no
optimizer state, credentials or local research datasets are included. Frozen
runtime sources and extracted compatibility components are recorded in
provenance.json; SHA256SUMS.json binds the published files.
Release validation on one RTX 4090 compared the original runtime and this package
in separate processes. On two saved inputs (current screenshot only, and full
history), input tensors, logits, and decoded actions matched exactly, with one
forward per input. All 504 exported adapter tensors also matched exactly. The
synthetic example's actual output is in examples/output.json. Peak allocated GPU
memory was about 9.1 GiB in this bounded check; this is not a maximum-memory claim
for every image/context length. See release-validation.json for the environment.