| --- |
| license: apache-2.0 |
| language: |
| - zh |
| - en |
| library_name: pytorch |
| base_model: jhu-clsp/mmBERT-base |
| datasets: |
| - vyokky/GUI-360 |
| - osunlp/Mind2Web |
| - ServiceNow/GroundCUA |
| - ServiceNow/VideoCUA |
| - Writer/omniact |
| - xlangai/AgentNet |
| tags: |
| - computer-use |
| - gui-agent |
| - desktop-agent |
| - windows |
| - ui-automation |
| - decision-model |
| - system-one |
| --- |
| |
| # S1-Fast |
|
|
| S1-Fast is the local decision model of [DesktopAgent](https://github.com/elidot007/desktop-agent), a desktop agent for Windows |
| business software. For each step it reads the window's element table from UI Automation (role, name, value, state, |
| box), one visual token per element and eight screen tokens, and answers all of the step's questions in a single |
| encoder pass: |
|
|
| | question | answer | |
| |---|---| |
| | `op` | one of 15 operations (click, set value, select, toggle, expand, scroll, type, hotkey, drag, wait, done, escalate, ask the user, ...) | |
| | `click_target`, `set_value_target`, `select_target`, `toggle_target`, `expand_target`, `scroll_target` | the element for each operation type, as a pointer over the candidates | |
| | `prev_ok`, `subgoal_done` | yes / no | |
| | `value` | the span of the goal or subgoal text to type, or none | |
| | `hotkey_key` | one of 255 key chords | |
|
|
| It has 333M parameters (an mmBERT-base encoder, 22 layers, plus cross-attention heads), takes about 20 ms per pass on |
| an H100 and 0.7–1.5 s per decision on a laptop CPU, and comes with its calibration and the execution gate that |
| decides which steps may run without verification. This is round 13b, the checkpoint of the paper *One Pass, Then |
| Verify: A Fast Local Decision Model for Desktop Agents on Windows Business Software* (arXiv link to follow). |
|
|
| ## Files |
|
|
| | file | | |
| |---|---| |
| | `model.safetensors` | weights, fp32 (1.3 GB) | |
| | `s1_config.json` | model and sequence configuration, question list, operation and hotkey vocabularies | |
| | `encoder/config.json`, `tokenizer/` | encoder configuration and tokenizer (mmBERT-base plus marker tokens) | |
| | `calibration.json` | temperatures and tier thresholds per question type, option count and modality | |
| | `step_gate.json` | step-confidence cuts per application class, fitted out of fold, and the cascade routes | |
| | `metrics.json` | validation metrics of the training run | |
|
|
| The image tower is [google/siglip2-base-patch16-224](https://huggingface.co/google/siglip2-base-patch16-224) |
| (frozen, not included). Without it the model runs on the element table alone. |
|
|
| ## Use |
|
|
| With DesktopAgent on Windows: |
|
|
| ```powershell |
| git clone https://github.com/elidot007/desktop-agent.git desktopagent; cd desktopagent |
| powershell -ExecutionPolicy Bypass -File training\setup.ps1 # venv, CPU PyTorch, this model and SigLIP2 |
| powershell -ExecutionPolicy Bypass -File training\serve.ps1 # decision service on 127.0.0.1:8765 |
| ``` |
|
|
| The service alone, on any OS (Python 3.10+): |
|
|
| ```bash |
| git clone https://github.com/elidot007/desktop-agent.git desktopagent && cd desktopagent/training |
| pip install torch --index-url https://download.pytorch.org/whl/cpu && pip install -e . |
| S1_API_KEY=<token> python -m s1.serve --checkpoint elidot007/S1-Fast --vision-model google/siglip2-base-patch16-224 |
| ``` |
|
|
| From Python: |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| from s1.infer import Predictor |
| |
| p = Predictor(snapshot_download("elidot007/S1-Fast"), vision_model="google/siglip2-base-patch16-224") |
| out = p.predict(snapshot, {"goal": "把金额填成1200", "subgoal": None, "history": []}, image=png_bytes) |
| ``` |
|
|
| `snapshot` is a UET snapshot of the window (`da observe --window <title> --compact`); the |
| [training README](https://github.com/elidot007/desktop-agent/blob/main/training/README.md) documents the HTTP API, the tiers, the |
| step gate, the optional System-2 planner and the S1-Accurate cascade. |
|
|
| ## Results |
|
|
| From the paper, measured with S1's own candidate selection and two-stage recall; they are not comparable with numbers |
| published for these datasets under other protocols. Step success means both the operation and the target element are |
| right. |
|
|
| | test set (held out) | no subgoal | closed-loop subgoal | hindsight subgoal | |
| |---|---|---|---| |
| | VideoCUA (unseen applications) | 66.4 | 67.3 | 86.7 | |
| | OmniACT | 82.6 | 79.7 | 90.3 | |
| | GUI-360° real trajectories | 68.2 | 67.2 | 83.0 | |
| | Mind2Web cross-task | 46.8 | 60.8 | 68.3 | |
| | AndroidControl (human low-level instruction) | – | 67.2 | – | |
| | macro average | | **68.4** | | |
|
|
| On Chinese business applications (no Chinese interface was in the training data): |
|
|
| | | step success | |
| |---|---| |
| | original apps (cn_v1, 559 questions), effect-equivalent scoring, goal only / with subgoal | 75.8 / 76.4 | |
| | generated apps, held-out control implementations: element ranked first / operation right | 97.3 / 9.0 | |
| |
| The model learned from public data to click a drop-down open, while the DesktopAgent executor selects a value in one |
| verifiable step; on control implementations it never saw, it finds the field but picks the wrong operation. |
| |
| With the shipped step gate, steps in desktop applications run on confidence alone at step confidence ≥ 0.9945; out of |
| fold this covered 14.7% (VideoCUA) and 17.8% (OmniACT) of steps at about 1% error. Office, browser and mobile windows |
| fail closed: their steps run only when the runtime can read the effect back. Deferring low-confidence steps to |
| [S1-Accurate](https://huggingface.co/elidot007/S1-Accurate-LoRA) raised the macro average from 68.0% to 74.4% on the |
| subsets where it was run. |
| |
| ## Training |
| |
| One epoch (31,099 steps) on two H100s, batch 32 per GPU, AdamW, encoder learning rate 5e-5 and heads 2e-4, soft |
| cross-entropy plus a Brier term, distillation from the previous round's checkpoint (weight 0.5, temperature 2), |
| modality dropout, hard negatives and oversampling of rare operations. The encoder was initialized from |
| [mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base); the heads were trained from scratch. |
| |
| | source | license | records | actions from | |
| |---|---|---|---| |
| | [GUI-360°](https://huggingface.co/datasets/vyokky/GUI-360), interface states | MIT | 100,000 (sampled) | goals from templates (Chinese and English) | |
| | GUI-360°, real trajectories | MIT | 79,134 | GPT-4o / GPT-4.1 driving the UFO agent | |
| | [Mind2Web](https://huggingface.co/datasets/osunlp/Mind2Web) (Deng et al., 2023) | CC BY 4.0 | 6,818 | crowdworkers | |
| | [GroundCUA](https://huggingface.co/datasets/ServiceNow/GroundCUA) | MIT | 50,000 (sampled) | human boxes, template goals | |
| | [VideoCUA](https://huggingface.co/datasets/ServiceNow/VideoCUA) | MIT | 26,766 | human | |
| | [OmniACT](https://huggingface.co/datasets/Writer/omniact) | MIT | 8,212 | human | |
| | [AndroidControl](https://github.com/google-research/google-research/tree/master/android_control) | Apache-2.0 | 66,788 | human | |
| | [AgentNet](https://huggingface.co/datasets/xlangai/AgentNet) | MIT | 291,956 | human | |
| |
| About 1.5 million records per epoch with copies and up-weighting. Please note what in this data was written by |
| models: |
| |
| - The tasks and actions of the GUI-360° real trajectories were produced by GPT-4o and GPT-4.1 driving the UFO agent, |
| according to the dataset's authors. |
| - Copies with Chinese goals, hindsight subgoals and closed-loop subgoals were written by Qwen3-30B-A3B-Instruct-2507 |
| (Apache-2.0), run locally. One AgentNet copy uses the dataset's own per-step descriptions, which its authors |
| generated with an LLM, as subgoals. |
| - VideoCUA and OmniACT labels were cleaned by two LLM judges that could only remove records. |
| |
| The interfaces in the training data are English (desktop, web, Office, Android); none is Chinese. |
| |
| ## Limitations |
| |
| - **Simulated applications only.** The Chinese results come from our own WinForms applications; commercial clients |
| such as Kingdee, Yonyou, WPS or DingTalk were not evaluated. |
| - **Operation conventions do not transfer** across control implementations (9.0% right operation on held-out |
| implementations), and thresholds fitted on validation data did not hold on test data. Fit and certify the gate in |
| your own domain before you let steps run unverified. |
| - **One step at a time.** S1 decides the next step; multi-step goals need a planner (the calling agent or a System-2 |
| LLM). |
| - **Single training run**, single reference labels on the public sets, candidates from our own pipeline. |
| - **Not a safety mechanism.** Irreversible actions (save, submit, approve, delete, pay) must be confirmed by a person. |
| The DesktopAgent runtime enforces this; if you use the model elsewhere, you have to. |
| |
| ## License |
| |
| Apache-2.0, see `LICENSE` and `NOTICE`. The encoder was initialized from mmBERT-base (MIT). The training data keep |
| their own licenses (above); you are responsible for checking that their terms, and those of the models that produced |
| parts of them, fit your use. |
| |
| ## Citation |
| |
| ```bibtex |
| @misc{tuoheti2026onepass, |
| title = {One Pass, Then Verify: A Fast Local Decision Model for Desktop Agents on Windows Business Software}, |
| author = {Yilihamu Tuoheti and Tonghe Wang}, |
| year = {2026}, |
| note = {arXiv preprint} |
| } |
| ``` |
| |
