README.md

metadata

license: apache-2.0
language:
  - zh
  - en
library_name: pytorch
base_model: jhu-clsp/mmBERT-base
datasets:
  - vyokky/GUI-360
  - osunlp/Mind2Web
  - ServiceNow/GroundCUA
  - ServiceNow/VideoCUA
  - Writer/omniact
  - xlangai/AgentNet
tags:
  - computer-use
  - gui-agent
  - desktop-agent
  - windows
  - ui-automation
  - decision-model
  - system-one

S1-Fast

S1-Fast is the local decision model of DesktopAgent, a desktop agent for Windows business software. For each step it reads the window's element table from UI Automation (role, name, value, state, box), one visual token per element and eight screen tokens, and answers all of the step's questions in a single encoder pass:

question	answer
op	one of 15 operations (click, set value, select, toggle, expand, scroll, type, hotkey, drag, wait, done, escalate, ask the user, ...)
click_target, set_value_target, select_target, toggle_target, expand_target, scroll_target	the element for each operation type, as a pointer over the candidates
prev_ok, subgoal_done	yes / no
value	the span of the goal or subgoal text to type, or none
hotkey_key	one of 255 key chords

It has 333M parameters (an mmBERT-base encoder, 22 layers, plus cross-attention heads), takes about 20 ms per pass on an H100 and 0.7–1.5 s per decision on a laptop CPU, and comes with its calibration and the execution gate that decides which steps may run without verification. This is round 13b, the checkpoint of the paper One Pass, Then Verify: A Fast Local Decision Model for Desktop Agents on Windows Business Software (arXiv link to follow).

Files

file
model.safetensors	weights, fp32 (1.3 GB)
s1_config.json	model and sequence configuration, question list, operation and hotkey vocabularies
encoder/config.json, tokenizer/	encoder configuration and tokenizer (mmBERT-base plus marker tokens)
calibration.json	temperatures and tier thresholds per question type, option count and modality
step_gate.json	step-confidence cuts per application class, fitted out of fold, and the cascade routes
metrics.json	validation metrics of the training run

The image tower is google/siglip2-base-patch16-224 (frozen, not included). Without it the model runs on the element table alone.

Use

With DesktopAgent on Windows:

git clone https://github.com/elidot007/desktop-agent.git desktopagent; cd desktopagent
powershell -ExecutionPolicy Bypass -File training\setup.ps1     # venv, CPU PyTorch, this model and SigLIP2
powershell -ExecutionPolicy Bypass -File training\serve.ps1     # decision service on 127.0.0.1:8765

The service alone, on any OS (Python 3.10+):

git clone https://github.com/elidot007/desktop-agent.git desktopagent && cd desktopagent/training
pip install torch --index-url https://download.pytorch.org/whl/cpu && pip install -e .
S1_API_KEY=<token> python -m s1.serve --checkpoint elidot007/S1-Fast --vision-model google/siglip2-base-patch16-224

From Python:

from huggingface_hub import snapshot_download
from s1.infer import Predictor

p = Predictor(snapshot_download("elidot007/S1-Fast"), vision_model="google/siglip2-base-patch16-224")
out = p.predict(snapshot, {"goal": "把金额填成1200", "subgoal": None, "history": []}, image=png_bytes)

snapshot is a UET snapshot of the window (da observe --window <title> --compact); the training README documents the HTTP API, the tiers, the step gate, the optional System-2 planner and the S1-Accurate cascade.

Results

From the paper, measured with S1's own candidate selection and two-stage recall; they are not comparable with numbers published for these datasets under other protocols. Step success means both the operation and the target element are right.

test set (held out)	no subgoal	closed-loop subgoal	hindsight subgoal
VideoCUA (unseen applications)	66.4	67.3	86.7
OmniACT	82.6	79.7	90.3
GUI-360° real trajectories	68.2	67.2	83.0
Mind2Web cross-task	46.8	60.8	68.3
AndroidControl (human low-level instruction)	–	67.2	–
macro average		68.4

On Chinese business applications (no Chinese interface was in the training data):

	step success
original apps (cn_v1, 559 questions), effect-equivalent scoring, goal only / with subgoal	75.8 / 76.4
generated apps, held-out control implementations: element ranked first / operation right	97.3 / 9.0

The model learned from public data to click a drop-down open, while the DesktopAgent executor selects a value in one verifiable step; on control implementations it never saw, it finds the field but picks the wrong operation.

With the shipped step gate, steps in desktop applications run on confidence alone at step confidence ≥ 0.9945; out of fold this covered 14.7% (VideoCUA) and 17.8% (OmniACT) of steps at about 1% error. Office, browser and mobile windows fail closed: their steps run only when the runtime can read the effect back. Deferring low-confidence steps to S1-Accurate raised the macro average from 68.0% to 74.4% on the subsets where it was run.

Training

One epoch (31,099 steps) on two H100s, batch 32 per GPU, AdamW, encoder learning rate 5e-5 and heads 2e-4, soft cross-entropy plus a Brier term, distillation from the previous round's checkpoint (weight 0.5, temperature 2), modality dropout, hard negatives and oversampling of rare operations. The encoder was initialized from mmBERT-base; the heads were trained from scratch.

source	license	records	actions from
GUI-360°, interface states	MIT	100,000 (sampled)	goals from templates (Chinese and English)
GUI-360°, real trajectories	MIT	79,134	GPT-4o / GPT-4.1 driving the UFO agent
Mind2Web (Deng et al., 2023)	CC BY 4.0	6,818	crowdworkers
GroundCUA	MIT	50,000 (sampled)	human boxes, template goals
VideoCUA	MIT	26,766	human
OmniACT	MIT	8,212	human
AndroidControl	Apache-2.0	66,788	human
AgentNet	MIT	291,956	human

About 1.5 million records per epoch with copies and up-weighting. Please note what in this data was written by models:

• The tasks and actions of the GUI-360° real trajectories were produced by GPT-4o and GPT-4.1 driving the UFO agent, according to the dataset's authors.
• Copies with Chinese goals, hindsight subgoals and closed-loop subgoals were written by Qwen3-30B-A3B-Instruct-2507 (Apache-2.0), run locally. One AgentNet copy uses the dataset's own per-step descriptions, which its authors generated with an LLM, as subgoals.
• VideoCUA and OmniACT labels were cleaned by two LLM judges that could only remove records.

The interfaces in the training data are English (desktop, web, Office, Android); none is Chinese.

Limitations

• Simulated applications only. The Chinese results come from our own WinForms applications; commercial clients such as Kingdee, Yonyou, WPS or DingTalk were not evaluated.
• Operation conventions do not transfer across control implementations (9.0% right operation on held-out implementations), and thresholds fitted on validation data did not hold on test data. Fit and certify the gate in your own domain before you let steps run unverified.
• One step at a time. S1 decides the next step; multi-step goals need a planner (the calling agent or a System-2 LLM).
• Single training run, single reference labels on the public sets, candidates from our own pipeline.
• Not a safety mechanism. Irreversible actions (save, submit, approve, delete, pay) must be confirmed by a person. The DesktopAgent runtime enforces this; if you use the model elsewhere, you have to.

License

Apache-2.0, see LICENSE and NOTICE. The encoder was initialized from mmBERT-base (MIT). The training data keep their own licenses (above); you are responsible for checking that their terms, and those of the models that produced parts of them, fit your use.

Citation

@misc{tuoheti2026onepass,
  title  = {One Pass, Then Verify: A Fast Local Decision Model for Desktop Agents on Windows Business Software},
  author = {Yilihamu Tuoheti and Tonghe Wang},
  year   = {2026},
  note   = {arXiv preprint}
}