---
license: apache-2.0
language:
- zh
- en
library_name: pytorch
base_model: jhu-clsp/mmBERT-base
datasets:
- vyokky/GUI-360
- osunlp/Mind2Web
- ServiceNow/GroundCUA
- ServiceNow/VideoCUA
- Writer/omniact
- xlangai/AgentNet
tags:
- computer-use
- gui-agent
- desktop-agent
- windows
- ui-automation
- decision-model
- system-one
---

# S1-Fast

S1-Fast is the local decision model of [DesktopAgent](https://github.com/elidot007/desktop-agent), a desktop agent for Windows
business software. For each step it reads the window's element table from UI Automation (role, name, value, state,
box), one visual token per element and eight screen tokens, and answers all of the step's questions in a single
encoder pass:

| question | answer |
|---|---|
| `op` | one of 15 operations (click, set value, select, toggle, expand, scroll, type, hotkey, drag, wait, done, escalate, ask the user, ...) |
| `click_target`, `set_value_target`, `select_target`, `toggle_target`, `expand_target`, `scroll_target` | the element for each operation type, as a pointer over the candidates |
| `prev_ok`, `subgoal_done` | yes / no |
| `value` | the span of the goal or subgoal text to type, or none |
| `hotkey_key` | one of 255 key chords |

It has 333M parameters (an mmBERT-base encoder, 22 layers, plus cross-attention heads), takes about 20 ms per pass on
an H100 and 0.7–1.5 s per decision on a laptop CPU, and comes with its calibration and the execution gate that
decides which steps may run without verification. This is round 13b, the checkpoint of the paper *One Pass, Then
Verify: A Fast Local Decision Model for Desktop Agents on Windows Business Software* (arXiv link to follow).

## Files

| file | |
|---|---|
| `model.safetensors` | weights, fp32 (1.3 GB) |
| `s1_config.json` | model and sequence configuration, question list, operation and hotkey vocabularies |
| `encoder/config.json`, `tokenizer/` | encoder configuration and tokenizer (mmBERT-base plus marker tokens) |
| `calibration.json` | temperatures and tier thresholds per question type, option count and modality |
| `step_gate.json` | step-confidence cuts per application class, fitted out of fold, and the cascade routes |
| `metrics.json` | validation metrics of the training run |

The image tower is [google/siglip2-base-patch16-224](https://huggingface.co/google/siglip2-base-patch16-224)
(frozen, not included). Without it the model runs on the element table alone.

## Use

With DesktopAgent on Windows:

```powershell
git clone https://github.com/elidot007/desktop-agent.git desktopagent; cd desktopagent
powershell -ExecutionPolicy Bypass -File training\setup.ps1     # venv, CPU PyTorch, this model and SigLIP2
powershell -ExecutionPolicy Bypass -File training\serve.ps1     # decision service on 127.0.0.1:8765
```

The service alone, on any OS (Python 3.10+):

```bash
git clone https://github.com/elidot007/desktop-agent.git desktopagent && cd desktopagent/training
pip install torch --index-url https://download.pytorch.org/whl/cpu && pip install -e .
S1_API_KEY=<token> python -m s1.serve --checkpoint elidot007/S1-Fast --vision-model google/siglip2-base-patch16-224
```

From Python:

```python
from huggingface_hub import snapshot_download
from s1.infer import Predictor

p = Predictor(snapshot_download("elidot007/S1-Fast"), vision_model="google/siglip2-base-patch16-224")
out = p.predict(snapshot, {"goal": "把金额填成1200", "subgoal": None, "history": []}, image=png_bytes)
```

`snapshot` is a UET snapshot of the window (`da observe --window <title> --compact`); the
[training README](https://github.com/elidot007/desktop-agent/blob/main/training/README.md) documents the HTTP API, the tiers, the
step gate, the optional System-2 planner and the S1-Accurate cascade.

## Results

From the paper, measured with S1's own candidate selection and two-stage recall; they are not comparable with numbers
published for these datasets under other protocols. Step success means both the operation and the target element are
right.

| test set (held out) | no subgoal | closed-loop subgoal | hindsight subgoal |
|---|---|---|---|
| VideoCUA (unseen applications) | 66.4 | 67.3 | 86.7 |
| OmniACT | 82.6 | 79.7 | 90.3 |
| GUI-360° real trajectories | 68.2 | 67.2 | 83.0 |
| Mind2Web cross-task | 46.8 | 60.8 | 68.3 |
| AndroidControl (human low-level instruction) | – | 67.2 | – |
| macro average | | **68.4** | |

On Chinese business applications (no Chinese interface was in the training data):

| | step success |
|---|---|
| original apps (cn_v1, 559 questions), effect-equivalent scoring, goal only / with subgoal | 75.8 / 76.4 |
| generated apps, held-out control implementations: element ranked first / operation right | 97.3 / 9.0 |

The model learned from public data to click a drop-down open, while the DesktopAgent executor selects a value in one
verifiable step; on control implementations it never saw, it finds the field but picks the wrong operation.

With the shipped step gate, steps in desktop applications run on confidence alone at step confidence ≥ 0.9945; out of
fold this covered 14.7% (VideoCUA) and 17.8% (OmniACT) of steps at about 1% error. Office, browser and mobile windows
fail closed: their steps run only when the runtime can read the effect back. Deferring low-confidence steps to
[S1-Accurate](https://huggingface.co/elidot007/S1-Accurate-LoRA) raised the macro average from 68.0% to 74.4% on the
subsets where it was run.

## Training

One epoch (31,099 steps) on two H100s, batch 32 per GPU, AdamW, encoder learning rate 5e-5 and heads 2e-4, soft
cross-entropy plus a Brier term, distillation from the previous round's checkpoint (weight 0.5, temperature 2),
modality dropout, hard negatives and oversampling of rare operations. The encoder was initialized from
[mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base); the heads were trained from scratch.

| source | license | records | actions from |
|---|---|---|---|
| [GUI-360°](https://huggingface.co/datasets/vyokky/GUI-360), interface states | MIT | 100,000 (sampled) | goals from templates (Chinese and English) |
| GUI-360°, real trajectories | MIT | 79,134 | GPT-4o / GPT-4.1 driving the UFO agent |
| [Mind2Web](https://huggingface.co/datasets/osunlp/Mind2Web) (Deng et al., 2023) | CC BY 4.0 | 6,818 | crowdworkers |
| [GroundCUA](https://huggingface.co/datasets/ServiceNow/GroundCUA) | MIT | 50,000 (sampled) | human boxes, template goals |
| [VideoCUA](https://huggingface.co/datasets/ServiceNow/VideoCUA) | MIT | 26,766 | human |
| [OmniACT](https://huggingface.co/datasets/Writer/omniact) | MIT | 8,212 | human |
| [AndroidControl](https://github.com/google-research/google-research/tree/master/android_control) | Apache-2.0 | 66,788 | human |
| [AgentNet](https://huggingface.co/datasets/xlangai/AgentNet) | MIT | 291,956 | human |

About 1.5 million records per epoch with copies and up-weighting. Please note what in this data was written by
models:

- The tasks and actions of the GUI-360° real trajectories were produced by GPT-4o and GPT-4.1 driving the UFO agent,
  according to the dataset's authors.
- Copies with Chinese goals, hindsight subgoals and closed-loop subgoals were written by Qwen3-30B-A3B-Instruct-2507
  (Apache-2.0), run locally. One AgentNet copy uses the dataset's own per-step descriptions, which its authors
  generated with an LLM, as subgoals.
- VideoCUA and OmniACT labels were cleaned by two LLM judges that could only remove records.

The interfaces in the training data are English (desktop, web, Office, Android); none is Chinese.

## Limitations

- **Simulated applications only.** The Chinese results come from our own WinForms applications; commercial clients
  such as Kingdee, Yonyou, WPS or DingTalk were not evaluated.
- **Operation conventions do not transfer** across control implementations (9.0% right operation on held-out
  implementations), and thresholds fitted on validation data did not hold on test data. Fit and certify the gate in
  your own domain before you let steps run unverified.
- **One step at a time.** S1 decides the next step; multi-step goals need a planner (the calling agent or a System-2
  LLM).
- **Single training run**, single reference labels on the public sets, candidates from our own pipeline.
- **Not a safety mechanism.** Irreversible actions (save, submit, approve, delete, pay) must be confirmed by a person.
  The DesktopAgent runtime enforces this; if you use the model elsewhere, you have to.

## License

Apache-2.0, see `LICENSE` and `NOTICE`. The encoder was initialized from mmBERT-base (MIT). The training data keep
their own licenses (above); you are responsible for checking that their terms, and those of the models that produced
parts of them, fit your use.

## Citation

```bibtex
@misc{tuoheti2026onepass,
  title  = {One Pass, Then Verify: A Fast Local Decision Model for Desktop Agents on Windows Business Software},
  author = {Yilihamu Tuoheti and Tonghe Wang},
  year   = {2026},
  note   = {arXiv preprint}
}
```