license: apache-2.0
library_name: pytorch
base_model: facebook/dinov2-base
datasets:
- jiamingzz/lgwm-data
tags:
- world-model
- gui-agent
- computer-use
- mobile-agent
- agent-safety
- jepa
- self-supervised
model-index:
- name: LGWM
results:
- task:
type: other
name: GUI transition verification
dataset:
name: RSWT-Bench
type: jiamingzz/lgwm-data
config: rswt_bench
split: test
metrics:
- type: roc_auc
value: 0.987
name: RSWT AUC (integrity score, training-free)
- type: roc_auc
value: 0.954
name: Harm AUC (linear harm head)
LGWM: Latent GUI World Model
Weights for Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents.
Jiaming Zhang, Xuan Wang, Fuyao Zhang, Yang Cao, Lingjuan Lyu, Wei Yang Bryan Lim
Project page · Code · Data · Paper
LGWM is an action-conditioned world model that predicts the next screen in representation space and checks it against the screen that actually appears. One cosine distance verifies every step of a GUI agent, with no judge model and no attack labels.
Key results
- 0.987 AUC on RSWT-Bench with a training-free integrity score, within one AUC point of Gemini 3.7 Flash (0.995) and GPT-5.6 Luna (0.988).
- 17 ms per decision on one A100. World models that render the future (gWorld, Code2World) score about ten AUC points lower and need over 90 s per transition.
- 0.954 AUC at separating harmful from benign violations with a linear head on the prediction residual, where prompted VLMs score between 0.51 and 0.61.
- 261M parameters in training, 173M at inference.
Files
| Directory | Model |
|---|---|
lgwm-online/ |
World model: ViT-B/14 screen encoder, action encoder and 12-layer predictor (173M parameters) |
lgwm-harm-head/ |
Linear harm head on the prediction residual |
Usage
git clone https://github.com/jiamingzhang94/lgwm && cd lgwm
pip install -c requirements-tested.txt -e '.[text]'
hf download jiamingzz/lgwm --local-dir weights
echo '{"action_type": "tap", "x": 0.5, "y": 0.5}' > action.json
python tools/score_transition.py --before before.png --after after.png \
--action action.json --harm-head weights/lgwm-harm-head
The command prints integrity, which is 1 − cos(predicted, observed), and
harm_score. Higher values indicate a less expected transition and a higher
predicted hijack probability.
Training
Self-supervised on 1.85M real GUI transitions from Android in the Wild, AndroidControl, GUIOdyssey, AMEX and MiniWoB++, starting from DINOv2 ViT-B/14: 120,000 updates at batch size 512 on four A100 GPUs. The harm head is fit on the 6,786-example supervised split of RSWT-Bench.
Citation
@article{zhang2026lgwm,
title = {Right Screen, Wrong Transition:
World Models as Verifiers for GUI Agents},
author = {Zhang, Jiaming and Wang, Xuan and Zhang, Fuyao and
Cao, Yang and Lyu, Lingjuan and Lim, Wei Yang Bryan},
journal = {arXiv preprint arXiv:2610.11942},
year = {2026}
}