---
license: apache-2.0
library_name: pytorch
base_model: facebook/dinov2-base
datasets:
- jiamingzz/lgwm-data
tags:
- world-model
- gui-agent
- computer-use
- mobile-agent
- agent-safety
- jepa
- self-supervised
model-index:
- name: LGWM
  results:
  - task:
      type: other
      name: GUI transition verification
    dataset:
      name: RSWT-Bench
      type: jiamingzz/lgwm-data
      config: rswt_bench
      split: test
    metrics:
    - type: roc_auc
      value: 0.987
      name: RSWT AUC (integrity score, training-free)
    - type: roc_auc
      value: 0.954
      name: Harm AUC (linear harm head)
---

# LGWM: Latent GUI World Model

Weights for **Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents**.

Jiaming Zhang, Xuan Wang, Fuyao Zhang, Yang Cao, Lingjuan Lyu, Wei Yang Bryan Lim

[Project page](https://jiamingzhang.netlify.app/lgwm/) · [Code](https://github.com/jiamingzhang94/lgwm) · [Data](https://huggingface.co/datasets/jiamingzz/lgwm-data) · [Paper](https://arxiv.org/abs/2610.11942)

![LGWM compares predicted and observed screens in one representation space. The same login screen is legitimate after one action and a hijack after another.](https://jiamingzhang.netlify.app/lgwm/assets/overview.png)

LGWM is an action-conditioned world model that predicts the next screen in
representation space and checks it against the screen that actually appears. One
cosine distance verifies every step of a GUI agent, with no judge model and no
attack labels.

## Key results

- **0.987 AUC** on RSWT-Bench with a training-free integrity score, within one AUC
  point of Gemini 3.7 Flash (0.995) and GPT-5.6 Luna (0.988).
- **17 ms** per decision on one A100. World models that render the future (gWorld,
  Code2World) score about ten AUC points lower and need over 90 s per transition.
- **0.954 AUC** at separating harmful from benign violations with a linear head on
  the prediction residual, where prompted VLMs score between 0.51 and 0.61.
- **261M** parameters in training, 173M at inference.

## Files

| Directory | Model |
| --- | --- |
| `lgwm-online/` | World model: ViT-B/14 screen encoder, action encoder and 12-layer predictor (173M parameters) |
| `lgwm-harm-head/` | Linear harm head on the prediction residual |

## Usage

```bash
git clone https://github.com/jiamingzhang94/lgwm && cd lgwm
pip install -c requirements-tested.txt -e '.[text]'
hf download jiamingzz/lgwm --local-dir weights

echo '{"action_type": "tap", "x": 0.5, "y": 0.5}' > action.json
python tools/score_transition.py --before before.png --after after.png \
  --action action.json --harm-head weights/lgwm-harm-head
```

The command prints `integrity`, which is 1 − cos(predicted, observed), and
`harm_score`. Higher values indicate a less expected transition and a higher
predicted hijack probability.

## Training

Self-supervised on 1.85M real GUI transitions from Android in the Wild,
AndroidControl, GUIOdyssey, AMEX and MiniWoB++, starting from DINOv2 ViT-B/14:
120,000 updates at batch size 512 on four A100 GPUs. The harm head is fit on the
6,786-example supervised split of RSWT-Bench.

## Citation

```bibtex
@article{zhang2026lgwm,
  title   = {Right Screen, Wrong Transition:
             World Models as Verifiers for GUI Agents},
  author  = {Zhang, Jiaming and Wang, Xuan and Zhang, Fuyao and
             Cao, Yang and Lyu, Lingjuan and Lim, Wei Yang Bryan},
  journal = {arXiv preprint arXiv:2610.11942},
  year    = {2026}
}
```