Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -1,17 +1,55 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
tags:
|
| 4 |
-
- gui-agents
|
| 5 |
- world-model
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
---
|
| 7 |
|
| 8 |
-
# LGWM
|
| 9 |
|
| 10 |
Weights for **Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents**.
|
| 11 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
| Directory | Model |
|
| 13 |
| --- | --- |
|
| 14 |
| `lgwm-online/` | LGWM world model |
|
| 15 |
| `lgwm-harm-head/` | Supervised harm head |
|
| 16 |
|
| 17 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
+
library_name: pytorch
|
| 4 |
+
base_model: facebook/dinov2-base
|
| 5 |
+
datasets:
|
| 6 |
+
- jiamingzz/lgwm-data
|
| 7 |
tags:
|
|
|
|
| 8 |
- world-model
|
| 9 |
+
- gui-agent
|
| 10 |
+
- computer-use
|
| 11 |
+
- agent-safety
|
| 12 |
+
- jepa
|
| 13 |
+
- self-supervised
|
| 14 |
---
|
| 15 |
|
| 16 |
+
# LGWM: A Latent World Model that Verifies GUI Agent Transitions
|
| 17 |
|
| 18 |
Weights for **Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents**.
|
| 19 |
|
| 20 |
+
[Code](https://github.com/jiamingzhang94/lgwm) 路 [Data](https://huggingface.co/datasets/jiamingzz/lgwm-data)
|
| 21 |
+
|
| 22 |
+
LGWM is a decoder-free, action-conditioned world model for GUI agents. Given the
|
| 23 |
+
current screenshot and an action, it predicts the representation of the next
|
| 24 |
+
screen, and compares it with the screen that actually appears. One cosine
|
| 25 |
+
distance tells whether the transition is the one the action should have produced.
|
| 26 |
+
|
| 27 |
+
## Key results
|
| 28 |
+
|
| 29 |
+
- **0.987 AUC** on RSWT-Bench with a training-free integrity score, on par with the
|
| 30 |
+
strongest closed-source VLMs.
|
| 31 |
+
- **17 ms** per decision on one A100, over three orders of magnitude faster than
|
| 32 |
+
generative GUI world models, and about ten AUC points more accurate.
|
| 33 |
+
- **0.954 AUC** at separating harmful from benign violations with a linear head on
|
| 34 |
+
the prediction residual, where prompted VLMs are near chance.
|
| 35 |
+
- **173M** parameters at inference (ViT-B encoder, action encoder and predictor).
|
| 36 |
+
|
| 37 |
+
## Files
|
| 38 |
+
|
| 39 |
| Directory | Model |
|
| 40 |
| --- | --- |
|
| 41 |
| `lgwm-online/` | LGWM world model |
|
| 42 |
| `lgwm-harm-head/` | Supervised harm head |
|
| 43 |
|
| 44 |
+
## Usage
|
| 45 |
+
|
| 46 |
+
```bash
|
| 47 |
+
git clone https://github.com/jiamingzhang94/lgwm && cd lgwm
|
| 48 |
+
pip install -c requirements-tested.txt -e '.[text]'
|
| 49 |
+
hf download jiamingzz/lgwm --local-dir weights
|
| 50 |
+
python tools/score_transition.py --before before.png --after after.png \
|
| 51 |
+
--action action.json --harm-head weights/lgwm-harm-head
|
| 52 |
+
```
|
| 53 |
+
|
| 54 |
+
`action.json` holds the action, for example `{"action_type": "tap", "x": 0.5, "y": 0.5}`.
|
| 55 |
+
The command prints `integrity` and `harm_score`.
|