jiamingzz commited on
Commit
8b03a53
路
verified 路
1 Parent(s): 4b0404f

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +41 -3
README.md CHANGED
@@ -1,17 +1,55 @@
1
  ---
2
  license: apache-2.0
 
 
 
 
3
  tags:
4
- - gui-agents
5
  - world-model
 
 
 
 
 
6
  ---
7
 
8
- # LGWM
9
 
10
  Weights for **Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents**.
11
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
12
  | Directory | Model |
13
  | --- | --- |
14
  | `lgwm-online/` | LGWM world model |
15
  | `lgwm-harm-head/` | Supervised harm head |
16
 
17
- Code and usage: https://github.com/jiamingzhang94/lgwm
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ library_name: pytorch
4
+ base_model: facebook/dinov2-base
5
+ datasets:
6
+ - jiamingzz/lgwm-data
7
  tags:
 
8
  - world-model
9
+ - gui-agent
10
+ - computer-use
11
+ - agent-safety
12
+ - jepa
13
+ - self-supervised
14
  ---
15
 
16
+ # LGWM: A Latent World Model that Verifies GUI Agent Transitions
17
 
18
  Weights for **Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents**.
19
 
20
+ [Code](https://github.com/jiamingzhang94/lgwm) 路 [Data](https://huggingface.co/datasets/jiamingzz/lgwm-data)
21
+
22
+ LGWM is a decoder-free, action-conditioned world model for GUI agents. Given the
23
+ current screenshot and an action, it predicts the representation of the next
24
+ screen, and compares it with the screen that actually appears. One cosine
25
+ distance tells whether the transition is the one the action should have produced.
26
+
27
+ ## Key results
28
+
29
+ - **0.987 AUC** on RSWT-Bench with a training-free integrity score, on par with the
30
+ strongest closed-source VLMs.
31
+ - **17 ms** per decision on one A100, over three orders of magnitude faster than
32
+ generative GUI world models, and about ten AUC points more accurate.
33
+ - **0.954 AUC** at separating harmful from benign violations with a linear head on
34
+ the prediction residual, where prompted VLMs are near chance.
35
+ - **173M** parameters at inference (ViT-B encoder, action encoder and predictor).
36
+
37
+ ## Files
38
+
39
  | Directory | Model |
40
  | --- | --- |
41
  | `lgwm-online/` | LGWM world model |
42
  | `lgwm-harm-head/` | Supervised harm head |
43
 
44
+ ## Usage
45
+
46
+ ```bash
47
+ git clone https://github.com/jiamingzhang94/lgwm && cd lgwm
48
+ pip install -c requirements-tested.txt -e '.[text]'
49
+ hf download jiamingzz/lgwm --local-dir weights
50
+ python tools/score_transition.py --before before.png --after after.png \
51
+ --action action.json --harm-head weights/lgwm-harm-head
52
+ ```
53
+
54
+ `action.json` holds the action, for example `{"action_type": "tap", "x": 0.5, "y": 0.5}`.
55
+ The command prints `integrity` and `harm_score`.