sodan commited on
Commit
fc2a18c
·
verified ·
1 Parent(s): 3f45dc9

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +69 -4
README.md CHANGED
@@ -2,12 +2,77 @@
2
  license: apache-2.0
3
  base_model: convaiinnovations/laya
4
  library_name: laya
5
- tags: [laya, computer-use, decision-model, calibrated]
 
 
6
  ---
7
  # laya-cua
8
 
9
- `convaiinnovations/laya` fine-tuned to choose the next computer-use action type from a screen state. It is a typed decision model: it scores ['click', 'type', 'key', 'scroll', 'navigate', 'move', 'wait', 'drag', 'done'] plus `task_done` and `needs_human`. It does not generate text, coordinates or typed strings.
10
 
11
- Training: RLCD via `laya.train`, 5316 items, 2 epochs. Held-out evaluation: see the notebook output; numbers are only valid for the dataset it was run on.
 
12
 
13
- Notebook and code: https://github.com/sodan/laya-cua
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2
  license: apache-2.0
3
  base_model: convaiinnovations/laya
4
  library_name: laya
5
+ datasets:
6
+ - xlangai/AgentNet
7
+ tags: [laya, computer-use, gui-agent, action-prediction]
8
  ---
9
  # laya-cua
10
 
11
+ A small (421M) model for computer-use agents. Given a task, a text description of the current screen and the last few steps, it answers two questions in one forward pass:
12
 
13
+ 1. **What kind of action comes next?** One of: click, type, key, scroll, move, wait, drag, done.
14
+ 2. **Is the task already finished?** (yes/no probability)
15
 
16
+ ## What it does NOT do
17
+ - It does not generate text, so it cannot write the string to type.
18
+ - It does not output click coordinates or element IDs.
19
+ - It does not read screenshots. The input is text (see format below).
20
+ - It is not a full agent. Use it as a fast router or sanity check next to whatever picks the target and the text (an LLM, a grounding model, a script).
21
+
22
+ ## Input and output
23
+ Input: a dict with `task` (instruction), `screen` (text description or accessibility tree of the current screen, trained on up to 1,800 characters) and `history` (up to the last 4 steps as short strings).
24
+ Output: `answers["action_type"]["choice"]` is one action name, `answers["task_done"]["noul"]` is the probability the task is done. The result also carries confidence values.
25
+
26
+ ## Usage
27
+ ```
28
+ pip install laya huggingface_hub
29
+ ```
30
+ ```python
31
+ import laya
32
+ from huggingface_hub import snapshot_download
33
+
34
+ ACTIONS = {
35
+ "click": "click or tap a UI element", "type": "type text into a focused field",
36
+ "key": "press a key or shortcut", "scroll": "scroll the view",
37
+ "navigate": "open a URL or switch app or tab", "move": "move the mouse without clicking (hover)",
38
+ "wait": "wait for the page or app to load", "drag": "drag and drop or select by dragging",
39
+ "done": "the task is already complete, stop",
40
+ }
41
+ QUESTIONS = {
42
+ "action_type": {"type": "choice", "instructions": "Which action type should the agent take next?", "criteria": ACTIONS},
43
+ "task_done": {"type": "noul", "instructions": "Is the task already fully complete on screen?"},
44
+ }
45
+ agent = laya.Agent(snapshot_download("sodan/laya-cua"), device="cuda") # or "cpu"
46
+ state = {"task": "Open the settings app and turn on dark mode",
47
+ "screen": "Settings window. Sidebar: Network, Display, Sound. Display page shows Theme: Light.",
48
+ "history": ["opened Settings"]}
49
+ res = agent.predict(state, QUESTIONS)
50
+ print(res["answers"]["action_type"]["choice"], res["answers"]["task_done"]["noul"])
51
+ ```
52
+ The calls (`laya.Agent`, `predict`, `answers[...]`) are the same ones used for the evaluation below, and loading this repo from the Hub was run and worked. The exact snippet above is being re-run end to end and this line will be updated once it has.
53
+
54
+ ## Training
55
+ - Base: convaiinnovations/laya (ModernBERT-large encoder plus a typed decision head). Method: the official `laya.train` RLCD loop on one free Colab T4.
56
+ - Data: OpenCUA AgentNet (xlangai/AgentNet, MIT), 400 tasks, one row per step: 5,822 train rows. Held-out: the same 334 rows (from 200 other tasks, whole tasks held out) used for every version. Labels come from the recorded action. `needs_human` has no label and was not trained.
57
+ - 2 epochs, same recipe as v0. The only change between v0 and v1 is more training tasks.
58
+
59
+ ## Held-out results (n=334, single run each)
60
+ | metric | v1 (current) | v0 (200 tasks) | base Laya | always "click" |
61
+ |---|---|---|---|---|
62
+ | action_type accuracy | 0.8084 | 0.7665 | 0.1856 | 0.7156 |
63
+ | task_done accuracy | 0.9611 | 0.9581 | 0.7066 | not computed |
64
+
65
+ ## Results log
66
+ | version | change | action_type | task_done | status |
67
+ |---|---|---|---|---|
68
+ | v0 | 200 tasks, 2 epochs | 0.7665 | 0.9581 | replaced |
69
+ | v1 | 400 tasks, 2 epochs | 0.8084 | 0.9611 | current (re-downloaded from this repo and re-evaluated: same numbers) |
70
+
71
+ ## Limits
72
+ - Small test set (334 rows), a single run per version, only 400 tasks of training data.
73
+ - v1 is about 9 points above always predicting "click" (0.8084 vs 0.7156). That is a measured gap on this one split, not a guarantee elsewhere.
74
+ - About 94% of held-out rows are "not done", so task_done accuracy is close to a trivial baseline.
75
+ - Trained on AgentNet text descriptions (mostly Ubuntu-style desktop tasks). Behavior on other screen formats, apps or phone UIs is untested.
76
+ - No other benchmarks were run.
77
+
78
+ License: Apache-2.0 (as the base). Data: AgentNet, MIT.