laya-cua

A small (421M) model for computer-use agents. Given a task, a text description of the current screen and the last few steps, it answers two questions in one forward pass:

  1. What kind of action comes next? One of: click, type, key, scroll, move, wait, drag, done.
  2. Is the task already finished? (yes/no probability)

What it does NOT do

  • It does not generate text, so it cannot write the string to type.
  • It does not output click coordinates or element IDs.
  • It does not read screenshots. The input is text (see format below).
  • It is not a full agent. Use it as a fast router or sanity check next to whatever picks the target and the text (an LLM, a grounding model, a script).

Input and output

Input: a dict with task (instruction), screen (text description or accessibility tree of the current screen, trained on up to 1,800 characters) and history (up to the last 4 steps as short strings). Output: answers["action_type"]["choice"] is one action name, answers["task_done"]["noul"] is the probability the task is done. The result also carries confidence values.

Usage

pip install laya huggingface_hub
import laya
from huggingface_hub import snapshot_download

ACTIONS = {
  "click": "click or tap a UI element", "type": "type text into a focused field",
  "key": "press a key or shortcut", "scroll": "scroll the view",
  "navigate": "open a URL or switch app or tab", "move": "move the mouse without clicking (hover)",
  "wait": "wait for the page or app to load", "drag": "drag and drop or select by dragging",
  "done": "the task is already complete, stop",
}
QUESTIONS = {
  "action_type": {"type": "choice", "instructions": "Which action type should the agent take next?", "criteria": ACTIONS},
  "task_done":   {"type": "noul",   "instructions": "Is the task already fully complete on screen?"},
}
agent = laya.Agent(snapshot_download("sodan/laya-cua"), device="cuda")  # or "cpu"
state = {"task": "Open the settings app and turn on dark mode",
         "screen": "Settings window. Sidebar: Network, Display, Sound. Display page shows Theme: Light.",
         "history": ["opened Settings"]}
res = agent.predict(state, QUESTIONS)
print(res["answers"]["action_type"]["choice"], res["answers"]["task_done"]["noul"])

The calls (laya.Agent, predict, answers[...]) are the same ones used for the evaluation below, and loading this repo from the Hub was run and worked. The exact snippet above is being re-run end to end and this line will be updated once it has.

Training

  • Base: convaiinnovations/laya (ModernBERT-large encoder plus a typed decision head). Method: the official laya.train RLCD loop on one free Colab T4.
  • Data: OpenCUA AgentNet (xlangai/AgentNet, MIT), 400 tasks, one row per step: 5,822 train rows. Held-out: the same 334 rows (from 200 other tasks, whole tasks held out) used for every version. Labels come from the recorded action. needs_human has no label and was not trained.
  • 2 epochs, same recipe as v0. The only change between v0 and v1 is more training tasks.

Held-out results (n=334, single run each)

metric v1 (current) v0 (200 tasks) base Laya always "click"
action_type accuracy 0.8084 0.7665 0.1856 0.7156
task_done accuracy 0.9611 0.9581 0.7066 not computed

Results log

version change action_type task_done status
v0 200 tasks, 2 epochs 0.7665 0.9581 replaced
v1 400 tasks, 2 epochs 0.8084 0.9611 current (re-downloaded from this repo and re-evaluated: same numbers)

Limits

  • Small test set (334 rows), a single run per version, only 400 tasks of training data.
  • v1 is about 9 points above always predicting "click" (0.8084 vs 0.7156). That is a measured gap on this one split, not a guarantee elsewhere.
  • About 94% of held-out rows are "not done", so task_done accuracy is close to a trivial baseline.
  • Trained on AgentNet text descriptions (mostly Ubuntu-style desktop tasks). Behavior on other screen formats, apps or phone UIs is untested.
  • No other benchmarks were run.

License: Apache-2.0 (as the base). Data: AgentNet, MIT.

Downloads last month
-
Safetensors
Model size
0.4B params
Tensor type
F16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for sodan/laya-cua

Finetuned
(253)
this model

Dataset used to train sodan/laya-cua