File size: 6,708 Bytes
6ec7f56
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
---
title: WebTask Arena
emoji: πŸ•ΉοΈ
colorFrom: blue
colorTo: gray
sdk: docker
app_port: 8000
pinned: false
license: mit
short_description: Deterministic, server-graded web tasks for GUI agents
---

# WebTask Arena

A controlled environment for studying **where GUI agents fail**.

> **Live demo:** this repo also runs as a Hugging Face Space β€”
> [xiningli/webtask-arena](https://huggingface.co/spaces/xiningli/webtask-arena) β€”
> with a **replay player** (step through recorded UI-TARS episodes: every
> screenshot, action, and model thought next to the server's verdict) and a
> **"beat the agent" mode** (play the same seeded episode yourself and get
> graded by the same oracle, subgoal by subgoal).

WebTask Arena is a set of deterministic, resettable, self-contained web tasks
with server-side reward functions, built for evaluating computer-use agents
(UI-TARS, Qwen2.5-VL, Claude computer use, ...) under conditions you can
actually reproduce: fixed seeds, versioned task logic, and no external
network dependencies.

## Why another environment?

Agent evaluations are noisy. Public benchmarks like OSWorld and WebArena are
invaluable for coverage, but hard to use for *controlled* failure analysis:
tasks depend on real software versions, rewards are sometimes brittle, and
runs are expensive. WebTask Arena takes the opposite trade-off β€” a small
number of fully-owned tasks where:

- **Every episode is seed-deterministic.** `POST /api/email-triage/reset
  {"seed": 42}` produces byte-identical state, instruction, and page, every
  time, on every machine.
- **Reward is computed server-side from ground-truth state**, never scraped
  from the DOM. The agent can't fool the grader, and neither can a flaky
  selector.
- **Success is decomposed into named subgoals** (`archived_all_from_target`,
  `no_extra_archived`, ...), so a failed episode tells you *which* capability
  broke, not just that one did.
- **Adversarial rendering is a flag, not a fork.** Every task can be run with
  artificial load latency (`latency_ms`) and delayed layout shift
  (`layout_shift`) to measure how much of an agent's success rate is an
  artifact of stable, instant pages.
- **Hermetic by construction.** No CDNs, no fonts, no analytics, no external
  requests. The container is the whole world.

## Tasks (v0)

| Task | Skills probed | Traps |
|---|---|---|
| `email-triage` | list scanning, per-row actions, precision | extra archives fail; star exactly one |
| `settings-panel` | navigation across sections, toggles, selects | draft changes don't count until **Save**; unrelated changes fail |
| `checkout-form` | multi-step forms, validation-error recovery | submitting wrong data "succeeds" in the UI but fails verification |

## Quickstart

```bash
docker compose up --build
# open http://localhost:8000
```

Drive an episode:

```bash
# start a fresh, seeded episode (optionally with adversarial rendering)
curl -X POST localhost:8000/api/email-triage/reset \
  -H 'Content-Type: application/json' \
  -d '{"seed": 42, "latency_ms": 800, "layout_shift": true}'
# -> {"task_id": "email-triage", "seed": 42, "instruction": "In the inbox, archive every email from ..."}

# ...point your agent at http://localhost:8000/task/email-triage ...

# grade the episode
curl localhost:8000/api/email-triage/verify
# -> {"success": false, "subgoals": {"archived_all_from_target": false, ...}}
```

`/api/*/verify` and `/api/*/state` are **oracle endpoints for the harness** β€”
never expose them to the agent under evaluation.

One episode per task per container; parallelize batch evals by running
multiple containers.

## Harness

The harness drives real browser episodes with Playwright: screenshot in,
[computer-use-style action](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool)
out (`left_click` at pixel coordinates, `type`, `key`, `scroll`, ...), executed
against a fixed 1280Γ—800 viewport. Every episode records `step_NNN.png`,
`actions.jsonl`, and `result.json` with per-subgoal outcomes.

```bash
pip install -r requirements-dev.txt && playwright install chromium
uvicorn server.app:app &

# self-test: the scripted oracle agent exercises the full pipeline
python -m harness.run --agent scripted --task all --seeds 0-4

# same tasks under adversarial rendering
python -m harness.run --agent scripted --task all --seeds 0-4 \
    --latency-ms 800 --layout-shift

# Claude computer-use adapter (needs ANTHROPIC_API_KEY + pip install anthropic)
python -m harness.run --agent claude --task all --seeds 0-4
```

The **scripted agent** is a DOM-oracle reference implementation: it parses the
natural-language instruction and emits ordinary click/type/key actions, but
locates targets via the DOM. It exists to validate the pipeline (env β†’
screenshots β†’ executor β†’ server state β†’ verify), not as a baseline β€” model
adapters never get DOM access. It passes 3/3 tasks across seeds under both
clean and adversarial rendering.

## Development

```bash
pip install -r requirements-dev.txt
pytest            # determinism + reward contract tests
uvicorn server.app:app --reload
```

Adding a task: implement the `Task` protocol (`server/registry.py`) β€” a
seeded `generate`, an `instruction`, UI action handlers, and a `verify` that
returns named subgoals β€” then register it. The contract tests in
`tests/test_environment.py` are parametrized over the registry.

## Roadmap

- [x] Playwright harness: screenshot loop, agent adapter interface, trajectory recording
- [x] Claude computer-use adapter (`computer_20251124`)
- [x] UI-TARS-1.5-7B evaluated against the arena β€” see
      [uitars-webtask-eval](https://github.com/xiningli/uitars-webtask-eval)
      (1/3 pass at seed 0, with a diagnosed state-tracking failure and an
      environment finding about native `<select>` under headless rendering)
- [ ] Qwen2.5-VL adapter; port the UI-TARS adapter into `harness/agents/`
- [ ] pass@1 / pass^k reporting with per-subgoal breakdowns across repeated seeds
- [ ] Failure taxonomy: grounding misclicks vs. perception errors vs. planning loops vs. premature termination
- [ ] More tasks: drag-to-reorder, infinite-scroll search, modal interruptions

## Related work

- [UI-TARS / UI-TARS-2](https://github.com/bytedance/UI-TARS) β€” native GUI agent models (ByteDance)
- [OSWorld](https://os-world.github.io/) β€” real-OS benchmark for multimodal agents
- [WebArena](https://webarena.dev/) / VisualWebArena β€” realistic self-hosted web benchmarks
- [ScreenSpot](https://github.com/njucckevin/SeeClick) β€” GUI grounding evaluation
- [Anthropic computer use](https://docs.anthropic.com/en/docs/agents-and-tools/computer-use) β€” reference agent loop

## License

MIT