Hugging Face's logo

Spaces:
xiningli
/
webtask-arena
Sleeping

App Files Community
webtask-arena / README.md
Xining Li
Deploy snapshot: causal-gate + flash-offer tasks (textbook/ omitted, binary PDF rejected by HF's plain-git push policy)
6ec7f56
preview code
|
Raw History Blame Contribute Delete
6.71 kB
metadata 
title: WebTask Arena
emoji: 🕹️
colorFrom: blue
colorTo: gray
sdk: docker
app_port: 8000
pinned: false
license: mit
short_description: Deterministic, server-graded web tasks for GUI agents

WebTask Arena

A controlled environment for studying where GUI agents fail.

Live demo: this repo also runs as a Hugging Face Space — xiningli/webtask-arena — with a replay player (step through recorded UI-TARS episodes: every screenshot, action, and model thought next to the server's verdict) and a "beat the agent" mode (play the same seeded episode yourself and get graded by the same oracle, subgoal by subgoal).

WebTask Arena is a set of deterministic, resettable, self-contained web tasks with server-side reward functions, built for evaluating computer-use agents (UI-TARS, Qwen2.5-VL, Claude computer use, ...) under conditions you can actually reproduce: fixed seeds, versioned task logic, and no external network dependencies.

Why another environment?

Agent evaluations are noisy. Public benchmarks like OSWorld and WebArena are invaluable for coverage, but hard to use for controlled failure analysis: tasks depend on real software versions, rewards are sometimes brittle, and runs are expensive. WebTask Arena takes the opposite trade-off — a small number of fully-owned tasks where:

  • Every episode is seed-deterministic. POST /api/email-triage/reset {"seed": 42} produces byte-identical state, instruction, and page, every time, on every machine.
  • Reward is computed server-side from ground-truth state, never scraped from the DOM. The agent can't fool the grader, and neither can a flaky selector.
  • Success is decomposed into named subgoals (archived_all_from_target, no_extra_archived, ...), so a failed episode tells you which capability broke, not just that one did.
  • Adversarial rendering is a flag, not a fork. Every task can be run with artificial load latency (latency_ms) and delayed layout shift (layout_shift) to measure how much of an agent's success rate is an artifact of stable, instant pages.
  • Hermetic by construction. No CDNs, no fonts, no analytics, no external requests. The container is the whole world.

Tasks (v0)

Task Skills probed Traps
email-triage list scanning, per-row actions, precision extra archives fail; star exactly one
settings-panel navigation across sections, toggles, selects draft changes don't count until Save; unrelated changes fail
checkout-form multi-step forms, validation-error recovery submitting wrong data "succeeds" in the UI but fails verification

Quickstart 

docker compose up --build
# open http://localhost:8000

Drive an episode:

# start a fresh, seeded episode (optionally with adversarial rendering)
curl -X POST localhost:8000/api/email-triage/reset \
  -H 'Content-Type: application/json' \
  -d '{"seed": 42, "latency_ms": 800, "layout_shift": true}'
# -> {"task_id": "email-triage", "seed": 42, "instruction": "In the inbox, archive every email from ..."}

# ...point your agent at http://localhost:8000/task/email-triage ...

# grade the episode
curl localhost:8000/api/email-triage/verify
# -> {"success": false, "subgoals": {"archived_all_from_target": false, ...}}

/api/*/verify and /api/*/state are oracle endpoints for the harness — never expose them to the agent under evaluation.

One episode per task per container; parallelize batch evals by running multiple containers.

Harness

The harness drives real browser episodes with Playwright: screenshot in, computer-use-style action out (left_click at pixel coordinates, type, key, scroll, ...), executed against a fixed 1280×800 viewport. Every episode records step_NNN.png, actions.jsonl, and result.json with per-subgoal outcomes.

pip install -r requirements-dev.txt && playwright install chromium
uvicorn server.app:app &

# self-test: the scripted oracle agent exercises the full pipeline
python -m harness.run --agent scripted --task all --seeds 0-4

# same tasks under adversarial rendering
python -m harness.run --agent scripted --task all --seeds 0-4 \
    --latency-ms 800 --layout-shift

# Claude computer-use adapter (needs ANTHROPIC_API_KEY + pip install anthropic)
python -m harness.run --agent claude --task all --seeds 0-4

The scripted agent is a DOM-oracle reference implementation: it parses the natural-language instruction and emits ordinary click/type/key actions, but locates targets via the DOM. It exists to validate the pipeline (env → screenshots → executor → server state → verify), not as a baseline — model adapters never get DOM access. It passes 3/3 tasks across seeds under both clean and adversarial rendering.

Development 

pip install -r requirements-dev.txt
pytest            # determinism + reward contract tests
uvicorn server.app:app --reload

Adding a task: implement the Task protocol (server/registry.py) — a seeded generate, an instruction, UI action handlers, and a verify that returns named subgoals — then register it. The contract tests in tests/test_environment.py are parametrized over the registry.

Roadmap

  • Playwright harness: screenshot loop, agent adapter interface, trajectory recording
  • Claude computer-use adapter (computer_20251124)
  • UI-TARS-1.5-7B evaluated against the arena — see uitars-webtask-eval (1/3 pass at seed 0, with a diagnosed state-tracking failure and an environment finding about native <select> under headless rendering)
  • Qwen2.5-VL adapter; port the UI-TARS adapter into harness/agents/
  • pass@1 / pass^k reporting with per-subgoal breakdowns across repeated seeds
  • Failure taxonomy: grounding misclicks vs. perception errors vs. planning loops vs. premature termination
  • More tasks: drag-to-reorder, infinite-scroll search, modal interruptions

Related work

License

MIT