WebArena-Pro: A Heterogeneous, Multimodal, Reproducible Benchmark for Web Agents
Abstract
Web agents powered by large language and vision-language models are increasingly applied to realistic browser work that spans heterogeneous applications, multimodal content, and stateful workflows. However, existing reproducible web-agent benchmarks cover only a small number of web applications drawn from a few software categories, and restrict modality to text and vision. Live benchmarks broaden site coverage but sacrifice reproducibility, since pages and data drift between runs. Moreover, existing benchmarks do not meaningfully evaluate whether agents can understand and use audio and video content embedded within web tasks. To address these gaps, we introduce WebArena-Pro, a benchmark comprising 300 tasks across 20 self-hosted web applications in six domain categories, spanning distinct interface conventions, workflows, and data models. Across the evaluated agents, the best performance is achieved by both Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5 at 62.3%, followed by OpenCode with Kimi 3.1 Pro at 60.7%. The strongest GenericAgent configurations, Gemini 3.1 Pro omni and Gemma4 31B, both achieve 30.3%. Among reproducible, human-curated web agent benchmarks, WebArena-Pro provides the broadest application coverage and the most comprehensive multimodal support to date. The benchmark treats audio and video as core observations alongside text and vision, with dedicated actions for extracting information from each. WebArena-Pro runs each task in isolation and supports reproducible, parallel evaluation. Tasks are authored through a dedicated annotator interface, filtered by LLM-assisted triage, and finally validated by humans before release. Project page: https://webarena-pro.github.io
Community
We're releasing WebArena-Pro, a benchmark for web agents covering 20 self-hosted applications across six domains, with 300 human-validated tasks spanning text, vision, audio, and video, and isolated task execution for reproducible, parallel evaluation. The benchmark supports multiple harness integrations, including BrowserGym and Harbor for running MCP and coding agents.
Hi authors, thanks for the benchmark. A question on Appendix C.1: 1,683 unsuccessful of 2,400 GenericAgent Test runs leaves 717 successes, but the eight GenericAgent rows of Table 5 give 497 (so 1,903 unsuccessful). Was C.1 run on different trajectories? Details in this GitHub issue: https://github.com/McGill-NLP/webarena-pro/issues/4
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- WebHarbor: Docking Real Websites for Evolving GUI Agent Environments (2026)
- Back to Basics: A Skill-Centric Approach to Generalist Web Agents (2026)
- EmailBench: A Benchmark for Evaluating LLM Agents on Enterprise Email and Productivity Tasks (2026)
- Same Tasks, Different Apps: Why Mobile GUI Agents Fail to Generalize? (2026)
- Agent World Model in the Wild: Computer-Use Agent Environment Synthesis (2026)
- MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environments (2026)
- WebUniverse: Scaling Verified Training Data for Agentic Web Development (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2611.17955 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper