Buckets:
| # OpenApp Environment | |
| *A web application simulation environment for OpenEnv that wraps the [OpenApps](https://github.com/facebookresearch/OpenApps) framework and BrowserGym.* | |
| ## Overview | |
| Agents interact with simulated web apps (calendar, todo list, messenger, maps) through BrowserGym browser actions: click, fill forms, navigate, scroll and type. | |
| ## Quick Start | |
| Build the image from `envs/openapp_env/` (about 5.7 GB, it includes Chromium and OpenApps): | |
| ```bash | |
| docker build -t openapp-env:latest -f server/Dockerfile . | |
| ``` | |
| The container runs two servers: OpenApps on port 5001 (internal) and the OpenEnv API on port 8000. You only talk to port 8000. | |
| ```python | |
| from openapp_env import OpenAppAction, OpenAppEnv | |
| with OpenAppEnv.from_docker_image("openapp-env:latest").sync() as client: | |
| result = client.reset() | |
| print(f"Starting URL: {result.observation.url}") | |
| result = client.step(OpenAppAction(action_type="goto", url="http://localhost:5001/calendar")) | |
| result = client.step(OpenAppAction(action_type="click", bid="add-event-btn")) | |
| result = client.step(OpenAppAction(action_type="fill", bid="event-title-input", text="Team Meeting")) | |
| print(f"Reward: {result.reward}, Done: {result.done}") | |
| ``` | |
| Element IDs (`bid`) come from the observation's `axtree_txt`. To use the async client, `await OpenAppEnv.from_docker_image(...)` instead of calling `.sync()`. | |
| A complete script is in [`examples/openapp_example.py`](https://github.com/huggingface/OpenEnv/blob/main/examples/openapp_example.py): | |
| ```bash | |
| python examples/openapp_example.py --mode docker --num-steps 20 | |
| ``` | |
| With the container running (`docker run -p 8000:8000 openapp-env:latest`), the web UI is at `http://localhost:8000/web` and the API docs at `http://localhost:8000/docs`. | |
| ## Environment Details | |
| ### Action | |
| **OpenAppAction** | |
| | `action_type` | Required fields | | |
| |---------------|-----------------| | |
| | `click` | `bid` (BrowserGym element ID) | | |
| | `fill` | `bid`, `text` | | |
| | `select_option` | `bid`, `value` | | |
| | `goto` | `url` | | |
| | `scroll` | `direction` (`"up"` or `"down"`) | | |
| | `send_keys` | `text` | | |
| | `noop` | none | | |
| ### Observation | |
| **OpenAppObservation** | |
| - `html`: current page HTML | |
| - `url`: current page URL | |
| - `open_pages_urls`, `active_page_index`: open tabs and the active one | |
| - `axtree_txt`: accessibility tree, with the element IDs to act on | |
| - `screenshot`: base64-encoded screenshot (optional) | |
| - `app_state`: state of the apps (calendar events, todos, messages, ...) | |
| - `task_info`: `{"task_name": ...}` when the server was started with a task name | |
| - `last_action_error`: error message if the last action failed | |
| - `metadata["cumulative_reward"]`: reward accumulated over the episode | |
| ### Reward | |
| The environment wraps OpenApps in a generic BrowserGym task that never scores or ends an episode by itself. Each step returns: | |
| - `-0.1` when the action fails or `action_type` is unknown | |
| - the BrowserGym task reward otherwise, which is `0.0` with the generic task | |
| The episode ends after `max_steps` steps (default 50). For task-specific rewards, score the episode yourself from `app_state`. See the [OpenApps documentation](https://facebookresearch.github.io/OpenApps/) for its tasks. | |
| ## Configuration | |
| The environment connects to an OpenApps server that is already running, at `OPENAPPS_URL` (the Docker image sets it to `http://localhost:5001`). It doesn't launch one itself. | |
| To change the other settings, construct `OpenAppEnvironment` (in `server/openapp_environment.py`) yourself: | |
| | Parameter | Default | Description | | |
| |-----------|---------|-------------| | |
| | `openapps_url` | `OPENAPPS_URL` | URL of the OpenApps server. `reset()` fails if `OPENAPPS_URL` is not set | | |
| | `headless` | `True` | Run the browser headless | | |
| | `task_name` | `None` | Task name, reported in `task_info` | | |
| | `apps_config` | `{}` | App configuration | | |
| | `max_steps` | `50` | Steps per episode | | |
| ## Running Without Docker | |
| Install the environment and a browser, then start OpenApps from a clone of its repository (the pip package doesn't ship `launch.py` and its Hydra configs): | |
| ```bash | |
| pip install -e envs/openapp_env | |
| playwright install chromium | |
| git clone https://github.com/facebookresearch/OpenApps.git | |
| cd OpenApps && uv sync | |
| uv run launch.py # headless | |
| uv run launch.py browsergym_env_args.headless=False # with a visible browser | |
| ``` | |
| In another terminal: | |
| ```bash | |
| export OPENAPPS_URL=http://localhost:5001 | |
| python examples/openapp_example.py --mode local | |
| ``` | |
| The apps are also reachable in your browser at `http://localhost:5001` (`/calendar`, `/todo`, `/messages`, `/maps`). The browser window is controlled by the OpenApps server, so start it with `browsergym_env_args.headless=False` to watch the agent. | |
| ## Troubleshooting | |
| - **`Container did not become ready`** behind an HTTP proxy: set `export NO_PROXY=localhost,127.0.0.1` and retry. | |
| - **Container exits immediately**: check `docker logs <container-id>`. Usually OpenApps failed to start (port conflict) or a dependency is missing (rebuild with `--no-cache`). | |
| - **`Connection refused to localhost:5001`** in local mode: start the OpenApps server first and set `OPENAPPS_URL`. | |
| - **Slow container**: it runs Chromium and the web apps. Give Docker 6 GB or more of memory and keep `headless=True`. | |
| ## Attribution | |
| - [OpenApps](https://github.com/facebookresearch/OpenApps): web application simulation framework | |
| - [BrowserGym](https://github.com/ServiceNow/BrowserGym): browser automation environment | |
| ## Citation | |
| If you use this environment in your research, please cite both OpenEnv and OpenApps: | |
| ```bibtex | |
| @article{ullrich2025openapps0, | |
| title = {OpenApps: Simulating Environment Variations to Measure UI-Agent Reliability}, | |
| author = {Karen Ullrich and Jingtong Su and Claudia Shi and Arjun Subramonian and Amir Bar and Ivan Evtimov and Nikolaos Tsilivis and Randall Balestriero and Julia Kempe and Mark Ibrahim}, | |
| year = {2025}, | |
| journal = {arXiv preprint arXiv: 2511.20766} | |
| } | |
| ``` | |
Xet Storage Details
- Size:
- 5.99 kB
- Xet hash:
- 106187bef6695b8a58d33f27071f224555f37d56d922358b87f37fa841d8ca5b
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.