Helium Browser Environment

This example shows the main pieces of an OpenEnv environment by wrapping a Chromium browser. A client sends one typed action at a time. The environment performs it with Helium and Selenium, then returns a typed observation with a fresh screenshot and the current URL.

The browser is useful here because the boundary is easy to see: OpenEnv owns the reset/step protocol, while the environment owns all browser-specific behavior. The example does not include an agent, so the OpenEnv parts stay in focus.

How to run the browser agent

The actual browser loop is browser.py. You can run it with following steps. First publish the environment. Make sure to authenticate yourself with a write token.

hf auth login
openenv push envs/helium_browser_env \
  --repo-id YOUR_USERNAME/helium-browser-env

The agent uses the token saved by hf auth login, or HF_TOKEN if it is set. Make sure that token can access your inference endpoint. Then deploy the model and provide your endpoint.

export HELIUM_BROWSER_IMAGE='hf.co/spaces/YOUR_USERNAME/helium-browser-env'
export BASE_URL='https://YOUR_ENDPOINT.endpoints.huggingface.cloud/v1'
export MODEL_ID="HF-MODEL-ID" # replace with the model you run endpoint with

Run browser.py loop inside Hugging Face Sandboxes.

import os
import runpy
from openenv.core.containers.runtime.hf_sandbox_provider import HFSandboxProvider

with HFSandboxProvider(
    image=os.environ["HELIUM_BROWSER_IMAGE"],
    flavor="cpu-basic",
    env_vars={"BROWSER_SANDBOX": "hf"},
) as provider:
    url = provider.start_container()
    provider.wait_for_ready(url, timeout_s=300)
    os.environ["BROWSER_BASE_URL"] = url
    runpy.run_path("envs/helium_browser_env/browser.py", run_name="__main__")

Rest of this tutorial goes through OpenEnv basics and how to build a multimodal browser agent.

How browser agents work

A natively multimodal browser agent operates in a loop:

1. Observe: receive the current page, as a screenshot.
2. Decide: choose one small action that moves the task forward.
3. Act: click, type, press a key, scroll, or navigate.
4. Observe again: inspect the page after the action instead of assuming it worked.
5. Stop: finish when the task is complete or the episode reaches a limit.

This example uses screenshots as the main observation. The agent sees the same 800×600 browser image that the environment uses for coordinates, then returns one structured action. A click such as x=320, y=180 means a click position in that image; it is not a CSS selector or a reference to the page’s HTML.

One action per step matters because web pages change. A click may open a menu, show a cookie banner, navigate to another page, or fail. Returning a fresh screenshot and URL after every action lets the caller react to what actually happened. It also produces a clear trajectory: observation, action, next observation.

That loop explains the design choices used later in this example:

• Typed actions give OpenEnv a small, validated set of actions for controlling the browser.
• A fixed viewport keeps screenshot pixels and click coordinates aligned.
• Real pointer events let clicks reach visual controls such as overlays and canvases without exposing DOM selectors to the agent.
• An isolated browser keeps temporary profiles and untrusted pages away from the user’s personal browser session.
• Step limits and done give every browser episode a definite boundary.

The agent or policy that chooses actions is intentionally outside this environment. OpenEnv provides the repeatable interaction contract; the caller decides how to turn an observation into the next BrowserAction.

Tasks

Websites always keep on changing so using a regular dataset doesn’t work. You can try the samples in this dataset as they are created late 2026. They are divided into three:

• Information: getting an information from a website by browsing it.
• Navigation: navigating a specific part of the website.
• Interaction: clicking, filtering etc.

The OpenEnv pieces

An OpenEnv environment has four small parts in this example:

These parts separate domain logic from transport. BrowserEnvironment knows how to drive a browser, but it does not define HTTP routes. BrowserClient knows how to exchange OpenEnv messages, but it does not import Helium or Selenium.

1. Define the data contract

models.py subclasses OpenEnv’s three base models with the actions an agent can take: “click”, “type”, “key”, “scroll”, “back”, “wait”, “finish”. x and y are clicking coordinates, dy is change in vertical direction (scroll dy pixels), text is text to be typed on e.g. search bars.

class BrowserAction(Action):
    op: Literal["click", "type", "key", "scroll", "back", "wait", "finish"]
    x: int | None = Field(default=None, ge=0, lt=800)
    y: int | None = Field(default=None, ge=0, lt=600)
    text: str = Field(default="", max_length=2000)
    dy: int = Field(default=0, ge=-1200, le=1200)


class BrowserObservation(Observation):
    screenshot: str
    url: str
    error: str = ""


class BrowserState(State):
    pass

Actions are taken by the agent and observations are the result of the actions in the environment. On the other hand, OpenEnv uses state to report episode metadata such as the episode ID and step count, without adding those fields to every browser observation.

Click coordinates are browser pixels: x is from 0 to 799 and y is from 0 to 599. Scroll values are limited to the range -1200 to 1200.

Supported keys are ENTER, TAB, ESCAPE, BACKSPACE, CTRL+A, UP, DOWN, LEFT, and RIGHT.

Every observation contains:

• screenshot: the current browser window as a base64-encoded PNG;
• url: the current page URL;
• error: an empty string or a short error code;
• done: whether the episode has ended.

The OpenEnv step result carries a reward of 0 because this example does not score browser actions.

The episode ends when it receives finish, reaches the configured step limit, encounters a renderer timeout, or detects a supported bot-verification page. An action error is reported by exception class name so the caller can inspect the next screenshot and decide what to do.

2. Implement the environment lifecycle

BrowserEnvironment subclasses OpenEnv’s Environment. Its methods have the same roles they would have in a game, simulator, or other environment:

• reset(...) starts a new episode and returns its first observation. Here it creates a temporary browser profile, opens start_url, and takes a screenshot.
• step(action) applies exactly one validated action and returns the next observation.
• state exposes the current BrowserState, including OpenEnv’s episode ID and step count.
• close() releases the resources owned by the episode.

This is the Gym-like core of OpenEnv. Browser setup and Helium calls are ordinary implementation details behind that interface.

3. Turn it into a server

server/app.py passes the environment class and its wire types to create_app:

app = create_app(
    BrowserEnvironment,
    BrowserAction,
    BrowserObservation,
    env_name="helium_browser_env",
    max_concurrent_envs=1,
)

create_app supplies the FastAPI application and the standard OpenEnv endpoints, including health, reset, step, state, and WebSocket access. The environment code only implements the lifecycle; it does not duplicate those routes.

The environment class is passed instead of a pre-built instance so OpenEnv can own its lifecycle. Concurrency is limited to one because Helium stores the active Selenium driver globally and Selenium drivers are not thread-safe.

4. Add the typed client

BrowserClient subclasses OpenEnv’s generic EnvClient with the action, observation, and state types:

class BrowserClient(EnvClient[BrowserAction, BrowserObservation, BrowserState]):
    def _step_payload(self, action):
        return action.model_dump()

    def _parse_result(self, payload): ...

    def _parse_state(self, payload):
        return BrowserState.model_validate(payload)

The base client handles the connection and the common reset, step, and state operations. This subclass only teaches it how this environment’s models map to JSON. Callers therefore work with BrowserAction and BrowserObservation, not untyped dictionaries.

How it fits together

BrowserClient -> OpenEnv server -> BrowserEnvironment -> Helium/Selenium -> Chromium
BrowserClient <- screenshot, URL, error, done <- BrowserEnvironment

For a step, BrowserClient serializes a BrowserAction. OpenEnv validates it, calls BrowserEnvironment.step(), and serializes the returned BrowserObservation. The client then rebuilds a typed StepResult containing the observation, reward, and done flag.

The OpenEnv loop

Here’s how OpenEnv loop works: once the model server is running, reset the browser to a URL and send one action at a time.

from helium_browser_env.client import BrowserClient
from helium_browser_env.models import BrowserAction

with BrowserClient(base_url="http://localhost:8000").sync() as browser:
    result = browser.reset(start_url="https://example.com")
    observation = result.observation

    result = browser.step(BrowserAction(op="scroll", dy=500))
    observation = result.observation

    print(observation.url)
    print(observation.error)
    print(result.done, result.reward)
    print(observation.screenshot)  # Base64-encoded PNG

It starts a fresh browser profile, opens the page, and returns the first observation. step performs one action and returns an OpenEnv StepResult. The result groups the next observation with the reward and episode termination flag.

Notes

The environment is designed to run inside its browser sandbox, not against a personal desktop browser. Chromium uses a temporary profile for each episode. Downloads and password storage are disabled, and a reset rejects URLs that contain a username or password.

Web pages are untrusted input. Do not enter credentials or sensitive data, and expect some sites to show automation challenges. This example provides the browser interaction loop; it does not provide tasks, scoring, or a reward function.