---
title: armnet eval
emoji: 🦾
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 6.20.0
python_version: "3.12"
app_file: app.py
pinned: false
short_description: Evaluate a robot policy on real hardware, no setup.
---

# armnet eval — evaluate a policy on real robots

[armnet.dev](https://armnet.dev/)

A deliberately constrained public demo of the armnet infrastructure. **No install,
no auth setup, no Docker.** Pick a task, paste (or select) a policy repo id, press
**Run**, and watch **20 rollouts** on a real SO-101 cell:

- **Live video** of the rollout, streamed from the cell.
- **Cell availability light**: a blinking dot — green when at least one BusyBox
  cell is free, yellow when every cell is busy, red when none are online — with
  a count such as `1 cell available`.
- **Auto-scored** per-rollout success from the instrumented BusyBox — no human
  judges, and the scene resets itself between rollouts.
- **Shared**: everyone who opens the Space follows the same fleet in real time.
  If more than one cell is running an eval, the Live tab names each policy and
  lets you pick which feed to watch.
- **Queue-aware**: while every cell is busy, submitters can queue another
  20-rollout eval. A client-side abuse guard limits this Space to 10 waiting
  evals; it does not cap the fleet-wide orchestrator queue. The Space only sees
  jobs owned by its API user and renders unrelated ones as `other job`. The Run
  button remains available while a Space eval is live, but the same policy repo
  cannot be running and/or queued more than once. Once its run finishes, it can
  be submitted again.
- **Leaderboard**: every run is recorded and pooled by policy repo + revision,
  ranked by the 95% lower-bound success rate (so more rollouts only help a
  genuinely good policy — spamming evals can't game it).

## Tasks

A selector above the tabs picks the task, and it scopes everything below it: the
dataset the instructions point at, the policy the Live tab defaults to, and the
board the Leaderboard shows. Both run on BusyBox cells assigned the `busybox`
environment — the orchestrator resolves the task slug to that environment when
it routes a job, and any free cell serving it can pick the work up.

| Task | Slug | Dataset | Reference policy |
| --- | --- | --- | --- |
| Push the green button | `push_green_button` | [`armnet/busybox_push_green_button`](https://huggingface.co/datasets/armnet/busybox_push_green_button) | [`…_push_green_button_diffusion`](https://huggingface.co/armnet/busybox_push_green_button_diffusion) |
| Flip left switch off | `busybox_left_toggle_down` | [`armnet/busybox_flip_left_switch_off`](https://huggingface.co/datasets/armnet/busybox_flip_left_switch_off) | [`…_flip_left_switch_off_diffusion`](https://huggingface.co/armnet/busybox_flip_left_switch_off_diffusion) |

Either task-specific dataset can be replaced with
[`armnet/busybox_multitask`](https://huggingface.co/datasets/armnet/busybox_multitask)
for language-conditioned multi-task training.

Adding a third is one entry in `TASKS` in `config.py`. The only hard requirement
is that it **resets itself** — a task needing a human to tidy up between rollouts
can't be run unattended by the public.

## Persistence

Gradio has no built-in database and a Space's disk is ephemeral, so the
leaderboard is stored in a Hugging Face **dataset** repo (`ARMNET_DEMO_LEADERBOARD_REPO`,
default `armnet/armnet-demo-leaderboard`): one JSONL line per completed
run, reloaded on restart. Reads are public; **writing new runs needs `HF_TOKEN`**
(the same token used to pull gated policies).

Each run records `source`, `cell_id`, `embodiment` and `task` alongside the
success counts. Each task gets **its own board**: rows are filtered to this
Space's embodiment (`lerobot/so-101`) and to the selected task exactly, since a
policy that presses buttons well says nothing about one that flips a switch.
Runs from every BusyBox cell in the demo fleet are pooled together. The board
shows a **Source** column: a verified checkmark ("result verified by Armnet")
when every pooled rollout for a policy came from this demo, otherwise
`script`/`mixed`. Armnet eval **scripts** can also write to the same dataset
(`source="script"`) via `ctx.submit_results_to_leaderboard()`; that path
currently relies on the eval container's HF token having write access to the
dataset — a server-side submission endpoint is planned so external users can
submit without that token.

## How it works

1. The Space submits a job to the armnet orchestrator (baked-in demo API key)
   naming only the embodiment (`lerobot/so-101`) and the selected task's slug,
   passing the policy repo id as a job arg (`use_rerun=True` turns on the video
   feed). No cell is named in the submission — the task is what routes it. If
   every matching cell is occupied, the job waits in the shared queue and the
   Space adopts it for live display when the scheduler dispatches it.
2. The orchestrator resolves that task to its **environment** (`busybox`) and
   routes the job to a cell assigned that environment. Cells are assigned an
   environment in the FMS, not a task, and serve every task within it — so if no
   cell is set to `busybox` the availability light goes red and nothing can be
   submitted.
3. The chosen cell pulls the pre-built eval image, loads the policy from the Hub,
   and runs the rollouts. It streams observation frames (Rerun packets) and
   per-rollout results back over the orchestrator websockets.
   When a queued Space eval starts, the Space attaches a live follower to that
   job even if another cell is already running.
4. The Space decodes those into per-cell live video + results, held in
   process-global slots so all viewers share the same feeds. Each browser
   receives the newest frame of the cell it is watching through one cancellable
   async Gradio stream, capped at 5 FPS; slow viewers skip stale frames instead
   of accumulating video latency. Frames are resized once process-wide rather
   than once per viewer.

## Space configuration

Set these in **Settings → Variables and secrets**:

| Variable | Required | Purpose |
| --- | --- | --- |
| `ARMNET_DEMO_EVAL_IMAGE` | no | Registry ref of the pre-built lerobot-eval runtime image (below). A current ref is baked into `config.py`; set this only to override it without editing code. A Space can't build Docker, so this is built + pushed once by an operator. |
| `ARMNET_API_KEY` | **yes** | Orchestrator API key for the demo (a low-privilege throwaway key). Provided as a secret — nothing is baked into the repo. |
| `HF_TOKEN` | recommended | The Space stores it once as a named orchestrator secret (`ARMNET_DEMO_HF_SECRET_NAME`, default `hf_token`) under the demo user and the eval references it as `$HF_TOKEN` to pull a gated policy; it also authenticates leaderboard writes. Without it the leaderboard is read-only and gated policies won't load. |
| `ARMNET_DEMO_N_EPISODES` | no | Rollouts per run (default 20). |
| `ARMNET_DEMO_MAX_QUEUED_JOBS` | no | Client-side cap on this Space's queued eval jobs (default 10); it does not limit the orchestrator queue. |
| `ARMNET_DEMO_LEADERBOARD_REPO` | no | HF dataset repo backing the leaderboard (default `armnet/armnet-demo-leaderboard`). |
| `ARMNET_DEMO_DATASET_NAMESPACE` | no | Owner of the recorded eval dataset (default `armnet`). Without this the eval uses whoever owns `HF_TOKEN`. |
| `ARMNET_DEMO_CELL_STATUS_POLL_S` | no | How often the shared availability lights refresh (default 5s). One process-wide poll per task, not per viewer. |
| `DEBUG_ARMNET_ORCHESTRATOR_URI` | no | Override the orchestrator URL (default: the one compiled into `armnet-client`). |

> No keys are baked into the repo — both `ARMNET_API_KEY` and `HF_TOKEN` are
> read from the Space's secrets. The demo API key is a low-privilege throwaway
> key; rotate/invalidate it freely.

### Deploying the Space itself

The same `Armnet Demo Eval Image` workflow uploads this directory (minus `tests/`)
to the Space on every push to `main`, so a fix to the app or its requirements goes
live without anyone pushing by hand. It compares the checkout against what the
Space actually holds and commits only the files that differ — a commit restarts
the Space, so an unchanged Space is left alone. Because that check is on content
rather than on which paths a commit touched, a deploy that failed is simply
retried by the next run.

To preview or do it by hand:

```bash
cd armnet
uv run python examples/ci_deploy_demo_space.py --dry-run   # what would change
uv run python examples/ci_deploy_demo_space.py             # deploy it
```

Files it finds on the Space but not in the checkout are reported, not deleted.

### Building the eval image

The policy is **not** baked into the image — it is loaded at runtime from the
repo id — so one image serves every policy, and building it needs no robot.

The `Armnet Demo Eval Image` workflow keeps this current: on every push to `main`
it works out the image ref from the runtime source, builds and pushes it if that
ref is not already in the registry, and sets `ARMNET_DEMO_EVAL_IMAGE` on the
Space. Because the tag is a hash of the build context, a commit that leaves the
eval runtime alone resolves to an image that already exists, so the build is
skipped and the Space is left running exactly what it was.

The workflow targets `armnet/armnet-eval` and needs the repository's `HF_TOKEN`
secret to have write access to that Space.

To do it by hand — to test a change before it lands on `main`, say:

```bash
cd armnet
# Builds runtime_lerobot_eval (which bakes in the molmoact2 plugin) and pushes it,
# or prints the existing ref if this source is already published.
uv run python examples/ci_publish_eval_image.py
```

Then set `ARMNET_DEMO_EVAL_IMAGE` on the Space to the ref it prints, or pass
`--space <owner>/<space>` to have it do that for you.

## Local dev

```bash
cd armnet/demo_space
ARMNET_DEMO_EVAL_IMAGE=<ref> python app.py
```