Spaces:
Running
Running
| title: armnet eval | |
| emoji: 🦾 | |
| colorFrom: indigo | |
| colorTo: purple | |
| sdk: gradio | |
| sdk_version: 6.20.0 | |
| python_version: "3.12" | |
| app_file: app.py | |
| pinned: false | |
| short_description: Evaluate a robot policy on real hardware, no setup. | |
| # armnet eval — evaluate a policy on real robots | |
| [armnet.dev](https://armnet.dev/) | |
| A deliberately constrained public demo of the armnet infrastructure. **No install, | |
| no auth setup, no Docker.** Pick a task, paste (or select) a policy repo id, press | |
| **Run**, and watch **20 rollouts** on a real SO-101 cell: | |
| - **Live video** of the rollout, streamed from the cell. | |
| - **Cell availability light**: a blinking dot — green when at least one BusyBox | |
| cell is free, yellow when every cell is busy, red when none are online — with | |
| a count such as `1 cell available`. | |
| - **Auto-scored** per-rollout success from the instrumented BusyBox — no human | |
| judges, and the scene resets itself between rollouts. | |
| - **Shared**: everyone who opens the Space follows the same fleet in real time. | |
| If more than one cell is running an eval, the Live tab names each policy and | |
| lets you pick which feed to watch. | |
| - **Queue-aware**: while every cell is busy, submitters can queue another | |
| 20-rollout eval. A client-side abuse guard limits this Space to 10 waiting | |
| evals; it does not cap the fleet-wide orchestrator queue. The Space only sees | |
| jobs owned by its API user and renders unrelated ones as `other job`. The Run | |
| button remains available while a Space eval is live, but the same policy repo | |
| cannot be running and/or queued more than once. Once its run finishes, it can | |
| be submitted again. | |
| - **Leaderboard**: every run is recorded and pooled by policy repo + revision, | |
| ranked by the 95% lower-bound success rate (so more rollouts only help a | |
| genuinely good policy — spamming evals can't game it). | |
| ## Tasks | |
| A selector above the tabs picks the task, and it scopes everything below it: the | |
| dataset the instructions point at, the policy the Live tab defaults to, and the | |
| board the Leaderboard shows. Both run on BusyBox cells assigned the `busybox` | |
| environment — the orchestrator resolves the task slug to that environment when | |
| it routes a job, and any free cell serving it can pick the work up. | |
| | Task | Slug | Dataset | Reference policy | | |
| | --- | --- | --- | --- | | |
| | Push the green button | `push_green_button` | [`armnet/busybox_push_green_button`](https://huggingface.co/datasets/armnet/busybox_push_green_button) | [`…_push_green_button_diffusion`](https://huggingface.co/armnet/busybox_push_green_button_diffusion) | | |
| | Flip left switch off | `busybox_left_toggle_down` | [`armnet/busybox_flip_left_switch_off`](https://huggingface.co/datasets/armnet/busybox_flip_left_switch_off) | [`…_flip_left_switch_off_diffusion`](https://huggingface.co/armnet/busybox_flip_left_switch_off_diffusion) | | |
| Either task-specific dataset can be replaced with | |
| [`armnet/busybox_multitask`](https://huggingface.co/datasets/armnet/busybox_multitask) | |
| for language-conditioned multi-task training. | |
| Adding a third is one entry in `TASKS` in `config.py`. The only hard requirement | |
| is that it **resets itself** — a task needing a human to tidy up between rollouts | |
| can't be run unattended by the public. | |
| ## Persistence | |
| Gradio has no built-in database and a Space's disk is ephemeral, so the | |
| leaderboard is stored in a Hugging Face **dataset** repo (`ARMNET_DEMO_LEADERBOARD_REPO`, | |
| default `armnet/armnet-demo-leaderboard`): one JSONL line per completed | |
| run, reloaded on restart. Reads are public; **writing new runs needs `HF_TOKEN`** | |
| (the same token used to pull gated policies). | |
| Each run records `source`, `cell_id`, `embodiment` and `task` alongside the | |
| success counts. Each task gets **its own board**: rows are filtered to this | |
| Space's embodiment (`lerobot/so-101`) and to the selected task exactly, since a | |
| policy that presses buttons well says nothing about one that flips a switch. | |
| Runs from every BusyBox cell in the demo fleet are pooled together. The board | |
| shows a **Source** column: a verified checkmark ("result verified by Armnet") | |
| when every pooled rollout for a policy came from this demo, otherwise | |
| `script`/`mixed`. Armnet eval **scripts** can also write to the same dataset | |
| (`source="script"`) via `ctx.submit_results_to_leaderboard()`; that path | |
| currently relies on the eval container's HF token having write access to the | |
| dataset — a server-side submission endpoint is planned so external users can | |
| submit without that token. | |
| ## How it works | |
| 1. The Space submits a job to the armnet orchestrator (baked-in demo API key) | |
| naming only the embodiment (`lerobot/so-101`) and the selected task's slug, | |
| passing the policy repo id as a job arg (`use_rerun=True` turns on the video | |
| feed). No cell is named in the submission — the task is what routes it. If | |
| every matching cell is occupied, the job waits in the shared queue and the | |
| Space adopts it for live display when the scheduler dispatches it. | |
| 2. The orchestrator resolves that task to its **environment** (`busybox`) and | |
| routes the job to a cell assigned that environment. Cells are assigned an | |
| environment in the FMS, not a task, and serve every task within it — so if no | |
| cell is set to `busybox` the availability light goes red and nothing can be | |
| submitted. | |
| 3. The chosen cell pulls the pre-built eval image, loads the policy from the Hub, | |
| and runs the rollouts. It streams observation frames (Rerun packets) and | |
| per-rollout results back over the orchestrator websockets. | |
| When a queued Space eval starts, the Space attaches a live follower to that | |
| job even if another cell is already running. | |
| 4. The Space decodes those into per-cell live video + results, held in | |
| process-global slots so all viewers share the same feeds. Each browser | |
| receives the newest frame of the cell it is watching through one cancellable | |
| async Gradio stream, capped at 5 FPS; slow viewers skip stale frames instead | |
| of accumulating video latency. Frames are resized once process-wide rather | |
| than once per viewer. | |
| ## Space configuration | |
| Set these in **Settings → Variables and secrets**: | |
| | Variable | Required | Purpose | | |
| | --- | --- | --- | | |
| | `ARMNET_DEMO_EVAL_IMAGE` | no | Registry ref of the pre-built lerobot-eval runtime image (below). A current ref is baked into `config.py`; set this only to override it without editing code. A Space can't build Docker, so this is built + pushed once by an operator. | | |
| | `ARMNET_API_KEY` | **yes** | Orchestrator API key for the demo (a low-privilege throwaway key). Provided as a secret — nothing is baked into the repo. | | |
| | `HF_TOKEN` | recommended | The Space stores it once as a named orchestrator secret (`ARMNET_DEMO_HF_SECRET_NAME`, default `hf_token`) under the demo user and the eval references it as `$HF_TOKEN` to pull a gated policy; it also authenticates leaderboard writes. Without it the leaderboard is read-only and gated policies won't load. | | |
| | `ARMNET_DEMO_N_EPISODES` | no | Rollouts per run (default 20). | | |
| | `ARMNET_DEMO_MAX_QUEUED_JOBS` | no | Client-side cap on this Space's queued eval jobs (default 10); it does not limit the orchestrator queue. | | |
| | `ARMNET_DEMO_LEADERBOARD_REPO` | no | HF dataset repo backing the leaderboard (default `armnet/armnet-demo-leaderboard`). | | |
| | `ARMNET_DEMO_DATASET_NAMESPACE` | no | Owner of the recorded eval dataset (default `armnet`). Without this the eval uses whoever owns `HF_TOKEN`. | | |
| | `ARMNET_DEMO_CELL_STATUS_POLL_S` | no | How often the shared availability lights refresh (default 5s). One process-wide poll per task, not per viewer. | | |
| | `DEBUG_ARMNET_ORCHESTRATOR_URI` | no | Override the orchestrator URL (default: the one compiled into `armnet-client`). | | |
| > No keys are baked into the repo — both `ARMNET_API_KEY` and `HF_TOKEN` are | |
| > read from the Space's secrets. The demo API key is a low-privilege throwaway | |
| > key; rotate/invalidate it freely. | |
| ### Deploying the Space itself | |
| The same `Armnet Demo Eval Image` workflow uploads this directory (minus `tests/`) | |
| to the Space on every push to `main`, so a fix to the app or its requirements goes | |
| live without anyone pushing by hand. It compares the checkout against what the | |
| Space actually holds and commits only the files that differ — a commit restarts | |
| the Space, so an unchanged Space is left alone. Because that check is on content | |
| rather than on which paths a commit touched, a deploy that failed is simply | |
| retried by the next run. | |
| To preview or do it by hand: | |
| ```bash | |
| cd armnet | |
| uv run python examples/ci_deploy_demo_space.py --dry-run # what would change | |
| uv run python examples/ci_deploy_demo_space.py # deploy it | |
| ``` | |
| Files it finds on the Space but not in the checkout are reported, not deleted. | |
| ### Building the eval image | |
| The policy is **not** baked into the image — it is loaded at runtime from the | |
| repo id — so one image serves every policy, and building it needs no robot. | |
| The `Armnet Demo Eval Image` workflow keeps this current: on every push to `main` | |
| it works out the image ref from the runtime source, builds and pushes it if that | |
| ref is not already in the registry, and sets `ARMNET_DEMO_EVAL_IMAGE` on the | |
| Space. Because the tag is a hash of the build context, a commit that leaves the | |
| eval runtime alone resolves to an image that already exists, so the build is | |
| skipped and the Space is left running exactly what it was. | |
| The workflow targets `armnet/armnet-eval` and needs the repository's `HF_TOKEN` | |
| secret to have write access to that Space. | |
| To do it by hand — to test a change before it lands on `main`, say: | |
| ```bash | |
| cd armnet | |
| # Builds runtime_lerobot_eval (which bakes in the molmoact2 plugin) and pushes it, | |
| # or prints the existing ref if this source is already published. | |
| uv run python examples/ci_publish_eval_image.py | |
| ``` | |
| Then set `ARMNET_DEMO_EVAL_IMAGE` on the Space to the ref it prints, or pass | |
| `--space <owner>/<space>` to have it do that for you. | |
| ## Local dev | |
| ```bash | |
| cd armnet/demo_space | |
| ARMNET_DEMO_EVAL_IMAGE=<ref> python app.py | |
| ``` | |