README.md

metadata

title: armnet eval
emoji: 🦾
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 6.20.0
python_version: '3.12'
app_file: app.py
pinned: false
short_description: Evaluate a robot policy on real hardware, no setup.

armnet eval — evaluate a policy on real robots

armnet.dev

A deliberately constrained public demo of the armnet infrastructure. No install, no auth setup, no Docker. Pick a task, paste (or select) a policy repo id, press Run, and watch 20 rollouts on a real SO-101 cell:

• Live video of the rollout, streamed from the cell.
• Cell availability light: a blinking dot — green when at least one BusyBox cell is free, yellow when every cell is busy, red when none are online — with a count such as 1 cell available.
• Auto-scored per-rollout success from the instrumented BusyBox — no human judges, and the scene resets itself between rollouts.
• Shared: everyone who opens the Space follows the same fleet in real time. If more than one cell is running an eval, the Live tab names each policy and lets you pick which feed to watch.
• Queue-aware: while every cell is busy, submitters can queue another 20-rollout eval. A client-side abuse guard limits this Space to 10 waiting evals; it does not cap the fleet-wide orchestrator queue. The Space only sees jobs owned by its API user and renders unrelated ones as other job. The Run button remains available while a Space eval is live, but the same policy repo cannot be running and/or queued more than once. Once its run finishes, it can be submitted again.
• Leaderboard: every run is recorded and pooled by policy repo + revision, ranked by the 95% lower-bound success rate (so more rollouts only help a genuinely good policy — spamming evals can't game it).

Tasks

A selector above the tabs picks the task, and it scopes everything below it: the dataset the instructions point at, the policy the Live tab defaults to, and the board the Leaderboard shows. Both run on BusyBox cells assigned the busybox environment — the orchestrator resolves the task slug to that environment when it routes a job, and any free cell serving it can pick the work up.

Task	Slug	Dataset	Reference policy
Push the green button	push_green_button	armnet/busybox_push_green_button	…_push_green_button_diffusion
Flip left switch off	busybox_left_toggle_down	armnet/busybox_flip_left_switch_off	…_flip_left_switch_off_diffusion

Either task-specific dataset can be replaced with armnet/busybox_multitask for language-conditioned multi-task training.

Adding a third is one entry in TASKS in config.py. The only hard requirement is that it resets itself — a task needing a human to tidy up between rollouts can't be run unattended by the public.

Persistence

Gradio has no built-in database and a Space's disk is ephemeral, so the leaderboard is stored in a Hugging Face dataset repo (ARMNET_DEMO_LEADERBOARD_REPO, default armnet/armnet-demo-leaderboard): one JSONL line per completed run, reloaded on restart. Reads are public; writing new runs needs HF_TOKEN (the same token used to pull gated policies).

Each run records source, cell_id, embodiment and task alongside the success counts. Each task gets its own board: rows are filtered to this Space's embodiment (lerobot/so-101) and to the selected task exactly, since a policy that presses buttons well says nothing about one that flips a switch. Runs from every BusyBox cell in the demo fleet are pooled together. The board shows a Source column: a verified checkmark ("result verified by Armnet") when every pooled rollout for a policy came from this demo, otherwise script/mixed. Armnet eval scripts can also write to the same dataset (source="script") via ctx.submit_results_to_leaderboard(); that path currently relies on the eval container's HF token having write access to the dataset — a server-side submission endpoint is planned so external users can submit without that token.

How it works

1. The Space submits a job to the armnet orchestrator (baked-in demo API key) naming only the embodiment (lerobot/so-101) and the selected task's slug, passing the policy repo id as a job arg (use_rerun=True turns on the video feed). No cell is named in the submission — the task is what routes it. If every matching cell is occupied, the job waits in the shared queue and the Space adopts it for live display when the scheduler dispatches it.
2. The orchestrator resolves that task to its environment (busybox) and routes the job to a cell assigned that environment. Cells are assigned an environment in the FMS, not a task, and serve every task within it — so if no cell is set to busybox the availability light goes red and nothing can be submitted.
3. The chosen cell pulls the pre-built eval image, loads the policy from the Hub, and runs the rollouts. It streams observation frames (Rerun packets) and per-rollout results back over the orchestrator websockets. When a queued Space eval starts, the Space attaches a live follower to that job even if another cell is already running.
4. The Space decodes those into per-cell live video + results, held in process-global slots so all viewers share the same feeds. Each browser receives the newest frame of the cell it is watching through one cancellable async Gradio stream, capped at 5 FPS; slow viewers skip stale frames instead of accumulating video latency. Frames are resized once process-wide rather than once per viewer.

Space configuration

Set these in Settings → Variables and secrets:

Variable	Required	Purpose
ARMNET_DEMO_EVAL_IMAGE	no	Registry ref of the pre-built lerobot-eval runtime image (below). A current ref is baked into config.py; set this only to override it without editing code. A Space can't build Docker, so this is built + pushed once by an operator.
ARMNET_API_KEY	yes	Orchestrator API key for the demo (a low-privilege throwaway key). Provided as a secret — nothing is baked into the repo.
HF_TOKEN	recommended	The Space stores it once as a named orchestrator secret (ARMNET_DEMO_HF_SECRET_NAME, default hf_token) under the demo user and the eval references it as $HF_TOKEN to pull a gated policy; it also authenticates leaderboard writes. Without it the leaderboard is read-only and gated policies won't load.
ARMNET_DEMO_N_EPISODES	no	Rollouts per run (default 20).
ARMNET_DEMO_MAX_QUEUED_JOBS	no	Client-side cap on this Space's queued eval jobs (default 10); it does not limit the orchestrator queue.
ARMNET_DEMO_LEADERBOARD_REPO	no	HF dataset repo backing the leaderboard (default armnet/armnet-demo-leaderboard).
ARMNET_DEMO_DATASET_NAMESPACE	no	Owner of the recorded eval dataset (default armnet). Without this the eval uses whoever owns HF_TOKEN.
ARMNET_DEMO_CELL_STATUS_POLL_S	no	How often the shared availability lights refresh (default 5s). One process-wide poll per task, not per viewer.
DEBUG_ARMNET_ORCHESTRATOR_URI	no	Override the orchestrator URL (default: the one compiled into armnet-client).

No keys are baked into the repo — both ARMNET_API_KEY and HF_TOKEN are read from the Space's secrets. The demo API key is a low-privilege throwaway key; rotate/invalidate it freely.

Deploying the Space itself

The same Armnet Demo Eval Image workflow uploads this directory (minus tests/) to the Space on every push to main, so a fix to the app or its requirements goes live without anyone pushing by hand. It compares the checkout against what the Space actually holds and commits only the files that differ — a commit restarts the Space, so an unchanged Space is left alone. Because that check is on content rather than on which paths a commit touched, a deploy that failed is simply retried by the next run.

To preview or do it by hand:

cd armnet
uv run python examples/ci_deploy_demo_space.py --dry-run   # what would change
uv run python examples/ci_deploy_demo_space.py             # deploy it

Files it finds on the Space but not in the checkout are reported, not deleted.

Building the eval image

The policy is not baked into the image — it is loaded at runtime from the repo id — so one image serves every policy, and building it needs no robot.

The Armnet Demo Eval Image workflow keeps this current: on every push to main it works out the image ref from the runtime source, builds and pushes it if that ref is not already in the registry, and sets ARMNET_DEMO_EVAL_IMAGE on the Space. Because the tag is a hash of the build context, a commit that leaves the eval runtime alone resolves to an image that already exists, so the build is skipped and the Space is left running exactly what it was.

The workflow targets armnet/armnet-eval and needs the repository's HF_TOKEN secret to have write access to that Space.

To do it by hand — to test a change before it lands on main, say:

cd armnet
# Builds runtime_lerobot_eval (which bakes in the molmoact2 plugin) and pushes it,
# or prints the existing ref if this source is already published.
uv run python examples/ci_publish_eval_image.py

Then set ARMNET_DEMO_EVAL_IMAGE on the Space to the ref it prints, or pass --space <owner>/<space> to have it do that for you.

Local dev

cd armnet/demo_space
ARMNET_DEMO_EVAL_IMAGE=<ref> python app.py