Spaces:
Running
Running
Update Armnet organization references
Browse files
README.md
CHANGED
|
@@ -25,6 +25,9 @@ no auth setup, no Docker.** Pick a task, paste (or select) a policy repo id, pre
|
|
| 25 |
- **Auto-scored** per-rollout success from the instrumented BusyBox β no human
|
| 26 |
judges, and the scene resets itself between rollouts.
|
| 27 |
- **Shared**: everyone who opens the Space follows the same eval in real time.
|
|
|
|
|
|
|
|
|
|
| 28 |
- **Leaderboard**: every run is recorded and pooled by policy repo + revision,
|
| 29 |
ranked by the 95% lower-bound success rate (so more rollouts only help a
|
| 30 |
genuinely good policy β spamming evals can't game it).
|
|
@@ -39,8 +42,12 @@ environment when it routes a job.
|
|
| 39 |
|
| 40 |
| Task | Slug | Dataset | Reference policy |
|
| 41 |
| --- | --- | --- | --- |
|
| 42 |
-
| Push the green button | `push_green_button` | [`
|
| 43 |
-
| Flip left switch off | `busybox_left_toggle_down` | [`
|
|
|
|
|
|
|
|
|
|
|
|
|
| 44 |
|
| 45 |
Adding a third is one entry in `TASKS` in `config.py`. The only hard requirement
|
| 46 |
is that it **resets itself** β a task needing a human to tidy up between rollouts
|
|
@@ -50,7 +57,7 @@ can't be run unattended by the public.
|
|
| 50 |
|
| 51 |
Gradio has no built-in database and a Space's disk is ephemeral, so the
|
| 52 |
leaderboard is stored in a Hugging Face **dataset** repo (`ARMNET_DEMO_LEADERBOARD_REPO`,
|
| 53 |
-
default `
|
| 54 |
run, reloaded on restart. Reads are public; **writing new runs needs `HF_TOKEN`**
|
| 55 |
(the same token used to pull gated policies).
|
| 56 |
|
|
@@ -71,7 +78,9 @@ submission endpoint is planned so external users can submit without that token.
|
|
| 71 |
1. The Space submits a job to the armnet orchestrator (baked-in demo API key)
|
| 72 |
naming only the embodiment (`lerobot/so-101`) and the selected task's slug,
|
| 73 |
passing the policy repo id as a job arg (`use_rerun=True` turns on the video
|
| 74 |
-
feed). No cell is named in the submission β the task is what routes it.
|
|
|
|
|
|
|
| 75 |
2. The orchestrator resolves that task to its **environment** (`busybox`) and
|
| 76 |
routes the job to a cell assigned that environment. Cells are assigned an
|
| 77 |
environment in the FMS, not a task, and serve every task within it β so if no
|
|
@@ -95,7 +104,8 @@ Set these in **Settings β Variables and secrets**:
|
|
| 95 |
| `ARMNET_API_KEY` | **yes** | Orchestrator API key for the demo (a low-privilege throwaway key). Provided as a secret β nothing is baked into the repo. |
|
| 96 |
| `HF_TOKEN` | recommended | The Space stores it once as a named orchestrator secret (`ARMNET_DEMO_HF_SECRET_NAME`, default `hf_token`) under the demo user and the eval references it as `$HF_TOKEN` to pull a gated policy; it also authenticates leaderboard writes. Without it the leaderboard is read-only and gated policies won't load. |
|
| 97 |
| `ARMNET_DEMO_N_EPISODES` | no | Rollouts per run (default 20). |
|
| 98 |
-
| `
|
|
|
|
| 99 |
| `ARMNET_DEMO_CELL_LABEL` | no | Name shown next to the availability light (default `Cell 1`). |
|
| 100 |
| `ARMNET_DEMO_CELL_STATUS_POLL_S` | no | How often the shared availability lights refresh (default 5s). One process-wide poll per task, not per viewer. |
|
| 101 |
| `DEBUG_ARMNET_ORCHESTRATOR_URI` | no | Override the orchestrator URL (default: the one compiled into `armnet-client`). |
|
|
@@ -136,8 +146,8 @@ Space. Because the tag is a hash of the build context, a commit that leaves the
|
|
| 136 |
eval runtime alone resolves to an image that already exists, so the build is
|
| 137 |
skipped and the Space is left running exactly what it was.
|
| 138 |
|
| 139 |
-
|
| 140 |
-
|
| 141 |
|
| 142 |
To do it by hand β to test a change before it lands on `main`, say:
|
| 143 |
|
|
|
|
| 25 |
- **Auto-scored** per-rollout success from the instrumented BusyBox β no human
|
| 26 |
judges, and the scene resets itself between rollouts.
|
| 27 |
- **Shared**: everyone who opens the Space follows the same eval in real time.
|
| 28 |
+
- **Queue-aware**: while the cell is busy, submitters can queue another
|
| 29 |
+
20-rollout eval. The public queue is capped at 10 jobs and shows policy repo
|
| 30 |
+
IDs only for Armnet eval images; unrelated jobs appear as `other job`.
|
| 31 |
- **Leaderboard**: every run is recorded and pooled by policy repo + revision,
|
| 32 |
ranked by the 95% lower-bound success rate (so more rollouts only help a
|
| 33 |
genuinely good policy β spamming evals can't game it).
|
|
|
|
| 42 |
|
| 43 |
| Task | Slug | Dataset | Reference policy |
|
| 44 |
| --- | --- | --- | --- |
|
| 45 |
+
| Push the green button | `push_green_button` | [`armnet/busybox_push_green_button`](https://huggingface.co/datasets/armnet/busybox_push_green_button) | [`β¦_push_green_button_diffusion`](https://huggingface.co/armnet/busybox_push_green_button_diffusion) |
|
| 46 |
+
| Flip left switch off | `busybox_left_toggle_down` | [`armnet/busybox_flip_left_switch_off`](https://huggingface.co/datasets/armnet/busybox_flip_left_switch_off) | [`β¦_flip_left_switch_off_diffusion`](https://huggingface.co/armnet/busybox_flip_left_switch_off_diffusion) |
|
| 47 |
+
|
| 48 |
+
Either task-specific dataset can be replaced with
|
| 49 |
+
[`armnet/busybox_multitask`](https://huggingface.co/datasets/armnet/busybox_multitask)
|
| 50 |
+
for language-conditioned multi-task training.
|
| 51 |
|
| 52 |
Adding a third is one entry in `TASKS` in `config.py`. The only hard requirement
|
| 53 |
is that it **resets itself** β a task needing a human to tidy up between rollouts
|
|
|
|
| 57 |
|
| 58 |
Gradio has no built-in database and a Space's disk is ephemeral, so the
|
| 59 |
leaderboard is stored in a Hugging Face **dataset** repo (`ARMNET_DEMO_LEADERBOARD_REPO`,
|
| 60 |
+
default `armnet/armnet-demo-leaderboard`): one JSONL line per completed
|
| 61 |
run, reloaded on restart. Reads are public; **writing new runs needs `HF_TOKEN`**
|
| 62 |
(the same token used to pull gated policies).
|
| 63 |
|
|
|
|
| 78 |
1. The Space submits a job to the armnet orchestrator (baked-in demo API key)
|
| 79 |
naming only the embodiment (`lerobot/so-101`) and the selected task's slug,
|
| 80 |
passing the policy repo id as a job arg (`use_rerun=True` turns on the video
|
| 81 |
+
feed). No cell is named in the submission β the task is what routes it. If
|
| 82 |
+
the cell is occupied, the job waits in the shared queue and the Space adopts
|
| 83 |
+
it for live display when the scheduler dispatches it.
|
| 84 |
2. The orchestrator resolves that task to its **environment** (`busybox`) and
|
| 85 |
routes the job to a cell assigned that environment. Cells are assigned an
|
| 86 |
environment in the FMS, not a task, and serve every task within it β so if no
|
|
|
|
| 104 |
| `ARMNET_API_KEY` | **yes** | Orchestrator API key for the demo (a low-privilege throwaway key). Provided as a secret β nothing is baked into the repo. |
|
| 105 |
| `HF_TOKEN` | recommended | The Space stores it once as a named orchestrator secret (`ARMNET_DEMO_HF_SECRET_NAME`, default `hf_token`) under the demo user and the eval references it as `$HF_TOKEN` to pull a gated policy; it also authenticates leaderboard writes. Without it the leaderboard is read-only and gated policies won't load. |
|
| 106 |
| `ARMNET_DEMO_N_EPISODES` | no | Rollouts per run (default 20). |
|
| 107 |
+
| `ARMNET_DEMO_MAX_QUEUED_JOBS` | no | Fleet-wide queued-job cap shown and enforced by the Space (default 10). |
|
| 108 |
+
| `ARMNET_DEMO_LEADERBOARD_REPO` | no | HF dataset repo backing the leaderboard (default `armnet/armnet-demo-leaderboard`). |
|
| 109 |
| `ARMNET_DEMO_CELL_LABEL` | no | Name shown next to the availability light (default `Cell 1`). |
|
| 110 |
| `ARMNET_DEMO_CELL_STATUS_POLL_S` | no | How often the shared availability lights refresh (default 5s). One process-wide poll per task, not per viewer. |
|
| 111 |
| `DEBUG_ARMNET_ORCHESTRATOR_URI` | no | Override the orchestrator URL (default: the one compiled into `armnet-client`). |
|
|
|
|
| 146 |
eval runtime alone resolves to an image that already exists, so the build is
|
| 147 |
skipped and the Space is left running exactly what it was.
|
| 148 |
|
| 149 |
+
The workflow targets `armnet/armnet-eval` and needs the repository's `HF_TOKEN`
|
| 150 |
+
secret to have write access to that Space.
|
| 151 |
|
| 152 |
To do it by hand β to test a change before it lands on `main`, say:
|
| 153 |
|