villekuosmanen commited on
Commit
f6e5166
Β·
verified Β·
1 Parent(s): 75f0ee4

Update Armnet organization references

Browse files
Files changed (1) hide show
  1. README.md +17 -7
README.md CHANGED
@@ -25,6 +25,9 @@ no auth setup, no Docker.** Pick a task, paste (or select) a policy repo id, pre
25
  - **Auto-scored** per-rollout success from the instrumented BusyBox β€” no human
26
  judges, and the scene resets itself between rollouts.
27
  - **Shared**: everyone who opens the Space follows the same eval in real time.
 
 
 
28
  - **Leaderboard**: every run is recorded and pooled by policy repo + revision,
29
  ranked by the 95% lower-bound success rate (so more rollouts only help a
30
  genuinely good policy β€” spamming evals can't game it).
@@ -39,8 +42,12 @@ environment when it routes a job.
39
 
40
  | Task | Slug | Dataset | Reference policy |
41
  | --- | --- | --- | --- |
42
- | Push the green button | `push_green_button` | [`villekuosmanen/busybox_push_green_button`](https://huggingface.co/datasets/villekuosmanen/busybox_push_green_button) | [`…_push_green_button_diffusion`](https://huggingface.co/villekuosmanen/busybox_push_green_button_diffusion) |
43
- | Flip left switch off | `busybox_left_toggle_down` | [`villekuosmanen/busybox_flip_left_switch_off`](https://huggingface.co/datasets/villekuosmanen/busybox_flip_left_switch_off) | [`…_flip_left_switch_off_diffusion`](https://huggingface.co/villekuosmanen/busybox_flip_left_switch_off_diffusion) |
 
 
 
 
44
 
45
  Adding a third is one entry in `TASKS` in `config.py`. The only hard requirement
46
  is that it **resets itself** β€” a task needing a human to tidy up between rollouts
@@ -50,7 +57,7 @@ can't be run unattended by the public.
50
 
51
  Gradio has no built-in database and a Space's disk is ephemeral, so the
52
  leaderboard is stored in a Hugging Face **dataset** repo (`ARMNET_DEMO_LEADERBOARD_REPO`,
53
- default `villekuosmanen/armnet-demo-leaderboard`): one JSONL line per completed
54
  run, reloaded on restart. Reads are public; **writing new runs needs `HF_TOKEN`**
55
  (the same token used to pull gated policies).
56
 
@@ -71,7 +78,9 @@ submission endpoint is planned so external users can submit without that token.
71
  1. The Space submits a job to the armnet orchestrator (baked-in demo API key)
72
  naming only the embodiment (`lerobot/so-101`) and the selected task's slug,
73
  passing the policy repo id as a job arg (`use_rerun=True` turns on the video
74
- feed). No cell is named in the submission β€” the task is what routes it.
 
 
75
  2. The orchestrator resolves that task to its **environment** (`busybox`) and
76
  routes the job to a cell assigned that environment. Cells are assigned an
77
  environment in the FMS, not a task, and serve every task within it β€” so if no
@@ -95,7 +104,8 @@ Set these in **Settings β†’ Variables and secrets**:
95
  | `ARMNET_API_KEY` | **yes** | Orchestrator API key for the demo (a low-privilege throwaway key). Provided as a secret β€” nothing is baked into the repo. |
96
  | `HF_TOKEN` | recommended | The Space stores it once as a named orchestrator secret (`ARMNET_DEMO_HF_SECRET_NAME`, default `hf_token`) under the demo user and the eval references it as `$HF_TOKEN` to pull a gated policy; it also authenticates leaderboard writes. Without it the leaderboard is read-only and gated policies won't load. |
97
  | `ARMNET_DEMO_N_EPISODES` | no | Rollouts per run (default 20). |
98
- | `ARMNET_DEMO_LEADERBOARD_REPO` | no | HF dataset repo backing the leaderboard (default `villekuosmanen/armnet-demo-leaderboard`). |
 
99
  | `ARMNET_DEMO_CELL_LABEL` | no | Name shown next to the availability light (default `Cell 1`). |
100
  | `ARMNET_DEMO_CELL_STATUS_POLL_S` | no | How often the shared availability lights refresh (default 5s). One process-wide poll per task, not per viewer. |
101
  | `DEBUG_ARMNET_ORCHESTRATOR_URI` | no | Override the orchestrator URL (default: the one compiled into `armnet-client`). |
@@ -136,8 +146,8 @@ Space. Because the tag is a hash of the build context, a commit that leaves the
136
  eval runtime alone resolves to an image that already exists, so the build is
137
  skipped and the Space is left running exactly what it was.
138
 
139
- It needs two repo settings: an `ARMNET_DEMO_SPACE_ID` variable holding
140
- `<owner>/<space>`, and an `HF_TOKEN` secret with write access to it.
141
 
142
  To do it by hand β€” to test a change before it lands on `main`, say:
143
 
 
25
  - **Auto-scored** per-rollout success from the instrumented BusyBox β€” no human
26
  judges, and the scene resets itself between rollouts.
27
  - **Shared**: everyone who opens the Space follows the same eval in real time.
28
+ - **Queue-aware**: while the cell is busy, submitters can queue another
29
+ 20-rollout eval. The public queue is capped at 10 jobs and shows policy repo
30
+ IDs only for Armnet eval images; unrelated jobs appear as `other job`.
31
  - **Leaderboard**: every run is recorded and pooled by policy repo + revision,
32
  ranked by the 95% lower-bound success rate (so more rollouts only help a
33
  genuinely good policy β€” spamming evals can't game it).
 
42
 
43
  | Task | Slug | Dataset | Reference policy |
44
  | --- | --- | --- | --- |
45
+ | Push the green button | `push_green_button` | [`armnet/busybox_push_green_button`](https://huggingface.co/datasets/armnet/busybox_push_green_button) | [`…_push_green_button_diffusion`](https://huggingface.co/armnet/busybox_push_green_button_diffusion) |
46
+ | Flip left switch off | `busybox_left_toggle_down` | [`armnet/busybox_flip_left_switch_off`](https://huggingface.co/datasets/armnet/busybox_flip_left_switch_off) | [`…_flip_left_switch_off_diffusion`](https://huggingface.co/armnet/busybox_flip_left_switch_off_diffusion) |
47
+
48
+ Either task-specific dataset can be replaced with
49
+ [`armnet/busybox_multitask`](https://huggingface.co/datasets/armnet/busybox_multitask)
50
+ for language-conditioned multi-task training.
51
 
52
  Adding a third is one entry in `TASKS` in `config.py`. The only hard requirement
53
  is that it **resets itself** β€” a task needing a human to tidy up between rollouts
 
57
 
58
  Gradio has no built-in database and a Space's disk is ephemeral, so the
59
  leaderboard is stored in a Hugging Face **dataset** repo (`ARMNET_DEMO_LEADERBOARD_REPO`,
60
+ default `armnet/armnet-demo-leaderboard`): one JSONL line per completed
61
  run, reloaded on restart. Reads are public; **writing new runs needs `HF_TOKEN`**
62
  (the same token used to pull gated policies).
63
 
 
78
  1. The Space submits a job to the armnet orchestrator (baked-in demo API key)
79
  naming only the embodiment (`lerobot/so-101`) and the selected task's slug,
80
  passing the policy repo id as a job arg (`use_rerun=True` turns on the video
81
+ feed). No cell is named in the submission β€” the task is what routes it. If
82
+ the cell is occupied, the job waits in the shared queue and the Space adopts
83
+ it for live display when the scheduler dispatches it.
84
  2. The orchestrator resolves that task to its **environment** (`busybox`) and
85
  routes the job to a cell assigned that environment. Cells are assigned an
86
  environment in the FMS, not a task, and serve every task within it β€” so if no
 
104
  | `ARMNET_API_KEY` | **yes** | Orchestrator API key for the demo (a low-privilege throwaway key). Provided as a secret β€” nothing is baked into the repo. |
105
  | `HF_TOKEN` | recommended | The Space stores it once as a named orchestrator secret (`ARMNET_DEMO_HF_SECRET_NAME`, default `hf_token`) under the demo user and the eval references it as `$HF_TOKEN` to pull a gated policy; it also authenticates leaderboard writes. Without it the leaderboard is read-only and gated policies won't load. |
106
  | `ARMNET_DEMO_N_EPISODES` | no | Rollouts per run (default 20). |
107
+ | `ARMNET_DEMO_MAX_QUEUED_JOBS` | no | Fleet-wide queued-job cap shown and enforced by the Space (default 10). |
108
+ | `ARMNET_DEMO_LEADERBOARD_REPO` | no | HF dataset repo backing the leaderboard (default `armnet/armnet-demo-leaderboard`). |
109
  | `ARMNET_DEMO_CELL_LABEL` | no | Name shown next to the availability light (default `Cell 1`). |
110
  | `ARMNET_DEMO_CELL_STATUS_POLL_S` | no | How often the shared availability lights refresh (default 5s). One process-wide poll per task, not per viewer. |
111
  | `DEBUG_ARMNET_ORCHESTRATOR_URI` | no | Override the orchestrator URL (default: the one compiled into `armnet-client`). |
 
146
  eval runtime alone resolves to an image that already exists, so the build is
147
  skipped and the Space is left running exactly what it was.
148
 
149
+ The workflow targets `armnet/armnet-eval` and needs the repository's `HF_TOKEN`
150
+ secret to have write access to that Space.
151
 
152
  To do it by hand β€” to test a change before it lands on `main`, say:
153