petkopetkov commited on
Commit
3f2be7f
·
verified ·
1 Parent(s): 21fed8e

Shorter model card

Browse files
Files changed (1) hide show
  1. README.md +20 -40
README.md CHANGED
@@ -1,67 +1,47 @@
1
  ---
2
  license: mit
3
  library_name: lerobot
4
- tags: [lerobot, envhub, robotics, simulation, benchmark, long-horizon, maniskill, sapien, widowx, ladder]
5
  ---
6
 
7
  # LADDER SimplerEnv-Long
8
 
9
- The SimplerEnv-Long suite of [LADDER](https://ladderrobotics.ai), a long-horizon robot manipulation benchmark, as a LeRobot [EnvHub](https://huggingface.co/docs/lerobot/envhub) environment. It has 54 WidowX tasks of 1 to 10 pick-and-place subgoals in the Bridge digital twin of SimplerEnv, on SAPIEN 3 / ManiSkill 3.
10
 
11
- The two other suites are separate repositories, because each simulator needs its own Python environment:
12
-
13
- | Suite | Repository | Simulator | Robot | Tasks | Episodes per task |
14
- |---|---|---|---|---|---|
15
- | RoboLab-Long | [`INSAIT-Institute/LADDER-RoboLab-Long`](https://huggingface.co/INSAIT-Institute/LADDER-RoboLab-Long) | Isaac Sim 6 / Isaac Lab 3 | Franka (DROID) | 44 | 8 |
16
- | SimplerEnv-Long | this one | SAPIEN 3 / ManiSkill 3 | WidowX | 54 | 24 |
17
- | RobotArena-∞-Long | [`INSAIT-Institute/LADDER-RobotArena-Long`](https://huggingface.co/INSAIT-Institute/LADDER-RobotArena-Long) | Genesis | WidowX | 47 | 8 |
18
 
19
  ## Evaluate a policy
20
 
21
- Serve the suite with the LeRobot simulator server and evaluate against it. The server runs in a Docker image built from this repository, so the simulator's dependencies stay out of your policy's environment. From a LeRobot checkout, with Docker and the NVIDIA Container Toolkit:
22
 
23
  ```bash
24
- pip install -e ".[dataset,evaluation,remote]" # plus your policy's extra, e.g. ".[smolvla]"
25
  lerobot-env-server --envhub=INSAIT-Institute/LADDER-SimplerEnv-Long@v1.0.0 --docker=true
26
  lerobot-eval --policy.path=<your WidowX checkpoint> \
27
  --env.type=envhub --env.hub_path=INSAIT-Institute/LADDER-SimplerEnv-Long@v1.0.0 \
28
- --env.server.endpoint=tcp/127.0.0.1:7448 \
29
- --env.task=all2_plate --eval.batch_size=24 --eval.n_episodes=24 --eval.use_async_envs=false
30
  ```
31
 
32
- - `--env.task` takes one task, a comma-separated list (run one after another: keep `--env.max_parallel_tasks=1`), or nothing for all 54. [TASKS.md](https://github.com/insait-institute/ladder/blob/main/TASKS.md) lists them.
33
- - Each task is one batched simulator world: keep `--eval.batch_size` at 24 and `--eval.use_async_envs=false`. Episode `i` is LADDER's object layout `i`, so 24 episodes are the whole task; more episodes repeat the same layouts.
34
- - `@v1.0.0` pins this repository to its release for LADDER v1.0.0 (or pin a commit): LeRobot runs its `env.py`.
35
-
36
- Two other ways to run it, as fast as the one above:
37
 
38
- - Without Docker: in a separate Python 3.12 environment, `pip install "lerobot[remote]>=0.7.0" -r requirements.txt` and start `lerobot-env-server --envhub=INSAIT-Institute/LADDER-SimplerEnv-Long@v1.0.0` there.
39
- - In-process: `pip install "lerobot[dataset,evaluation]>=0.7.0" -r requirements.txt` next to your policy, then run `lerobot-eval` as above without `--env.server.endpoint` and with `--env.trust_remote_code=true`. Only when this repository's pinned versions suit your policy.
40
-
41
- ## Observations, actions and scores
42
 
43
  | | |
44
  |---|---|
45
- | Observation | `pixels.main`: the third-person camera, 480x640 RGB; `agent_pos`: the end-effector pose in the robot base frame, `[x, y, z, rx, ry, rz]` (metres, axis-angle) in the BridgeData v2 end-effector convention, and the gripper (1 open, 0 closed) |
46
- | Action | `[dx, dy, dz, drx, dry, drz, gripper]`. The translation is in the base frame and the axis-angle rotation in the end-effector frame, both applied to the commanded pose (not the observed one), as LADDER's Bridge policy is run; the gripper target is 1 open, 0 closed |
47
  | Control rate | 5 Hz |
48
- | Step budget | 100 steps per subgoal, the task's own budget |
49
- | Instruction | `task_description`, e.g. "put the carrot on the plate and then the spoon on the towel" |
50
- | Success | `info["is_success"]`: LADDER's task success (`pc_success` in LeRobot) |
51
- | Progress | `info["metrics"]["subgoal_progress"]` (and `subgoals_done`): LADDER's subgoal score at the end of the episode, each subgoal met in the final state and, for an object moved twice, in order; all of them once the task succeeded. `reward` is its change at each step, so `avg_sum_reward` in LeRobot is the mean final progress |
52
-
53
- Options, passed as `--env.make_env_kwargs='{...}'` (`--sim.kwargs='{make_env_kwargs: {...}}'` for the server): `front_camera_name` (default `main`). The camera renders at 480x640, as in LADDER.
54
-
55
- ## How it relates to LADDER
56
-
57
- `ladder/` is LADDER's own code at release [`v1.0.0`](https://github.com/insait-institute/ladder/releases/tag/v1.0.0), unchanged: the task definitions, scene layouts and success predicates. `env.py` builds the suite's environment with the settings of LADDER's own server config (`widowx_composition.yaml`). The simulation assets and SAPIEN's PhysX library are downloaded from that release on first use (about 160 MB, checked by sha256) into `~/.cache/ladder/envhub` (`LADDER_CACHE` moves it).
58
-
59
- The full suite is all 54 tasks with 24 episodes each. LADDER's own harness (`scripts/eval.sh`) stays the reference for leaderboard results; see [EVALUATION.md](https://github.com/insait-institute/ladder/blob/main/EVALUATION.md).
60
 
61
- ## Checked against LADDER
62
 
63
- - The reset frames match LADDER's recorded episodes: 36.6 dB PSNR on each of the 24 layouts, each closest to its own layout.
64
- - LADDER's Bridge policy, run through this repository's Docker image with LADDER's own policy server and client settings, on 4 tasks (`all1_towel`, `split2`, `ordered_plate_2`, `spoons2`; 96 episodes), succeeded in 60 and 64 of 96 episodes in two runs. LADDER's published episodes of the same tasks: 59 of 96.
65
 
66
  ## Citation
67
 
@@ -77,4 +57,4 @@ The full suite is all 54 tasks with 24 episodes each. LADDER's own harness (`scr
77
 
78
  ## License
79
 
80
- MIT, as LADDER. The downloaded assets keep their own licenses (see LADDER's [THIRD_PARTY_NOTICES.md](https://github.com/insait-institute/ladder/blob/main/THIRD_PARTY_NOTICES.md)); ManiSkill's WidowX model is fetched from [haosulab/ManiSkill-WidowX250S](https://github.com/haosulab/ManiSkill-WidowX250S).
 
1
  ---
2
  license: mit
3
  library_name: lerobot
4
+ tags: [lerobot, envhub, robotics, simulation, benchmark, ladder, widowx, maniskill, sapien]
5
  ---
6
 
7
  # LADDER SimplerEnv-Long
8
 
9
+ 54 WidowX tasks of 1 to 10 pick-and-place subgoals in the Bridge digital twin of SimplerEnv, on SAPIEN 3 / ManiSkill 3. It is one of the three suites of [LADDER](https://ladderrobotics.ai), a long-horizon manipulation benchmark, packaged as a LeRobot [EnvHub](https://huggingface.co/docs/lerobot/envhub) environment. Each simulator needs its own Python environment, so each suite is its own repository:
10
 
11
+ | Suite | Repository | Simulator | Robot | Tasks |
12
+ |---|---|---|---|---|
13
+ | RoboLab-Long | [`INSAIT-Institute/LADDER-RoboLab-Long`](https://huggingface.co/INSAIT-Institute/LADDER-RoboLab-Long) | Isaac Sim 6 | Franka (DROID) | 44 |
14
+ | SimplerEnv-Long | this one | SAPIEN 3 / ManiSkill 3 | WidowX | 54 |
15
+ | RobotArena-∞-Long | [`INSAIT-Institute/LADDER-RobotArena-Long`](https://huggingface.co/INSAIT-Institute/LADDER-RobotArena-Long) | Genesis | WidowX | 47 |
 
 
16
 
17
  ## Evaluate a policy
18
 
19
+ From a LeRobot checkout, with Docker and the NVIDIA Container Toolkit:
20
 
21
  ```bash
22
+ pip install -e ".[dataset,evaluation,remote]" # plus your policy's extra
23
  lerobot-env-server --envhub=INSAIT-Institute/LADDER-SimplerEnv-Long@v1.0.0 --docker=true
24
  lerobot-eval --policy.path=<your WidowX checkpoint> \
25
  --env.type=envhub --env.hub_path=INSAIT-Institute/LADDER-SimplerEnv-Long@v1.0.0 \
26
+ --env.server.endpoint=tcp/127.0.0.1:7448 --env.task=all2_plate \
27
+ --eval.batch_size=24 --eval.n_episodes=24 --eval.use_async_envs=false
28
  ```
29
 
30
+ - `--env.task` takes a task, a comma-separated list, or nothing for the whole suite ([task list](https://github.com/insait-institute/ladder/blob/main/TASKS.md)). A task's 24 episodes run as one batch, so keep the batch size at 24.
31
+ - To run it in-process instead, `pip install "lerobot[dataset,evaluation]>=0.7.0" -r requirements.txt` and replace `--env.server.endpoint` with `--env.trust_remote_code=true`.
 
 
 
32
 
33
+ ## Interface
 
 
 
34
 
35
  | | |
36
  |---|---|
37
+ | Observation | `pixels.main`: 480x640 RGB; `agent_pos`: end-effector pose `[x, y, z, rx, ry, rz]` (base frame, axis-angle, BridgeData convention) and gripper (1 open) |
38
+ | Action | `[dx, dy, dz, drx, dry, drz, gripper]`: translation in the base frame, rotation in the end-effector frame, applied to the commanded pose as LADDER runs its Bridge policy; gripper 1 open, 0 closed |
39
  | Control rate | 5 Hz |
40
+ | Score | `pc_success`: LADDER's task success. `avg_sum_reward`: its subgoal progress at the end of the episode (also `info["metrics"]["subgoal_progress"]`) |
 
 
 
 
 
 
 
 
 
 
 
41
 
42
+ ## Relation to LADDER
43
 
44
+ `ladder/` is LADDER's code at release [v1.0.0](https://github.com/insait-institute/ladder/releases/tag/v1.0.0), unchanged, run with the settings of LADDER's own server. The simulation assets (160 MB) are downloaded from that release on first use. Its reset frames match LADDER's recorded episodes, and LADDER's Bridge policy run through it succeeds as often as in LADDER's published episodes (60 and 64 of 96 against 59 on four tasks). LADDER's own harness stays the reference for the [leaderboard](https://ladderrobotics.ai).
 
45
 
46
  ## Citation
47
 
 
57
 
58
  ## License
59
 
60
+ MIT, as LADDER. Downloaded assets keep their own licenses ([THIRD_PARTY_NOTICES.md](https://github.com/insait-institute/ladder/blob/main/THIRD_PARTY_NOTICES.md)).