| --- |
| license: mit |
| library_name: lerobot |
| tags: [lerobot, envhub, robotics, simulation, benchmark, ladder, franka, droid, isaac-sim, isaac-lab] |
| --- |
| |
| # LADDER RoboLab-Long |
|
|
| 44 tasks of 1 to 10 subgoals for a Franka arm with a Robotiq gripper (the DROID setup), built on [RoboLab](https://github.com/NVlabs/RoboLab) and Isaac Sim 6 / Isaac Lab 3. It is one of the three suites of [LADDER](https://ladderrobotics.ai), a long-horizon manipulation benchmark, packaged as a LeRobot [EnvHub](https://huggingface.co/docs/lerobot/envhub) environment. Each simulator needs its own Python environment, so each suite is its own repository: |
|
|
| | Suite | Repository | Simulator | Robot | Tasks | |
| |---|---|---|---|---| |
| | RoboLab-Long | this one | Isaac Sim 6 | Franka (DROID) | 44 | |
| | SimplerEnv-Long | [`INSAIT-Institute/LADDER-SimplerEnv-Long`](https://huggingface.co/INSAIT-Institute/LADDER-SimplerEnv-Long) | SAPIEN 3 / ManiSkill 3 | WidowX | 54 | |
| | RobotArena-∞-Long | [`INSAIT-Institute/LADDER-RobotArena-Long`](https://huggingface.co/INSAIT-Institute/LADDER-RobotArena-Long) | Genesis | WidowX | 47 | |
|
|
| ## Evaluate a policy |
|
|
| From a LeRobot checkout, with Docker, the NVIDIA Container Toolkit and an RTX GPU: |
|
|
| ```bash |
| pip install -e ".[dataset,evaluation,remote]" # plus your policy's extra |
| lerobot-env-server --envhub=INSAIT-Institute/LADDER-RoboLab-Long@v1.0.0 --docker=true |
| lerobot-eval --policy.path=<your DROID checkpoint> \ |
| --env.type=envhub --env.hub_path=INSAIT-Institute/LADDER-RoboLab-Long@v1.0.0 \ |
| --env.server.endpoint=tcp/127.0.0.1:7448 --env.task=comp_bin_2 \ |
| --eval.batch_size=8 --eval.n_episodes=8 --eval.use_async_envs=false |
| ``` |
|
|
| - The first `--docker=true` builds the simulator image (about 29 GB, 10 to 20 minutes); later starts reuse it. |
| - `--env.task` takes a task, a comma-separated list, or nothing for the whole suite ([task list](https://github.com/insait-institute/ladder/blob/main/TASKS.md)). A task's 8 episodes run as one batch: keep the batch size at 8 on a 48 GB GPU, 2 on a 24 GB one. |
| - It runs only behind the server, like LeRobot's own RoboLab integration: Isaac Sim needs NumPy 2.3, which LeRobot does not accept. The server starts Isaac Sim before it reports ready (about 5 minutes the first time, compiling shaders). |
|
|
| ## Interface |
|
|
| | | | |
| |---|---| |
| | Observation | `pixels.main` (over-the-shoulder) and `pixels.wrist`, 224x224 RGB; `agent_pos`: 7 joint positions and gripper (0 open, 1 closed) | |
| | Action | 7 absolute joint positions and gripper (above 0.5 closes), as LADDER runs pi0.5 (`pi05_droid_jointpos`) | |
| | Control rate | 15 Hz | |
| | Score | `pc_success`: LADDER's task success. `avg_sum_reward`: its subgoal progress at the end of the episode (also `info["metrics"]["subgoal_progress"]`) | |
|
|
| ## Relation to LADDER |
|
|
| `ladder/` is LADDER's code at release [v1.0.0](https://github.com/insait-institute/ladder/releases/tag/v1.0.0), unchanged, run with the settings of LADDER's own RoboLab server; `requirements.txt` is LADDER's lock of that environment. Isaac Lab, RoboLab and the assets (about 1 GB) are downloaded at the versions LADDER pins on first use. Its reset frames match LADDER's recorded episodes, and pi0.5 run through it succeeds about as often as in LADDER's published episodes (19 and 16 of 32 against 21 on four tasks, within run-to-run variation). LADDER's own harness stays the reference for the [leaderboard](https://ladderrobotics.ai). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{ladder2026, |
| title = {{LADDER}: A Long-Horizon Robot Manipulation Benchmark}, |
| author = {Petkov, Petko and Albanese, Giuliano and Nikolov, Nikolay and Van Gool, Luc and Tremblay, Jonathan and Paudel, Danda Pani}, |
| journal = {arXiv preprint arXiv:2610.18053}, |
| year = {2026}, |
| url = {https://arxiv.org/abs/2610.18053} |
| } |
| ``` |
|
|
| ## License |
|
|
| MIT, as LADDER. RoboLab is Apache-2.0 and its assets mostly CC BY-NC-SA 4.0 ([THIRD_PARTY_NOTICES.md](https://github.com/insait-institute/ladder/blob/main/THIRD_PARTY_NOTICES.md)). Running it accepts the [NVIDIA Isaac Sim license](https://docs.isaacsim.omniverse.nvidia.com/latest/common/NVIDIA_Omniverse_License_Agreement.html) (`env.py` sets `OMNI_KIT_ACCEPT_EULA=YES`). |
|
|
