diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000000000000000000000000000000000000..7a60b85e148f80966a550e5ab6a762a907c69ca6 --- /dev/null +++ b/.gitignore @@ -0,0 +1,2 @@ +__pycache__/ +*.pyc diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000000000000000000000000000000000000..51f39ed5f26f312e49b8921940d660f9dcb8c962 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 INSAIT + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md new file mode 100644 index 0000000000000000000000000000000000000000..576293196218ee84492710ea8fdb078b9bbf48e0 --- /dev/null +++ b/README.md @@ -0,0 +1,76 @@ +--- +license: mit +library_name: lerobot +tags: [lerobot, envhub, robotics, simulation, benchmark, long-horizon, isaac-sim, isaac-lab, franka, droid, ladder] +--- + +# LADDER RoboLab-Long + +The RoboLab-Long suite of [LADDER](https://ladderrobotics.ai), a long-horizon robot manipulation benchmark, as a LeRobot [EnvHub](https://huggingface.co/docs/lerobot/envhub) environment. It has 44 tasks of 1 to 10 subgoals for a Franka arm with a Robotiq gripper (the DROID setup), built on [RoboLab](https://github.com/NVlabs/RoboLab) and Isaac Sim 6 / Isaac Lab 3. + +The two other suites are separate repositories, because each simulator needs its own Python environment: + +| Suite | Repository | Simulator | Robot | Tasks | Episodes per task | +|---|---|---|---|---|---| +| RoboLab-Long | this one | Isaac Sim 6 / Isaac Lab 3 | Franka (DROID) | 44 | 8 | +| SimplerEnv-Long | [`INSAIT-Institute/LADDER-SimplerEnv-Long`](https://huggingface.co/INSAIT-Institute/LADDER-SimplerEnv-Long) | SAPIEN 3 / ManiSkill 3 | WidowX | 54 | 24 | +| RobotArena-∞-Long | [`INSAIT-Institute/LADDER-RobotArena-Long`](https://huggingface.co/INSAIT-Institute/LADDER-RobotArena-Long) | Genesis | WidowX | 47 | 8 | + +## Evaluate a policy + +Serve it with the LeRobot simulator server, from a LeRobot checkout, with Docker and the NVIDIA Container Toolkit, on an NVIDIA RTX GPU with 48 GB for 8 parallel episodes. Isaac Sim 6 needs NumPy 2.3, which LeRobot itself does not accept, so like LeRobot's own RoboLab integration it runs only in the server, apart from your policy's environment: + +```bash +pip install -e ".[dataset,evaluation,remote]" # plus your policy's extra, e.g. ".[pi]" +lerobot-env-server --envhub=INSAIT-Institute/LADDER-RoboLab-Long@v1.0.0 --docker=true +lerobot-eval --policy.path= \ + --env.type=envhub --env.hub_path=INSAIT-Institute/LADDER-RoboLab-Long@v1.0.0 \ + --env.server.endpoint=tcp/127.0.0.1:7448 \ + --env.task=comp_bin_2 --eval.batch_size=8 --eval.n_episodes=8 --eval.use_async_envs=false +``` + +- `--env.task` takes one task, a comma-separated list (run one after another: keep `--env.max_parallel_tasks=1`), or nothing for all 44. [TASKS.md](https://github.com/insait-institute/ladder/blob/main/TASKS.md) lists them. +- Each task is one Isaac Lab world of 8 parallel envs: keep `--eval.batch_size` at 8 (2 on a 24 GB GPU) and `--eval.use_async_envs=false`. The server downloads the assets and starts Isaac Sim (compiling its shaders on a new machine) before it reports ready; each task's world is then built when the client opens it. +- `@v1.0.0` pins this repository to its release for LADDER v1.0.0 (or pin a commit): LeRobot runs its `env.py`. + +## Observations, actions and scores + +| | | +|---|---| +| Observation | `pixels.main`: the over-the-shoulder camera and `pixels.wrist`: the wrist camera, 224x224 RGB; `agent_pos`: 7 joint positions and the gripper (0 open, 1 closed) | +| Action | 7 absolute joint positions and the gripper (above 0.5 closes it), as LADDER runs pi0.5 (`pi05_droid_jointpos`) | +| Control rate | 15 Hz | +| Step budget | the task's own budget, from RoboLab's episode length | +| Instruction | `task_description`, the task's instruction | +| Success | `info["is_success"]`: LADDER's task success (`pc_success` in LeRobot) | +| Progress | `info["metrics"]["subgoal_progress"]` (and `subgoals_done`): LADDER's subgoal score at the end of the episode, each subgoal met in the final state and, for an object moved twice, in order; all of them once the task succeeded. `reward` is its change at each step, so `avg_sum_reward` in LeRobot is the mean final progress | + +Options, passed as `--sim.kwargs='{make_env_kwargs: {...}}'` to the server: `observation_height`, `observation_width` (default 224x224, as LADDER runs pi0.5), `front_camera_name` (default `main`), `wrist_camera_name` (default `wrist`). + +## How it relates to LADDER + +`ladder/` is LADDER's own code at release [`v1.0.0`](https://github.com/insait-institute/ladder/releases/tag/v1.0.0), unchanged: the task definitions, scene layouts and success predicates. `env.py` builds the suite's environment with the settings of LADDER's own RoboLab server (`env_server.py`). `requirements.txt` is LADDER's own lock of that environment, every package pinned to its version, less Isaac Lab and two packages only Isaac Lab needs. Isaac Lab's wheel (the version LADDER pins), RoboLab's source (the commit LADDER pins, with LADDER's one-line asset-directory patch), openpi's client and the RoboLab assets are downloaded on first use (about 1 GB, checked by sha256) into `~/.cache/ladder/envhub` (`LADDER_CACHE` moves it). Isaac Lab is loaded from its wheel rather than installed because its metadata pins `transformers==4.57.6`, for an image encoder RoboLab never uses, and that version needs a `huggingface-hub` older than the LeRobot server accepts. + +The full suite is all 44 tasks with 8 episodes each. LADDER's own harness (`scripts/eval.sh`) stays the reference for leaderboard results; see [EVALUATION.md](https://github.com/insait-institute/ladder/blob/main/EVALUATION.md). + +## Checked against LADDER + +- The reset frames match LADDER's recorded episodes (34.5 dB PSNR on the main camera, 33.2 dB on the wrist camera). +- pi0.5 (`pi05_droid_jointpos`, LADDER's checkpoint), run through this repository's Docker image with LADDER's own client settings, on 4 tasks (`comp_bin_2`, `comp_transfer_3`, `comp_bin_6`, `comp_route_3`; 32 episodes), succeeded in 19 and 16 of 32 episodes in two runs. LADDER's published episodes of the same tasks: 21 of 32, and LADDER's earlier runs of pi0.5 on them vary as much (`comp_bin_6`: 3 and 5 of 8). +- In a new container the server is ready after about 5 minutes (downloads, Isaac Sim's first start, one world built to compile shaders); a client's first reset then takes about 20 seconds. + +## Citation + +```bibtex +@article{ladder2026, + title = {{LADDER}: A Long-Horizon Robot Manipulation Benchmark}, + author = {Petkov, Petko and Albanese, Giuliano and Nikolov, Nikolay and Van Gool, Luc and Tremblay, Jonathan and Paudel, Danda Pani}, + journal = {arXiv preprint arXiv:2610.18053}, + year = {2026}, + url = {https://arxiv.org/abs/2610.18053} +} +``` + +## License + +MIT, as LADDER. RoboLab is Apache-2.0 (NVIDIA); the RoboLab assets are mostly CC BY-NC-SA 4.0. The downloaded files keep their own licenses (see LADDER's [THIRD_PARTY_NOTICES.md](https://github.com/insait-institute/ladder/blob/main/THIRD_PARTY_NOTICES.md)). **Isaac Sim** is distributed under the [NVIDIA Isaac Sim license](https://docs.isaacsim.omniverse.nvidia.com/latest/common/NVIDIA_Omniverse_License_Agreement.html); `env.py` sets `OMNI_KIT_ACCEPT_EULA=YES`, so running this environment means you accept it. diff --git a/_ladder_envhub.py b/_ladder_envhub.py new file mode 100644 index 0000000000000000000000000000000000000000..1f9328102243b913c4c86c30f1ec77aa08cd4356 --- /dev/null +++ b/_ladder_envhub.py @@ -0,0 +1,493 @@ +# Copyright (c) 2026 INSAIT. MIT License (see LICENSE). +"""Code shared by the three LADDER EnvHub repositories. + +LADDER (https://github.com/insait-institute/ladder) is a long-horizon manipulation benchmark with three +suites, each on its own simulator. Each suite is published as its own EnvHub repository, because the +three simulators need incompatible Python environments. This module holds what they share: fetching the +pinned benchmark files, the batched ``gym.vector.VectorEnv`` around a LADDER task, and the action and +observation conversions. + +The benchmark's own code (task definitions, scene layouts, success predicates) is the ``ladder/`` +package next to this file, copied unchanged from the LADDER release named in ``LADDER_RELEASE``. +Third-party code and the simulation assets are downloaded from that release on first use and checked +against their sha256. +""" + +from __future__ import annotations + +import contextlib +import fcntl +import hashlib +import logging +import os +import shutil +import sys +import tarfile +import tempfile +import urllib.request +import zipfile +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import gymnasium as gym +import numpy as np +from gymnasium import spaces +from gymnasium.vector.utils import batch_space + +logger = logging.getLogger(__name__) + +LADDER_RELEASE = "v1.0.0" +LADDER_COMMIT = "6c88eeadc8f5666fdf65f10b9c06bb7bdf1f7ca4" +_RELEASE_URL = f"https://github.com/insait-institute/ladder/releases/download/{LADDER_RELEASE}" + + +@dataclass(frozen=True) +class Download: + """One pinned archive. ``strip`` drops a leading directory, ``prefix`` places the files under one.""" + + url: str + sha256: str + strip: str = "" + prefix: str = "" + + +DOWNLOADS: dict[str, Download] = { + "simpler_assets": Download( + f"{_RELEASE_URL}/simpler_assets.tar.gz", + "dfa45de8a2670cc6f05bb9db28cdf16b6ebabc24f41fe10a3cbe0ff1bde53a9f", + ), + # ManiSkill's WidowX model, from its upstream repository (it has no license to redistribute it). + "widowx": Download( + "https://github.com/haosulab/ManiSkill-WidowX250S/archive/6a12bb745d964aab9fc897f0999cad53aa29000c.tar.gz", + "c440d733291e28fd257d0e6a6fb71b8575d155fd7bde92f6779743a52d8fedd0", + strip="ManiSkill-WidowX250S-6a12bb745d964aab9fc897f0999cad53aa29000c", + prefix="data/robots/widowx", + ), + "sapien_physx": Download( + f"{_RELEASE_URL}/sapien_physx_gpu_linux.zip", + "0353b3b0c5d45d3ad85136de38ac89c216ddfea8f2306966bfafb93cd6cd839c", + ), + "robot_arena_assets": Download( + f"{_RELEASE_URL}/robot_arena_assets.tar.gz", + "4e9ab964bc1fe2dd71570307265dc0ab0e7edb8466ddfe46056db7d90c0b2187", + strip="robot_arena_data", + ), + "robolab_assets": Download( + f"{_RELEASE_URL}/robolab_assets.tar.gz", + "19da3abcc8d94126791a78c3cb06cb3b78776e8513c73be2ad0961206b209d8d", + ), + "robolab_src": Download( + f"{_RELEASE_URL}/robolab_src.tar.gz", + "ffcd150871ead600dd84b9302459a66b9a1e59f4ed2cc7a07a44ac6c0b27b4a5", + strip="RoboLab-27b6212df31799f72883bd79c8629607d89782de", + ), + # Isaac Lab's own wheel, as LADDER pins it. It is loaded from here rather than installed: its metadata pins + # transformers==4.57.6 (for an optional Theia image encoder RoboLab never imports), which needs + # huggingface-hub<1, and the LeRobot server needs huggingface-hub>=1.31. + "isaaclab": Download( + "https://pypi.nvidia.com/isaaclab/isaaclab-3.0.0b2-py3-none-any.whl", + "386fcbe043ae98c880468f30872ffcdbfb37f26dc6756b9dc2bbe0226947eb11", + ), + "openpi_src": Download( + f"{_RELEASE_URL}/openpi_src.tar.gz", + "2e36927c2c77ffaf111a16e5533d6d4ac26ea00fb47c1650f95a96e5d9f078de", + strip="openpi-aa6420561529593114160d05e5ad155792b272f3", + ), +} + + +def cache_dir() -> Path: + """Where downloads are kept: ``$LADDER_CACHE``, else ``~/.cache/ladder/envhub``.""" + root = os.environ.get("LADDER_CACHE") or os.path.join(os.path.expanduser("~"), ".cache", "ladder", "envhub") + return Path(root) / LADDER_RELEASE + + +def _sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as f: + for block in iter(lambda: f.read(1 << 20), b""): + digest.update(block) + return digest.hexdigest() + + +def fetch(name: str) -> Path: + """Return the extracted directory of a pinned download, downloading it once per machine. + + Concurrent processes (e.g. several eval lanes) wait on a file lock and reuse one copy. + """ + spec = DOWNLOADS[name] + target = cache_dir() / name + if (target / ".complete").exists(): + return target + target.parent.mkdir(parents=True, exist_ok=True) + with open(target.parent / f".{name}.lock", "w") as lock: + fcntl.flock(lock, fcntl.LOCK_EX) + if (target / ".complete").exists(): + return target + with tempfile.TemporaryDirectory(dir=target.parent, prefix=f".{name}.") as tmp: + archive = Path(tmp) / spec.url.rsplit("/", 1)[-1] + logger.info("LADDER: downloading %s", spec.url) + with urllib.request.urlopen(spec.url, timeout=60) as response, archive.open("wb") as out: + shutil.copyfileobj(response, out, length=1 << 20) + actual = _sha256(archive) + if actual != spec.sha256: + raise RuntimeError(f"{spec.url}: sha256 {actual}, expected {spec.sha256}") + extracted = Path(tmp) / "x" + if archive.suffix in (".zip", ".whl"): + with zipfile.ZipFile(archive) as z: + z.extractall(extracted) + else: + with tarfile.open(archive) as t: + t.extractall(extracted, filter="data") + archive.unlink() + root = extracted / spec.strip if spec.strip else extracted + staged = Path(tmp) / "staged" + destination = staged / spec.prefix if spec.prefix else staged + destination.parent.mkdir(parents=True, exist_ok=True) + root.rename(destination) + (staged / ".complete").touch() + shutil.rmtree(target, ignore_errors=True) + staged.rename(target) + return target + + +def add_to_path(*paths: Path) -> None: + for path in paths: + if str(path) not in sys.path: + sys.path.insert(0, str(path)) + + +# ---------------------------------------------------------------------------------------------- poses + +# LADDER's poses are in its canonical end-effector frame. For the WidowX, observations and actions use the +# BridgeData v2 end-effector frame instead, as LADDER's Bridge policy does: the canonical frame turned 180 degrees +# about its x axis (CANONICAL_TO_BRIDGE_ROTATION in LADDER's vlam_robot_inference.py). +CANONICAL_TO_BRIDGE = np.diag([1.0, -1.0, -1.0]) + +def axisangle_to_matrix(rotvec: np.ndarray) -> np.ndarray: + rv = np.asarray(rotvec, dtype=np.float64).reshape(3) + theta = float(np.linalg.norm(rv)) + if theta < 1e-12: + return np.eye(3) + kx, ky, kz = rv / theta + k = np.array([[0.0, -kz, ky], [kz, 0.0, -kx], [-ky, kx, 0.0]]) + return np.eye(3) + np.sin(theta) * k + (1.0 - np.cos(theta)) * (k @ k) + + +def matrix_to_axisangle(rot: np.ndarray) -> np.ndarray: + r = np.asarray(rot, dtype=np.float64).reshape(3, 3) + theta = float(np.arccos(np.clip((np.trace(r) - 1.0) * 0.5, -1.0, 1.0))) + if theta < 1e-7: + return np.zeros(3, dtype=np.float32) + axis = np.array([r[2, 1] - r[1, 2], r[0, 2] - r[2, 0], r[1, 0] - r[0, 1]]) + norm = float(np.linalg.norm(axis)) + return ((axis / norm) * theta).astype(np.float32) if norm >= 1e-12 else np.zeros(3, dtype=np.float32) + + +WIDOWX_STATE_NAMES = ("x", "y", "z", "rx", "ry", "rz", "gripper") +WIDOWX_DELTA_ACTION_NAMES = ("dx", "dy", "dz", "drx", "dry", "drz", "gripper") +DROID_JOINT_NAMES = (*(f"joint_{i}.pos" for i in range(7)), "gripper.pos") + + +def in_env_server() -> bool: + """Whether this process is LeRobot's env server (`lerobot-env-server`), not an in-process evaluation. + + The server opens a task within a client's 120 s deadline, so slow first builds are worth doing before it reports + ready; in-process there is no deadline and LeRobot passes no task, so they would only add start-up time. + """ + return "lerobot_env_server" in sys.argv[0] or "lerobot-env-server" in sys.argv[0] + + +def one_task(task: str | None, known: list[str]) -> str: + """The single task ``make_env`` builds: the suite's simulator keeps one scene per process.""" + names = select_tasks(task, known) if task else [] + if len(names) != 1: + raise ValueError( + "make_env builds one task (pass task=, --env.task=); LeRobot's default path " + f"(vector_env_fns) runs several, one after another. Available: {known}" + ) + return names[0] + + +def select_tasks(task: str | None, known: list[str]) -> list[str]: + """``task`` is a task name, a comma-separated list, or empty/"all" for every task.""" + if not task or task in ("all", "*"): + return list(known) + names = [t.strip() for t in task.split(",") if t.strip()] + unknown = [n for n in names if n not in known] + if unknown: + raise ValueError(f"Unknown LADDER tasks {unknown}. Available: {known}") + return names + + +# ---------------------------------------------------------------------------------------------- env + + +class LadderVectorEnv(gym.vector.VectorEnv): + """``n_envs`` parallel episodes of one LADDER task, stepped as one batch by the suite's simulator. + + Observations follow LeRobot's EnvHub conventions: ``pixels`` (one image per camera) and + ``agent_pos``. ``info["is_success"]`` is LADDER's task success, latched once reached. + ``info["metrics"]`` holds LADDER's subgoal score of the current state (``subgoals_done``, and + ``subgoal_progress``, its fraction), which LeRobot reads at the end of each episode; ``reward`` is + the change in ``subgoal_progress``, so an episode's rewards sum to its final progress. + Episode ``i`` of rollout ``r`` is LADDER episode ``r * n_envs + i``. + + Subclasses implement ``_backend_reset(episode_ids)`` and ``_backend_step(actions)`` with the + suite's own LADDER environment; both return one LADDER ``RobotObservation`` (or its dict) per env. + """ + + suite: str = "" + robot: str = "widowx" # or "franka" + default_image_size: tuple[int, int] = (480, 640) + resizable: bool = True # whether the simulator renders at observation_height x observation_width + episodes_per_task: int = 8 # distinct LADDER episodes; later rollouts repeat them + fps: int = 5 + + def __init__( + self, + task: str, + n_envs: int = 1, + max_episode_steps: int | None = None, + observation_height: int | None = None, + observation_width: int | None = None, + front_camera_name: str = "main", + wrist_camera_name: str = "wrist", + seed: int = 42, + ) -> None: + if not self.resizable and (observation_height or observation_width): + raise ValueError(f"{self.suite} renders at {self.default_image_size}; resize the frames in your policy") + self.task = task + self.task_description = task + self.num_envs = int(n_envs) + self._max_episode_steps = int(max_episode_steps or self.task_budget(task)) + self._front = front_camera_name + self._wrist = wrist_camera_name + self._seed = int(seed) + self._frame = CANONICAL_TO_BRIDGE if self.robot == "widowx" else np.eye(3) + self._h = int(observation_height or self.default_image_size[0]) + self._w = int(observation_width or self.default_image_size[1]) + self.metadata = { + "render_modes": ["rgb_array"], + "render_fps": self.fps, + "autoreset_mode": gym.vector.AutoresetMode.DISABLED, + } + self.render_mode = "rgb_array" + self.closed = False + + image = spaces.Box(0, 255, shape=(self._h, self._w, 3), dtype=np.uint8) + cameras = {self._front: image} + if self.robot == "franka": + cameras[self._wrist] = image + state_dim = 8 if self.robot == "franka" else 7 + self.single_observation_space = spaces.Dict( + { + "pixels": spaces.Dict(cameras), + "agent_pos": spaces.Box(-np.inf, np.inf, shape=(state_dim,), dtype=np.float32), + } + ) + # Franka: absolute joint targets and the gripper; WidowX: pose deltas and the gripper. + self.single_action_space = spaces.Box(-np.inf, np.inf, shape=(state_dim,), dtype=np.float32) + self.observation_space = batch_space(self.single_observation_space, self.num_envs) + self.action_space = batch_space(self.single_action_space, self.num_envs) + + self._rollout = 0 + self._step_count = 0 + self._target_poses = [np.eye(4, dtype=np.float32) for _ in range(self.num_envs)] + self._joints: list[np.ndarray | None] = [None] * self.num_envs + self._frames = [np.zeros((self._h, self._w, 3), dtype=np.uint8) for _ in range(self.num_envs)] + self._success = np.zeros(self.num_envs, dtype=bool) + self._progress = np.zeros(self.num_envs, dtype=np.float32) + self._first_met: list[dict[int, int]] = [{} for _ in range(self.num_envs)] + self._met_after_ready: list[set[int]] = [set() for _ in range(self.num_envs)] + self._last_achieved: list[list[bool] | None] = [None] * self.num_envs + + def goal_sources(self) -> list[str | None] | None: + """The object each subgoal moves, for LADDER's ordering of an object's legs; None if unordered.""" + return None + + # Subclass API ------------------------------------------------------------------------------- + @classmethod + def task_budget(cls, task: str) -> int: + raise NotImplementedError + + def _backend_reset(self, episode_ids: list[int]) -> list[Any]: + raise NotImplementedError + + def _backend_step(self, actions: list[Any]) -> list[Any]: + raise NotImplementedError + + def _backend_close(self) -> None: + pass + + # Gymnasium API ------------------------------------------------------------------------------ + def reset(self, *, seed: Any = None, options: dict[str, Any] | None = None): + episode_ids = [self._rollout * self.num_envs + i for i in range(self.num_envs)] + if max(episode_ids) >= self.episodes_per_task and not getattr(self, "_warned_repeat", False): + self._warned_repeat = True + logger.warning( + "LADDER %s has %d distinct episodes per task; episodes past them repeat them", + self.suite, self.episodes_per_task, + ) + self._rollout += 1 + self._step_count = 0 + self._success[:] = False + self._progress[:] = 0.0 + self._first_met = [{} for _ in range(self.num_envs)] + self._met_after_ready = [set() for _ in range(self.num_envs)] + self._last_achieved = [None] * self.num_envs + obs, _, _, _, info = self._pack(self._backend_reset(episode_ids)) + return obs, info + + def step(self, actions: np.ndarray): + actions = np.asarray(actions, dtype=np.float32).reshape(self.num_envs, -1) + raw = self._backend_step([self._decode_action(i, a) for i, a in enumerate(actions)]) + self._step_count += 1 + return self._pack(raw) + + def render(self) -> tuple[np.ndarray, ...]: + return tuple(self._frames) + + def call(self, name: str, *args: Any, **kwargs: Any) -> tuple[Any, ...]: + if name == "render": + return self.render() + attr = getattr(self, name) + value = attr(*args, **kwargs) if callable(attr) else attr + return (value,) * self.num_envs + + def get_attr(self, name: str) -> tuple[Any, ...]: + return (getattr(self, name),) * self.num_envs + + def close(self, **kwargs: Any) -> None: + if not self.closed: + self._backend_close() + self.closed = True + + # Conversions -------------------------------------------------------------------------------- + def _pack(self, raw_obs: list[Any]): + fronts, wrists, states = [], [], [] + progress = np.zeros(self.num_envs, dtype=np.float32) + truncated = np.zeros(self.num_envs, dtype=bool) + completed = np.zeros(self.num_envs, dtype=np.int32) + for i, raw in enumerate(raw_obs): + if not isinstance(raw, dict): + raw = {"images": raw.images, "state": vars(raw.state), "metadata": raw.metadata} + images = {k: np.asarray(v, dtype=np.uint8) for k, v in (raw.get("images") or {}).items()} + state = raw.get("state") or {} + meta = raw.get("metadata") or {} + eef = np.asarray(state.get("eef_pose", np.eye(4)), dtype=np.float32) + target = state.get("eef_pose_from_target") + gripper = float(np.asarray(state.get("gripper_state", [1.0])).reshape(-1)[0]) + joints = state.get("joints") + self._target_poses[i] = np.asarray(target if target is not None else eef, dtype=np.float32) + self._joints[i] = None if joints is None else np.asarray(joints, dtype=np.float32).reshape(-1)[:7] + + if meta.get("instruction"): + self.task_description = str(meta["instruction"]).strip() + front = images.get("main", next(iter(images.values()), None)) + if front is None: + front = np.zeros((self._h, self._w, 3), dtype=np.uint8) + fronts.append(front) + self._frames[i] = front + if self.robot == "franka": + wrists.append(images.get("wrist", np.zeros_like(front))) + states.append(np.concatenate([self._joints[i], [1.0 - gripper]]).astype(np.float32)) + else: + states.append(np.concatenate([eef[:3, 3], matrix_to_axisangle(eef[:3, :3] @ self._frame), [gripper]])) + + self._success[i] |= bool(meta.get("done", False)) + oracle = meta.get("oracle") if isinstance(meta.get("oracle"), dict) else {} + achieved = [bool(v) for v in oracle.get("achieved", [])] + completed[i] = self._subgoals_done(i, achieved) + progress[i] = completed[i] / len(achieved) if achieved else float(self._success[i]) + truncated[i] = bool(meta.get("truncated", False)) or self._step_count >= self._max_episode_steps + + pixels = {self._front: np.stack(fronts)} + if self.robot == "franka": + pixels[self._wrist] = np.stack(wrists) + obs = {"pixels": pixels, "agent_pos": np.stack(states).astype(np.float32)} + reward = progress - self._progress + self._progress = progress + info = { + "is_success": self._success.copy(), + "metrics": {"subgoals_done": completed.astype(np.float32), "subgoal_progress": progress.copy()}, + } + return obs, reward, self._success.copy(), truncated, info + + def _subgoals_done(self, i: int, achieved: list[bool]) -> int: + """LADDER's subgoal score (scripts/leaderboard.py ``subgoals_done``) of the episode so far, ending now. + + Every subgoal once the task succeeded; otherwise the subgoals met now, each counted only if it was + met after every earlier subgoal of the same object was first met (a relay's legs are ordered). + """ + if self._success[i]: + return len(achieved) + step, first, after = self._step_count, self._first_met[i], self._met_after_ready[i] + sources = self.goal_sources() or [None] * len(achieved) + # LADDER scores its goal trace, which records the steps where the achieved vector changes. + changed = self._last_achieved[i] != achieved + self._last_achieved[i] = list(achieved) + for g, met in enumerate(achieved if changed else []): + if met: + first.setdefault(g, step) + for g, met in enumerate(achieved if changed else []): + earlier = [leg for leg in range(g) if sources[g] is not None and sources[leg] == sources[g]] + if met and all(leg in first for leg in earlier) and step > max((first[leg] for leg in earlier), default=-1): + after.add(g) + return sum(1 for g, met in enumerate(achieved) if met and g in after) + + def _decode_action(self, i: int, raw: np.ndarray) -> Any: + from ladder.components.inference.api.core.action import ( + EndEffectorPoseAction, + JointAnglesAction, + ManipulatorAction, + ) + + if self.robot == "franka": + # As LADDER runs pi05_droid_jointpos: 7 joint targets, and the gripper closed above 0.5. + return ManipulatorAction( + arm_action=JointAnglesAction(angles=raw[:7].astype(np.float32)), + gripper_action=0.0 if float(raw[7]) > 0.5 else 1.0, + ) + # As LADDER's clients run the Bridge policy: the rotation in the end-effector frame, the translation in the + # base frame, both applied to the commanded pose rather than the observed one + # (EndEffectorPoseDeltaMode.ROT_EEF_TRANS_BASE), in the BridgeData end-effector frame. + frame = self._frame + target = self._target_poses[i].astype(np.float64) + new_target = np.eye(4, dtype=np.float32) + new_target[:3, 3] = target[:3, 3] + raw[:3].astype(np.float64) + new_target[:3, :3] = target[:3, :3] @ frame @ axisangle_to_matrix(raw[3:6]) @ frame.T + self._target_poses[i] = new_target + return ManipulatorAction( + arm_action=EndEffectorPoseAction(translation=new_target[:3, 3].copy(), rotation=new_target[:3, :3].copy()), + gripper_action=float(np.clip(raw[6], 0.0, 1.0)), + ) + + +def lerobot_sim(robot: str, fps: int) -> dict[str, Any]: + """The ``LEROBOT_SIM`` description of a suite.""" + if robot == "franka": + names = list(DROID_JOINT_NAMES) + return {"fps": fps, "control_mode": "position", "state_names": names, "action_names": names} + return { + "fps": fps, + "control_mode": "delta", + "state_names": list(WIDOWX_STATE_NAMES), + "action_names": list(WIDOWX_DELTA_ACTION_NAMES), + } + + +@contextlib.contextmanager +def ignore_signal_in_thread(): + """Isaac Sim installs signal handlers, which only the main thread may do.""" + import signal + import threading + + original = signal.signal + if threading.current_thread() is not threading.main_thread(): + signal.signal = lambda *a, **kw: None # type: ignore[assignment] + try: + yield + finally: + signal.signal = original # type: ignore[assignment] diff --git a/env.py b/env.py new file mode 100644 index 0000000000000000000000000000000000000000..60d2195cd6be5f3f107c559bbdec11bbf340ed97 --- /dev/null +++ b/env.py @@ -0,0 +1,273 @@ +# Copyright (c) 2026 INSAIT. MIT License (see LICENSE). +"""LADDER RoboLab-Long for LeRobot EnvHub: 44 long-horizon Franka (DROID) tasks on Isaac Sim 6 / Isaac Lab 3. + +Each task is one suite of LeRobot's ``{suite: {task_id: env}}``; its 8 episodes run as 8 parallel envs +in one Isaac Lab world. Isaac Sim starts once per process. See README.md. +""" + +import os +import re +import sys + +REPO_ID = "INSAIT-Institute/LADDER-RoboLab-Long" + +_HERE = os.path.dirname(os.path.abspath(__file__)) +if not os.path.exists(os.path.join(_HERE, "_ladder_envhub.py")): + # `make_env("")` downloads only this file: fetch the rest of the same revision. + from huggingface_hub import snapshot_download + + _revision = os.path.basename(_HERE) + _HERE = snapshot_download(REPO_ID, revision=_revision if re.fullmatch(r"[0-9a-f]{40}", _revision) else None) +if _HERE not in sys.path: + sys.path.insert(0, _HERE) + +import atexit # noqa: E402 +import logging # noqa: E402 +from concurrent.futures import ThreadPoolExecutor # noqa: E402 +from functools import partial # noqa: E402 +from typing import Any # noqa: E402 + +import _ladder_envhub as lh # noqa: E402 + +EPISODES_PER_TASK = 8 +CONTROL_HZ = 15 +LEROBOT_SIM = lh.lerobot_sim("franka", fps=CONTROL_HZ) + + +def _add_sources() -> None: + """Isaac Lab, RoboLab (patched to read its assets from ROBOLAB_ASSET_DIR) and openpi_client, as LADDER pins them.""" + if _Isaac.sources: + return + robolab = lh.fetch("robolab_src") + constants = robolab / "robolab" / "constants.py" + text = constants.read_text() + patched = 'ASSET_DIR = os.environ.get("ROBOLAB_ASSET_DIR") or os.path.join(PACKAGE_DIR, "assets")' + if patched not in text: + # LADDER's build_support/python/patches/robolab_constants_asset_dir.patch. + text = text.replace('ASSET_DIR = os.path.join(PACKAGE_DIR, "assets")', patched, 1) + text = text.replace( + " return os.path.join(PACKAGE_DIR, relative_path)", + ' normalized = relative_path.replace(os.sep, "/")\n' + ' if normalized.startswith("assets/"):\n' + ' return os.path.join(ASSET_DIR, *normalized[len("assets/"):].split("/"))\n' + " return os.path.join(PACKAGE_DIR, relative_path)", + 1, + ) + # Atomically: another process may be importing it. + staged = constants.with_name(f".constants.{os.getpid()}.py") + staged.write_text(text) + os.replace(staged, constants) + if patched not in constants.read_text(): + raise RuntimeError(f"{constants}: LADDER's asset-directory patch no longer applies") + # The extracted wheel is an installed package on sys.path, with its dist-info (see _ladder_envhub.DOWNLOADS). + lh.add_to_path(lh.fetch("isaaclab"), robolab, lh.fetch("openpi_src") / "packages" / "openpi-client" / "src") + os.environ.setdefault("ROBOLAB_ASSET_DIR", str(lh.fetch("robolab_assets") / "assets")) + _Isaac.sources = True + + +def task_names() -> list[str]: + """The suite's task names, from LADDER's own registry.""" + from ladder.components.inference.robots.robolab.composition.specs import SPECS + + return sorted(SPECS) + + +class _Isaac: + """Isaac Sim starts once per process and runs on one thread; one task's world is kept at a time.""" + + thread = ThreadPoolExecutor(max_workers=1, thread_name_prefix="ladder-isaac") + sources = False + app: Any = None + env: Any = None + env_cfg: Any = None + oracle: Any = None + key: Any = None + + +def _start_isaac() -> None: + if _Isaac.app is not None: + return + _add_sources() + # Running this environment accepts the NVIDIA Isaac Sim EULA (see README); Kit asks on stdin otherwise. + os.environ.setdefault("OMNI_KIT_ACCEPT_EULA", "YES") + os.environ.setdefault("ACCEPT_EULA", "Y") + import cv2 # noqa: F401 + from isaaclab.app import AppLauncher + + root = logging.getLogger() + levels = {handler: handler.level for handler in root.handlers} + with lh.ignore_signal_in_thread(): + _Isaac.app = AppLauncher(headless=True, enable_cameras=True).app + for handler, level in levels.items(): # Kit lowers the levels of existing handlers + handler.setLevel(level) + # Kit hangs in its own shutdown; leave without it, keeping the exit status. + atexit.register( + lambda: ( + sys.stdout.flush(), + sys.stderr.flush(), + os._exit(0 if getattr(sys, "last_value", None) is None else 1), + ) + ) + import carb.settings + import robolab.constants + + # As LADDER's env_server.py: LADDER scores subgoals itself, and renderer failures stay visible. + robolab.constants.ENABLE_SUBTASK_PROGRESS_CHECKING = False + carb.settings.get_settings().set("/persistent/app/usd/muteUsdDiagnostics", False) + + +class RoboLabLong(lh.LadderVectorEnv): + suite = "robolab" + robot = "franka" + default_image_size = (224, 224) + episodes_per_task = EPISODES_PER_TASK + fps = CONTROL_HZ + + @classmethod + def task_budget(cls, task: str) -> int: + from ladder.components.inference.robots.robolab.composition.specs import SPECS + + return int(SPECS[task].episode_length_s * CONTROL_HZ) + + def __init__(self, *args: Any, **kwargs: Any) -> None: + super().__init__(*args, **kwargs) + self._env = None + self._raw = None + self._converter = None + # LeRobot's server builds an env (one env of the first task) before it reports ready. Isaac Sim starts here, + # and the first env of a process builds its world here too: that compiles the shaders and loads the assets + # every task shares, which on a new machine or container would otherwise fall in a client's first open. + # A client's own batch size gets its world on its first reset. + self._fresh = _Isaac.thread.submit(self._start).result() + + def _start(self) -> bool: + _start_isaac() + return _Isaac.env is None and self._build() + + def goal_sources(self) -> list[str | None] | None: + # The object each goal places (a relay's legs are separate goals), as LADDER's privileged.py names it. + return [ + next((geometry[0] for _, geometry in reversed(list(goal.legs)) if geometry is not None), None) + for goal in _Isaac.oracle or [] + ] + + def _backend_reset(self, episode_ids: list[int]) -> list[Any]: + return _Isaac.thread.submit(self._reset).result() + + def _backend_step(self, actions: list[Any]) -> list[Any]: + if _Isaac.env is not self._env: + raise RuntimeError("Another task's world replaced this one: run one task at a time (--env.max_parallel_tasks=1)") + return _Isaac.thread.submit(self._step, actions).result() + + def _build(self) -> bool: + """Build this task's world at this batch size, unless the process holds it; True if it was built.""" + from ladder.components.inference.robots.robolab.composition.specs import task_class_name, task_target + from ladder.components.inference.robots.robolab.oracle import build_oracle + from robolab.core.environments.runtime import create_env + from robolab.registrations.droid.auto_env_registrations_jointpos import auto_register_droid_envs + + self._class_name = task_class_name(self.task) + key = (self._class_name, self.num_envs) + if _Isaac.key == key: + return False + if _Isaac.env is not None: + _Isaac.env.close() + _Isaac.env = _Isaac.key = None + auto_register_droid_envs(task_dirs=None, task=[task_target(self.task)], cameras=None) + # The defaults of LADDER's env_server.py (RoboLab's add_common_eval_args). + _Isaac.env, _Isaac.env_cfg = create_env( + self._class_name, + device="cuda:0", + num_envs=self.num_envs, + instruction_type="default", + policy="ladder_runner", + renderer="realtime", + rendering_mode="balanced", + ) + _Isaac.oracle = build_oracle(getattr(_Isaac.env.cfg, "subtasks", None), _Isaac.env_cfg.instruction) + _Isaac.key = key + if int(_Isaac.env.max_episode_length) != self._max_episode_steps: + logging.getLogger(__name__).warning( + "%s: Isaac Lab budget %d steps, expected %d", self.task, _Isaac.env.max_episode_length, + self._max_episode_steps, + ) + return True + + def _reset(self) -> list[Any]: + from ladder.components.inference.robots.robolab.droid_conversions import DroidJointposConverter + + # As LADDER's env_server.py: a new world is reset as it is, a kept one first unfreezes its finished envs. + if not self._build() and not self._fresh: + _Isaac.env.reset_eval_state() + self._fresh = False + self._env = _Isaac.env + self._env.reset() + self._raw, _ = self._env.reset() # RoboLab's own double reset (robolab.eval.episode.run_episode) + # DroidJointposConverter takes (width, height), as LADDER's --image-size. + self._converter = DroidJointposConverter(num_envs=self.num_envs, image_size=(self._w, self._h)) + return self._observations() + + def _step(self, actions: list[Any]) -> list[Any]: + import omni.kit.app + import omni.timeline + import torch + + joints = self._raw["proprio_obs"]["arm_joint_pos"].detach().cpu().numpy() + commands = self._converter.actions_to_jointpos(actions, joints) + timeline, kit = omni.timeline.get_timeline_interface(), omni.kit.app.get_app() + while not timeline.is_playing(): + kit.update() + self._raw, _, _, _, _ = self._env.step(torch.as_tensor(commands, device=self._env.device)) + return self._observations() + + def _observations(self) -> list[Any]: + images, proprio = self._raw["image_obs"], self._raw["proprio_obs"] + results = self._env.get_env_results() + active = set(self._env.active_env_ids) + # During a step, _step_count is not yet incremented; LADDER's server counts the step first. + out_of_time = self._step_count + 1 >= int(self._env.max_episode_length) + achieved = None + if _Isaac.oracle: + achieved = [[bool(g.check(self._env, env_id=i)) for g in _Isaac.oracle] for i in range(self.num_envs)] + observations = [] + for i in range(self.num_envs): + success = bool(results[i]["success"]) + metadata = { + "env_name": self._class_name, + "robot_name": "franka", + "instruction": _Isaac.env_cfg.instruction, + "done": success, + "truncated": out_of_time or i not in active, + "oracle": {"achieved": achieved[i] if achieved is not None else [success]}, + } + observations.append( + self._converter.observation_dict( + env_i=i, + main_image=images["over_shoulder_left_camera"][i].detach().cpu().numpy(), + wrist_image=images["wrist_cam"][i].detach().cpu().numpy(), + joints=proprio["arm_joint_pos"][i].detach().cpu().numpy(), + gripper_position=proprio["gripper_pos"][i].detach().cpu().numpy(), + metadata=metadata, + ) + ) + return observations + + +def vector_env_fns( + task: str = "", + n_envs: int = EPISODES_PER_TASK, + **options: Any, +) -> dict[str, dict[int, Any]]: + """One factory per task, ``{task: {0: factory}}``: every task, or the comma-separated ``task``. + + LeRobot builds only the tasks it runs (``--env.task``), so listing all 44 costs nothing. + """ + names = lh.select_tasks(task, task_names()) + _add_sources() # download before the server reports ready + return {name: {0: partial(RoboLabLong, name, n_envs, **options)} for name in names} + + +def make_env(n_envs: int = EPISODES_PER_TASK, use_async_envs: bool = False, task: str = "", **options: Any): + """One task as a batched vector env (``--env.use_native_batch``); LeRobot otherwise uses ``vector_env_fns``.""" + name = lh.one_task(task, task_names()) + return {name: {0: vector_env_fns(name, n_envs, **options)[name][0]()}} diff --git a/ladder/components/inference/api/core/action.py b/ladder/components/inference/api/core/action.py new file mode 100644 index 0000000000000000000000000000000000000000..f33a720394df9618dbcabacb699be64f0664047c --- /dev/null +++ b/ladder/components/inference/api/core/action.py @@ -0,0 +1,228 @@ +from dataclasses import dataclass +from enum import Enum +from typing import Any, Dict, List, Optional, Tuple + +import numpy as np +import numpy.typing as npt + +from ladder.components.inference.api.core.observation import EFFECTOR_NAMES + + +class ArmActionType(str, Enum): + NONE = "none" + JOINT_ANGLES = "joint_angles" + JOINT_VELOCITIES = "joint_velocities" + END_EFFECTOR_POSE = "end_effector_pose" + + +class EndEffectorPoseDeltaMode(str, Enum): + ROT_EEF_TRANS_BASE = "rot_eef_trans_base" + ROT_EEF_TRANS_EEF = "rot_eef_trans_eef" + ROT_EEF_BASE_ALIGNED_TRANS_BASE = "rot_eef_base_aligned_trans_base" + + +@dataclass(kw_only=True) +class ArmAction: + type: ArmActionType = ArmActionType.NONE + + def __post_init__(self): + assert self.type != ArmActionType.NONE + + @staticmethod + def from_dict(arm_action_dict: Dict[str, Any]) -> "ArmAction": + try: + action_type = ArmActionType(arm_action_dict["type"]) + except KeyError: + raise ValueError("Action type not specified.") + except ValueError: + raise ValueError(f"Invalid action type. {arm_action_dict['type']}") + + try: + action_cls = { + ArmActionType.JOINT_ANGLES: JointAnglesAction, + ArmActionType.JOINT_VELOCITIES: JointVelocitiesAction, + ArmActionType.END_EFFECTOR_POSE: EndEffectorPoseAction, + }.get(action_type) + except KeyError: + raise ValueError(f"No action class found for action type {action_type}") + return action_cls(**arm_action_dict) + + +@dataclass(kw_only=True) +class JointAnglesAction(ArmAction): + type: ArmActionType = ArmActionType.JOINT_ANGLES + angles: npt.NDArray[np.float32] # [num_dofs] + + +@dataclass(kw_only=True) +class JointVelocitiesAction(ArmAction): + type: ArmActionType = ArmActionType.JOINT_VELOCITIES + velocities: npt.NDArray[np.float32] # [num_dofs] + + +@dataclass(kw_only=True) +class EndEffectorPoseAction(ArmAction): + """ + The end effector absolute pose is always with respect + to the robot base frame. + """ + + type: ArmActionType = ArmActionType.END_EFFECTOR_POSE + translation: npt.NDArray[np.float32] # [3] + rotation: npt.NDArray[np.float32] # [3, 3] + + def __post_init__(self): + super().__post_init__() + assert self.rotation.shape == (3, 3) + assert self.translation.shape == (3,) + + def to_homogeneous_matrix(self) -> npt.NDArray[np.float32]: + return np.block([[self.rotation, self.translation.reshape(3, 1)], [0, 0, 0, 1]]) + + @staticmethod + def from_homogeneous_matrix(matrix: npt.NDArray[np.float32]) -> "EndEffectorPoseAction": + return EndEffectorPoseAction(translation=matrix[:3, 3], rotation=matrix[:3, :3]) + + def apply_delta( + self, + delta_rotation: npt.NDArray[np.float32], + delta_translation: npt.NDArray[np.float32], + delta_mode: str | EndEffectorPoseDeltaMode, + ) -> "EndEffectorPoseAction": + """ + Constructs a new pose action by applying a delta rotation and a delta translation + (in the specified reference frame) to the current end effector pose. + + In the following we refer to `self.rotation` and `self.translation` as R_be and t_be. + + Let [R_be, t_be] be the current rotation and translation of the end effector frame with respect + to the robot base frame, i.e. the arguments passed `apply_delta`. + + Case 1: delta_mode == EndEffectorPoseDeltaMode.ROT_EEF_TRANS_BASE + + The delta pose is [DR, Dt], where DR represents a rotations in the current end effector frame and + Dt represents a translation in the robot base frame. + Then, `apply_delta` will return: + + [R, t] = [R_be @ DR, t_be + Dt] + + Case 2: translation_reference_frame == EndEffectorPoseDeltaMode.ROT_EEF_TRANS_EEF + + The delta pose is [R_e_e1, t_e], where t_e represents a translation in the end effector frame. + Then, `apply_delta` will return: + + [R, t] = [R_be @ R_e_e1, t_be + R_be @ t_e] + + Case 3: delta_mode == EndEffectorPoseDeltaMode.ROT_EEF_BASE_ALIGNED_TRANS_BASE + + The delta pose is [DR, Dt], where DR represents a rotation in the axis aligned end effector + frame and Dt represents a translation in the robot base frame. + + Then, `apply_delta` will return: + + [R, t] = [DR @ R_be, t_be + Dt] + + Args: + delta_rotation: The delta rotation of the end effector. + delta_translation: The delta translation of the end effector. + delta_mode: The mode of the delta pose. + + Returns: + A new pose action representing the end effector pose after applying the delta pose. + """ + delta_mode = EndEffectorPoseDeltaMode(delta_mode) + + if delta_mode == EndEffectorPoseDeltaMode.ROT_EEF_TRANS_BASE: + rotation = self.rotation @ delta_rotation + translation = delta_translation + self.translation + elif delta_mode == EndEffectorPoseDeltaMode.ROT_EEF_TRANS_EEF: + rotation = self.rotation @ delta_rotation + translation = self.rotation @ delta_translation + self.translation + elif delta_mode == EndEffectorPoseDeltaMode.ROT_EEF_BASE_ALIGNED_TRANS_BASE: + rotation = delta_rotation @ self.rotation + translation = delta_translation + self.translation + else: + raise ValueError(f"Invalid delta mode: {delta_mode}") + return EndEffectorPoseAction(translation=translation, rotation=rotation) + + +class RobotActionType(str, Enum): + NONE = "none" + SINGLE_MANIPULATOR = "single_manipulator" + DUAL_MANIPULATOR = "dual_manipulator" + + +@dataclass(kw_only=True) +class RobotAction: + type: RobotActionType = RobotActionType.NONE + + def __post_init__(self): + assert self.type != RobotActionType.NONE + + @staticmethod + def from_dict(action_dict: Dict[str, Any]) -> "RobotAction": + try: + action_type = RobotActionType(action_dict["type"]) + except KeyError: + raise ValueError("Robot action type not specified.") + except ValueError: + raise ValueError(f"Invalid robot action type. {action_dict['type']}") + action_cls = { + RobotActionType.SINGLE_MANIPULATOR: ManipulatorAction, + RobotActionType.DUAL_MANIPULATOR: DualManipulatorAction, + }.get(action_type) + if action_cls is None: + raise ValueError(f"No action class found for robot action type {action_type}") + return action_cls.from_dict(action_dict) + + +@dataclass(kw_only=True) +class ManipulatorAction(RobotAction): + """A single arm + its gripper (the former RobotAction).""" + + type: RobotActionType = RobotActionType.SINGLE_MANIPULATOR + arm_action: Optional[ArmAction] = None + gripper_action: Optional[float] = None + + def __post_init__(self): + super().__post_init__() + if self.arm_action is None and self.gripper_action is None: + raise ValueError("At least arm_action or gripper_action must be specified.") + + @staticmethod + def from_dict(action_dict: Dict[str, Any]) -> "ManipulatorAction": + arm_action = None + if arm_action_dict := action_dict.get("arm_action", None): + arm_action = ArmAction.from_dict(arm_action_dict) + return ManipulatorAction( + arm_action=arm_action, + gripper_action=action_dict.get("gripper_action", None), + ) + + +@dataclass(kw_only=True) +class DualManipulatorAction(RobotAction): + type: RobotActionType = RobotActionType.DUAL_MANIPULATOR + right: ManipulatorAction + left: ManipulatorAction + + @staticmethod + def from_dict(action_dict: Dict[str, Any]) -> "DualManipulatorAction": + return DualManipulatorAction( + right=ManipulatorAction.from_dict(action_dict["right"]), + left=ManipulatorAction.from_dict(action_dict["left"]), + ) + + +def effector_actions(action: RobotAction) -> List[ManipulatorAction]: + """One entry per effector, right before left, mirroring + :func:`ladder.components.inference.api.core.observation.effector_states`.""" + if isinstance(action, DualManipulatorAction): + return [action.right, action.left] + assert isinstance(action, ManipulatorAction), f"{type(action).__name__} commands no manipulator" + return [action] + + +def named_effector_actions(action: RobotAction) -> List[Tuple[str, ManipulatorAction]]: + """:func:`effector_actions` paired with the effector names.""" + return list(zip(EFFECTOR_NAMES, effector_actions(action), strict=False)) diff --git a/ladder/components/inference/api/core/instructions.py b/ladder/components/inference/api/core/instructions.py new file mode 100644 index 0000000000000000000000000000000000000000..3158abb9d7fc0820d37be30502397628feed6722 --- /dev/null +++ b/ladder/components/inference/api/core/instructions.py @@ -0,0 +1,100 @@ +import dataclasses +from collections import OrderedDict +from typing import List, Optional, Sequence, Tuple + +IN = "in" +ON = "on" + +_NUMBER = { + 1: "one", + 2: "two", + 3: "three", + 4: "four", + 5: "five", + 6: "six", + 7: "seven", + 8: "eight", + 9: "nine", + 10: "ten", +} + + +@dataclasses.dataclass(frozen=True) +class Placement: + """One leg: which object, and where it has to end up.""" + + noun: str + preposition: str # ON or IN, per the receptacle + receptacle: str + + +def _join(phrases: Sequence[str]) -> str: + if len(phrases) == 1: + return phrases[0] + return ", ".join(phrases[:-1]) + " and " + phrases[-1] + + +def _quantify(noun: str, count: int, definite: bool = True) -> str: + if not definite: # the same type is wanted in two places, so "the spoon" would name one twice + return f"{_NUMBER[count]} {noun}" + ("" if count == 1 else "s") + if count == 1: + return f"the {noun}" + if count == 2: + return f"both {noun}s" + return f"the {_NUMBER.get(count, str(count))} {noun}s" + + +def _clauses(placements: Sequence[Placement]) -> List[Tuple[str, str, str]]: + """(objects, preposition, receptacle) per destination, in first-mention order.""" + grouped: "OrderedDict[Tuple[str, str], OrderedDict[str, int]]" = OrderedDict() + for p in placements: + key = (p.receptacle, p.preposition) + grouped.setdefault(key, OrderedDict()) + grouped[key][p.noun] = grouped[key].get(p.noun, 0) + 1 + spread = { + noun + for noun in {n for counts in grouped.values() for n in counts} + if sum(noun in counts for counts in grouped.values()) > 1 + } + return [ + (_join([_quantify(n, c, n not in spread) for n, c in counts.items()]), preposition, receptacle) + for (receptacle, preposition), counts in grouped.items() + ] + + +def compose( + placements: Sequence[Placement], + *, + sequential: bool = False, + source: Optional[str] = None, + excluded: Sequence[str] = (), +) -> str: + """A composition task's instruction, in the style all three benchmarks share. + + Objects are named except where the task hands the system objects to leave alone: resolving + "everything except the red block" against the scene is what a scope task tests, while a bare + "everything" reads the same at two objects as at ten. ``sequential`` keeps one clause per leg, + for a task scored on the order or on the route taken. + """ + if excluded: + counts: "OrderedDict[str, int]" = OrderedDict() + for noun in excluded: + counts[noun] = counts.get(noun, 0) + 1 + what = "everything except " + _join([_quantify(n, c) for n, c in counts.items()]) + _, preposition, receptacle = _clauses(placements)[0] + body = f"put {what} {preposition} the {receptacle}" + elif source is not None: + what, _, receptacle = _clauses(placements)[0] + body = f"move {what} from the {source} to the {receptacle}" + elif sequential: + steps = [f"put the {p.noun} {p.preposition} the {p.receptacle}" for p in placements] + body = steps[0] if len(steps) == 1 else "first " + ", then ".join(steps) + else: + body = _join([f"put {what} {prep} the {where}" for what, prep, where in _clauses(placements)]) + # A one-leg task's instruction IS its only subtask, so the two match rather than differ by a stop. + return f"{body}." if len(placements) == 1 else body + + +def subtask(placement: Placement) -> str: + """One commanded goal, the unit an orchestrator hands the policy.""" + return compose([placement]) diff --git a/ladder/components/inference/api/core/observation.py b/ladder/components/inference/api/core/observation.py new file mode 100644 index 0000000000000000000000000000000000000000..5ceacd561876a83d211301b36e8b0a7c4010d1d4 --- /dev/null +++ b/ladder/components/inference/api/core/observation.py @@ -0,0 +1,166 @@ +import dataclasses +import functools +from enum import Enum +from typing import Any, Dict, List, Optional, Tuple + +import numpy as np +import numpy.typing as npt + + +class RobotStateType(str, Enum): + NONE = "none" + SINGLE_MANIPULATOR = "single_manipulator" + DUAL_MANIPULATOR = "dual_manipulator" + + +@dataclasses.dataclass(kw_only=True, frozen=True) +class RobotState: + type: RobotStateType = RobotStateType.NONE + + def __post_init__(self): + assert self.type != RobotStateType.NONE + + @staticmethod + def from_dict(state_dict: Dict[str, Any]) -> "RobotState": + try: + state_type = RobotStateType(state_dict["type"]) + except KeyError: + raise ValueError("Robot state type not specified.") + except ValueError: + raise ValueError(f"Invalid robot state type. {state_dict['type']}") + state_cls = { + RobotStateType.SINGLE_MANIPULATOR: ManipulatorState, + RobotStateType.DUAL_MANIPULATOR: DualManipulatorState, + }.get(state_type) + if state_cls is None: + raise ValueError(f"No state class found for robot state type {state_type}") + return state_cls.from_dict(state_dict) + + +@dataclasses.dataclass(kw_only=True, frozen=True) +class ManipulatorState(RobotState): + """State of a single arm + its gripper (the former flat RobotObservation fields).""" + + type: RobotStateType = RobotStateType.SINGLE_MANIPULATOR + eef_pose: Optional[npt.NDArray[np.float32]] = None # [4, 4] + eef_pose_from_target: Optional[npt.NDArray[np.float32]] = None # [4, 4] + gripper_state: Optional[npt.NDArray[np.float32]] = None # [1] + joints: Optional[npt.NDArray[np.float32]] = None # [num_dofs] + + @staticmethod + def from_dict(state_dict: Dict[str, Any]) -> "ManipulatorState": + return ManipulatorState( + eef_pose=state_dict.get("eef_pose"), + eef_pose_from_target=state_dict.get("eef_pose_from_target"), + gripper_state=state_dict.get("gripper_state"), + joints=state_dict.get("joints"), + ) + + +@dataclasses.dataclass(kw_only=True, frozen=True) +class DualManipulatorState(RobotState): + type: RobotStateType = RobotStateType.DUAL_MANIPULATOR + right: ManipulatorState + left: ManipulatorState + + @staticmethod + def from_dict(state_dict: Dict[str, Any]) -> "DualManipulatorState": + return DualManipulatorState( + right=ManipulatorState.from_dict(state_dict["right"]), + left=ManipulatorState.from_dict(state_dict["left"]), + ) + + +EFFECTOR_NAMES: Tuple[str, ...] = ("right", "left") + + +def effector_states(state: RobotState) -> List[ManipulatorState]: + """One entry per effector. A single manipulator is one effector""" + if isinstance(state, DualManipulatorState): + return [state.right, state.left] + assert isinstance( + state, ManipulatorState + ), f"{type(state).__name__} carries no manipulator to drive a VLAM with" + return [state] + + +def named_effector_states(state: RobotState) -> List[Tuple[str, ManipulatorState]]: + """:func:`effector_states` paired with :data:`EFFECTOR_NAMES`, so a single arm is ``right``.""" + return list(zip(EFFECTOR_NAMES, effector_states(state), strict=False)) + + +@dataclasses.dataclass(kw_only=True, frozen=True) +class RobotObservation: + images: Optional[Dict[str, npt.NDArray[np.uint8]]] # images are [h, w, c] + state: RobotState + timestamps: Optional[npt.NDArray[np.float64]] # [1] + metadata: Dict[str, Any] = dataclasses.field(default_factory=dict) + + @staticmethod + def from_dict(obs_dict: Dict[str, Any]) -> "RobotObservation": + return RobotObservation( + images=obs_dict.get("images"), + state=RobotState.from_dict(obs_dict["state"]), + timestamps=obs_dict.get("timestamps"), + metadata=obs_dict.get("metadata", {}), + ) + + @functools.cached_property + def cameras(self) -> Dict[str, "CameraFrame"]: + """The depth cameras, decoded from the plain dicts that crossed the msgpack wire.""" + payload = self.metadata.get(METADATA_CAMERAS_KEY) or {} + return { + str(name): CameraFrame( + depth_m=np.asarray(raw["depth_m"], dtype=np.float32), + intrinsics=np.asarray(raw["intrinsics"], dtype=np.float64).reshape(3, 3), + base_from_camera=np.asarray(raw["base_from_camera"], dtype=np.float64).reshape(4, 4), + ) + for name, raw in payload.items() + } + + +# Camera frames in ``RobotObservation.metadata``, keyed by camera name. +METADATA_CAMERAS_KEY = "cameras" + + +def encode_camera(depth_m: Any, intrinsics: Any, base_from_camera: Any) -> Dict[str, Any]: + """One depth camera's frame and calibration, for ``METADATA_CAMERAS_KEY``. + + ``depth_m`` is metres along the optical axis (not ray length), NaN where the sensor returned + nothing. ``intrinsics`` is a 3x3 pinhole matrix in pixels, OpenCV convention (+Z forward, +Y + down). ``base_from_camera`` takes camera points into the frame the policy commands in. + """ + return { + "depth_m": np.asarray(depth_m, dtype=np.float32), + "intrinsics": np.asarray(intrinsics, dtype=np.float32).reshape(3, 3), + "base_from_camera": np.asarray(base_from_camera, dtype=np.float32).reshape(4, 4), + } + + +@dataclasses.dataclass(frozen=True) +class CameraFrame: + depth_m: np.ndarray # [H, W] float32 metres, NaN where invalid + intrinsics: np.ndarray # [3, 3] pinhole, pixels + base_from_camera: np.ndarray # [4, 4] camera frame into the canonical robot base frame + + @property + def shape(self) -> tuple: + return tuple(np.asarray(self.depth_m).shape[:2]) + + def unproject_camera(self, pixels_xy: np.ndarray) -> np.ndarray: + """Points in the camera's own frame, which is the frame a grasp sampler works in.""" + pixels = np.asarray(pixels_xy, dtype=np.float64).reshape(-1, 2) + depth = np.asarray(self.depth_m, dtype=np.float64) + columns = np.clip(np.round(pixels[:, 0]).astype(int), 0, depth.shape[1] - 1) + rows = np.clip(np.round(pixels[:, 1]).astype(int), 0, depth.shape[0] - 1) + z = depth[rows, columns] + intrinsics = np.asarray(self.intrinsics, dtype=np.float64) + fx, fy = intrinsics[0, 0], intrinsics[1, 1] + cx, cy = intrinsics[0, 2], intrinsics[1, 2] + camera = np.stack([(pixels[:, 0] - cx) * z / fx, (pixels[:, 1] - cy) * z / fy, z], axis=1) + return np.where((np.isfinite(z) & (z > 0.0))[:, None], camera, np.nan) + + def unproject(self, pixels_xy: np.ndarray) -> np.ndarray: + camera = self.unproject_camera(pixels_xy) + transform = np.asarray(self.base_from_camera, dtype=np.float64) + return camera @ transform[:3, :3].T + transform[:3, 3] diff --git a/ladder/components/inference/api/core/privileged_geometry.py b/ladder/components/inference/api/core/privileged_geometry.py new file mode 100644 index 0000000000000000000000000000000000000000..e8194b49d577cee2fbf650231a8ded63de84f55f --- /dev/null +++ b/ladder/components/inference/api/core/privileged_geometry.py @@ -0,0 +1,218 @@ +from typing import Any, Dict, List, Optional, Sequence, Tuple + +import numpy as np + +from ladder.components.inference.api.core.privileged_state import ( + encode_condition, + encode_object, + encode_state, +) + +_NEIGHBOUR_GAP_M = 0.005 # so two objects sharing a destination do not touch +_IN_CLEARANCE_M = 0.005 # drop height above a container's floor + + +def to_base(world_to_base: np.ndarray, points: np.ndarray) -> np.ndarray: + """Map world-frame points ``[..., 3]`` into the robot base frame the policy commands in.""" + flat = np.asarray(points, dtype=np.float64).reshape(-1, 3) + homogeneous = np.concatenate([flat, np.ones((len(flat), 1))], axis=1) + return (homogeneous @ np.asarray(world_to_base, dtype=np.float64).T)[:, :3].reshape(points.shape) + + +def spread_offsets( + half_extents: Sequence[np.ndarray], dest_half: np.ndarray +) -> Tuple[np.ndarray, np.ndarray]: + """Where to lay N objects on one destination, as XY offsets plus the layer each one sits in. + + Piling is a real failure mode: every arrival dropped on the same point knocks the last one off. + The constraint is each object's FOOTPRINT, not its centre. A grid that only keeps centres + inside the destination still aims every object so its own body overhangs the rim, which for a + container means setting it down on the wall, where a delivery is credited and then lost. + + When a grid sized for the largest object cannot hold them all, each object's own footprint is + laid in rows along the destination's longer side instead, and what still does not fit piles on + the floor. Stacked layers are avoided because they aim each later arrival higher above the + destination, and an object dropped from that height bounces out of a container. + """ + count = len(half_extents) + offsets = np.zeros((count, 2)) + layers = np.zeros(count, dtype=int) + if count <= 1: + return offsets, layers + + source_half = np.array([max(h[0] for h in half_extents), max(h[1] for h in half_extents)]) + step = 2.0 * source_half + _NEIGHBOUR_GAP_M + # The centre of an object at the extreme cell must sit this far in for its edge to clear the rim. + usable = np.maximum(np.asarray(dest_half[:2], dtype=np.float64) - _NEIGHBOUR_GAP_M - source_half, 0.0) + cells = np.maximum(np.floor(2.0 * usable / step).astype(int) + 1, 1) + + per_layer = int(cells[0] * cells[1]) + if count > per_layer: + return _row_offsets(half_extents, np.asarray(dest_half[:2], dtype=np.float64)), layers + for index in range(count): + cell, layers[index] = index % per_layer, index // per_layer + grid_xy = np.array([cell % cells[0], cell // cells[0]], dtype=np.float64) + offsets[index] = (grid_xy - (cells - 1) / 2.0) * step + return offsets, layers + + +def _row_offsets(half_extents: Sequence[np.ndarray], dest_half: np.ndarray) -> np.ndarray: + """XY offsets laying each object's own footprint in rows along ``dest_half``'s longer axis.""" + along = 0 if dest_half[0] >= dest_half[1] else 1 + across = 1 - along + length = 2.0 * float(dest_half[along]) - _NEIGHBOUR_GAP_M + rows: List[List[int]] = [[]] + used = 0.0 + for index, half in enumerate(half_extents): + width = 2.0 * float(half[along]) + _NEIGHBOUR_GAP_M + if rows[-1] and used + width > length: + rows.append([]) + used = 0.0 + rows[-1].append(index) + used += width + depths = [max(2.0 * float(half_extents[i][across]) for i in row) + _NEIGHBOUR_GAP_M for row in rows] + offsets = np.zeros((len(half_extents), 2)) + row_start = -0.5 * sum(depths) + for row, depth in zip(rows, depths, strict=True): + widths = [2.0 * float(half_extents[i][along]) + _NEIGHBOUR_GAP_M for i in row] + start = -0.5 * sum(widths) + for index, width in zip(row, widths, strict=True): + offsets[index, along] = start + 0.5 * width + offsets[index, across] = row_start + 0.5 * depth + start += width + row_start += depth + # Rows past the destination's edge are pulled back in: those objects pile on the others. + limit = np.maximum(dest_half[None, :] - np.asarray(half_extents)[:, :2] - _NEIGHBOUR_GAP_M, 0.0) + return np.clip(offsets, -limit, limit) + + +def goal_point( + relation: str, + dest_center: np.ndarray, + dest_min: np.ndarray, + dest_max: np.ndarray, + source_half: np.ndarray, + offset_xy: np.ndarray, + layer: int = 0, +) -> Tuple[np.ndarray, float]: + """Where a source object's CENTRE must end up, and the EEF height to release it at. + + ``on`` rests it on the destination's top face; ``in`` drops it just above the floor of the + container, which is what ``is_inside`` scores. Those differ for a walled destination: the object + must END at the floor, but it must be RELEASED above the rim, since carrying it down inside fouls the + gripper on the wall and on anything already delivered. Releasing above the rim lets it fall the + last centimetres, which is how the container gets loaded. + """ + # "on" rests the object on the top face; "in" aims just above the container's AABB floor, + # which ``is_inside`` credits. + rim_z = float(dest_max[2]) + float(source_half[2]) + z = rim_z + if relation == "in": + z = float(dest_min[2]) + float(source_half[2]) + _IN_CLEARANCE_M + # A stacked layer rests on the layer below it. + z += layer * 2.0 * float(source_half[2]) + point = np.array([dest_center[0] + offset_xy[0], dest_center[1] + offset_xy[1], z], dtype=np.float32) + return point, max(rim_z, z) + + +def build_payloads( + *, + names: Sequence[str], + centers: np.ndarray, + mins: np.ndarray, + maxs: np.ndarray, + grasped: np.ndarray, + goals: Sequence[Dict[str, Any]], + n_envs: int, + long_axis_yaws: Optional[np.ndarray] = None, +) -> List[Dict[str, Any]]: + """One privileged payload per env. Arrays are ``[n_objects, n_envs, 3]`` in the base frame. + + Each entry of ``goals`` is ``{"source": idx, "destination": idx | None, "relation": str, + "satisfied": [n_envs] bool}``, optionally with ``"predicate"`` (default ``"pick_and_place"``; + ``pickup``/``grasp``/``reach`` end the motion early) and ``"point"``, an explicit goal + position, ``[3]`` or ``[n_envs, 3]``, for a goal whose destination is a place rather than an + object, which is how a region or a relative-placement task states where the object must end up. + """ + halves = 0.5 * (maxs - mins) + per_destination: Dict[Optional[int], List[int]] = {} + for index, goal in enumerate(goals): + per_destination.setdefault(goal["destination"], []).append(index) + + last_destination = {goal["source"]: goal["destination"] for goal in goals} + + payloads = [] + for env in range(n_envs): + offsets = {} + for destination, indices in per_destination.items(): + if destination is None: + continue + # One slot per object, not per goal: a relay or revisit brings the same object back to + # the same destination, and a slot of its own per visit aims it at the rim. Only objects + # left here need slots side by side; when every visitor moves on again (a relay's pot), + # they are there one at a time, so each takes the centre. + sources = list(dict.fromkeys(goals[i]["source"] for i in indices)) + if not any(last_destination[source] == destination for source in sources): + offsets.update({i: (np.zeros(2), 0) for i in indices}) + continue + spread, layers = spread_offsets( + [halves[source, env] for source in sources], halves[destination, env] + ) + slots = dict(zip(sources, zip(spread, layers, strict=True), strict=True)) + offsets.update({i: slots[goals[i]["source"]] for i in indices}) + + conditions = [] + for index, goal in enumerate(goals): + source = goal["source"] + target = encode_object( + names[source], + centers[source, env], + mins[source, env], + maxs[source, env], + display_name=names[source], + grasped=bool(grasped[source, env]), + long_axis_yaw_rad=(None if long_axis_yaws is None else float(long_axis_yaws[source, env])), + ) + destination = goal["destination"] + point, release_z = None, None + explicit = goal.get("point") + if explicit is not None: + explicit = np.asarray(explicit, dtype=np.float64) + point = explicit[env] if explicit.ndim == 2 else explicit + release_z = float(point[2]) + elif destination is not None: + point, release_z = goal_point( + goal["relation"], + centers[destination, env], + mins[destination, env], + maxs[destination, env], + halves[source, env], + offsets[index][0], + layer=int(offsets[index][1]), + ) + conditions.append( + encode_condition( + goal.get("predicate", "pick_and_place"), + target, + relation=goal["relation"], + goal_point_m=point, + release_z_m=release_z, + destination=(names[destination] if destination is not None else None), + grasped=bool(grasped[source, env]), + satisfied=bool(goal["satisfied"][env]), + ) + ) + objects = [ + encode_object( + name, + centers[i, env], + mins[i, env], + maxs[i, env], + display_name=name, + grasped=bool(grasped[i, env]), + long_axis_yaw_rad=(None if long_axis_yaws is None else float(long_axis_yaws[i, env])), + ) + for i, name in enumerate(names) + ] + payloads.append(encode_state(conditions, objects)) + return payloads diff --git a/ladder/components/inference/api/core/privileged_state.py b/ladder/components/inference/api/core/privileged_state.py new file mode 100644 index 0000000000000000000000000000000000000000..049e199c599b005a4ccb97f5c0093c8ba1fa5410 --- /dev/null +++ b/ladder/components/inference/api/core/privileged_state.py @@ -0,0 +1,192 @@ +import dataclasses +from typing import Any, Dict, List, Optional + +import numpy as np + +# Key under which the privileged payload lives in ``observation.metadata``. +METADATA_PRIVILEGED_KEY = "privileged" + + +@dataclasses.dataclass(frozen=True) +class PrivilegedObject: + """A scene object's ground-truth geometry in the canonical base frame.""" + + name: str + center_m: np.ndarray # [3] object center (canonical frame) + aabb_min_m: np.ndarray # [3] axis-aligned bounding-box min corner + aabb_max_m: np.ndarray # [3] axis-aligned bounding-box max corner + # Name as the instructions refer to it, for grounding a free-text subtask. + display_name: str = "" + grasped: bool = False # privileged: is THIS object currently grasped + # Direction of the object's longest horizontal extent (radians about base Z, modulo pi), where the + # server publishes it. The AABB cannot say it for an object lying diagonally. + long_axis_yaw_rad: Optional[float] = None + + @property + def top_z_m(self) -> float: + """Z of the object's top surface (for placing another object on it).""" + return float(self.aabb_max_m[2]) + + @property + def half_extent_m(self) -> np.ndarray: + return 0.5 * (self.aabb_max_m - self.aabb_min_m) + + @property + def height_m(self) -> float: + return float(self.aabb_max_m[2] - self.aabb_min_m[2]) + + +@dataclasses.dataclass(frozen=True) +class PrivilegedCondition: + """One success condition with its target/goal resolved to ground truth. + + A canonical (single-condition) task has exactly one of these; a long-horizon + task has one per sub-goal, evaluated in order against ``satisfied``. + """ + + predicate: str # e.g. "pick_and_place", "stack", "reach", "pickup" + relation: Optional[str] # "on" | "in" | "left" | ... | None + target: PrivilegedObject + # Where the target's centre must end up, base frame. None for reach/grasp/pickup. + goal_point_m: Optional[np.ndarray] + grasped: bool # privileged: is the target currently grasped + satisfied: bool # privileged: does this condition currently hold + # Lowest release height clearing the destination's walls; above the rim for a walled one, + # so the object falls in rather than the gripper reaching down inside. + release_z_m: Optional[float] = None + # The scene object the target must end up on or in, by name; None for a goal stated as a point. + destination: Optional[str] = None + + +@dataclasses.dataclass(frozen=True) +class PrivilegedState: + conditions: List[PrivilegedCondition] + # Every scene object, for grounding a subtask in ``subtask`` mode. Empty when structured. + objects: List[PrivilegedObject] = dataclasses.field(default_factory=list) + + def first_unsatisfied(self) -> Optional[PrivilegedCondition]: + """The first condition not yet satisfied (drives sequential execution).""" + for condition in self.conditions: + if not condition.satisfied: + return condition + return None + + def find_object(self, phrase: str) -> Optional[PrivilegedObject]: + """Resolve a noun phrase from a subtask to a scene object by display name. + + Prefers an exact display-name match, then the longest display name contained in the phrase + (so "green block" beats "block"). Returns ``None`` if nothing matches. + """ + text = phrase.strip().lower() + exact = [o for o in self.objects if o.display_name and o.display_name.lower() == text] + if exact: + return exact[0] + contained = [o for o in self.objects if o.display_name and o.display_name.lower() in text] + if contained: + return max(contained, key=lambda o: len(o.display_name)) + return None + + +def _arr(value: Any) -> np.ndarray: + return np.asarray(value, dtype=np.float64).reshape(-1) + + +def _object_from_dict(data: Dict[str, Any]) -> PrivilegedObject: + return PrivilegedObject( + name=str(data["name"]), + center_m=_arr(data["center_m"]), + aabb_min_m=_arr(data["aabb_min_m"]), + aabb_max_m=_arr(data["aabb_max_m"]), + display_name=str(data.get("display_name", "")), + grasped=bool(data.get("grasped", False)), + long_axis_yaw_rad=( + float(data["long_axis_yaw_rad"]) if data.get("long_axis_yaw_rad") is not None else None + ), + ) + + +def from_metadata(metadata: Optional[Dict[str, Any]]) -> Optional[PrivilegedState]: + """Parse the privileged payload from observation metadata, or ``None`` if absent. + + ``None`` lets the policy fail loudly when privileged state was expected but the + server did not emit it (a misconfiguration), rather than silently no-op. + """ + if not metadata: + return None + payload = metadata.get(METADATA_PRIVILEGED_KEY) + if not payload: + return None + conditions: List[PrivilegedCondition] = [] + for raw in payload["conditions"]: + goal = raw.get("goal_point_m") + conditions.append( + PrivilegedCondition( + predicate=str(raw["predicate"]), + relation=(str(raw["relation"]) if raw.get("relation") is not None else None), + target=_object_from_dict(raw["target"]), + goal_point_m=(_arr(goal) if goal is not None else None), + release_z_m=(float(raw["release_z_m"]) if raw.get("release_z_m") is not None else None), + destination=(str(raw["destination"]) if raw.get("destination") else None), + grasped=bool(raw["grasped"]), + satisfied=bool(raw["satisfied"]), + ) + ) + objects = [_object_from_dict(o) for o in payload.get("objects", [])] + return PrivilegedState(conditions=conditions, objects=objects) + + +def encode_object( + name: str, + center_m: np.ndarray, + aabb_min_m: np.ndarray, + aabb_max_m: np.ndarray, + *, + display_name: str = "", + grasped: bool = False, + long_axis_yaw_rad: Optional[float] = None, +) -> Dict[str, Any]: + """Build the msgpack-safe dict for one object (poses in canonical frame).""" + encoded = { + "name": str(name), + "center_m": np.asarray(center_m, dtype=np.float32).reshape(3), + "aabb_min_m": np.asarray(aabb_min_m, dtype=np.float32).reshape(3), + "aabb_max_m": np.asarray(aabb_max_m, dtype=np.float32).reshape(3), + "display_name": str(display_name), + "grasped": bool(grasped), + } + if long_axis_yaw_rad is not None and np.isfinite(long_axis_yaw_rad): + encoded["long_axis_yaw_rad"] = float(long_axis_yaw_rad) + return encoded + + +def encode_condition( + predicate: str, + target: Dict[str, Any], + *, + relation: Optional[str] = None, + goal_point_m: Optional[np.ndarray] = None, + release_z_m: Optional[float] = None, + destination: Optional[str] = None, + grasped: bool = False, + satisfied: bool = False, +) -> Dict[str, Any]: + """Build the msgpack-safe dict for one success condition.""" + return { + "predicate": str(predicate), + "relation": (str(relation) if relation is not None else None), + "target": target, + "goal_point_m": ( + None if goal_point_m is None else np.asarray(goal_point_m, dtype=np.float32).reshape(3) + ), + "release_z_m": (None if release_z_m is None else float(release_z_m)), + "destination": (str(destination) if destination is not None else None), + "grasped": bool(grasped), + "satisfied": bool(satisfied), + } + + +def encode_state( + conditions: List[Dict[str, Any]], objects: Optional[List[Dict[str, Any]]] = None +) -> Dict[str, Any]: + """Build the full privileged payload for ``metadata[METADATA_PRIVILEGED_KEY]``.""" + return {"conditions": list(conditions), "objects": list(objects or [])} diff --git a/ladder/components/inference/api/core/request_reply.py b/ladder/components/inference/api/core/request_reply.py new file mode 100644 index 0000000000000000000000000000000000000000..48ec5c0a9a26af8a7d8b87e564b8114b1ddc078a --- /dev/null +++ b/ladder/components/inference/api/core/request_reply.py @@ -0,0 +1,262 @@ +from abc import abstractmethod +from typing import Any, Dict, Optional, TypeVar + +import zmq +from typing_extensions import Protocol + +from ladder.components.inference.api.core.requests import RequestType, parse_request_type +from ladder.components.inference.api.core.serialization import deserialize, serialize +from ladder.core.common.logger import get_logger +from ladder.core.databib.config import Config, Configurable + +LOGGER = get_logger(__name__) + + +class CallbackProtocol(Protocol): + def __call__(self, message: Dict) -> Dict: + ... + + +class RequestReplyServerConfig(Config): + pass + + +RequestReplyServerConfigT = TypeVar('RequestReplyServerConfigT', bound=RequestReplyServerConfig) + + +class RequestReplyServer(Configurable[RequestReplyServerConfigT]): + """ + Abstract base class for a request-reply server that listens + for requests, dispatches them to the provided callback, and + sends the output back to the client. + + Methods: + start(on_request_cb: CallbackProtocol): Begins listening for requests + """ + + @abstractmethod + def start(self, on_request_cb: CallbackProtocol) -> None: + pass + + +class RequestReplyClientConfig(Config): + pass + + +RequestReplyClientConfigT = TypeVar('RequestReplyClientConfigT', bound=RequestReplyClientConfig) + + +class RequestReplyClient(Configurable[RequestReplyClientConfigT]): + """ + Abstract base class for a request-reply client that sends + requests to the server and receives the output from the server. + + Methods: + send(request: Dict) -> Optional[Dict]: Send a request to the server. + """ + + @abstractmethod + def send(self, request: Dict[str, Any]) -> Optional[Dict[str, Any]]: + pass + + +class ZmqRequestReplyServerConfig(RequestReplyServerConfig): + port: int # Port number to listen on + recv_timeout_ms: int = 5000 # Timeout in milliseconds for calls to socket.recv + + +class ZmqRequestReplyServer(RequestReplyServer[ZmqRequestReplyServerConfig]): + """ + ZeroMQ-based implementation of RequestReplyServer. + + Methods: + start(on_request_cb: CallbackProtocol): Begins listening for requests. + + Notes: + Designed to be used in conjunction with the ZmqRequestReplyClient class. + """ + + def __init__(self, config: ZmqRequestReplyServerConfig): + super().__init__(config) + self._context = zmq.Context() + self._init_socket() + + def _init_socket(self): + self.socket = self._context.socket(zmq.REP) + self.socket.bind(f"tcp://*:{self.config.port}") + self.socket.setsockopt(zmq.SNDHWM, 5) + self.socket.setsockopt(zmq.LINGER, 0) + self.socket.setsockopt(zmq.RCVTIMEO, self.config.recv_timeout_ms) + + def start(self, on_request_cb: CallbackProtocol) -> None: + """ + Starts listening for requests over the socket, blocking on the current thread. + When a request is received, the on_request_cb function is called with the request + as an argument, and the output is sent back as a reply to the client. + + Args: + on_request_cb (CallbackProtocol): Function to be called when a request is received. + + Notes: this function is blocking + """ + LOGGER.info("Starting ZMQ request reply server") + while True: + LOGGER.debug("Waiting for new request") + try: + message = self.socket.recv() + except zmq.Again: + LOGGER.warn("ZMQ receive timeout. Continuing.") + continue + except zmq.ZMQError as e: + LOGGER.error(f"A ZMQ error occured while receiving. Error: {e}") + raise e + LOGGER.debug(f"Received new request") + + LOGGER.debug(f"Processing new request") + message = deserialize(message) + res = on_request_cb(message) + res = serialize(res) + LOGGER.debug(f"Request processed") + + LOGGER.debug(f"Sending reply") + try: + self.socket.send(res) + except zmq.ZMQError as e: + LOGGER.error(f"A ZMQ error occured while sending. Error: {e}") + raise e + LOGGER.debug(f"Reply sent") + + # If the request is an exit request, we stop the loop and + # let the process exit gracefully. + if parse_request_type(message) == RequestType.EXIT: + LOGGER.info("Received EXIT request. Stopping server.") + break + + def __del__(self): + if self.socket: + self.socket.close() + self._context.term() + + +class ZmqRequestReplyClientConfig(RequestReplyClientConfig): + ip: str # IP address of the server + port: int # Port number of the server + recv_timeout_ms: int = 15000 # Timeout in milliseconds for calls to socket.recv + send_num_retries: int = 3 # Number of times to retry sending a message + # If True, recv timeouts are raised as ``zmq.Again`` to the caller after the broken + # REQ socket has been reset. Use when a timeout indicates the worker should fail and + # be respawned by its supervisor (e.g. a remote inference client). Default False + # keeps the "log and return None" behaviour. + propagate_timeout: bool = False + + +class ZmqRequestReplyClient(RequestReplyClient[ZmqRequestReplyClientConfig]): + """ + ZeroMQ-based implementation of RequestReplyClient. + + It communicates with the server using a ZeroMQ in REQ mode. + + Methods: + send(request: dict) -> Optional[Dict]: Send a request to the server and + receive a reply. + + Notes: + This class is designed to be used in conjunction with the + ZmqRequestReplyServer class. + """ + + def __init__(self, config: ZmqRequestReplyClientConfig): + super().__init__(config) + self._closed = False + self.context = zmq.Context() + self.socket = None + self._init_socket() + + def _init_socket(self): + """ + Reset the socket connection, this is needed when REQ is in a broken state. + """ + if self.socket: + LOGGER.info(f"Closing and deleting existing socket") + self.socket.close() + del self.socket + self.socket = self.context.socket(zmq.REQ) + self.socket.setsockopt(zmq.RCVTIMEO, self.config.recv_timeout_ms) + # this option is needed, among other things, to prevent context.term() + # from handing if socket.recv() is interrupted e.g. by a KeyboardInterrupt + self.socket.setsockopt(zmq.LINGER, 0) + + LOGGER.info(f"Connecting to {self.config.ip}:{self.config.port}") + self.socket.connect(f"tcp://{self.config.ip}:{self.config.port}") + LOGGER.info(f"Connected to {self.config.ip}:{self.config.port}") + + def send(self, request: Dict[str, Any]) -> Optional[Dict[str, Any]]: + if self.socket is None or self.socket.closed: + LOGGER.error(f"The socket is closed, request not sent.") + return None + + try: + LOGGER.debug(f"Sending request") + sent = False + request_serialized = serialize(request) + for i in range(self.config.send_num_retries): + try: + self.socket.send(request_serialized) + sent = True + LOGGER.debug(f"Request sent.") + break + except zmq.ZMQError as e: + LOGGER.error( + f"Failed to send message to {self.config.ip}:{self.config.port}. Error: {e}." + ) + + LOGGER.debug(f"Attempt {i} failed. Retrying.") + + if not sent: + LOGGER.warn("Number of retries exceeded. Aborting.") + return None + + # If the request is an exit request, we don't wait for a reply. + if request.get("type") == RequestType.EXIT: + return None + + LOGGER.debug(f"Waiting for reply") + try: + message = self.socket.recv() + except zmq.Again: + # REQ socket is now stuck in "waiting for reply", so any subsequent send + # would fail with ZMQError. Reset before deciding how to surface the timeout. + self._init_socket() + if self.config.propagate_timeout: + raise + LOGGER.warn("ZMQ receive timeout. Aborting.") + return None + except zmq.ZMQError as e: + LOGGER.error(f"A ZMQ error occured while receiving. Error: {e}.") + raise e + + return deserialize(message) + + except KeyboardInterrupt: + # Ctrl+C might be sent mid-episode. In this case, if the exception is raised inside the try + # block, the socket will fall into a broken state. Here we reinitialize it to keep the + # communication working. + self._init_socket() + raise + + def close(self) -> None: + """Idempotently tear down the socket and ZMQ context. Safe to call multiple times.""" + if self._closed: + return + if self.socket: + try: + self.socket.disconnect(f"tcp://{self.config.ip}:{self.config.port}") + except zmq.ZMQError: + # Socket already disconnected (e.g. _init_socket re-created it after a timeout). + pass + self.socket.close() + self.context.term() + self._closed = True + + def __del__(self): + self.close() diff --git a/ladder/components/inference/api/core/requests.py b/ladder/components/inference/api/core/requests.py new file mode 100644 index 0000000000000000000000000000000000000000..d898fa8beb869558617ae43c804a3a96e23463ad --- /dev/null +++ b/ladder/components/inference/api/core/requests.py @@ -0,0 +1,21 @@ +from enum import Enum +from typing import Any, Dict + + +class RequestType(str, Enum): + RESET = "reset" + GET_OBSERVATIONS = "get_observations" + EXEC_ACTION = "exec_action" + CONNECT = "connect" + EXIT = "exit" + + +def parse_request_type(request: Dict[str, Any]) -> RequestType: + try: + request_type = RequestType(request["type"]) + except KeyError: + raise ValueError("Invalid request: request type not specified.") + except ValueError: + raise ValueError("Invalid request type.") + + return request_type diff --git a/ladder/components/inference/api/core/serialization.py b/ladder/components/inference/api/core/serialization.py new file mode 100644 index 0000000000000000000000000000000000000000..ef82a7d4cca4b341a114054d26cee2c139334635 --- /dev/null +++ b/ladder/components/inference/api/core/serialization.py @@ -0,0 +1,15 @@ +from typing import Any, Dict + +import msgpack +import msgpack_numpy + +# apply msgpack_numpy monkey-patch to enable automatic numpy serialization +msgpack_numpy.patch() + + +def serialize(data: Dict[str, Any]) -> bytes: + return msgpack.packb(data) + + +def deserialize(message: bytes) -> Dict[str, Any]: + return msgpack.unpackb(message) diff --git a/ladder/components/inference/robots/robolab/README.md b/ladder/components/inference/robots/robolab/README.md new file mode 100644 index 0000000000000000000000000000000000000000..d955b394c2309de168af8ec9820e5ef6dc9a235f --- /dev/null +++ b/ladder/components/inference/robots/robolab/README.md @@ -0,0 +1,167 @@ +# RoboLab benchmark integration + +Bazel-packages NVIDIA's [RoboLab](https://github.com/NVlabs/RoboLab) manipulation benchmark +(DROID/Franka), from [a fork](https://github.com/psi-robotics/RoboLab) that upgrades it to Isaac Sim +6.0.0.1 and Isaac Lab 3.0.0b2. + +- **`env_server.py`** serves RoboLab envs over + ladder's RobotServer ZMQ protocol to the standard `run_model` + `SimplerEpisodeRunner` stack, + so runner-side features (HLC configs, RecorderWrapper, dashboard-format outputs) + apply to RoboLab exactly as to RobotArena/SimplerEnv. One Isaac app serves the whole + (task x run) sweep. Model/behavior config: + `models/vlams/configs/robolab/vlam_robolab.yaml`. It lives with the other per-benchmark `run_model` + configs (`vlam_simpler.yaml`, `vlam_robot_arena_*.yaml`) because `run_model`'s bazel target + packages that directory and the config imports `vlam_simpler.yaml`. + +The launcher takes any policy, not only our checkpoints. Two variables cover one that holds its +own model (an external model under `models/external/`, or the scripted policy): `RUN_MODEL` names +the `run_model` target to launch, which is the model's own when it brings its own pip environment, +and `INFERENCE_SERVER=0` says there is nothing to serve. Everything else is identical, so a policy +either way is scored by the same checker, recorder and metric layer, and pairs with the HLC systems +below. + +RoboLab runs in a py3.12 + isaac env (numpy 2.x) and the model in the vlams torch env, so the +path from env to policy crosses one ZMQ socket into `run_model` (which talks to the shared +`batch_inference_server`), mirroring RobotArena's process layout. + +`droid_conversions.py` owns the DROID conventions (gripper inversion both directions, +maintained EE targets with `controls_on_targets`, `resize_with_pad(224)` matching RoboLab's own +client, and a `pytorch_kinematics` Franka FK/IK in `franka_kinematics.py` over ladder's panda +URDF (torch/numpy-2 native). RoboLab publishes `base_link`/`eef_frame` poses offset +from ladder's canonical `panda_link8`, so staying in joint space + the canonical flange frame +keeps the integration frame-convention-free. + +## Packaging (see `http-assets.MODULE.bazel`) + +- **Code**: `@robolab_repo`: the RoboLab source archive (sha256-pinned, code only). A `constants.py` + patch makes it resolve assets from `ROBOLAB_ASSET_DIR`. +- **Assets**: `@robolab_assets_archive`: the ~1 GB asset tree the tasks load, staged into + `assets/` by `copy_to_directory`. First build on a fresh machine fetches it once. +- **openpi client**: `@openpi_client_repo` vendors `openpi_client` as source (its `numpy<2` pin + conflicts with isaacsim's numpy 2.x; the source runs fine on 2.x). + +## Running an eval + +`scripts/vlam_robolab.sh` (env-var driven, ports auto-allocated). GPU node with RT cores; +Isaac Sim 6 needs driver 595. + +```bash +# one of our checkpoints, behind the shared inference server (the default) +SESSION_NAME=sess_... CKPT_NAME=model-checkpoint-ema-050000.pt \ +TASKS="BananaOnPlateTask" NUM_ENVS=8 NUM_RUNS=1 \ +./ladder/components/inference/robots/robolab/scripts/vlam_robolab.sh +# an external model -- one model per lane, no inference server +RUN_MODEL=models/external/openpi:run_model INFERENCE_SERVER=0 TASKS="comp_bin_2" \ +LADDER_CONFIG=ladder/components/inference/models/external/openpi/configs/robolab/composition/openpi_robolab_composition.yaml \ +./ladder/components/inference/robots/robolab/scripts/vlam_robolab.sh +``` + +Manual invocation of the individual pieces (the launcher does all of this for you): + +```bash +# env server: serves RoboLab envs over the RobotServer ZMQ protocol +bazel run //ladder/components/inference/robots/robolab:env_server -- \ + --port 5556 --task BananaOnPlateTask --num-envs 8 --num-runs 1 --headless +# dashboard export: map a run to episode_results.jsonl +bazel run //ladder/components/inference/robots/robolab:export_robolab_results -- \ + --episode-log /all/episode_log.yaml --output-dir +``` + +Runs write `results.yaml`, `episode_log.yaml` and per-env mp4s, and +(via `export_robolab_results.py`, run automatically at the end of a run) RoboLab's +`episode_results.jsonl`, so RoboLab's dashboard can read the results too. Env-side diagnostics (hdf5 +trajectories, SPARC/path-length metrics, event streams) are the one thing only RoboLab's own runner +ever wrote. + +An external model runs in its own pip environment, which bazel provisions, and holds one copy of +the weights per lane rather than one per card. That costs roughly a lane's worth of VRAM, so run +fewer lanes per card. Its own package documents its caches +(`models/external/openpi/README.md` for pi0.5). + +## Hierarchical eval systems + +The ladder path drives the shared HLC stack (`api/client/hlc`), the same one RobotArena and +SimplerEnv use, so a system is defined ONCE in `api/client/hlc/configs/` and each benchmark ships +one eval config per system, **named after it**. `LADDER_CONFIG` selects the system; the RoboLab +configs live in `models/vlams/configs/robolab/composition/` and each imports its shared system plus +this benchmark's plan cache and RoboLab's own timing: + +| `LADDER_CONFIG` (`…/configs/robolab/composition/`) | Shared system | Plan | Switches on | +|---|---|---|---| +| `vlam_robolab_composition.yaml` | none (flat, no HLC) | none (raw instruction) | n/a | +| `…_oracle.yaml` | `oracle` | from the env | per-goal `achieved` | +| `…_plan_gripper.yaml` | `plan_gripper` | cached | the gripper cycle | +| `…_plan_periodic.yaml` | `plan_periodic` | cached | a step clock (no feedback) | +| `…_plan_verifier.yaml` | `plan_verifier` | cached | a VLM check | +| `…_plan_verifier_interval.yaml` | `plan_verifier_interval` | cached | a VLM check on a clock | +| `…_plan_verifier_local.yaml` | `plan_verifier_local` | cached | a VLM check, in-process Qwen3-VL | +| `…_plan_verifier_local_interval.yaml` | `plan_verifier_local` | cached | the local check, on a clock | +| `…_plan_verifier_noretry.yaml` | `plan_verifier_noretry` | cached | a VLM check, no retry | +| `…_plan_verifier_retry.yaml` | `plan_verifier_retry` | cached | a VLM check, retry in place | +| `…_stepwise_gripper.yaml` | `stepwise_gripper` | none (one at a time) | the gripper cycle | +| `…_stepwise_interval.yaml` | `stepwise_interval` | none (one at a time) | a step clock | +| `…_volo.yaml` | `volo` | cached | VoLo's own tool loop | + +`…_plan_verifier_live.yaml` is the one file that is not a system: it is `_plan_verifier` with +`cache_path` omitted, so the VLM decomposes the reset frame. This is the pass that mints a plan for a newly +added task. `composition/language/` holds the same paraphrase probes the other two benchmarks ship. +This is the same set, file for file, as SimplerEnv's and RobotArena's composition configs, so a +system compares across all three. + +```bash +SESSION_NAME=sess_... CKPT_NAME=model-checkpoint-ema-015000.pt \ +LADDER_CONFIG=ladder/components/inference/models/vlams/configs/robolab/composition/vlam_robolab_composition_plan_verifier.yaml \ +TASKS="comp_bin_2" ./ladder/components/inference/robots/robolab/scripts/vlam_robolab.sh +``` + +**Plans are a config concern, not the launcher's.** Only the `_plan_*` systems read +`composition/decomp_cache.json` (zero API calls at eval time), via `hlc_config.source.cache_path`. +Flat and `_oracle` need no plans at all and the `_stepwise_*` systems hold none by design. The cache +sits beside the composition package exactly as it does on the other two benchmarks; RoboLab's is keyed +by task class name (`CompBin4Task`). + +The oracle needs no cache: `oracle.py` synthesizes the per-goal `metadata['oracle']` signal from the +task's OWN subtask spec: one `(command, check)` pair per score-carrying goal, rendered from the +goal's terminating predicate (verb + object/target nouns). It is the RoboLab-side analogue of SimplerEnv's +`oracle_subtasks`. Commands and `achieved` are built in one pass so `oracle.achieved[i]` lines up +with commanded subtask `i` by construction. + +RoboLab's timescale is the one thing that does not carry over from the other two benchmarks: a Franka +pick-and-place takes ~250 steps against ~50 on SimplerEnv/RA, so every advancer window +(`stuck_patience_steps`, `attempt_stall_steps`, `interval`, `verify_interval`) is set from RoboLab's +own measured median subtask rather than inherited. A window shorter than one placement abandons every +goal mid-reach and reads as a difficulty wall that is not there. + +## Throughput + +`NUM_ENVS` parallel Isaac envs are stacked into one batched model forward per decode, sharing the +fixed model cost across N episodes. Raise it for throughput (`MAX_BATCH_SIZE >= NUM_ENVS`). +Per-step wall is dominated by the upstream Isaac render, not the model. + +## Parity contract vs the native runner + +Validated against RoboLab's native runner: +clean-start success 86.4% ladder vs 86.7% native; frame-level camera/action-input equivalence +verified; the ladder path ran as fast as native or up to 30% faster. + +Identical by construction (any change here must keep both paths in lockstep): +- Task iteration = `run_evaluation`'s loop: `create_env` per task (same args, seed 0 default), + `reset_eval_state` between runs, double `env.reset()`, timeline pump before each step. +- Env-side semantics untouched: success predicates, frozen envs, per-task episode caps, + the ep_len<=2 physics-artifact re-reset. +- Model I/O: same frames (`resize_with_pad(224)`, both cameras), gripper flips + binarize, + maintained-EE-target + clamped IK (`droid_conversions.py`), same `ActionChunkingController` + cadence. + +Known, deliberate differences: +- Native-only features: `--num-episodes-adaptive` sampling; resume-skip of completed cells + (the ladder runner appends); hdf5 trajectories + event streams. Trajectory metrics + (`ee_sparc`, `joint_sparc_mean`, `ee_isj`, `joint_isj`, `ee_path_length`, `joint_rmse_mean`, + `ee_speed_max/mean`) are native-parity: the env server samples the same articulation state + the native hdf5 recorder writes and calls robolab's own `compute_episode_metrics` at each + env's terminal step, so `export_robolab_results.py` fills `metrics`; `events`/`timing` stay + empty. +- The ladder runner queries the model for frozen envs too (results-neutral; the batched + server amortizes it, and the ladder path is still faster than native overall). +- Ladder videos are the 224x224 letterboxed model-input frames (native records 720p half-scale); + each env's recording ends at its own terminal step. diff --git a/ladder/components/inference/robots/robolab/composition/README.md b/ladder/components/inference/robots/robolab/composition/README.md new file mode 100644 index 0000000000000000000000000000000000000000..05c03aae019cfe7ddfe235c36c3d6a3c97a49f10 --- /dev/null +++ b/ladder/components/inference/robots/robolab/composition/README.md @@ -0,0 +1,146 @@ +# RoboLab long-horizon composition benchmark + +Systematic long-horizon manipulation tasks for RoboLab, mirroring the extended SimplerEnv / RobotArena +composition benchmarks: the same task-type families (single-destination, split across receptacles, +distractors, ordered sequence) scaled along an N-goal axis (2 → 10), to measure how flat VLAs, +a privileged oracle, and the HLC orchestrators degrade as the horizon grows. + +## Design + +Every composition task is one `CompositionSpec` (`specs.py`): a subset of the object catalog +(`catalog.py`), each object assigned to a receptacle, with an optional ordering constraint. The N-goal +count is `len(goals)`. `factory.make_task(spec_name)` turns a spec into a RoboLab `Task` +subclass, and `tasks.py` defines one such class per spec. RoboLab addresses a single one of them as +`::` (`specs.task_target`), so a spec name is all a caller passes. + +## The task set + +**44 tasks, and `SPECS` is the whole benchmark.** There is no calibration tier, no easy/hard split +and no held-out set: one list, pinned name by name in `test/test_composition_benchmark.py`, so a task +cannot enter or leave the suite as a side effect of editing a generator. + +| family | tasks | goals | +|---|---|---| +| single pick-and-place (bin, plate) | 2 | 1 | +| single-destination into a bin | 3 | 2, 6, 10 | +| distractor (`everything except ...`) | 2 | 2, 4 | +| route (two receptacles, plate + bin) | 10 | 2-10 | +| split to a plate | 7 | 2-8 | +| ordered into a bin | 9 | 2-10 | +| relay (table → plate → bin) | 4 | 4, 6, 8, 10 | +| revisit | 3 | 3, 6, 9 | +| transfer (off the plate into the bin) | 2 | 2, 3 | +| identical instances (counting) | 2 | 2, 4 | + +Every task names the objects it asks for. A collective phrasing (`put everything in the bin`) is word +for word identical at two objects and at ten, so a ladder built on it measures how many placements +fit in the step budget rather than whether the instruction was understood. + +`FRUIT_LADDER` is the object order the N-goal tasks draw from, and the pomegranate is deliberately +last: it is the only object that both a DROID-trained checkpoint and π0.5 fail on, so at any +earlier position it would cap every longer rung by the asset rather than by the horizon. The rest of +the order is deliberately NOT ranked by any one model's per-object success rates, so the benchmark is +not fitted to a particular model under test. + +## Where objects start + +A graspable's distance from the robot base matters more than anything else in this suite: objects +far from the base are often out of comfortable reach. + +`_scatter` maximises the minimum separation between objects, which on its own pushes objects to the +far edge of `_OBJ_RECT` (0.83 m), out of comfortable reach. The pairwise constraint already +guarantees nothing overlaps at worst-case jitter, so every spec is also laid out against +`layout.REACH_MAX` (0.62 m) at worst-case jitter. Transfer and relay lay their objects out on a +receptacle, so their source is slid toward the base until it complies instead of being positioned by +hand. Worst start distance across the registry: 0.619 m. + +The bound is on an object's CENTRE, which makes it exactly invariant to the reset's full-yaw +randomization (yaw rotates an object about its own centre). Position jitter is handled by evaluating +at the far corner of the jitter box, so the bound holds for every episode rather than on average. The +price is per-episode variation at the long rungs: jitter tapers from 0.07 m at one to three goals down +to 0.02 m at eight to ten, since the tighter band holds less. + +`factory._author_usda` writes one scene USD per task into `$TMPDIR/robolab_composition_scenes/`: +a bare tabletop rig (`fixtures/table_oak.usd` + `fixtures/franka_table.usd`, ground plane, physics +material) with every graspable and receptacle attached as a direct child of `world` through a USD +**payload**. Nothing is hand-authored and no asset is re-vendored: the payloads point straight into +RoboLab's own `ASSET_DIR`, so the whole benchmark stays a ladder-side addition with no fork edit. + +Composing the scene rather than selecting movers out of a pre-made one is what makes the per-object +scoring possible: `import_scene` scrapes those direct children as single rigid bodies, so a contact +sensor resolves one body per name. A runtime spawn of a raw catalog USD nests its bodies instead, +which breaks the per-object contact filter every goal predicate is built on. + +The pools the specs draw from, both in `catalog.py`: + +| pool | entries | +|---|---| +| `CATALOG` movers | 10 produce (banana, two oranges, lemon, lime, avocado, pomegranate, lychee, onion, pumpkin) + 2 coloured blocks (red, blue), decoys only | +| `RECEPTACLE_TYPES` | `bin` (in), `bowl` (in), `plate` (on) | + +Only known-graspable produce is used as a mover: RoboLab's droid policy is grasp-floored on +blocks/cans, so a benchmark built from those would measure grasp, not orchestration. Every catalog +entry is verified single-rigid-body and confirmed to render (`objaverse/apple_01` is excluded because it +spawns invisible). + +## Why the oracle works for free + +A spec's `subtasks` is one `pick_and_place[_on_surface]` per receptacle group with `logical="all"`. +`oracle.build_oracle` expands a `logical="all"` group into one goal per object, so the privileged +plan is exactly the per-object N-goal sequence, with no per-scene oracle authoring. The `_plan_*` +systems read `composition/decomp_cache.json`, which carries an instruction-echo plan per task; +`_plan_verifier_live` needs no cache (it decomposes the reset frame live). + +## Inspecting + +`python3 -m ladder.components.inference.robots.robolab.composition.specs`, run from the repo root, +prints the full task manifest (name, N, family, scene, instruction). + +## Running an eval + +These run through the SAME launcher as RoboLab's own tasks: `TASKS` takes composition task **names** +(env_server resolves a `comp_*` name to its Task class in `tasks.py`), and +`LADDER_CONFIG` selects the hierarchical system, exactly the shape `SCENES`/`ENV_NAMES` + +`RA_CLIENT_CONFIG`/`SIMPLER_CONFIG` have on the RobotArena and SimplerEnv launchers. + +```bash +SESSION_NAME= CKPT_NAME=model-checkpoint-ema-015000.pt \ +LADDER_CONFIG=ladder/components/inference/models/vlams/configs/robolab/composition/vlam_robolab_composition_oracle.yaml \ +TASKS="comp_bin_6 comp_split_plate_4" \ +NUM_ENVS=8 NUM_RUNS=1 VIDEO=True \ +bash ladder/components/inference/robots/robolab/scripts/vlam_robolab.sh +``` + +`LADDER_CONFIG` selects the system: `vlam_robolab_composition.yaml` is flat (the model gets only the +composed instruction), and one `_` variant exists per hierarchical system in +`api/client/hlc/configs/`, named after it: `_oracle`, `_plan_gripper`, `_plan_periodic`, +`_plan_verifier`, `_plan_verifier_interval`, `_plan_verifier_local`, `_stepwise_gripper`, +`_stepwise_interval`. `_plan_verifier_live` is not a system of its own: it is `_plan_verifier` with +`cache_path` omitted, so the VLM decomposes the reset frame. Each writes to its own +`robolab_composition*/` directory. The full table, including the RoboLab-only variants +(`_plan_verifier_retry`, `_plan_verifier_noretry`), is in the +[package README](../README.md#hierarchical-eval-systems). + +**Task selection.** `TASKS` is the only env control. Pass a subset, or the whole registry: + +```bash +# every composition task, in registry order (44 of them) +TASKS=$(python3 -c 'from ladder.components.inference.robots.robolab.composition.specs \ + import SPECS; print(" ".join(SPECS))') + +# longest budget first, which shortens the tail: a lane owns a slice for the life of one Isaac app, +# so a lane dealt the long tasks finishes last +TASKS=$(python3 -c 'from ladder.components.inference.robots.robolab.composition.specs \ + import SPECS; print(" ".join(sorted(SPECS, key=lambda n: -SPECS[n].episode_length_s)))') +``` + +**Language probes.** Two independent knobs on the runner, the same pair the other two composition +benchmarks use, in `models/vlams/configs/robolab/composition/language/`. +`runner_config.instruction_override` rewords the TASK instruction, keyed by the env_name env_server +reports, i.e. the RoboLab Task CLASS name (`CompBin4Task`, not `comp_bin_4`); a key matching no task is +silently ignored, so check names against `task_class_name`. RoboLab additionally has an env-side instruction axis +(`INSTRUCTION_TYPE=collective|enumerated`, from each spec's own `instruction` dict), which is +independent of both. + +Per-episode `episode_meta.yaml` records the subtask switches (`instruction_log`), so the dashboard +shows the subtask timeline. diff --git a/ladder/components/inference/robots/robolab/composition/catalog.py b/ladder/components/inference/robots/robolab/composition/catalog.py new file mode 100644 index 0000000000000000000000000000000000000000..3642443e091c1cdeb0cf7b58c20a157e0c6a513a --- /dev/null +++ b/ladder/components/inference/robots/robolab/composition/catalog.py @@ -0,0 +1,113 @@ +import dataclasses +from typing import Dict, Tuple + +# How an object satisfies its goal on a receptacle. +IN = "in" # object_in_container: bins, bowls, boxes +ON = "on" # object_on_top: plates and other flat surfaces + + +def verb(kind: str) -> str: + """The preposition a receptacle of this kind takes.""" + return ON if kind == ON else IN + + +@dataclasses.dataclass(frozen=True) +class ObjectType: + """A graspable catalog object. ``usd`` is relative to the RoboLab ASSET_DIR (the factory + joins it); ``rest_z`` is the object origin's resting height above the table top (~half the + bbox height) so it spawns sitting on the surface; ``radius`` is the footprint bounding radius + (half the larger XY bbox dim, from the sim) used to keep it clear of receptacles and edges after + jitter. ``category`` drives collective phrasing ("all the fruit") and goal-vs-distractor choice.""" + + usd: str + noun: str + rest_z: float + radius: float + category: str # fruit | vegetable | block + # Visual attribute a plan names the object by ("the yellow lemon"); empty when the noun alone + # is what a viewer would say (an orange, a red block). The reference plans render it. + colour: str = "" + + +@dataclasses.dataclass(frozen=True) +class ReceptacleType: + usd: str + kind: str # IN | ON + noun: str + rest_z: float + footprint: Tuple[float, float] # (half_x, half_y) top-surface extent (m), from the sim bbox + colour: str = "" + + +# Graspable objects. The DROID VLA is grasp-limited to produce-like objects, so the pool is +# fruit/veg; blocks are decoys only. ``radius`` / receptacle ``footprint`` are measured from the sim +# bbox (usd_utils.get_dimensions), so the placement clearances match the real geometry. +# Every asset here is verified single-rigid-body AND confirmed to render (objaverse/apple_01 is +# excluded -- it spawns invisible and its nested collision shapes break the per-object contact filter). +CATALOG: Dict[str, ObjectType] = { + "banana": ObjectType("objects/ycb/banana.usd", "banana", 0.020, 0.089, "fruit", colour="yellow"), + "orange": ObjectType("objects/fruits_veggies/orange1.usd", "orange", 0.042, 0.036, "fruit"), + "lemon": ObjectType("objects/fruits_veggies/lemon1.usd", "lemon", 0.026, 0.038, "fruit", colour="yellow"), + "lime": ObjectType("objects/fruits_veggies/lime.usd", "lime", 0.030, 0.038, "fruit", colour="green"), + "avocado": ObjectType("objects/fruits_veggies/avocado.usd", "avocado", 0.046, 0.031, "fruit"), + "pomegranate": ObjectType( + "objects/fruits_veggies/pomegranate.usd", "pomegranate", 0.058, 0.058, "fruit", colour="red" + ), + "lychee": ObjectType("objects/fruits_veggies/lychee.usd", "lychee", 0.025, 0.021, "fruit"), + "orange2": ObjectType("objects/fruits_veggies/orange2.usd", "orange", 0.037, 0.036, "fruit"), + "onion": ObjectType( + "objects/fruits_veggies/red_onion.usd", "onion", 0.045, 0.030, "vegetable", colour="red" + ), + "pumpkin": ObjectType( + "objects/vomp/pumpkinsmall/pumpkinsmall.usd", "pumpkin", 0.038, 0.038, "vegetable", colour="orange" + ), + "red_block": ObjectType("objects/basic/red_block.usd", "red block", 0.024, 0.030, "block"), + "blue_block": ObjectType("objects/basic/blue_block.usd", "blue block", 0.024, 0.030, "block"), +} + +RECEPTACLE_TYPES: Dict[str, ReceptacleType] = { + # A roomy bin (30x20cm) and a wide 30cm bowl -- both hold a long banana and several fruits. + # The flat plate is the transfer/relay source (objects resting ON it stay easiest to grasp out). + "bin": ReceptacleType( + "objects/vomp/bin_a05/bin_a05.usd", IN, "bin", 0.080, (0.149, 0.100), colour="grey" + ), + "bowl": ReceptacleType( + "objects/vomp/blackandbrassbowl_large/blackandbrassbowl_large.usd", + IN, + "bowl", + 0.075, + (0.151, 0.151), + colour="black", + ), + "plate": ReceptacleType( + "objects/vomp/plate_large/plate_large.usd", ON, "plate", 0.013, (0.166, 0.166), colour="white" + ), +} + +# Ladder pool, best-grasp-first. The N-goal single-destination tasks take the first N. Nine distinct +# produce types plus a second orange for the tenth rung (so N=10 has a "both oranges" duplicate, like +# RoboLab's own multi-banana tasks). +FRUIT_LADDER: Tuple[str, ...] = ( + "banana", + "orange", + "lemon", + "lime", + "avocado", + "onion", + "lychee", + "pumpkin", + "orange2", + # Last, because it is the one object that policies consistently fail to place. Earlier in the + # order it would cap every longer rung by an asset rather than by the horizon the suite exists + # to measure. The rest of the order is deliberately not ranked by any one model's per-object + # success rates. + "pomegranate", +) + +# ------------------------------ workspace layout ------------------------------ +# Franka base at world origin, +x forward, table top at z=0. Graspables scatter over a reachable band; +# receptacles are PINNED at the deep back (the reset randomizes only the free table objects). Authored +# positions are the per-episode means; the reset jitters each free object in x/y + yaw (see factory). +# _TABLE is the measured table_oak top extent (from the scene USD); its front ~0.18 m is unreachable +# behind the mount. Every placement is checked so that, at the WORST jitter+footprint, an object never +# leaves the table and never enters a receptacle's rectangle -- see _scatter. diff --git a/ladder/components/inference/robots/robolab/composition/check_scene_rendered.py b/ladder/components/inference/robots/robolab/composition/check_scene_rendered.py new file mode 100644 index 0000000000000000000000000000000000000000..96c384254308efbb48438a0a507e2dad191aa79a --- /dev/null +++ b/ladder/components/inference/robots/robolab/composition/check_scene_rendered.py @@ -0,0 +1,145 @@ +#!/usr/bin/env python3 +"""Flag RoboLab episodes whose scene stopped being rendered. + +The composition rig composes every object and receptacle onto the base rig as a USD payload, and +Hydra can evict streamed payload geometry mid-episode: the reset frame shows the full scene, then +the table, robot and objects vanish while the static room backdrop keeps rendering. Physics is +unaffected, so nothing errors -- but the recorder writes the same observation the policy is served, +so the policy is blind for the rest of the episode and the cell scores as an ordinary failure. + +Frame 0 is the reset frame and is rendered before any step, so it is the per-episode reference: +an episode is flagged when the table occupies less than `--min-ratio` of the area it occupied then. +""" +import argparse +import colorsys +import glob +import os +import subprocess +import sys +import tempfile +from typing import List, Optional, Tuple + +from PIL import Image + +# The tabletop, in HSV. The rig's table is the one large light-wood surface in frame; the office +# backdrop behind it has no comparable region, which is what makes presence separable at all. +_HUE = (0.03, 0.13) +_MIN_SAT = 0.30 +_MIN_VAL = 0.30 + + +def table_fraction(path: str) -> float: + image = Image.open(path).convert("RGB").resize((96, 96)) + hits = 0 + for red, green, blue in image.getdata(): + hue, sat, val = colorsys.rgb_to_hsv(red / 255, green / 255, blue / 255) + if _HUE[0] <= hue <= _HUE[1] and sat >= _MIN_SAT and val >= _MIN_VAL: + hits += 1 + return hits / (96 * 96) + + +def _frame(video: str, index: int, out: str) -> Optional[str]: + subprocess.run( + [ + "ffmpeg", + "-loglevel", + "error", + "-y", + "-i", + video, + "-vf", + f"select=eq(n\\,{index})", + "-vframes", + "1", + out, + ], + check=False, + ) + return out if os.path.exists(out) and os.path.getsize(out) else None + + +def scan(video: str, samples: int, workdir: str) -> Tuple[float, List[float]]: + """(reference table fraction at reset, table fraction at each sampled step).""" + count = subprocess.run( + [ + "ffprobe", + "-v", + "error", + "-count_frames", + "-select_streams", + "v:0", + "-show_entries", + "stream=nb_read_frames", + "-of", + "csv=p=0", + video, + ], + capture_output=True, + text=True, + check=False, + ).stdout.strip() + total = int(count) if count.isdigit() else 0 + if total < 2: + return 0.0, [] + reference = _frame(video, 0, os.path.join(workdir, "ref.png")) + if reference is None: + return 0.0, [] + # Skip the first few steps: the payloads are still resolving and a frame there is not evidence. + indices = [int(total * (i + 1) / (samples + 1)) for i in range(samples)] + later = [] + for position, index in enumerate(indices): + path = _frame(video, index, os.path.join(workdir, f"f{position}.png")) + if path is not None: + later.append(table_fraction(path)) + return table_fraction(reference), later + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("root", help="campaign directory (scanned for episodes/*/main.mp4)") + parser.add_argument("--samples", type=int, default=4, help="frames to check per episode") + parser.add_argument( + "--min-ratio", + type=float, + default=0.30, + help="flag when the table shrinks below this share of its reset area", + ) + parser.add_argument( + "--min-absolute", + type=float, + default=0.02, + help="table area below this is no scene at all, whatever the reset frame showed", + ) + parser.add_argument("--quiet", action="store_true", help="only print the summary") + args = parser.parse_args() + + videos = sorted(glob.glob(os.path.join(args.root, "**", "episodes", "*", "main.mp4"), recursive=True)) + if not videos: + print(f"no episode videos under {args.root}", file=sys.stderr) + return 2 + + flagged = [] + with tempfile.TemporaryDirectory() as workdir: + for video in videos: + reference, later = scan(video, args.samples, workdir) + if not later: + continue + # Two ways to be blind, and the absolute test is not redundant: an episode whose payloads + # were already gone at reset has a degenerate REFERENCE, so its ratio looks healthy. + worst_absolute = min(later) + worst_ratio = min(later) / reference if reference > 0 else 0.0 + if worst_absolute < args.min_absolute or worst_ratio < args.min_ratio: + flagged.append((video, worst_absolute)) + if not args.quiet: + print( + f"BLIND table={worst_absolute:.3f} ratio={worst_ratio:4.2f} " + f"{os.path.relpath(video, args.root)}" + ) + print( + f"\n{len(flagged)}/{len(videos)} episodes lost their scene ({100 * len(flagged) / len(videos):.1f}%)" + ) + return 1 if flagged else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/ladder/components/inference/robots/robolab/composition/decomp_cache.json b/ladder/components/inference/robots/robolab/composition/decomp_cache.json new file mode 100644 index 0000000000000000000000000000000000000000..9b77c8b9279df4f3974799e1c0a260ca48260a9f --- /dev/null +++ b/ladder/components/inference/robots/robolab/composition/decomp_cache.json @@ -0,0 +1,500 @@ +{ + "_meta": { + "role": "reference plan", + "model": "ground_truth", + "generated": "2026-09-09", + "note": "NOT a model output: the plan per task from specs.decomp_cache_entries(), naming objects by colour and type. Every entry is {subtasks, ordered, revisits}." + }, + "CompBin10Task": { + "subtasks": [ + "put the yellow banana in the grey bin.", + "put the orange in the grey bin.", + "put the yellow lemon in the grey bin.", + "put the green lime in the grey bin.", + "put the avocado in the grey bin.", + "put the red onion in the grey bin.", + "put the lychee in the grey bin.", + "put the orange pumpkin in the grey bin.", + "put the orange in the grey bin.", + "put the red pomegranate in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompBin1Task": { + "subtasks": [ + "put the yellow banana in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompBin2DistractorTask": { + "subtasks": [ + "put the yellow banana in the grey bin.", + "put the orange in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompBin2Task": { + "subtasks": [ + "put the yellow banana in the grey bin.", + "put the orange in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompBin4DistractorTask": { + "subtasks": [ + "put the yellow banana in the grey bin.", + "put the orange in the grey bin.", + "put the yellow lemon in the grey bin.", + "put the green lime in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompBin6Task": { + "subtasks": [ + "put the yellow banana in the grey bin.", + "put the orange in the grey bin.", + "put the yellow lemon in the grey bin.", + "put the green lime in the grey bin.", + "put the avocado in the grey bin.", + "put the red onion in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompIdentical2Task": { + "subtasks": [ + "put the orange in the grey bin.", + "put the orange in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompIdentical4Task": { + "subtasks": [ + "put the green lime in the grey bin.", + "put the green lime in the grey bin.", + "put the green lime in the grey bin.", + "put the green lime in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompOrderedBin10Task": { + "subtasks": [ + "put the orange pumpkin in the grey bin.", + "put the red onion in the grey bin.", + "put the green lime in the grey bin.", + "put the avocado in the grey bin.", + "put the yellow lemon in the grey bin.", + "put the red pomegranate in the grey bin.", + "put the lychee in the grey bin.", + "put the yellow banana in the grey bin.", + "put the orange in the grey bin.", + "put the orange in the grey bin." + ], + "ordered": true, + "revisits": false + }, + "CompOrderedBin2Task": { + "subtasks": [ + "put the yellow banana in the grey bin.", + "put the orange in the grey bin." + ], + "ordered": true, + "revisits": false + }, + "CompOrderedBin3Task": { + "subtasks": [ + "put the yellow banana in the grey bin.", + "put the yellow lemon in the grey bin.", + "put the orange in the grey bin." + ], + "ordered": true, + "revisits": false + }, + "CompOrderedBin4Task": { + "subtasks": [ + "put the orange in the grey bin.", + "put the yellow lemon in the grey bin.", + "put the green lime in the grey bin.", + "put the yellow banana in the grey bin." + ], + "ordered": true, + "revisits": false + }, + "CompOrderedBin5Task": { + "subtasks": [ + "put the avocado in the grey bin.", + "put the orange in the grey bin.", + "put the yellow lemon in the grey bin.", + "put the green lime in the grey bin.", + "put the yellow banana in the grey bin." + ], + "ordered": true, + "revisits": false + }, + "CompOrderedBin6Task": { + "subtasks": [ + "put the avocado in the grey bin.", + "put the orange in the grey bin.", + "put the yellow lemon in the grey bin.", + "put the green lime in the grey bin.", + "put the red onion in the grey bin.", + "put the yellow banana in the grey bin." + ], + "ordered": true, + "revisits": false + }, + "CompOrderedBin7Task": { + "subtasks": [ + "put the avocado in the grey bin.", + "put the orange in the grey bin.", + "put the yellow lemon in the grey bin.", + "put the green lime in the grey bin.", + "put the red onion in the grey bin.", + "put the yellow banana in the grey bin.", + "put the lychee in the grey bin." + ], + "ordered": true, + "revisits": false + }, + "CompOrderedBin8Task": { + "subtasks": [ + "put the orange in the grey bin.", + "put the orange pumpkin in the grey bin.", + "put the lychee in the grey bin.", + "put the yellow lemon in the grey bin.", + "put the green lime in the grey bin.", + "put the yellow banana in the grey bin.", + "put the red onion in the grey bin.", + "put the avocado in the grey bin." + ], + "ordered": true, + "revisits": false + }, + "CompOrderedBin9Task": { + "subtasks": [ + "put the red onion in the grey bin.", + "put the orange in the grey bin.", + "put the orange pumpkin in the grey bin.", + "put the lychee in the grey bin.", + "put the yellow lemon in the grey bin.", + "put the green lime in the grey bin.", + "put the yellow banana in the grey bin.", + "put the avocado in the grey bin.", + "put the orange in the grey bin." + ], + "ordered": true, + "revisits": false + }, + "CompPlate1Task": { + "subtasks": [ + "put the yellow banana on the white plate." + ], + "ordered": false, + "revisits": false + }, + "CompRelay2Task": { + "subtasks": [ + "put the yellow lemon on the white plate.", + "put the yellow lemon in the grey bin.", + "put the green lime on the white plate.", + "put the green lime in the grey bin." + ], + "ordered": true, + "revisits": true + }, + "CompRelay3Task": { + "subtasks": [ + "put the yellow lemon on the white plate.", + "put the yellow lemon in the grey bin.", + "put the green lime on the white plate.", + "put the green lime in the grey bin.", + "put the yellow banana on the white plate.", + "put the yellow banana in the grey bin." + ], + "ordered": true, + "revisits": true + }, + "CompRelay4Task": { + "subtasks": [ + "put the yellow lemon on the white plate.", + "put the yellow lemon in the grey bin.", + "put the green lime on the white plate.", + "put the green lime in the grey bin.", + "put the yellow banana on the white plate.", + "put the yellow banana in the grey bin.", + "put the orange on the white plate.", + "put the orange in the grey bin." + ], + "ordered": true, + "revisits": true + }, + "CompRelay5Task": { + "subtasks": [ + "put the yellow lemon on the white plate.", + "put the yellow lemon in the grey bin.", + "put the green lime on the white plate.", + "put the green lime in the grey bin.", + "put the yellow banana on the white plate.", + "put the yellow banana in the grey bin.", + "put the orange on the white plate.", + "put the orange in the grey bin.", + "put the avocado on the white plate.", + "put the avocado in the grey bin." + ], + "ordered": true, + "revisits": true + }, + "CompRevisit1Task": { + "subtasks": [ + "put the yellow lemon in the grey bin.", + "put the yellow lemon on the white plate.", + "put the yellow lemon in the grey bin." + ], + "ordered": true, + "revisits": true + }, + "CompRevisit2Task": { + "subtasks": [ + "put the yellow lemon in the grey bin.", + "put the yellow lemon on the white plate.", + "put the yellow lemon in the grey bin.", + "put the green lime in the grey bin.", + "put the green lime on the white plate.", + "put the green lime in the grey bin." + ], + "ordered": true, + "revisits": true + }, + "CompRevisit3Task": { + "subtasks": [ + "put the yellow lemon in the grey bin.", + "put the yellow lemon on the white plate.", + "put the yellow lemon in the grey bin.", + "put the green lime in the grey bin.", + "put the green lime on the white plate.", + "put the green lime in the grey bin.", + "put the yellow banana in the grey bin.", + "put the yellow banana on the white plate.", + "put the yellow banana in the grey bin." + ], + "ordered": true, + "revisits": true + }, + "CompRoute10Task": { + "subtasks": [ + "put the yellow banana on the white plate.", + "put the yellow lemon on the white plate.", + "put the green lime on the white plate.", + "put the orange in the grey bin.", + "put the avocado in the grey bin.", + "put the red onion in the grey bin.", + "put the lychee in the grey bin.", + "put the orange pumpkin in the grey bin.", + "put the orange in the grey bin.", + "put the red pomegranate in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompRoute2BinfirstTask": { + "subtasks": [ + "put the orange in the grey bin.", + "put the yellow lemon on the white plate." + ], + "ordered": false, + "revisits": false + }, + "CompRoute2Task": { + "subtasks": [ + "put the yellow lemon on the white plate.", + "put the orange in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompRoute3Task": { + "subtasks": [ + "put the yellow lemon on the white plate.", + "put the green lime on the white plate.", + "put the orange in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompRoute4Task": { + "subtasks": [ + "put the yellow lemon on the white plate.", + "put the green lime on the white plate.", + "put the orange in the grey bin.", + "put the avocado in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompRoute5Task": { + "subtasks": [ + "put the yellow lemon on the white plate.", + "put the green lime on the white plate.", + "put the yellow banana on the white plate.", + "put the orange in the grey bin.", + "put the avocado in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompRoute6Task": { + "subtasks": [ + "put the yellow lemon on the white plate.", + "put the green lime on the white plate.", + "put the yellow banana on the white plate.", + "put the orange in the grey bin.", + "put the avocado in the grey bin.", + "put the red onion in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompRoute7Task": { + "subtasks": [ + "put the yellow banana on the white plate.", + "put the yellow lemon on the white plate.", + "put the green lime on the white plate.", + "put the orange in the grey bin.", + "put the avocado in the grey bin.", + "put the red onion in the grey bin.", + "put the lychee in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompRoute8Task": { + "subtasks": [ + "put the yellow banana on the white plate.", + "put the yellow lemon on the white plate.", + "put the green lime on the white plate.", + "put the orange in the grey bin.", + "put the avocado in the grey bin.", + "put the red onion in the grey bin.", + "put the lychee in the grey bin.", + "put the orange pumpkin in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompRoute9Task": { + "subtasks": [ + "put the yellow banana on the white plate.", + "put the yellow lemon on the white plate.", + "put the green lime on the white plate.", + "put the orange in the grey bin.", + "put the avocado in the grey bin.", + "put the red onion in the grey bin.", + "put the lychee in the grey bin.", + "put the orange pumpkin in the grey bin.", + "put the orange in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompSplitPlate2Task": { + "subtasks": [ + "put the avocado on the white plate.", + "put the orange in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompSplitPlate3Task": { + "subtasks": [ + "put the avocado on the white plate.", + "put the yellow banana on the white plate.", + "put the orange in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompSplitPlate4Task": { + "subtasks": [ + "put the avocado on the white plate.", + "put the yellow banana on the white plate.", + "put the orange in the grey bin.", + "put the green lime in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompSplitPlate5Task": { + "subtasks": [ + "put the avocado on the white plate.", + "put the yellow banana on the white plate.", + "put the orange in the grey bin.", + "put the red onion in the grey bin.", + "put the lychee in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompSplitPlate6Task": { + "subtasks": [ + "put the avocado on the white plate.", + "put the yellow banana on the white plate.", + "put the yellow lemon on the white plate.", + "put the orange in the grey bin.", + "put the green lime in the grey bin.", + "put the red onion in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompSplitPlate7Task": { + "subtasks": [ + "put the avocado on the white plate.", + "put the yellow banana on the white plate.", + "put the yellow lemon on the white plate.", + "put the orange in the grey bin.", + "put the green lime in the grey bin.", + "put the red onion in the grey bin.", + "put the lychee in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompSplitPlate8Task": { + "subtasks": [ + "put the avocado on the white plate.", + "put the yellow banana on the white plate.", + "put the yellow lemon on the white plate.", + "put the green lime on the white plate.", + "put the orange in the grey bin.", + "put the red onion in the grey bin.", + "put the lychee in the grey bin.", + "put the orange pumpkin in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompTransfer2Task": { + "subtasks": [ + "put the orange in the grey bin.", + "put the yellow lemon in the grey bin." + ], + "ordered": false, + "revisits": false + }, + "CompTransfer3Task": { + "subtasks": [ + "put the orange in the grey bin.", + "put the yellow lemon in the grey bin.", + "put the green lime in the grey bin." + ], + "ordered": false, + "revisits": false + } +} diff --git a/ladder/components/inference/robots/robolab/composition/entities.py b/ladder/components/inference/robots/robolab/composition/entities.py new file mode 100644 index 0000000000000000000000000000000000000000..7c2e7567fc3b4801921481719f15e1ebd0d33137 --- /dev/null +++ b/ladder/components/inference/robots/robolab/composition/entities.py @@ -0,0 +1,41 @@ +import dataclasses +from typing import Optional + + +@dataclasses.dataclass(frozen=True) +class PlacedObject: + """A graspable instance: a unique prim name, its catalog type, its authored (x, y), and the + receptacle prim it must end up on/in (``None`` for a distractor that must be left behind). + ``start_on`` names a receptacle the object is spawned resting on/in at episode start (transfer + tasks); ``None`` means it starts on the bare table.""" + + prim: str + _: dataclasses.KW_ONLY # the rest are keyword-only, so a construction site names what it sets + type_id: str + x: float + y: float + receptacle: Optional[str] = None + start_on: Optional[str] = None + + +@dataclasses.dataclass(frozen=True) +class PlacedReceptacle: + prim: str + _: dataclasses.KW_ONLY + type_id: str + x: float + y: float + + +@dataclasses.dataclass(frozen=True) +class Goal: + obj_id: str + receptacle: str + noun: str + + +@dataclasses.dataclass(frozen=True) +class Receptacle: + obj_id: str + kind: str + noun: str diff --git a/ladder/components/inference/robots/robolab/composition/export_scenes.py b/ladder/components/inference/robots/robolab/composition/export_scenes.py new file mode 100644 index 0000000000000000000000000000000000000000..ea573b575591df7499ee6125e84da8b8027cadae --- /dev/null +++ b/ladder/components/inference/robots/robolab/composition/export_scenes.py @@ -0,0 +1,194 @@ +import argparse +import json +import logging +import os +import sys +import traceback + +os.environ.setdefault("OMNI_KIT_ACCEPT_EULA", "YES") +# composition/ is one level under robolab/; the runfiles-staged asset tree is robolab/assets. +os.environ.setdefault( + "ROBOLAB_ASSET_DIR", + os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "assets"), +) + +import cv2 # noqa: F401 -- must import before isaaclab. Do not remove +from isaaclab.app import AppLauncher + +parser = argparse.ArgumentParser( + description="Render each RoboLab composition scene's start state from the over-shoulder and wrist " + "cameras, and write the scene records the decomposition benchmark is assembled from." +) +# NOT --out: add_common_eval_args contributes --output-folder-name, which makes a bare --out an +# ambiguous abbreviation and aborts argument parsing. +parser.add_argument("--out-dir", required=True, help="Output directory for the PNGs and scenes.json.") +parser.add_argument( + "--settle-updates", + type=int, + default=8, + help="Kit updates after reset to let the tiled cameras render a fresh frame.", +) +parser.add_argument( + "--scenes", + nargs="*", + default=None, + help="render only these tasks (Task class names); default renders every composition spec", +) + +from robolab.eval.runner import add_common_eval_args # noqa: E402 + +add_common_eval_args(parser) +AppLauncher.add_app_launcher_args(parser) + +args_cli, _ = parser.parse_known_args() +args_cli.enable_cameras = True + +app_launcher = AppLauncher(args_cli) +simulation_app = app_launcher.app + +import numpy as np # noqa: E402 +import robolab.constants # noqa: E402 +import torch # noqa: E402 +from robolab.core.environments.factory import get_envs # noqa: E402 +from robolab.core.environments.runtime import create_env # noqa: E402 +from robolab.registrations.droid.auto_env_registrations_jointpos import auto_register_droid_envs # noqa: E402 + +from ladder.components.inference.robots.robolab.composition.specs import ( # noqa: E402 + SPECS, + task_class_name, + task_target, +) + +# The composition tasks live in their own modules, so they are registered by target path the way +# env_server does it. Without this the exporter renders whatever else is registered (RoboLab's +# own tasks), and every one of them fails the spec lookup below -- frames written, no records. +# --task-dirs must NOT gate this: it defaults to DEFAULT_TASK_SUBFOLDERS, i.e. it is never empty. +if not args_cli.task: + args_cli.task = [task_target(name) for name in SPECS] + +# stdlib logging, as this package's other composition entry points use: ladder.core.common.logger +# needs //ladder/core/common:platform, which only env_server's target declares. +LOGGER = logging.getLogger("export_scenes") +# The plans the plan_* eval arms are served, so the benchmark's ground truth is what an eval uses. +CACHE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "decomp_cache.json") +# The two views env_server publishes to a policy, so a planner scored on these frames sees what it +# would see at reset in an eval. +CAMERAS = {"main": "over_shoulder_left_camera", "wrist": "wrist_cam"} + +robolab.constants.ENABLE_SUBTASK_PROGRESS_CHECKING = False +# contact sensors ENABLED: env.step()'s termination check needs them (eval keeps them too) + +try: + auto_register_droid_envs(task_dirs=args_cli.task_dirs, task=args_cli.task) +except Exception as exc: # noqa: BLE001 -- diagnostic: registration failing silently loses every task + print(f"[export_scenes] auto_register raised: {type(exc).__name__}: {exc}", flush=True) + + +def _to_bgr(image) -> np.ndarray: + arr = np.asarray(image) + if arr.dtype != np.uint8: # float renders arrive in [0, 1] + arr = (np.clip(arr, 0.0, 1.0) * 255).astype(np.uint8) + return cv2.cvtColor(arr[..., :3], cv2.COLOR_RGB2BGR) # drop alpha if present + + +def main() -> None: + with open(CACHE) as handle: + plans = json.load(handle) + plans.pop("_meta", None) # provenance header, not a scene + by_class = {task_class_name(name): spec for name, spec in SPECS.items()} + os.makedirs(args_cli.out_dir, exist_ok=True) + # get_envs(task=...) filters REGISTERED NAMES, and the composition tasks are asked for by target + # path ("tasks.py::CompBin2Task"), which matches no name -- it returns empty rather than raising. + # auto_register above registered them under their class names, so filter the full registry. + registered = get_envs() + tasks = [task for task in registered if task in by_class] + if args_cli.scenes is not None: + wanted = set(args_cli.scenes) + tasks = [task for task in tasks if task in wanted] + LOGGER.info(f"{len(registered)} env(s) registered, {len(tasks)} of them composition tasks") + LOGGER.info(f"rendering {len(tasks)} composition task(s): {tasks}") + + scenes, env = [], None + for task in tasks: + try: + if env is not None: + env.close() + env, env_cfg = create_env( + task, + device=args_cli.device, + num_envs=1, + instruction_type=args_cli.instruction_type, + policy="ladder_runner", + renderer=args_cli.renderer, + rendering_mode=args_cli.rendering_type, + ) + env.reset() + obs, _ = env.reset() # native double reset (robolab.eval.episode.run_episode) + # Step the full render pipeline: the reset-return image can predate the sampled poses. + hold = torch.zeros_like(env.action_manager.action) + for _ in range(args_cli.settle_updates): + obs, _, _, _, _ = env.step(hold) + + # Frame names, not paths: they sit next to scenes.json, so the directory stays relocatable. + frames = {label: f"{task}__{label}.png" for label in CAMERAS} + for label, camera in CAMERAS.items(): + cv2.imwrite( + os.path.join(args_cli.out_dir, frames[label]), + _to_bgr(obs["image_obs"][camera][0].detach().cpu().numpy()), + ) + + spec = by_class[task] + # A cache value is a plain list or {"subtasks", "ordered"} (see DecomposerConfig); + # storing the dict whole made len(subtasks) the key count. + plan = plans[task] + subtasks = plan["subtasks"] if isinstance(plan, dict) else plan + scenes.append( + { + "id": f"robolab/{task}", + "benchmark": "robolab_comp", + "scene": task, + "instruction": env_cfg.instruction, + "subtasks": subtasks, + "n_goals": len(spec.goal_objects), + "ordered": spec.ordered, + "revisits": bool(spec.via_legs), + # Part of the published schema. The registry carries the family directly; a + # distractor is an object the scene places with no receptacle to reach. + "family": spec.family, + "distractors": [o.type_id for o in spec.objects if o.receptacle is None], + # Scene contents, so a task that changes what it PLACES invalidates samples cut + # against it. The wording fields cannot see such a change. + "objects": sorted([o.prim for o in spec.objects] + [r.prim for r in spec.receptacles]), + "frames": frames, + } + ) + LOGGER.info(f"rendered {task} instruction={env_cfg.instruction!r}") + except Exception: # noqa: BLE001 -- one unrenderable scene must not cost the others + LOGGER.warning(f"{task}: FAILED {traceback.format_exc(limit=2)}") + + if env is not None: + env.close() + # Each scene is rendered under its own try, so a total failure still reaches here with an + # empty list and would publish an empty benchmark under a zero exit code. + if not scenes: + raise RuntimeError(f"no scene rendered out of {len(tasks)}; see the warnings above") + manifest = os.path.join(args_cli.out_dir, "scenes.json") + if args_cli.scenes is not None and os.path.isfile(manifest): + merged = {record["id"]: record for record in json.load(open(manifest))} + merged.update({record["id"]: record for record in scenes}) + scenes = sorted(merged.values(), key=lambda record: record["id"]) + with open(manifest, "w") as handle: + json.dump(scenes, handle, indent=1) + LOGGER.info(f"{len(scenes)}/{len(tasks)} scene(s) -> {args_cli.out_dir}/scenes.json") + simulation_app.close() + + +if __name__ == "__main__": + logging.basicConfig(level=logging.INFO, force=True) + try: + main() + except Exception as e: # noqa: BLE001 -- top-level handler: close the sim app before exiting + print(f"[export_scenes] Terminated with error: {e}") + traceback.print_exc() + simulation_app.close() + sys.exit(1) diff --git a/ladder/components/inference/robots/robolab/composition/factory.py b/ladder/components/inference/robots/robolab/composition/factory.py new file mode 100644 index 0000000000000000000000000000000000000000..bfaab8dfde1e44f34d9bd5b019bba8cd5f24622b --- /dev/null +++ b/ladder/components/inference/robots/robolab/composition/factory.py @@ -0,0 +1,378 @@ +import functools +import math +import os +import pathlib +import tempfile +from collections import OrderedDict +from typing import Dict, List, Optional + +import isaaclab.envs.mdp as mdp +import robolab.constants +import torch +from isaaclab.managers import EventTermCfg as EventTerm, TerminationTermCfg as DoneTerm +from isaaclab.utils import configclass +from robolab.core.events.reset_pose import reset_pose_uniform +from robolab.core.scenes.utils import import_scene +from robolab.core.task.conditionals import ( + object_in_container, + object_on_top, + objects_placed_in_container_in_order, + pick_and_place, + pick_and_place_on_surface, +) +from robolab.core.task.task import Task +from robolab.core.world.world_state import get_world + +from ladder.components.inference.robots.robolab.composition.catalog import CATALOG, ON, RECEPTACLE_TYPES +from ladder.components.inference.robots.robolab.composition.specs import ( + SPECS, + CompositionSpec, + advance_legs, + chain_complete, + task_class_name, +) + +# Authored scene USDs go to a fixed scratch dir, off the read-only bazel-runfiles tree. +_SCENE_DIR = pathlib.Path(tempfile.gettempdir()) / "robolab_composition_scenes" + +# A bare tabletop rig (base_empty.usda: table + franka mount + ground + physics) with each graspable +# and receptacle added as a direct child of `world` via a USD payload. import_scene scrapes exactly +# those direct children as single rigid bodies, so contact sensors resolve one body per name -- +# unlike a runtime spawn of the raw catalog USD, whose nested bodies break the per-object filters. +_RIG_TEMPLATE = """#usda 1.0 +( + defaultPrim = "world" + metersPerUnit = 1 + upAxis = "Z" +) + +def Xform "world" +{{ + def Material "PhysicsMaterial" ( + prepend apiSchemas = ["PhysicsMaterialAPI", "PhysxMaterialAPI"] + ) + {{ + float physics:dynamicFriction = 2 + float physics:staticFriction = 2 + uniform token physxMaterial:frictionCombineMode = "max" + }} + + def "table" ( + prepend payload = @{fixtures}/table_oak.usd@ + ) + {{ + double3 xformOp:translate = (0.5471367990922231, 0, -0.35) + uniform token[] xformOpOrder = ["xformOp:translate"] + }} + + def "franka_table" ( + prepend payload = @{fixtures}/franka_table.usd@ + ) + {{ + quatd xformOp:orient = (6.123233995736766e-17, 0, 0, 1) + double3 xformOp:translate = (-0.087, 0, 0) + uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:orient"] + }} + + def Xform "GroundPlane" + {{ + token visibility = "invisible" + double3 xformOp:translate = (0, 0, -0.697) + uniform token[] xformOpOrder = ["xformOp:translate"] + + def Plane "CollisionPlane" ( + prepend apiSchemas = ["PhysicsCollisionAPI"] + ) + {{ + uniform token axis = "Z" + uniform token purpose = "guide" + }} + }} +{objects}}} +""" + +_OBJECT_TEMPLATE = """ def "{name}" ( + prepend payload = @{usd}@ + ) + {{ + quatf xformOp:orient = (1, 0, 0, 0) + double3 xformOp:translate = ({x}, {y}, {z}) + uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:orient"] + }} +""" + + +def placed(env, obj, receptacle, kind, env_id): + """Per-object placement predicate: on-top for ON receptacles, containment otherwise.""" + if kind == ON: + return object_on_top( + env, + object=[obj], + reference_object=receptacle, + logical="all", + require_gripper_detached=True, + env_id=env_id, + ) + return object_in_container( + env, + object=[obj], + container=receptacle, + logical="all", + require_gripper_detached=True, + env_id=env_id, + ) + + +def legs_success(env, objects, legs, dest, dest_kind, env_id=None): + """Every object must have completed ``legs`` IN ORDER (latched, cleared on reset) and currently + rest in ``dest``. Ordering is what makes the intermediate hops real: without it an object put + straight into the last receptacle, or one that visits the stops backwards, would pass. A revisit + names its final receptacle among the legs too, so only the order tells the two visits apart.""" + world = get_world(env) + objects, legs = list(objects), list(legs) + state_key = f"legs::{'->'.join(name for name, _ in legs)}->{dest}::{','.join(objects)}" + + def _make_state(): + latched = { + o: [torch.zeros(env.num_envs, dtype=torch.bool, device=env.device) for _ in legs] for o in objects + } + + def reset(env_ids): + for flags in latched.values(): + for flag in flags: + flag[env_ids] = False + + return {"latched": latched, "__reset__": reset} + + latched = world.get_or_init_predicate_state(state_key, _make_state)["latched"] + result = None + for obj in objects: + holding = [placed(env, obj, name, kind, None) for name, kind in legs] + latched[obj] = advance_legs(latched[obj], holding) + ok = chain_complete(latched[obj], placed(env, obj, dest, dest_kind, None)) + result = ok if result is None else result & ok + return result if env_id is None else result[env_id] + + +def _latched(predicate, key: str): + """Make a placement predicate stay satisfied once achieved, on episode-scoped predicate state. + + A relay leg is a milestone, not an end state: lifting the object to start leg 2 makes the leg-1 + check false again, so an advancer reading the raw predicate walks back onto a leg it finished and + the task becomes unwinnable. Terminal goals are deliberately NOT wrapped: an object knocked out + of its destination must un-achieve so the advancer can repair it. The wrapper carries the inner + predicate's ``__name__`` and bound nouns, which the oracle renders each subtask's command from.""" + base = getattr(predicate, "func", predicate) + + def latched(env, env_id=None, **_bound_nouns_for_rendering): + world = get_world(env) + + def _make_state(): + flag = torch.zeros(env.num_envs, dtype=torch.bool, device=env.device) + + def reset(env_ids): + flag[env_ids] = False + + return {"flag": flag, "__reset__": reset} + + flag = world.get_or_init_predicate_state(key, _make_state)["flag"] + flag |= predicate(env, env_id=None) + return flag if env_id is None else flag[env_id] + + latched.__name__ = getattr(base, "__name__", "latched") + return functools.partial(latched, **(getattr(predicate, "keywords", None) or {})) + + +def _latch_final_condition(subtask, key: str): + """Latch each object's score-carrying condition -- the one the oracle publishes as ``achieved``.""" + for obj, conditions in subtask.conditions.items(): + predicate, score = conditions[-1] + conditions[-1] = (_latched(predicate, f"{key}::{obj}"), score) + return subtask + + +def _author_usda(spec: CompositionSpec) -> str: + """Write the task's scene USD (base rig + one payload prim per object/receptacle) and return it.""" + asset_dir = robolab.constants.ASSET_DIR + prims = [] + for obj in spec.objects: + t = CATALOG[obj.type_id] + z = t.rest_z + if obj.start_on: # spawn on the source receptacle's rim, dropped slightly so it settles + z = 2 * RECEPTACLE_TYPES[obj.start_on].rest_z + t.rest_z + 0.02 + prims.append( + _OBJECT_TEMPLATE.format(name=obj.prim, usd=os.path.join(asset_dir, t.usd), x=obj.x, y=obj.y, z=z) + ) + for recep in spec.receptacles: + rt = RECEPTACLE_TYPES[recep.type_id] + prims.append( + _OBJECT_TEMPLATE.format( + name=recep.prim, usd=os.path.join(asset_dir, rt.usd), x=recep.x, y=recep.y, z=rt.rest_z + ) + ) + usda = _RIG_TEMPLATE.format(fixtures=os.path.join(asset_dir, "fixtures"), objects="".join(prims)) + _SCENE_DIR.mkdir(parents=True, exist_ok=True) + path = _SCENE_DIR / f"{spec.name}.usda" + # Publish ATOMICALLY: every process rewrites these on import while running eval processes read + # them at env-creation time. A reader landing inside a truncating write loads an empty stage + # (no table, no robot, no objects), which still burns its step budget and scores as a failure. + tmp = _SCENE_DIR / f".{spec.name}.{os.getpid()}.usda.tmp" + tmp.write_text(usda) + os.replace(tmp, path) + return str(path) + + +def _events_cls(spec: CompositionSpec): + """Per-episode reset randomization: every graspable jittered by +/-``spec.jitter`` in x/y with a + full random yaw, collision-checked. Receptacles are omitted, so ``reset_to_default_otherwise`` + pins them and the clearances the object positions were solved against stay valid.""" + j = spec.jitter + event = EventTerm( + func=reset_pose_uniform, + mode="reset", + params={ + "pose_range": {"x": (-j, j), "y": (-j, j), "z": (0.0, 0.0), "yaw": (-math.pi, math.pi)}, + "velocity_range": {}, + "asset_cfg": spec.free_object_list, + "reset_to_default_otherwise": True, + "use_collision_check": True, + }, + ) + return configclass(type("CompositionEvents", (), {"randomize_init_pose": event})) + + +def _grouped_goals(spec: CompositionSpec) -> "OrderedDict[str, List[str]]": + """receptacle_id -> [object_id, ...], in goal order (also the oracle plan order).""" + groups: "OrderedDict[str, List[str]]" = OrderedDict() + for goal in spec.goals: + groups.setdefault(goal.receptacle, []).append(goal.obj_id) + return groups + + +def composition_success( + env, + groups: Dict[str, List[str]], + kinds: Dict[str, str], + forbidden: Optional[Dict[str, List[str]]] = None, + env_id=None, +): + """Whole-task success: a conjunction over receptacle groups, each an ``all`` placement check. + ``forbidden`` (distractor tasks) additionally requires no named decoy ended up in a receptacle.""" + result = None + for recep_id, objs in groups.items(): + if kinds[recep_id] == ON: + group_ok = object_on_top( + env, + object=objs, + reference_object=recep_id, + logical="all", + require_gripper_detached=True, + env_id=env_id, + ) + else: + group_ok = object_in_container( + env, + object=objs, + container=recep_id, + logical="all", + require_gripper_detached=True, + env_id=env_id, + ) + result = group_ok if result is None else result & group_ok + for recep_id, objs in (forbidden or {}).items(): + intruded = object_in_container(env, object=objs, container=recep_id, logical="any", env_id=env_id) + result = result & (~intruded) + return result + + +def _terminations_cls(spec: CompositionSpec): + groups = _grouped_goals(spec) + if spec.via_legs: + rm = spec.receptacle_map + dest = spec.receptacles_used[0] # single by construction + success = DoneTerm( + func=legs_success, + params={ + "objects": [g.obj_id for g in spec.goals], + "legs": [(name, rm[name].kind) for name in spec.via_legs], + "dest": dest, + "dest_kind": rm[dest].kind, + }, + ) + elif spec.ordered: # single-receptacle by construction; enforce first-placement order + success = DoneTerm( + func=objects_placed_in_container_in_order, + params={ + "objects": [g.obj_id for g in spec.goals], + "container": spec.goals[0].receptacle, + "require_gripper_detached": True, + }, + ) + else: + kinds = {recep: spec.receptacle_map[recep].kind for recep in groups} + params: Dict[str, object] = {"groups": dict(groups), "kinds": kinds} + distractors = [o.prim for o in spec.distractor_objects] + if distractors: # single-receptacle by construction; decoys must stay out of it + params["forbidden"] = {spec.receptacles_used[0]: distractors} + success = DoneTerm(func=composition_success, params=params) + return configclass( + type( + "CompositionTerminations", + (), + { + "time_out": DoneTerm(func=mdp.time_out, time_out=True), + "success": success, + }, + ) + ) + + +def _place_subtask(objs, recep_id, kind, score): + if kind == ON: + return pick_and_place_on_surface(object=objs, surface=recep_id, logical="all", score=score) + return pick_and_place(object=objs, container=recep_id, logical="all", score=score) + + +def _subtasks(spec: CompositionSpec): + """The oracle plan: one placement subtask PER GOAL OBJECT, in plan order. A single ``all`` group + over every object gives the advancer no per-object achieved flag, and the policy then stalls after + the first object once N >= 3. Relay emits the two-leg sequence (via, then dest) per object.""" + rm = spec.receptacle_map + if spec.via_legs: + score = 1.0 / ((1 + len(spec.via_legs)) * len(spec.goals)) + subtasks = [] + for i, g in enumerate(spec.goals): + for leg, name in enumerate(spec.via_legs): + # Every leg but the last is LATCHED: starting the next one necessarily undoes it. + subtasks.append( + _latch_final_condition( + _place_subtask([g.obj_id], name, rm[name].kind, score), + f"{spec.name}::via{i}_{leg}", + ) + ) + subtasks.append(_place_subtask([g.obj_id], g.receptacle, rm[g.receptacle].kind, score)) + return subtasks + score = 1.0 / len(spec.goals) + return [_place_subtask([g.obj_id], g.receptacle, rm[g.receptacle].kind, score) for g in spec.goals] + + +def make_task(spec_name: str): + """Build the RoboLab ``Task`` subclass for a composition spec. Called at task-file import time, + inside the env_server sim process, where the RoboLab asset tree is available.""" + spec = SPECS[spec_name] + name = task_class_name(spec_name) + return type( + name, + (Task,), + { + "scene": import_scene(_author_usda(spec), tuple(spec.contact_object_list)), + "events": _events_cls(spec), + "instruction": spec.instruction, + "terminations": _terminations_cls(spec), + "subtasks": _subtasks(spec), + "contact_object_list": spec.contact_object_list, + "episode_length_s": spec.episode_length_s, + "attributes": ["composition", spec.family, f"n{spec.n}"], + "task_name": name, + }, + ) diff --git a/ladder/components/inference/robots/robolab/composition/layout.py b/ladder/components/inference/robots/robolab/composition/layout.py new file mode 100644 index 0000000000000000000000000000000000000000..581bc86a532d577e5c2183c624d3b3add854a489 --- /dev/null +++ b/ladder/components/inference/robots/robolab/composition/layout.py @@ -0,0 +1,182 @@ +import math +from typing import Dict, List, Optional, Tuple + +_TABLE = (0.197, 0.897, -0.500, 0.500) # xmin, xmax, ymin, ymax of the tabletop +# Bare table that must stay visible beyond an object's own footprint, even at worst-case jitter. An +# object sitting at the lip is both very hard to grasp (no room for the fingers to come round it) and +# a physics hazard -- a nudge tips it off the table and the episode is unwinnable from that moment. +_EDGE_KEEP = 0.05 +_RECEP_MARGIN = 0.02 # keep free objects this far outside a receptacle footprint beyond (radius + jitter) +_PAIR_MARGIN = 0.01 # slack beyond touching between two objects at worst-case jitter +_RIM_KEEP = 0.02 # bare receptacle surface beyond an object resting ON it (transfer sources) +_OBJ_RECT = (0.33, 0.72, -0.42, 0.42) # arm-reachable band; the front ~0.18 m sits behind the mount +_WORKSPACE_CENTER = (0.50, 0.0) +_GRID_NX, _GRID_NY = 33, 41 +RECEP_ANCHOR_1 = (0.70, 0.0) # single receptacle: deep centre +RECEP_ANCHOR_2A = (0.72, 0.26) # split/transfer/relay: deep, pushed to the back edge to free the band +RECEP_ANCHOR_2B = (0.72, -0.26) +# How far from the base a GRASPABLE may start: grasp success drops sharply for objects starting much +# beyond ~0.63 m. Applies to start positions only; placing INTO a bin at 0.70 m is fine. 0.62 not +# 0.65 because the greedy placement is not monotone in the bound: 0.65 cannot seat ten objects at any +# ladder jitter, 0.62 seats them at 0.02. +REACH_MAX = 0.62 +# Transfer and relay put graspables on a receptacle rather than the table, so the SOURCE has to obey +# the same bound: at RECEP_ANCHOR_2A the objects would rest at 0.72-0.80 m, out of comfortable reach. The +# destination stays deep. +SOURCE_ANCHOR = (0.45, 0.26) +# Jitter is DERIVED per task: each scene takes the largest value here for which a provably safe layout +# exists (nothing leaves the table, enters a receptacle or overlaps a neighbour at worst-case jitter). +JITTER_LADDER = (0.07, 0.06, 0.05, 0.04, 0.03, 0.02) + +# Episode budget (s) keyed by PICK-PLACE OPERATIONS, taken from RoboLab's own medians over the 95 +# benchmark tasks with a placement conditional. Forward-filled so more work never means less time; the +# tail past nine extrapolates at the ~33 s/op their large tasks settle to. +ROBOLAB_BUDGET_S = { + 1: 60, + 2: 90, + 3: 180, + 4: 180, + 5: 180, + 6: 180, + 7: 240, + 8: 270, + 9: 300, + 10: 330, + 11: 330, + 12: 330, +} + + +def dist(a: Tuple[float, float], b: Tuple[float, float]) -> float: + return math.hypot(a[0] - b[0], a[1] - b[1]) + + +def _candidate_cells( + radius: float, jitter: float, reach_max: Optional[float] = None +) -> List[Tuple[float, float]]: + """Candidate centres for one object: grid points inside the arm-reachable band that are also far + enough from every table edge that neither the jitter nor the object's own footprint can bring it + within ``_EDGE_KEEP`` of the lip, and (when ``reach_max`` is given) within that distance of the + base even at worst-case jitter. Empty when the object cannot be placed safely at all.""" + m = radius + jitter + _EDGE_KEEP + xmin, xmax = max(_TABLE[0] + m, _OBJ_RECT[0]), min(_TABLE[1] - m, _OBJ_RECT[1]) + ymin, ymax = max(_TABLE[2] + m, _OBJ_RECT[2]), min(_TABLE[3] - m, _OBJ_RECT[3]) + if xmax <= xmin or ymax <= ymin: + return [] + xs = [xmin + (xmax - xmin) * i / (_GRID_NX - 1) for i in range(_GRID_NX)] + ys = [ymin + (ymax - ymin) * j / (_GRID_NY - 1) for j in range(_GRID_NY)] + cells = [(x, y) for x in xs for y in ys] + if reach_max is None: + return cells + return [c for c in cells if math.hypot(abs(c[0]) + jitter, abs(c[1]) + jitter) <= reach_max] + + +def _clear_of_receptacles( + c: Tuple[float, float], radius: float, jitter: float, reserved: List[Tuple[float, float, float, float]] +) -> bool: + """The object never overlaps a (pinned) receptacle's rectangular footprint, even at the worst jitter: + reserved entries are (x, y, half_x, half_y). A rectangular keep-out (not a circle) is what lets + objects sit beside a wide bin on the open table instead of being excluded from the whole back.""" + for rx, ry, hx, hy in reserved: + if ( + abs(c[0] - rx) < hx + radius + jitter + _RECEP_MARGIN + and abs(c[1] - ry) < hy + radius + jitter + _RECEP_MARGIN + ): + return False + return True + + +def _scatter( + radii: List[float], + reserved: List[Tuple[float, float, float, float]], + jitter: float, + reach_max: Optional[float] = None, +) -> Optional[List[Tuple[float, float]]]: + """Authored (x, y) per object (indexed as ``radii``) for which NO per-episode jitter can push an + object off the table, under a pinned receptacle, or into a neighbour. Objects are placed + largest-first (the banana needs the most room) by farthest-point selection -- start nearest the + workspace centre, then repeatedly add the candidate maximising the minimum distance to those + already chosen -- among cells satisfying every hard constraint for that object's own radius. + + The pairwise bound is what stops the scene from tearing itself apart: two objects can close by + ``2 * jitter``, so authoring them merely "far apart" (maximising the minimum distance without + asserting one) lets high-N scenes spawn interpenetrating, and the simulator resolves the overlap + by ejecting an object, sometimes off the table. + + Returns None when the scene is over-full at this jitter, so callers can retry lower.""" + placed: Dict[int, Tuple[float, float]] = {} + for oi in sorted(range(len(radii)), key=lambda i: -radii[i]): + r = radii[oi] + cand = [ + c + for c in _candidate_cells(r, jitter, reach_max) + if _clear_of_receptacles(c, r, jitter, reserved) + and all(dist(c, p) >= r + radii[j] + 2 * jitter + _PAIR_MARGIN for j, p in placed.items()) + ] + if not cand: + return None + if not placed: + placed[oi] = min(cand, key=lambda c: dist(c, _WORKSPACE_CENTER)) + else: + placed[oi] = max(cand, key=lambda c: min(dist(c, p) for p in placed.values())) + return [placed[i] for i in range(len(radii))] + + +def fit_scatter( + radii: List[float], + reserved: List[Tuple[float, float, float, float]], + reach_max: Optional[float] = None, +) -> Tuple[float, List[Tuple[float, float]]]: + """The most per-episode variation this scene can safely carry: the largest jitter on the ladder + that still admits a provably safe layout, together with that layout.""" + for jitter in JITTER_LADDER: + positions = _scatter(radii, reserved, jitter, reach_max) + if positions is not None: + return jitter, positions + raise ValueError( + f"no safe layout for {len(radii)} objects (max radius {max(radii):.3f}) clear of " + f"{len(reserved)} receptacle(s) within reach {reach_max} at any jitter down to " + f"{JITTER_LADDER[-1]}" + ) + + +def pull_into_reach( + anchor: Tuple[float, float], offsets: List[Tuple[float, float]], jitter: float, reach_max: Optional[float] +) -> Tuple[float, float]: + """Slide a source receptacle straight toward the base until every object resting on it obeys + ``reach_max`` at worst-case jitter. Objects on a receptacle are laid out by ``fit_on_receptacle``, + which knows nothing about reach, so a hand-picked anchor would otherwise silently decide how far + out they start.""" + if reach_max is None: + return anchor + scale = 1.0 + for _ in range(200): + ax, ay = anchor[0] * scale, anchor[1] * scale + worst = max(math.hypot(abs(ax + dx) + jitter, abs(ay + dy) + jitter) for dx, dy in offsets) + if worst <= reach_max: + return (ax, ay) + scale -= 0.005 + raise ValueError(f"cannot bring {anchor} within reach {reach_max}") + + +def fit_on_receptacle( + radii: List[float], footprint: Tuple[float, float] +) -> Tuple[float, List[Tuple[float, float]]]: + """Same idea for objects that start resting ON a receptacle (the transfer sources): offsets from + the receptacle centre, on a concentric ring, plus the largest jitter for which every object stays + ``_RIM_KEEP`` inside the rim and no two can overlap. Randomizing these within the source, rather + than pinning them, keeps the source intact while still giving each episode a fresh draw.""" + n = len(radii) + for jitter in JITTER_LADDER: + usable = min(footprint) - _RIM_KEEP - jitter - max(radii) + if usable <= 0: + continue + if n == 1: + return jitter, [(0.0, 0.0)] + need = max(radii[i] + radii[j] for i in range(n) for j in range(n) if i != j) + ring = (need + 2 * jitter + _PAIR_MARGIN) / (2 * math.sin(math.pi / n)) + if ring <= usable: + return jitter, [ + (ring * math.cos(2 * math.pi * k / n), ring * math.sin(2 * math.pi * k / n)) for k in range(n) + ] + raise ValueError(f"cannot fit {n} objects on a {footprint} receptacle with any ladder jitter") diff --git a/ladder/components/inference/robots/robolab/composition/specs.py b/ladder/components/inference/robots/robolab/composition/specs.py new file mode 100644 index 0000000000000000000000000000000000000000..91916f2c3edaa97558be5835cb43cf8f63922981 --- /dev/null +++ b/ladder/components/inference/robots/robolab/composition/specs.py @@ -0,0 +1,656 @@ +import dataclasses +import os +from collections import OrderedDict +from typing import Dict, List, Optional, Tuple + +from ladder.components.inference.api.core.instructions import Placement, compose, subtask +from ladder.components.inference.robots.robolab.composition.catalog import ( + CATALOG, + FRUIT_LADDER, + RECEPTACLE_TYPES, + verb, +) +from ladder.components.inference.robots.robolab.composition.entities import ( + Goal, + PlacedObject, + PlacedReceptacle, + Receptacle, +) +from ladder.components.inference.robots.robolab.composition.layout import ( + JITTER_LADDER, + REACH_MAX, + RECEP_ANCHOR_1, + RECEP_ANCHOR_2A, + RECEP_ANCHOR_2B, + ROBOLAB_BUDGET_S, + SOURCE_ANCHOR, + dist, + fit_on_receptacle, + fit_scatter, + pull_into_reach, +) + +# The primary N-scaling axis. It starts at 2 because the N=1 rung is SINGLES below, and stops at 10, +# the longest rung the tabletop scatters within reach. +# +# DENSE, deliberately, and it is the CONSTRAINED families that sweep it: route and ordered_bin run +# every rung from 2 to 10. That is what makes the length curve readable, because a rung's score then +# varies with length alone rather than with which family happens to reach that far. +GOAL_LADDER = range(2, 11) +# The single-destination control family is sparse where the others are dense: short, middle and long. +# It is the family the direct instruction already handles, so it is the length axis's control rather +# than a discriminator, and intermediate rungs would add episodes without adding a question. +BIN_LADDER = (2, 6, 10) +SINGLES: Tuple[Tuple[str, str], ...] = (("comp_bin_1", "bin"), ("comp_plate_1", "plate")) + +# ------------------------------ scene instances ------------------------------ + + +def task_class_name(spec_name: str) -> str: + """comp_bin_6 -> CompBin6Task (the RoboLab Task subclass name / env task_name).""" + return "".join(part.capitalize() for part in spec_name.split("_")) + "Task" + + +def task_target(spec_name: str) -> str: + """The RoboLab task identifier for a spec: the shared tasks module, addressing one class in it. + A bare path would resolve to whichever Task that module defines first.""" + tasks_module = os.path.join(os.path.dirname(os.path.abspath(__file__)), "tasks.py") + return f"{tasks_module}::{task_class_name(spec_name)}" + + +@dataclasses.dataclass(frozen=True) +class CompositionSpec: + """One long-horizon composition task, fully described physically: the exact graspable + instances on the table (goals + distractors), the receptacle instances, and an optional + ordering constraint. This is the declarative unit the factory turns into a programmatic + RoboLab ``Task`` (one RigidObjectCfg per instance on a bare tabletop) and the + oracle/decomposer read as the plan. The visible object count is exactly ``len(objects)``.""" + + name: str + family: str # single_dest | split | distractor | ordered | transfer | relay | revisit | identical + objects: Tuple[PlacedObject, ...] + receptacles: Tuple[PlacedReceptacle, ...] + ordered: bool = False + # The receptacles each object passes through BEFORE its final one, in order. A relay names one + # ("table -> plate -> bin"); a revisit names two, the second of which is the final receptacle + # again ("table -> bin -> plate -> bin"). Empty for every family that places each object once. + via_legs: Tuple[str, ...] = () + jitter: float = JITTER_LADDER[0] # per-episode uniform x/y jitter (m); DERIVED, see fit_scatter + + def __post_init__(self) -> None: + # The reset is ONE event with ONE pose_range for every asset it randomizes, so a spec cannot + # mix objects scattered on the table with objects confined to a source receptacle -- those + # need different jitters to stay on their respective surfaces. Every family satisfies this + # (transfer puts all of its graspables on the source); this keeps it true. + if len({o.start_on is not None for o in self.objects}) > 1: + raise ValueError( + f"{self.name}: mixes table-scattered and source-resting graspables, which cannot " + f"share one reset jitter" + ) + + @property + def source(self) -> Optional[str]: + """Transfer tasks: the receptacle the objects start resting on (named in the instruction).""" + return next((o.start_on for o in self.objects if o.start_on), None) + + @property + def goal_objects(self) -> List[PlacedObject]: + return [o for o in self.objects if o.receptacle is not None] + + @property + def distractor_objects(self) -> List[PlacedObject]: + return [o for o in self.objects if o.receptacle is None] + + @property + def n(self) -> int: + return len(self.goal_objects) + + @property + def n_ops(self) -> int: + """Pick-and-place operations, the unit the budget scales on. Each intermediate leg is one + more move of every object: a relay moves each twice, a revisit three times.""" + return (1 + len(self.via_legs)) * self.n + + @property + def episode_length_s(self) -> int: + """From RoboLab's own benchmark tasks. A larger budget does not convert into more completed + episodes: horizon length is the binding constraint, not the clock.""" + return ROBOLAB_BUDGET_S[min(self.n_ops, max(ROBOLAB_BUDGET_S))] + + @property + def receptacle_map(self) -> Dict[str, Receptacle]: + return { + r.prim: Receptacle(r.prim, RECEPTACLE_TYPES[r.type_id].kind, RECEPTACLE_TYPES[r.type_id].noun) + for r in self.receptacles + } + + @property + def goals(self) -> List[Goal]: + return [Goal(o.prim, o.receptacle, CATALOG[o.type_id].noun) for o in self.goal_objects] + + @property + def receptacles_used(self) -> List[str]: + seen: List[str] = [] + for g in self.goals: + if g.receptacle not in seen: + seen.append(g.receptacle) + return seen + + @property + def contact_object_list(self) -> List[str]: + """Every entity tracked for pose/contact: graspables then receptacles. No ``table`` -- the + tabletop is part of the base USD, not a tracked rigid body.""" + return [o.prim for o in self.objects] + [r.prim for r in self.receptacles] + + @property + def free_object_list(self) -> List[str]: + """The graspables the reset randomizes: all of them. The spec's jitter is sized so no draw + can leave the intended surface. Receptacles are never listed, so they stay pinned.""" + return [o.prim for o in self.objects] + + @property + def instruction(self) -> str: + """The task's command, in the wording shared with SimplerEnv and RobotArena.""" + rm = self.receptacle_map + stops = [*self.via_legs, self.goals[0].receptacle] if self.via_legs else None + if stops is not None: + placements = [ + Placement(g.noun, verb(rm[stop].kind), rm[stop].noun) for g in self.goals for stop in stops + ] + else: + placements = [ + Placement(g.noun, verb(rm[g.receptacle].kind), rm[g.receptacle].noun) for g in self.goals + ] + return compose( + placements, + sequential=self.ordered or bool(self.via_legs), + source=rm[self.source].noun if self.source else None, + excluded=[CATALOG[o.type_id].noun for o in self.distractor_objects], + ) + + +# ------------------------------ instruction composition ------------------------------ + + +def _instances( + type_ids: List[str], + positions: List[Tuple[float, float]], + receptacle_of: Dict[int, Optional[str]], + start_on: Optional[str] = None, +) -> Tuple[PlacedObject, ...]: + """Build unique-primed object instances. A type used once keeps its bare name ("orange"); a + repeated type is suffixed ("orange_1", "orange_2"). Prim names are internal ids only; the + instruction noun always comes from the catalog, so suffixes never leak into language.""" + counts: Dict[str, int] = {} + for t in type_ids: + counts[t] = counts.get(t, 0) + 1 + seen: Dict[str, int] = {} + objs: List[PlacedObject] = [] + for i, t in enumerate(type_ids): + if counts[t] == 1: + prim = t + else: + seen[t] = seen.get(t, 0) + 1 + prim = f"{t}_{seen[t]}" + objs.append( + PlacedObject( + prim, + type_id=t, + x=positions[i][0], + y=positions[i][1], + receptacle=receptacle_of[i], + start_on=start_on, + ) + ) + return tuple(objs) + + +def single_dest( + name: str, + n: int, + receptacle: str = "bin", + pool: Tuple[str, ...] = FRUIT_LADDER, + family: str = "single_dest", + distractor_types: Tuple[str, ...] = (), + ordered: bool = False, + reach_max: Optional[float] = None, +) -> CompositionSpec: + """Put the first ``n`` pool objects into one receptacle; optional decoy distractors that must + stay put. The core N-goal ladder and its distractor variants.""" + rt = RECEPTACLE_TYPES[receptacle] + recep = PlacedReceptacle(receptacle, type_id=receptacle, x=RECEP_ANCHOR_1[0], y=RECEP_ANCHOR_1[1]) + reserved = [(recep.x, recep.y, rt.footprint[0], rt.footprint[1])] + type_ids = list(pool[:n]) + list(distractor_types) + jitter, positions = fit_scatter([CATALOG[t].radius for t in type_ids], reserved, reach_max) + if ordered: + # An ordered rung only tests ordering if the required order FIGHTS the policy's own + # preference, and policies tend to go nearest-first. Requiring the farthest object first + # makes that prior maximally unhelpful, so a rung measures instruction-following rather than + # which layout fit_scatter happened to produce. Pairs move together, so every object keeps + # the slot fit_scatter sized for its radius. + goals = sorted(zip(type_ids[:n], positions[:n], strict=True), key=lambda tp: -dist(tp[1], (0.0, 0.0))) + type_ids = [t for t, _ in goals] + list(distractor_types) + positions = [p for _, p in goals] + list(positions[n:]) + receptacle_of = {i: (receptacle if i < n else None) for i in range(len(type_ids))} + return CompositionSpec( + name=name, + family=family, + objects=_instances(type_ids, positions, receptacle_of), + receptacles=(recep,), + ordered=ordered, + jitter=jitter, + ) + + +def split( + name: str, + groups: List[Tuple[Tuple[str, ...], str]], + family: str = "split", + reach_max: Optional[float] = None, + anchors: Optional[List[Tuple[float, float]]] = None, + reserve_as: Optional[Dict[str, str]] = None, +) -> CompositionSpec: + """Objects partitioned across two receptacles. ``groups`` = [(object_types, receptacle_id), ...]. + Receptacles sit at the two back anchors; every object is a goal (no distractors). + + ``groups`` fixes BOTH the plan order and, by position, which anchor each receptacle takes. + ``anchors`` decouples them, so a variant can reorder the plan while every receptacle and object + stays exactly where the base rung puts it. + + ``reserve_as`` maps a receptacle id to the type whose footprint the scatter should reserve for + it. Object placement is a function of the reserved boxes, so two rungs differing only in + receptacle otherwise get different object positions, and the resulting change in transport + distance would dominate the receptacle effect. Reserving the bowl AS a plate makes the twins + identical in layout, leaving the receptacle as the only difference. Name the LARGER type: a smaller reservation would let an object spawn overlapping + the bigger rim. Only the differing slot is remapped, so the shared bin stays where it is.""" + anchors = anchors or [RECEP_ANCHOR_2A, RECEP_ANCHOR_2B] + receps = tuple( + PlacedReceptacle(rid, type_id=rid, x=anchors[k][0], y=anchors[k][1]) + for k, (_, rid) in enumerate(groups) + ) + reserve_as = reserve_as or {} + reserved = [(r.x, r.y, *RECEPTACLE_TYPES[reserve_as.get(r.type_id, r.type_id)].footprint) for r in receps] + type_ids: List[str] = [] + recep_ids: List[str] = [] + for types, rid in groups: + type_ids += list(types) + recep_ids += [rid] * len(types) + jitter, positions = fit_scatter([CATALOG[t].radius for t in type_ids], reserved, reach_max) + receptacle_of = {i: recep_ids[i] for i in range(len(type_ids))} + return CompositionSpec( + name=name, + family=family, + objects=_instances(type_ids, positions, receptacle_of), + receptacles=receps, + jitter=jitter, + ) + + +def transfer( + name: str, + object_types: Tuple[str, ...], + source: str = "plate", + dest: str = "bin", + source_anchor: Tuple[float, float] = RECEP_ANCHOR_2A, + reach_max: Optional[float] = None, +) -> CompositionSpec: + """Objects start resting ON/IN the ``source`` receptacle and must be moved to ``dest``. Success + only requires them in ``dest`` (the source is a scene fixture, not a goal); the "from the source" + is expressed by the initial state + instruction (mirrors SimplerEnv's transfer family). The + objects are laid out on a ring within the source and re-drawn every episode, with a jitter sized + so no draw can leave the source's top surface.""" + dst = PlacedReceptacle(dest, type_id=dest, x=RECEP_ANCHOR_2B[0], y=RECEP_ANCHOR_2B[1]) + radii = [CATALOG[t].radius for t in object_types] + jitter, offsets = fit_on_receptacle(radii, RECEPTACLE_TYPES[source].footprint) + anchor = pull_into_reach(source_anchor, offsets, jitter, reach_max) + src = PlacedReceptacle(source, type_id=source, x=anchor[0], y=anchor[1]) + positions = [(src.x + dx, src.y + dy) for dx, dy in offsets] + objects = _instances( + list(object_types), positions, {i: dest for i in range(len(object_types))}, start_on=source + ) + return CompositionSpec( + name=name, family="transfer", objects=objects, receptacles=(src, dst), jitter=jitter + ) + + +def relay( + name: str, + object_types: Tuple[str, ...], + via: str = "plate", + dest: str = "bin", + via_anchor: Tuple[float, float] = RECEP_ANCHOR_2A, + reach_max: Optional[float] = None, +) -> CompositionSpec: + """Objects start on the table and must be relayed table -> ``via`` -> ``dest``. Success is a + latched milestone (each object must pass through ``via`` before its placement in ``dest`` counts), + so a direct table->dest delivery does not satisfy it (mirrors SimplerEnv's relay family).""" + via_r = PlacedReceptacle(via, type_id=via, x=via_anchor[0], y=via_anchor[1]) + dst_r = PlacedReceptacle(dest, type_id=dest, x=RECEP_ANCHOR_2B[0], y=RECEP_ANCHOR_2B[1]) + reserved = [ + (via_r.x, via_r.y, *RECEPTACLE_TYPES[via].footprint), + (dst_r.x, dst_r.y, *RECEPTACLE_TYPES[dest].footprint), + ] + jitter, positions = fit_scatter([CATALOG[t].radius for t in object_types], reserved, reach_max) + objects = _instances(list(object_types), positions, {i: dest for i in range(len(object_types))}) + return CompositionSpec( + name=name, + family="relay", + objects=objects, + receptacles=(via_r, dst_r), + ordered=True, + via_legs=(via,), + jitter=jitter, + ) + + +def revisit( + name: str, + object_types: Tuple[str, ...], + home: str = "bin", + away: str = "plate", + away_anchor: Tuple[float, float] = RECEP_ANCHOR_2A, + reach_max: Optional[float] = None, +) -> CompositionSpec: + """Objects go table -> ``home`` -> ``away`` -> ``home``: the last leg returns each object to a + receptacle it has already left, which no end state can witness, since the first visit is undone + by the second leg. Only the ORDER separates the two visits to ``home``, so the legs latch in + sequence (see :func:`advance_legs`).""" + away_r = PlacedReceptacle(away, type_id=away, x=away_anchor[0], y=away_anchor[1]) + home_r = PlacedReceptacle(home, type_id=home, x=RECEP_ANCHOR_2B[0], y=RECEP_ANCHOR_2B[1]) + reserved = [ + (away_r.x, away_r.y, *RECEPTACLE_TYPES[away].footprint), + (home_r.x, home_r.y, *RECEPTACLE_TYPES[home].footprint), + ] + jitter, positions = fit_scatter([CATALOG[t].radius for t in object_types], reserved, reach_max) + objects = _instances(list(object_types), positions, {i: home for i in range(len(object_types))}) + return CompositionSpec( + name=name, + family="revisit", + objects=objects, + receptacles=(away_r, home_r), + ordered=True, + via_legs=(home, away), + jitter=jitter, + ) + + +def identical( + name: str, obj_type: str, n: int, receptacle: str = "bin", reach_max: Optional[float] = None +) -> CompositionSpec: + """``n`` instances of the SAME object into one receptacle -- a counting task ("put the 3 lemons in + the bin"). Tracks multi-instance / matched-count success, as in the SimplerEnv/RobotArena + extended benchmarks.""" + return single_dest( + name, n, receptacle=receptacle, pool=(obj_type,) * n, family="identical", reach_max=reach_max + ) + + +# --------------------------------- benchmark matrix --------------------------------- + + +def _build_specs() -> "OrderedDict[str, CompositionSpec]": + specs: List[CompositionSpec] = [] + + for single_name, single_recep in SINGLES: + specs.append(single_dest(single_name, 1, receptacle=single_recep, reach_max=REACH_MAX)) + + # Family A - single-destination ladder into a bin, the primary N-scaling axis. Three rungs, + # short/middle/long, rather than every N: this is the family the direct instruction already + # handles, so it is the length axis's control, not a discriminator, and intermediate rungs add + # episodes without adding a question. + for n in BIN_LADDER: + specs.append(single_dest(f"comp_bin_{n}", n, reach_max=REACH_MAX)) + + # Family B - distractors present: put the fruit in the bin while non-fruit blocks sit on the + # table and must be left behind (goals are pure fruit so "all the fruit" is unambiguous). + # + # Deliberately only TWO rungs. Exclusion reads as a composition constraint but mostly tests + # low-level grounding (a policy told "put the lemon in the bin" may take the block anyway), and + # ground-truth subtasks barely help, so more rungs would add floor rather than discrimination. + specs.append( + single_dest( + "comp_bin_2_distractor", + 2, + family="distractor", + distractor_types=("red_block",), + reach_max=REACH_MAX, + ) + ) + specs.append( + single_dest( + "comp_bin_4_distractor", + 4, + family="distractor", + distractor_types=("red_block", "blue_block"), + reach_max=REACH_MAX, + ) + ) + + # Family D - ordered sequence into a bin (distinct objects so the required order is meaningful). + # The sharpest discriminator the suite has: a flat policy can place every object and still fail + # by delivering them in an arbitrary order. It spans N rather than sitting at one rung because a + # policy that ignores + # the order has an exactly known success rate, 1/N! -- 1/2, 1/6, 1/24, 1/120 -- so the rungs form + # a null curve that a real ordering capability must beat, and that a decomposing system should + # hold flat against. N=2 is a weak rung on its own (a coin flip) and is kept as that curve's + # anchor, not as a test. + # Swept over the full GOAL_LADDER so the long end of the length axis is not held by one family. + for n in GOAL_LADDER: + specs.append( + single_dest(f"comp_ordered_bin_{n}", n, family="ordered", ordered=True, reach_max=REACH_MAX) + ) + + # Family E - transfer: objects start on the plate, move them all to the bin. + for types in (("orange", "lemon"), ("orange", "lemon", "lime")): + specs.append( + transfer( + f"comp_transfer_{len(types)}", + types, + source_anchor=SOURCE_ANCHOR, + reach_max=REACH_MAX, + ) + ) + + # Family F - relay: table -> plate -> bin, milestone-enforced. The milestone needs an object that + # RESTS on the plate, which rules out the orange (a sphere), the avocado and the onion (both + # taller than they are wide). It does NOT rule out the banana: at radius/rest_z 4.45 it is the + # FLATTEST thing in the catalog, ahead of lemon (1.46) and lime (1.27). + # + # A third object does not have to SHARE the plate (the banana's 17.8 cm long axis does not fit + # beside others on a 16.6 cm plate): advance_legs chains leg k on every earlier leg latching, so + # an object may be relayed on its own and the plate is only ever asked to hold one. Co-residency + # is permitted (an early placement that still holds when the leg before it latches counts), not + # required. So comp_relay_3 is solvable object-by-object, but NOT by the collective reading + # "first put everything on the plate". + # NOTE ON THE NAMES: this family is parameterised by OBJECT count while goal counts are + # placements, so comp_relay_N is a 2N-goal task (comp_revisit_N a 3N-goal one). The names are + # kept for stability; goal counts are placements everywhere else. + for objs in ( + ("lemon", "lime"), + ("lemon", "lime", "banana"), + ("lemon", "lime", "banana", "orange"), + ("lemon", "lime", "banana", "orange", "avocado"), + ): + specs.append(relay(f"comp_relay_{len(objs)}", objs, via_anchor=SOURCE_ANCHOR, reach_max=REACH_MAX)) + # revisit, as in the SimplerEnv composition benchmark. One object keeps the chain + # short enough that the budget still covers three moves of it. + for objs in (("lemon",), ("lemon", "lime")): + specs.append( + revisit(f"comp_revisit_{len(objs)}", objs, away_anchor=SOURCE_ANCHOR, reach_max=REACH_MAX) + ) + + # Family H - routing with the plate side held to objects that rest flat by construction. Family + # C-b covers the plate/bin split; these vary the goal count on a nested ladder + # (2 subset 4 subset 6) with the receptacle question already settled. + for n, (to_plate, to_bin) in ( + (2, (("lemon",), ("orange",))), + (3, (("lemon", "lime"), ("orange",))), + (4, (("lemon", "lime"), ("orange", "avocado"))), + (5, (("lemon", "lime", "banana"), ("orange", "avocado"))), + (6, (("lemon", "lime", "banana"), ("orange", "avocado", "onion"))), + ): + specs.append( + split( + f"comp_route_{n}", [(to_plate, "plate"), (to_bin, "bin")], family="route", reach_max=REACH_MAX + ) + ) + + # The long end of the routing ladder, so the long rungs are not held by comp_bin_N alone. These + # take FRUIT_LADDER[:n] (the SAME objects comp_bin_n uses at that rung) and split them across two receptacles, so route_n against + # bin_n is one instruction naming two destinations against one naming a single destination, with + # the object set held fixed. The plate side is the three objects that rest flat by construction + # (banana 4.45, lemon 1.46, lime 1.27 on radius/rest_z); everything else goes to the walled bin, + # where shape does not matter. Nests with comp_route_6, which is already FRUIT_LADDER[:6]. + _PLATE_SAFE = ("banana", "lemon", "lime") + for n in range(7, 11): + pool = FRUIT_LADDER[:n] + to_plate = tuple(o for o in pool if o in _PLATE_SAFE) + to_bin = tuple(o for o in pool if o not in _PLATE_SAFE) + specs.append( + split( + f"comp_route_{n}", + [(to_plate, "plate"), (to_bin, "bin")], + family="route", + reach_max=REACH_MAX, + ) + ) + + # comp_route_2 with the PLAN ORDER reversed and nothing else. Explicit anchors undo the swap + # `groups` would otherwise cause, so both receptacles and both objects stay exactly where + # comp_route_2 puts them. Isolates whether a completed goal is lost because it sits exposed for + # the whole of the next subtask, or because the policy re-engages an object it has placed. + specs.append( + split( + "comp_route_2_binfirst", + [(("orange",), "bin"), (("lemon",), "plate")], + family="route", + reach_max=REACH_MAX, + anchors=[RECEP_ANCHOR_2B, RECEP_ANCHOR_2A], + ) + ) + + # A third rung for revisit, a family the direct instruction handles poorly. + specs.append( + revisit("comp_revisit_3", ("lemon", "lime", "banana"), away_anchor=SOURCE_ANCHOR, reach_max=REACH_MAX) + ) + + # Family C-b - the SPLIT family on a plate. A bin paired with a BOWL does not measure routing: + # both receptacles are dark and deep, the scene reads as one container, and policies confuse + # them. A plate is visually distinct from the bin, so these rungs test routing itself. + # + # Rungs 5, 7 and 8 exist to give the LENGTH axis a third routing family: without them 5 and 7 + # carried only comp_ordered_bin_N and comp_route_N, so a step in the ladder there was a step in + # two families rather than a property of the horizon. The plate group is capped at the four + # objects that rest stably on a flat unwalled surface (avocado, banana, lemon, lime); everything + # rounder rolls off and would score the asset rather than the routing. That cap is also why the + # family stops at eight: nine would need a fifth plate object, and the only candidates left are + # the pomegranate, which policies rarely place, and orange2, whose subtask renders identically + # to the first orange's. comp_route_N carries the + # ladder at nine and ten. + for n, (to_plate, to_bin) in ( + (2, (("avocado",), ("orange",))), + (3, (("avocado", "banana"), ("orange",))), + (4, (("avocado", "banana"), ("orange", "lime"))), + (5, (("avocado", "banana"), ("orange", "onion", "lychee"))), + (6, (("avocado", "banana", "lemon"), ("orange", "lime", "onion"))), + (7, (("avocado", "banana", "lemon"), ("orange", "lime", "onion", "lychee"))), + (8, (("avocado", "banana", "lemon", "lime"), ("orange", "onion", "lychee", "pumpkin"))), + ): + specs.append( + split( + f"comp_split_plate_{n}", + [(to_plate, "plate"), (to_bin, "bin")], + family="split_plate", + reach_max=REACH_MAX, + ) + ) + + # Family G - identical instances (counting): N copies of one object into the bin. + # TWO RUNGS ONLY, and deliberately. Duplicates is a family the direct instruction already + # solves about as well as the privileged oracle, so rungs past four would add length to the suite without adding anything an orchestrator could win. A family the + # flat policy handles reliably serves as a type control, not as a ladder. + for n, obj in ((2, "orange"), (4, "lime")): + specs.append(identical(f"comp_identical_{n}", obj, n, reach_max=REACH_MAX)) + + return OrderedDict((s.name, s) for s in specs) + + +SPECS: "OrderedDict[str, CompositionSpec]" = _build_specs() + + +def advance_legs(latched, holding): + """Advance a per-object leg chain by one observation, returning the updated latch flags. + + ``latched[k]`` says leg k was completed at some earlier step, ``holding[k]`` that its placement + holds right now. A leg latches only once every leg before it has, which is what separates a + relay from two unrelated placements: an object dropped straight into the last receptacle never + passed through the intermediate one, and one that visits them in the wrong order did not either. + Latches never clear, since finishing leg k+1 necessarily undoes leg k. + + Operates on anything supporting ``|`` and ``&`` elementwise, so the same rule serves the batched + boolean tensors the simulator produces and the plain booleans the tests use. + """ + updated, prior = [], None + for was, now in zip(latched, holding, strict=True): + step = (was | now) if prior is None else (was | (now & prior)) + updated.append(step) + prior = step + return updated + + +def chain_complete(latched, holding_final): + """A leg chain is done when every intermediate leg has latched AND the final placement holds + NOW: an object knocked back out of its last receptacle has not completed the task.""" + done = holding_final + for was in latched: + done = done & was + return done + + +def _plan_name(colour: str, noun: str) -> str: + return f"{colour} {noun}".strip() + + +def decomp_plan(spec: CompositionSpec) -> Dict[str, object]: + """The cached plan for a spec, one command per goal in order, naming objects by colour and type. + + The colour separates look-alikes: told a bare noun, a policy can confuse e.g. the lemon and the + orange. The task's own instruction stays on plain nouns. + """ + receptacle_type = {r.prim: RECEPTACLE_TYPES[r.type_id] for r in spec.receptacles} + + def command(obj: PlacedObject, receptacle_prim: str) -> str: + source, target = CATALOG[obj.type_id], receptacle_type[receptacle_prim] + return subtask( + Placement( + _plan_name(source.colour, source.noun), + verb(target.kind), + _plan_name(target.colour, target.noun), + ) + ) + + if spec.via_legs: + legs = [command(o, stop) for o in spec.goal_objects for stop in (*spec.via_legs, o.receptacle)] + return {"subtasks": legs, "ordered": True, "revisits": True} + return { + "subtasks": [command(o, o.receptacle) for o in spec.goal_objects], + "ordered": bool(spec.ordered), + "revisits": False, + } + + +def decomp_cache_entries() -> Dict[str, object]: + """{task_class_name: plan} for every composition spec, to merge into decomp_cache.json.""" + return {task_class_name(name): decomp_plan(spec) for name, spec in SPECS.items()} + + +if __name__ == "__main__": + for spec in SPECS.values(): + instr = spec.instruction + objs = ", ".join(f"{o.prim}@({o.x:.2f},{o.y:.2f})" for o in spec.objects) + print( + f"{spec.name:22s} n={spec.n:2d} fam={spec.family:11s} " + f"objs={len(spec.objects):2d} len={spec.episode_length_s:4d}s" + ) + print(f" instruction: {instr}") + print(f" layout: {objs}") diff --git a/ladder/components/inference/robots/robolab/composition/tasks.py b/ladder/components/inference/robots/robolab/composition/tasks.py new file mode 100644 index 0000000000000000000000000000000000000000..724112bcb138ac2e03394c9f3a291907d0e7fb65 --- /dev/null +++ b/ladder/components/inference/robots/robolab/composition/tasks.py @@ -0,0 +1,9 @@ +from ladder.components.inference.robots.robolab.composition.factory import make_task +from ladder.components.inference.robots.robolab.composition.specs import SPECS, task_class_name + +# One RoboLab Task subclass per composition spec, all in this module: RoboLab discovers tasks by +# scanning a module for Task subclasses, and env_server addresses a single one as +# "::" (see specs.task_target). Built from SPECS rather than written out so a +# spec can never be added without its task. +for _spec_name in SPECS: + globals()[task_class_name(_spec_name)] = make_task(_spec_name) diff --git a/ladder/components/inference/robots/robolab/droid_conversions.py b/ladder/components/inference/robots/robolab/droid_conversions.py new file mode 100644 index 0000000000000000000000000000000000000000..76ffd48d8d848cc54ad5162e27723845abc1b860 --- /dev/null +++ b/ladder/components/inference/robots/robolab/droid_conversions.py @@ -0,0 +1,161 @@ +import time +from typing import Any, Dict, List, Optional, Tuple + +import numpy as np +from openpi_client import image_tools + +from ladder.components.inference.api.core.action import EndEffectorPoseAction, JointAnglesAction, RobotAction +from ladder.components.inference.api.core.observation import RobotStateType +from ladder.components.inference.robots.robolab.franka_kinematics import FrankaKinematics +from ladder.core.common.logger import get_logger + +LOGGER = get_logger(__name__) + + +# Every Franka joint limit is within +-3.1 rad, so anything past this is a diverged articulation +# that merely has not reached NaN yet. Checking only for non-finite values would let huge joint +# readings through IK and feed the runaway back into the sim. +_JOINT_SANE_RAD = 6.5 + + +# Per-env maintained EE target + FK/IK between ladder EE-pose actions and 8-DoF jointpos. +class DroidJointposConverter: + def __init__( + self, + num_envs: int, + max_joint_step_rad: float = 0.5, + image_size: Tuple[int, int] = (224, 224), + ) -> None: + self._kinematics = FrankaKinematics() + self._max_joint_step_rad = max_joint_step_rad + self._image_width, self._image_height = image_size + # Per-env maintained EE target (4x4); None => seed from the observed pose. + self._target_poses: List[Optional[np.ndarray]] = [None] * num_envs + self._degenerate: List[bool] = [False] * num_envs # logged once per env per episode + # Last finite in-range joint vector per env: the freeze target for a diverged env. + # Commanding zeros instead would be a full-authority PD slam toward the upright pose, + # which can turn a marginal trip into a runaway that carries into the next episode. + self._last_sane: List[Optional[np.ndarray]] = [None] * num_envs + + def observation_dict( + self, + env_i: int, + main_image: np.ndarray, + wrist_image: np.ndarray, + joints: np.ndarray, + gripper_position: np.ndarray, + metadata: Dict[str, Any], + scene_image: Optional[np.ndarray] = None, + right_image: Optional[np.ndarray] = None, + ) -> Dict[str, Any]: + # RobotObservation field dict (what RobotEnv reconstructs client-side). + joints = np.asarray(joints, dtype=np.float32) + eef_pose = self._kinematics.forward(joints).astype(np.float32) + if self._target_poses[env_i] is None: + self._target_poses[env_i] = eef_pose + return { + # Native-path parity: RoboLab's client resizes with the same call before serving. + # The size is the consumer's, not the sim's. + # "scene" is the depth camera's own framing, unpadded: "main" is letterboxed to a + # square, so image coordinates on it do not map onto the depth frame. A consumer that + # reads depth through the image needs "scene"; the padded inputs are unchanged. + "images": { + "main": image_tools.resize_with_pad(main_image, self._image_height, self._image_width), + "wrist": image_tools.resize_with_pad(wrist_image, self._image_height, self._image_width), + **({"scene": scene_image} if scene_image is not None else {}), + **( + {} + if right_image is None + else { + "right": image_tools.resize_with_pad( + right_image, self._image_height, self._image_width + ) + } + ), + }, + "state": { + "type": RobotStateType.SINGLE_MANIPULATOR.value, + "eef_pose": eef_pose, + "eef_pose_from_target": self._target_poses[env_i], + "gripper_state": 1.0 + - np.atleast_1d(np.asarray(gripper_position, dtype=np.float32)), # RoboLab 0=open + "joints": joints, + }, + "timestamps": np.array([time.time()], dtype=np.float64), + "metadata": metadata, + } + + # Ladder action -> [7 clamped joint targets, binary gripper] (RoboLab wire). An end-effector + # pose is solved with inverse kinematics; joint angles are already the commanded quantity. Both + # go through the same divergence guard and per-step clamp below. + def actions_to_jointpos(self, actions: List[RobotAction], current_joints: np.ndarray) -> np.ndarray: + current = np.asarray(current_joints, dtype=np.float32) + solve = [ + env_i + for env_i, action in enumerate(actions) + if isinstance(action.arm_action, EndEffectorPoseAction) + and np.all(np.isfinite(current[env_i])) + and np.abs(current[env_i]).max() <= _JOINT_SANE_RAD + ] + inverses = {} + if solve: + try: + poses = np.stack([actions[env_i].arm_action.to_homogeneous_matrix() for env_i in solve]) + inverses = dict(zip(solve, self._kinematics.inverse(poses, current[solve]), strict=True)) + except Exception as error: # noqa: BLE001 + LOGGER.warn( + f"envs {solve}: IK solve failed ({error!r}); freezing at current joints this step" + ) + return np.stack( + [ + self.action_to_jointpos(env_i, action, current[env_i], inverses.get(env_i, current[env_i])) + for env_i, action in enumerate(actions) + ] + ) + + def action_to_jointpos( + self, env_i: int, action: RobotAction, current_joints: np.ndarray, inverse: np.ndarray + ) -> np.ndarray: + arm_action = action.arm_action + if isinstance(arm_action, JointAnglesAction): + target_pose = None + elif isinstance(arm_action, EndEffectorPoseAction): + target_pose = np.asarray(arm_action.to_homogeneous_matrix(), dtype=np.float64) + self._target_poses[env_i] = target_pose.astype(np.float32) + else: + raise TypeError(f"unsupported arm action: {type(arm_action).__name__}") + current = np.asarray(current_joints, dtype=np.float32) + # An exploded env feeds NaN joints -> singular Jacobian -> LinAlgError that would kill the + # whole multi-env server; command a constant for just that env instead (its episode times + # out on its own), and freeze it at the current pose on any solver failure. + if not np.all(np.isfinite(current)) or np.abs(current).max() > _JOINT_SANE_RAD: + if not self._degenerate[env_i]: + self._degenerate[env_i] = True + LOGGER.warn( + f"env {env_i}: diverged joints (max |q| = {np.nanmax(np.abs(current)):.3g} rad); " + "freezing at last sane pose until reset" + ) + LOGGER.warn( + f"env {env_i}: joint vector at trip: " + f"{np.array2string(current, precision=2, max_line_width=300)}" + ) + frozen = self._last_sane[env_i] + if frozen is None: + frozen = np.clip(np.nan_to_num(current), -_JOINT_SANE_RAD, _JOINT_SANE_RAD) + return np.concatenate([frozen, [0.0]]).astype(np.float32) + self._degenerate[env_i] = False + self._last_sane[env_i] = current.copy() + if target_pose is None: + inverse = np.asarray(arm_action.angles, dtype=np.float32) + if not np.all(np.isfinite(inverse)): + LOGGER.warn(f"env {env_i}: IK returned non-finite solution; freezing at current joints this step") + inverse = current + joint_step = np.clip( + inverse - current, + -self._max_joint_step_rad, + self._max_joint_step_rad, + ) + # Native-path parity: the serve flips (ladder 1.0=open -> RoboLab 0=open), then RoboLab's + # client binarizes the chunk's gripper at 0.5 (Pi0DroidJointposClient._postprocess_chunk). + gripper = 1.0 if (1.0 - float(action.gripper_action)) > 0.5 else 0.0 + return np.concatenate([current + joint_step, [gripper]]).astype(np.float32) diff --git a/ladder/components/inference/robots/robolab/env_server.py b/ladder/components/inference/robots/robolab/env_server.py new file mode 100644 index 0000000000000000000000000000000000000000..402a1cad9bc837e7a486d86dab2374b148848935 --- /dev/null +++ b/ladder/components/inference/robots/robolab/env_server.py @@ -0,0 +1,553 @@ +import argparse +import os +import sys + +os.environ.setdefault("OMNI_KIT_ACCEPT_EULA", "YES") +# @robolab_repo ships no assets; point its constants.py at the runfiles-staged tree before any import. +os.environ.setdefault( + "ROBOLAB_ASSET_DIR", + os.path.join(os.path.dirname(os.path.abspath(__file__)), "assets"), +) + +import cv2 # must be imported before isaaclab. Do not move +from isaaclab.app import AppLauncher + + +def _build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description="Serve RoboLab envs over ladder's RobotServer protocol.") + parser.add_argument("--port", type=int, required=True, help="ZMQ request-reply port.") + parser.add_argument( + "--expose-privileged", + action="store_true", + help="Publish ground-truth composition geometry for the scripted completability policy.", + ) + parser.add_argument( + "--publish-depth", + action="store_true", + help="Render depth and publish it with the camera calibration, for a camera-based policy.", + ) + parser.add_argument( + "--no-render", + action="store_true", + help="Render no camera, for a policy that reads no image (the scripted one). Physics is unchanged.", + ) + parser.add_argument( + "--exterior-cameras", + type=int, + choices=(1, 2), + default=1, + help="Exterior views to render and publish; 2 adds the over-shoulder right camera.", + ) + parser.add_argument( + "--image-size", + type=int, + nargs=2, + metavar=("WIDTH", "HEIGHT"), + default=(224, 224), + help="Size of the published camera images; set it to what the policy was trained on.", + ) + return parser + + +# AppLauncher must launch the sim app before the sim-dependent (# noqa: E402) imports below. +parser = _build_parser() + +from robolab.eval.runner import add_common_eval_args # noqa: E402 + +add_common_eval_args(parser) +AppLauncher.add_app_launcher_args(parser) + +args_cli, _ = parser.parse_known_args() +args_cli.enable_cameras = True + +app_launcher = AppLauncher(args_cli) +simulation_app = app_launcher.app + +import carb.settings # noqa: E402 +import numpy as np # noqa: E402 +import omni.kit.app # noqa: E402 +import omni.timeline # noqa: E402 +import robolab.constants # noqa: E402 +import torch # noqa: E402 +from robolab.core.environments.factory import get_envs # noqa: E402 +from robolab.core.environments.runtime import create_env # noqa: E402 +from robolab.core.metrics.compute_metrics import compute_episode_metrics # noqa: E402 +from robolab.registrations.droid.auto_env_registrations_jointpos import auto_register_droid_envs # noqa: E402 +from robolab.registrations.droid.camera_presets import WRIST_LEFT, WRIST_LEFT_RIGHT # noqa: E402 +from robolab.variations.camera import with_depth # noqa: E402 + +from ladder.components.inference.api.core.action import RobotAction # noqa: E402 +from ladder.components.inference.api.core.observation import METADATA_CAMERAS_KEY, encode_camera # noqa: E402 +from ladder.components.inference.api.core.privileged_state import METADATA_PRIVILEGED_KEY # noqa: E402 +from ladder.components.inference.api.core.request_reply import ( # noqa: E402 + ZmqRequestReplyServer, + ZmqRequestReplyServerConfig, +) +from ladder.components.inference.api.core.requests import RequestType # noqa: E402 +from ladder.components.inference.robots.robolab.composition.specs import ( # noqa: E402 + SPECS, + decomp_cache_entries, + task_class_name, + task_target, +) +from ladder.components.inference.robots.robolab.droid_conversions import DroidJointposConverter # noqa: E402 +from ladder.components.inference.robots.robolab.oracle import build_oracle # noqa: E402 +from ladder.components.inference.robots.robolab.privileged import privileged_state # noqa: E402 +from ladder.core.common.logger import get_logger # noqa: E402 + +LOGGER = get_logger(__name__) + +# MODE=ladder never uses RoboLab's own per-step subtask recorder: subtask boundaries are logged by +# ladder's RecorderWrapper (run_model side) and the oracle's per-goal achieved flags are computed here +# in _get_observations, not by that recorder. Leaving it enabled also crashes on the droid robot +# (its get_wrong_object_grabbed -> gripper_fully_closed path looks up an articulation named "robot", +# which world_state does not resolve), so keep it off regardless of the shared --enable-subtask flag. +robolab.constants.ENABLE_SUBTASK_PROGRESS_CHECKING = False + +# Kit mutes USD diagnostics by default, which hides renderer failures such as payload geometry +# being evicted mid-episode. +carb.settings.get_settings().set("/persistent/app/usd/muteUsdDiagnostics", False) + +# A composition task is named by its spec (comp_bin_2); everything else is a RoboLab task name or +# path, passed through. Both register and are then served under the Task class name. +_requested = args_cli.task or [] +# One view leaves the camera set at robolab's own default; two matches the layout some DROID +# policies were post-trained on. The second costs GPU, so it is opt-in. +_CAMERAS = WRIST_LEFT_RIGHT if args_cli.exterior_cameras == 2 else None +if args_cli.publish_depth: + # with_depth extends each camera's data_types, leaving the rgb-only registration alone. + _CAMERAS = [with_depth(camera) for camera in _CAMERAS or WRIST_LEFT] +auto_register_droid_envs( + task_dirs=None if _requested and all(t in SPECS for t in _requested) else args_cli.task_dirs, + task=[task_target(t) if t in SPECS else t for t in _requested] or None, + cameras=_CAMERAS, +) +_TASKS = [task_class_name(t) if t in SPECS else t for t in _requested] +# Task class name -> whether its spec is scored as a sequence, for the oracle metadata below. +_ORDERED_TASKS = {task_class_name(name): spec.ordered for name, spec in SPECS.items()} +# Task class name -> whether a repeated subtask moves the SAME object again (a relay/revisit leg). +_REVISIT_TASKS = {task_class_name(name): bool(spec.via_legs) for name, spec in SPECS.items()} +# Task class name -> the spec's own plan. The oracle publishes THIS wording, so the privileged arm +# and the planner arms (served the same list from decomp_cache.json) command identical text and +# differ only in where the plan came from. +_PLAN_SUBTASKS = {name: entry["subtasks"] for name, entry in decomp_cache_entries().items()} + + +def _half(frame) -> np.ndarray: + """The camera's own frame at half size and UNPADDED, to pair with the depth it publishes. + + Only the aspect ratio has to match the depth frame for normalized coordinates to map, and + halving each side preserves it exactly while quartering what goes over the wire. + """ + image = frame.detach().cpu().numpy() + return cv2.resize(image, (image.shape[1] // 2, image.shape[0] // 2), interpolation=cv2.INTER_AREA) + + +def _transform(position: np.ndarray, quaternion_wxyz: np.ndarray) -> np.ndarray: + """A 4x4 pose from a position and a wxyz quaternion, the shape Isaac reports.""" + w, x, y, z = (float(v) for v in np.asarray(quaternion_wxyz, dtype=np.float64).reshape(4)) + matrix = np.eye(4) + matrix[:3, :3] = np.array( + [ + [1 - 2 * (y * y + z * z), 2 * (x * y - w * z), 2 * (x * z + w * y)], + [2 * (x * y + w * z), 1 - 2 * (x * x + z * z), 2 * (y * z - w * x)], + [2 * (x * z - w * y), 2 * (y * z + w * x), 1 - 2 * (x * x + y * y)], + ] + ) + matrix[:3, 3] = np.asarray(position, dtype=np.float64).reshape(3) + return matrix + + +class RobolabServer: + def __init__(self, args: argparse.Namespace) -> None: + self._args = args + task_envs = get_envs(task=_TASKS) if _TASKS else get_envs() + self._batches = [(task, run) for task in task_envs for run in range(args.num_runs)] + self._batch_index = 0 + self._task = None + self._env = None + self._env_cfg = None + self._converter = None + self._raw_obs = None + self._step_count = 0 + self._oracle = None # (oracle command strings, per-goal checks) for the current task + self._prev_achieved = None # last logged privileged achieved vector, per batch + self._metrics_dt = None + self._ee_body_index = None + self._traj = None # per-env post-step trajectory samples for the current batch + self._metrics = None # per-env computed metric dicts (see _reset) + self._timeline = omni.timeline.get_timeline_interface() + self._kit_app = omni.kit.app.get_app() + LOGGER.info( + f"Serving {len(self._batches)} batches: {len(task_envs)} task(s) x {args.num_runs} run(s)" + ) + + def handle(self, message): + request_type = message.get("type") + if request_type == RequestType.RESET.value: + return self._reset() + if request_type == RequestType.GET_OBSERVATIONS.value: + return self._get_observations() + if request_type == RequestType.EXEC_ACTION.value: + return self._execute_actions(message["data"]) + return {"status": "ok"} # CONNECT / EXIT + + def _reset(self): + if self._batch_index >= len(self._batches): + return {"status": "finished"} + task, run_index = self._batches[self._batch_index] + self._batch_index += 1 + + if task != self._task: + if self._env is not None: + self._env.close() + self._env, self._env_cfg = create_env( + task, + device=self._args.device, + num_envs=self._args.num_envs, + instruction_type=self._args.instruction_type, + policy="ladder_runner", + renderer=self._args.renderer, + rendering_mode=self._args.rendering_type, + ) + self._task = task + if self._args.no_render: + # Isaac Lab's own switch: skips the Kit app loop and the RTX camera render, not physics. + getattr(self._env, "unwrapped", self._env).render_enabled = False + # Env-owned oracle: the per-goal checks from the task's own subtask spec, paired below + # with the composition spec's plan (or, for RoboLab's own tasks, commands rendered from + # the goal predicates). + self._oracle = build_oracle(getattr(self._env.cfg, "subtasks", None), self._env_cfg.instruction) + # The achieved vector is per-task, so a lane moving to a task with a different goal + # count would die on the strict zip below. Start the new task with no history. + self._prev_achieved = None + # Trajectory-metric inputs, matching the native hdf5 recorder: dt = sim.dt x decimation, + # EE body resolved like PostStepEndEffectorPoseRecorder (exact "base_link", then substring). + unwrapped = getattr(self._env, "unwrapped", self._env) + self._metrics_dt = float(unwrapped.cfg.sim.dt * unwrapped.cfg.decimation) + body_names = unwrapped.scene["robot"].data.body_names + self._ee_body_index = ( + body_names.index("base_link") + if "base_link" in body_names + else next((i for i, name in enumerate(body_names) if "base_link" in name), None) + ) + if self._ee_body_index is None: + LOGGER.warn(f"no base_link EE body among {body_names}; trajectory metrics disabled") + else: + self._env.reset_eval_state() + obs, _ = self._env.reset() + obs, _ = self._env.reset() # native double reset (robolab.eval.episode.run_episode) + self._raw_obs = obs + self._step_count = 0 + # Per batch: the next task's oracle has its own number of goals, and an episode starts with + # nothing achieved, so a vector carried over here would both mis-diff and mis-zip. + self._prev_achieved = None + self._converter = DroidJointposConverter( + num_envs=self._env.num_envs, image_size=tuple(args_cli.image_size) + ) + self._traj = [ + { + "ee_position": [], + "ee_linear_velocity": [], + "joint_position": [], + "joint_velocity": [], + "actions": [], + } + for _ in range(self._env.num_envs) + ] + # None = episode still running; {} = closed but too short for metrics; else the metric dict. + self._metrics = [None] * self._env.num_envs + self._diverged = set() + self._wedged = set() + LOGGER.info( + f"Batch {self._batch_index}/{len(self._batches)} task={task} run={run_index} " + f"instruction={self._env_cfg.instruction!r}" + ) + return {"status": "ok", "n_envs": self._env.num_envs} + + _MAIN_CAMERA = "over_shoulder_left_camera" + _base_logged = False + + def _camera_payloads(self, image_obs): + """Per-env calibrated depth frames for the main camera, or None when depth is off. + + ``"depth"`` aliases ``distance_to_image_plane``, so the buffer is already metres along the + optical axis, not ray length. Poses are relative to the articulation root, the frame + ``eef_pose`` is computed in. + """ + if not self._args.publish_depth: + return None + key = next( + (k for k in image_obs if self._MAIN_CAMERA in k and "depth" in k.lower()), + None, + ) + if key is None: + raise RuntimeError( + f"publish_depth is set but no depth observation was found; image_obs has " + f"{sorted(image_obs)}" + ) + depth = image_obs[key].detach().cpu().numpy().astype(np.float32) + if depth.ndim == 4 and depth.shape[-1] == 1: + depth = depth[..., 0] + sensor = self._env.unwrapped.scene[self._MAIN_CAMERA] + intrinsics = sensor.data.intrinsic_matrices.detach().cpu().numpy() + cam_pos = sensor.data.pos_w.detach().cpu().numpy() + # quat_w_ros is a Warp quatf: ordered XYZW despite the name. + cam_quat_xyzw = sensor.data.quat_w_ros.detach().cpu().numpy() + cam_quat = np.concatenate([cam_quat_xyzw[:, 3:4], cam_quat_xyzw[:, :3]], axis=1) + # root_quat_w is XYZW too. + robot = self._env.unwrapped.scene["robot"] + base_pos = robot.data.root_pos_w.detach().cpu().numpy() + base_quat_xyzw = robot.data.root_quat_w.detach().cpu().numpy() + if not self._base_logged: + self._base_logged = True + # Logged once: a wrong base frame misses every reach while every number looks fine. + origin = self._env.unwrapped.scene.env_origins.detach().cpu().numpy()[0] + LOGGER.info( + f"camera base frame: root={np.round(base_pos[0] - origin, 4).tolist()} rel env_origin, " + f"quat_xyzw={np.round(base_quat_xyzw[0], 4).tolist()}, bodies={list(robot.data.body_names)[:6]}" + ) + base_quat = np.concatenate([base_quat_xyzw[:, 3:4], base_quat_xyzw[:, :3]], axis=1) + payloads = [] + for env_i in range(len(depth)): + k = intrinsics[env_i] if intrinsics.ndim == 3 else intrinsics + world_from_base = _transform(base_pos[env_i], base_quat[env_i]) + world_from_camera = _transform(cam_pos[env_i], cam_quat[env_i]) + payloads.append( + encode_camera( + depth_m=depth[env_i].astype(np.float32), + intrinsics=k, + base_from_camera=np.linalg.inv(world_from_base) @ world_from_camera, + ) + ) + return payloads + + def _get_observations(self): + if self._env is None or self._raw_obs is None: + return {"status": "error", "message": "Environment not initialized"} + image_obs = self._raw_obs["image_obs"] + cameras = self._camera_payloads(image_obs) + proprio = self._raw_obs["proprio_obs"] + results = self._env.get_env_results() + active = set(self._env.active_env_ids) + # Belt against a never-freezing env; RoboLab's own time_out termination freezes first. + max_steps_reached = self._step_count >= int(self._env.max_episode_length) + commands, achieved_all, plan_subtasks = None, None, None + if self._oracle: + commands = [goal.command for goal in self._oracle] + plan_subtasks = _PLAN_SUBTASKS.get(self._task) + if plan_subtasks is not None and len(plan_subtasks) != len(commands): + LOGGER.warn( + f"{self._task}: spec plan has {len(plan_subtasks)} subtasks but the env oracle has " + f"{len(commands)} goals; publishing the env's own commands" + ) + plan_subtasks = None + plan_subtasks = plan_subtasks or commands + try: + achieved_all = [ + [bool(goal.check(self._env, env_id=env_i)) for goal in self._oracle] + for env_i in range(self._env.num_envs) + ] + except Exception as error: + LOGGER.exception( # noqa: PLE1205 + f"Per-goal oracle checks failed for {self._task}; falling back to single-goal", error + ) + achieved_all = None + # Log every change of the privileged achieved vector. This is the oracle progress trace the + # eval protocol reports per-goal progress from, and it is the only way to tell a goal that + # was never reached from one that was reached and then UN-achieved: a goal flipping + # True -> False means its predicate is instantaneous rather than a latched milestone, which + # sends the advancer backwards onto work it already finished. + if achieved_all is not None: + prev = self._prev_achieved + if prev != achieved_all: + LOGGER.info(f"ORACLE_ACHIEVED {achieved_all}") + if prev is not None: + for env_i, (was, now) in enumerate(zip(prev, achieved_all, strict=True)): + lost = [i for i, (a, b) in enumerate(zip(was, now, strict=True)) if a and not b] + if lost: + LOGGER.warn(f"ORACLE_UNACHIEVED env {env_i}: goals {lost} went True->False") + self._prev_achieved = [list(a) for a in achieved_all] + + privileged = ( + privileged_state(getattr(self._env, "unwrapped", self._env), self._oracle, achieved_all) + if self._args.expose_privileged and self._oracle and achieved_all is not None + else None + ) + + data = [] + for env_i in range(self._env.num_envs): + success = bool(results[env_i]["success"]) + truncated = max_steps_reached or env_i not in active + if (success or truncated) and self._metrics[env_i] is None: + # Episode just closed: native metric set over the trajectory up to this step. + traj = {key: np.asarray(values) for key, values in self._traj[env_i].items()} + self._metrics[env_i] = ( + compute_episode_metrics(traj, dt=self._metrics_dt) if len(traj["ee_position"]) else None + ) or {} + metadata = { + "env_name": self._task, + "robot_name": "franka", + "instruction": self._env_cfg.instruction, + "done": success, + "truncated": truncated, + # Privileged ground truth for oracle HLCs (PrivilegedDecomposer/Advancer): the + # synthesized per-goal commands paired with each goal's task-spec condition + # (robolab/oracle.py). A task with no subtask spec publishes the + # single-goal form: the instruction with the env-side success predicate. + "oracle": { + "subtasks": plan_subtasks, + "achieved": achieved_all[env_i], + # A composition spec scored as a sequence (relay, revisit, ordered) publishes it, + # so the oracle arm walks the chain in the order its cached plan already gives + # the planner arms. RoboLab's own tasks carry no such spec and stay unordered. + "ordered": _ORDERED_TASKS.get(self._task, False), + "revisits": _REVISIT_TASKS.get(self._task, False), + } + if achieved_all is not None + else {"subtasks": [self._env_cfg.instruction], "achieved": [success]}, + } + if self._metrics[env_i]: + metadata["metrics"] = self._metrics[env_i] + if privileged is not None: + metadata[METADATA_PRIVILEGED_KEY] = privileged[env_i] + if cameras is not None: + metadata[METADATA_CAMERAS_KEY] = {"scene": cameras[env_i]} + data.append( + self._converter.observation_dict( + env_i=env_i, + main_image=image_obs["over_shoulder_left_camera"][env_i].detach().cpu().numpy(), + wrist_image=image_obs["wrist_cam"][env_i].detach().cpu().numpy(), + scene_image=(_half(image_obs["over_shoulder_left_camera"][env_i]) if cameras else None), + right_image=( + None + if "over_shoulder_right_camera" not in image_obs + else image_obs["over_shoulder_right_camera"][env_i].detach().cpu().numpy() + ), + joints=proprio["arm_joint_pos"][env_i].detach().cpu().numpy(), + gripper_position=proprio["gripper_pos"][env_i].detach().cpu().numpy(), + metadata=metadata, + ) + ) + return {"status": "ok", "data": data[0] if self._env.num_envs == 1 else data} + + def _execute_actions(self, payload): + if self._env is None or self._raw_obs is None: + return {"status": "error", "message": "Environment not initialized"} + if isinstance(payload, dict) and "actions" in payload: + actions = [RobotAction.from_dict(action_dict) for action_dict in payload["actions"]] + else: + actions = [RobotAction.from_dict(payload)] + if len(actions) != self._env.num_envs: + return { + "status": "error", + "message": f"expected {self._env.num_envs} actions, got {len(actions)}", + } + + joints = self._raw_obs["proprio_obs"]["arm_joint_pos"].detach().cpu().numpy() + commands = self._converter.actions_to_jointpos(actions, joints) + while not self._timeline.is_playing(): + self._kit_app.update() + obs, _, _, _, _ = self._env.step(torch.as_tensor(commands, device=self._env.device)) + self._raw_obs = obs + self._step_count += 1 + self._record_trajectory(commands) + self._log_divergence(obs, commands) + return {"success": True} + + def _record_trajectory(self, commands: np.ndarray) -> None: + """One post-step sample per still-running env, from the same articulation state the native + hdf5 recorder writes (PostStepEndEffectorPoseRecorder: env-local EE position, world EE + linear velocity), the inputs of robolab's compute_episode_metrics.""" + if self._ee_body_index is None: + return + unwrapped = getattr(self._env, "unwrapped", self._env) + robot = unwrapped.scene["robot"] + ee_pos = robot.data.body_pos_w[:, self._ee_body_index] - unwrapped.scene.env_origins + ee_pos = ee_pos.detach().cpu().numpy() + ee_vel = robot.data.body_lin_vel_w[:, self._ee_body_index].detach().cpu().numpy() + joint_pos = robot.data.joint_pos.detach().cpu().numpy() + joint_vel = robot.data.joint_vel.detach().cpu().numpy() + for env_i in range(self._env.num_envs): + if self._metrics[env_i] is not None: + continue # episode closed: trajectory stays at its terminal step, like a native demo + traj = self._traj[env_i] + traj["ee_position"].append(ee_pos[env_i]) + traj["ee_linear_velocity"].append(ee_vel[env_i]) + traj["joint_position"].append(joint_pos[env_i]) + traj["joint_velocity"].append(joint_vel[env_i]) + traj["actions"].append(commands[env_i]) + + def _log_divergence(self, obs, commands): + """Dump the physics state the first time an env's joints go non-finite. + + The blow-up is silent, so this is the only place the cause is still visible: a joint-velocity + ramp over several steps means the position drive was driven unstable, a one-step jump means a + contact impulse, and objects going non-finite alongside the arm means the whole scene + diverged rather than the articulation alone. + """ + joints = obs["proprio_obs"]["arm_joint_pos"] + scene = getattr(self._env, "unwrapped", self._env) + scene = getattr(scene, "scene", None) + robot = scene["robot"] if scene is not None else None + if robot is not None: + # Gripper wedge detector: the Robotiq joints are physically bounded at ~1.05 rad, so + # a linkage past 2.0 rad is bent/wrecked even though it is far below the 6.5-rad ARM + # sanity threshold. Phrased as "diverged joints" so report filters that exclude + # diverged episodes pick these up unchanged. + grip_q = robot.data.joint_pos[:, 7:] + wedged = torch.isfinite(grip_q).all(dim=-1) & ( + torch.nan_to_num(grip_q).abs().max(dim=-1).values > _GRIP_WEDGE_RAD + ) + for env_i in wedged.nonzero(as_tuple=False).flatten().tolist(): + if env_i in self._wedged or env_i in self._diverged: + continue + self._wedged.add(env_i) + LOGGER.warn( + f"env {env_i}: diverged joints (gripper linkage wedged, max |q| = " + f"{float(grip_q[env_i].abs().max()):.3g} rad); flagging episode" + ) + finite = torch.isfinite(joints).all(dim=-1) + for env_i in (~finite).nonzero().flatten().tolist(): + if env_i in self._diverged: + continue + self._diverged.add(env_i) + objs = "n/a" + if scene is not None: + try: + objs = ( + ",".join( + name + for name, obj in scene.rigid_objects.items() + if not torch.isfinite(obj.data.root_pos_w[env_i]).all() + ) + or "none" + ) + except Exception as error: # noqa: BLE001 -- diagnostic only; the message carries the cause + objs = f"unreadable ({error!r})" + LOGGER.warn( + f"DIVERGENCE env {env_i} step {self._step_count} task {self._task}: " + f"nonfinite_objects=[{objs}] " + f"commanded={commands[env_i].round(3).tolist()}" + ) + + +# Gripper joint magnitude that means the linkage is bent/wrecked (its physical limits are +# ~1.05 rad) while staying far below the arm sanity threshold. Detects the wedge class that +# degrades episodes without an arm explosion. +_GRIP_WEDGE_RAD = 2.0 + + +def main() -> None: + server = ZmqRequestReplyServer(ZmqRequestReplyServerConfig(port=args_cli.port, recv_timeout_ms=120000)) + server.start(RobolabServer(args_cli).handle) + simulation_app.close() + + +if __name__ == "__main__": + try: + main() + except Exception as error: + LOGGER.exception("[env_server] Terminated with error", error) # noqa: PLE1205 + simulation_app.close() + sys.exit(1) diff --git a/ladder/components/inference/robots/robolab/export_robolab_results.py b/ladder/components/inference/robots/robolab/export_robolab_results.py new file mode 100644 index 0000000000000000000000000000000000000000..0eeaaae6dcc8507c3ad4a925b90ff7fd8616f620 --- /dev/null +++ b/ladder/components/inference/robots/robolab/export_robolab_results.py @@ -0,0 +1,56 @@ +import argparse +import json +import os + +import yaml + + +def main() -> None: + parser = argparse.ArgumentParser( + description="Map a ladder-runner campaign (episode_log.yaml) to RoboLab's episode_results.jsonl." + ) + parser.add_argument("--episode-log", required=True, help="episode_log.yaml from the ladder runner.") + parser.add_argument("--output-dir", required=True, help="Directory for episode_results.jsonl.") + parser.add_argument("--policy", default="ladder_vlam", help="Policy label stamped into records.") + args = parser.parse_args() + + with open(args.episode_log) as f: + episode_log = yaml.safe_load(f) or [] + + # Batches arrive in order per task; number them 0.. like RoboLab's run index. + batch_to_run = {} + per_task_runs = {} + records = [] + for entry in episode_log: + task = entry["env_name"] + batch_key = (task, entry.get("batch", entry.get("episode"))) + if batch_key not in batch_to_run: + batch_to_run[batch_key] = per_task_runs.get(task, 0) + per_task_runs[task] = batch_to_run[batch_key] + 1 + run_index = batch_to_run[batch_key] + records.append( + { + "task_name": task, + "run_name": f"{task}_{run_index}", + "run": run_index, + "env_id": entry.get("batch_env_index", 0), + "policy": args.policy, + "instruction": entry.get("instruction", ""), + "success": bool(entry["success"]), + "episode_step": entry["num_steps"], + "metrics": entry.get("metrics", {}), + "events": {}, + "timing": {}, + } + ) + + os.makedirs(args.output_dir, exist_ok=True) + output_path = os.path.join(args.output_dir, "episode_results.jsonl") + with open(output_path, "w") as f: + for record in records: + f.write(json.dumps(record) + "\n") + print(f"Wrote {len(records)} records to {output_path}") + + +if __name__ == "__main__": + main() diff --git a/ladder/components/inference/robots/robolab/franka_kinematics.py b/ladder/components/inference/robots/robolab/franka_kinematics.py new file mode 100644 index 0000000000000000000000000000000000000000..61b07066c0183bc7b35923c97c77936f0d057a2c --- /dev/null +++ b/ladder/components/inference/robots/robolab/franka_kinematics.py @@ -0,0 +1,72 @@ +from pathlib import Path + +import numpy as np +import pytorch_kinematics as pk +import torch +from pytorch_kinematics.ik import delta_pose +from pytorch_kinematics.transforms import rotation_conversions + +from ladder.core.common.platform import REPO_ROOT + +# panda_link8 (flange) is ladder's canonical DROID EE frame; pytorch_kinematics keeps this torch/numpy-2 native. +_PANDA_URDF = Path(REPO_ROOT) / "ladder/components/urdf/robots/panda/robot.urdf" +_EEF_LINK = "panda_link8" + +# Franka Panda joint position limits (rad). +_JOINT_LIMITS_LOWER = [-2.7437, -1.7837, -2.9007, -3.0421, -2.8065, 0.5445, -3.0159] +_JOINT_LIMITS_UPPER = [2.7437, 1.7837, 2.9007, -0.1518, 2.8065, 4.5169, 3.0159] + + +class FrankaKinematics: + def __init__(self) -> None: + with open(_PANDA_URDF) as urdf_file: + self._chain = pk.build_serial_chain_from_urdf(urdf_file.read(), _EEF_LINK).to(dtype=torch.float32) + self._joint_limits = torch.tensor([_JOINT_LIMITS_LOWER, _JOINT_LIMITS_UPPER], dtype=torch.float32).T + self._solver = pk.PseudoInverseIK( + self._chain, + retry_configs=torch.zeros(1, 7), + joint_limits=self._joint_limits, + max_iterations=100, + pos_tolerance=1e-4, + rot_tolerance=1e-3, + early_stopping_any_converged=True, + ) + + def forward(self, joint_position: np.ndarray) -> np.ndarray: + joints = torch.as_tensor(np.array(joint_position, dtype=np.float32)).reshape(1, -1) + pose = self._chain.forward_kinematics(joints, end_only=True).get_matrix()[0].numpy() + # A diverged sim step can report non-finite joints; a neutral pose keeps the rotation orthonormal + # so one diverged env doesn't crash the whole batched collate. + return pose if np.isfinite(pose).all() else np.eye(4, dtype=np.float32) + + def inverse(self, target_poses: np.ndarray, seed_joint_positions: np.ndarray) -> np.ndarray: + q = torch.as_tensor(np.array(seed_joint_positions, dtype=np.float32)) + target = torch.as_tensor(np.array(target_poses, dtype=np.float32)).reshape(-1, 4, 4) + target_pos = target[:, :3, 3] + target_wxyz = rotation_conversions.matrix_to_quaternion(target[:, :3, :3]) + remaining = torch.ones(len(q), dtype=torch.bool) + err_min, no_improve = None, None + with torch.no_grad(): + for _ in range(self._solver.max_iterations): + if not remaining.any(): + break + jacobian, pose = self._chain.jacobian(q, ret_eef_pose=True) + dx, pos_diff, rot_diff = delta_pose(pose.view(-1, 1, 4, 4), target_pos, target_wxyz) + dq = self._solver.compute_dq(jacobian, dx).squeeze(2) + q = torch.where(remaining.unsqueeze(1), q + self._solver.lr * dq, q) + converged = (pos_diff.squeeze(2).norm(dim=-1) < self._solver.pos_tolerance) & ( + rot_diff.squeeze(2).norm(dim=-1) < self._solver.rot_tolerance + ) + remaining &= ~converged + err = dx.squeeze(2).norm(dim=-1) + if no_improve is None: + no_improve = torch.zeros_like(err) + elif err_min is None: + err_min = err.clone() + else: + improved = err < err_min + err_min[improved] = err[improved] + no_improve[improved] = 0 + no_improve[~improved] += 1 + remaining &= no_improve <= self._solver.early_stopping_no_improvement_patience + return q.numpy() diff --git a/ladder/components/inference/robots/robolab/oracle.py b/ladder/components/inference/robots/robolab/oracle.py new file mode 100644 index 0000000000000000000000000000000000000000..0cdf3db519da0c7de4912d62dcf725e1c148d543 --- /dev/null +++ b/ladder/components/inference/robots/robolab/oracle.py @@ -0,0 +1,145 @@ +import dataclasses +import re +from typing import Callable, Dict, List, Optional, Sequence, Tuple + +from ladder.core.common.logger import get_logger + +LOGGER = get_logger(__name__) + +# Goal-terminating predicate name -> command template, filled from the predicate's bound +# noun kwargs. Only the placement/stack predicates that CARRY a goal's score need an entry; +# any other terminating predicate falls back to the whole-task instruction (logged once). +# Extend by adding one line when a new goal predicate is used. +_TEMPLATES: Dict[str, str] = { + "object_on_top": "put the {object} on the {reference_object}.", + "object_in_container": "put the {object} in the {container}.", + "object_inside": "put the {object} in the {container}.", + "object_on_bottom": "put the {object} in the {reference_object}.", +} + + +# Placement predicate -> (moved-object kwarg, destination kwarg, relation). These are the +# predicates that say WHERE something has to end up, which is what a privileged policy drives to. +_PLACEMENTS: Dict[str, Tuple[str, str, str]] = { + "object_on_top": ("object", "reference_object", "on"), + "object_in_container": ("object", "container", "in"), + "object_inside": ("object", "container", "in"), + "object_on_bottom": ("object", "reference_object", "in"), +} + + +@dataclasses.dataclass(frozen=True) +class OracleGoal: + """One score-carrying goal: its command string, the check that credits it, and its legs. + + ``legs`` is the ordered chain the goal is scored through -- a relay's intermediate stops, then + the final placement. Each leg pairs its own predicate with the geometry it names, so a + privileged policy can drive the chain and a text planner can read the command, both off the + same enumeration. + """ + + command: str + check: Callable + legs: Sequence[Tuple[Callable, Optional[Tuple[str, str, str]]]] + + +def goal_geometry(predicate: Callable) -> Optional[Tuple[str, str, str]]: + """(moved object, destination, relation) for a placement predicate, else None. + + Read off the predicate's own bound kwargs -- the same source ``_render`` draws its nouns from -- + so the geometry a policy drives to and the condition the task scores agree by construction. + """ + func = getattr(predicate, "func", predicate) + kwargs = getattr(predicate, "keywords", None) or {} + name = getattr(func, "__name__", "") + if name == "stacked": + objects = kwargs.get("objects") or [] + return (objects[0], objects[1], "on") if len(objects) >= 2 else None + entry = _PLACEMENTS.get(name) + if entry is None: + return None + obj_key, dest_key, relation = entry + if isinstance(kwargs.get(obj_key), str) and isinstance(kwargs.get(dest_key), str): + return (kwargs[obj_key], kwargs[dest_key], relation) + return None + + +def _noun(asset_id: str, instruction: str) -> str: + """Asset id -> a readable noun, grounded in the task instruction's own words. The per-token + instance-index suffix is stripped first ("lime01" -> "lime", "orange_01" -> "orange") so that + duplicated objects ground to their base noun. Keep the id tokens that appear in the instruction + ("bin_a06" + "...into the bin" -> "bin"; "plate_large" + "...on the plate" -> "plate"); when the + instruction never names the object (an abstract task), fall back to the stripped id + ("spam_can" -> "spam can").""" + tokens = [re.sub(r"\d+$", "", token) for token in asset_id.lower().split("_")] + tokens = [token for token in tokens if token] + words = set(re.findall(r"[a-z]+", instruction.lower())) + grounded = [token for token in tokens if token in words] + if grounded: + return " ".join(grounded) + return " ".join(tokens) or asset_id + + +def _render(predicate: Callable, instruction: str) -> str: + """One command string for a goal-terminating predicate, from its name + bound nouns; each + noun is grounded in the task ``instruction`` (which is also the no-template fallback).""" + func = getattr(predicate, "func", predicate) # unwrap functools.partial + kwargs = getattr(predicate, "keywords", None) or {} + name = getattr(func, "__name__", "") + if name == "stacked": + objects = [_noun(obj, instruction) for obj in kwargs.get("objects", [])] + return "stack the " + " on the ".join(objects) + "." if objects else instruction + template = _TEMPLATES.get(name) + if template is None: + LOGGER.warn(f"robolab oracle: no command template for predicate {name!r}; using task instruction") + return instruction + nouns = {key: _noun(value, instruction) for key, value in kwargs.items() if isinstance(value, str)} + try: + return template.format(**nouns) + except KeyError: + LOGGER.warn(f"robolab oracle: template for {name!r} missing a noun; using task instruction") + return instruction + + +def build_oracle(subtasks, task_instruction: str) -> List[OracleGoal]: + """Synthesize the privileged oracle plan from the env's OWN subtask spec: one goal per + score-carrying condition, in goal order. The command is rendered from the goal's terminating + predicate (verb + bound object/target nouns) and the check IS that predicate, so a plan and the + credit it is scored against stay aligned by construction. This is the RoboLab-side analogue of + SimplerEnv's ``oracle_subtasks`` / ``oracle_goal_keys`` (derived from the env's goals, not a + hand-authored plan); the server just forwards the result. + + The goal enumeration mirrors what RoboLab scores against: a ``logical="all"`` multi-object group + yields one goal per object (its final, score-carrying condition); ``"any"`` collapses to one + group-level goal; ``"choose"`` yields K goals (goal i-1 = "at least i of the group placed"). + Returns ``[]`` when the task declares no subtask spec, so the caller publishes the single + whole-task goal instead.""" + goals: List[OracleGoal] = [] + for subtask in subtasks or []: + chains = list(subtask.conditions.values()) + finals = [conditions[-1][0] for conditions in chains] + if subtask.logical == "all" and len(finals) > 1: + for chain in chains: + goals.append(_goal(chain[-1][0], task_instruction, chain)) + elif subtask.logical == "any": + check = lambda env, env_id=None, fs=finals: any(f(env, env_id=env_id) for f in fs) # noqa: E731 + goals.append(_goal(finals[0], task_instruction, chains[0], check=check)) + elif subtask.logical == "choose": + for need in range(1, (subtask.K or 1) + 1): + index = min(need - 1, len(finals) - 1) + check = lambda env, env_id=None, fs=finals, need=need: ( # noqa: E731 + sum(bool(f(env, env_id=env_id)) for f in fs) >= need + ) + goals.append(_goal(finals[index], task_instruction, chains[index], check=check)) + else: + goals.append(_goal(finals[0], task_instruction, chains[0])) + return goals + + +def _goal(predicate: Callable, instruction: str, chain, check: Optional[Callable] = None) -> OracleGoal: + """One goal from its terminating predicate and the ordered condition chain reaching it.""" + return OracleGoal( + command=_render(predicate, instruction), + check=check if check is not None else predicate, + legs=[(leg[0], goal_geometry(leg[0])) for leg in chain], + ) diff --git a/ladder/components/inference/robots/robolab/privileged.py b/ladder/components/inference/robots/robolab/privileged.py new file mode 100644 index 0000000000000000000000000000000000000000..3594a17e3d05d742580046054aabcb16cbed110f --- /dev/null +++ b/ladder/components/inference/robots/robolab/privileged.py @@ -0,0 +1,151 @@ +from typing import Any, Dict, List, Sequence, Tuple + +import numpy as np +import torch +from robolab.core.world.world_state import get_world + +from ladder.components.inference.api.core.privileged_geometry import build_payloads, to_base +from ladder.components.inference.robots.robolab.oracle import OracleGoal + +_ROBOT = "robot" +_BASE_LINK = "panda_link0" +_GRIPPER = "gripper" +_IGNORED = {"table"} # scenery registered as a rigid object, as upstream's EventTracker skips it + +Leg = Tuple[str, str, str] + + +_LONG_ASPECT = 1.2 # a footprint this much longer than wide has a long axis to grip across + + +def _numpy(value) -> np.ndarray: + return value.detach().cpu().numpy() if isinstance(value, torch.Tensor) else np.asarray(value) + + +def _quat_to_matrix(quat: np.ndarray) -> np.ndarray: + """Rotation from an (x, y, z, w) quaternion -- the order this articulation state reports. + + Verified against the sim: the arm's base sits square on the env origin, reporting [0, 0, 0, 1], + and the end effector resolves to the same x and y through forward kinematics as it does in + env-local coordinates. Reading that quaternion as (w, x, y, z) instead invents a 180 degree yaw, + which puts the whole scene behind the robot and sends the policy driving out of the workspace. + """ + x, y, z, w = (float(v) for v in quat) + return np.array( + [ + [1 - 2 * (y * y + z * z), 2 * (x * y - w * z), 2 * (x * z + w * y)], + [2 * (x * y + w * z), 1 - 2 * (x * x + z * z), 2 * (y * z - w * x)], + [2 * (x * z - w * y), 2 * (y * z + w * x), 1 - 2 * (x * x + y * y)], + ] + ) + + +def _base_transforms(world, n_envs: int) -> np.ndarray: + """Env-local -> robot base, one 4x4 per env. + + Object geometry comes out of WorldState relative to each env's scene origin, while the policy + reads an eef pose forward kinematics produced in the arm's own base frame. Publishing the two in + different frames fails silently: every number looks plausible and every reach misses by the mount + offset. + """ + poses = _numpy(world.get_articulation_link_pose(_ROBOT, _BASE_LINK)) + origins = _numpy(world.env.scene.env_origins) + + transforms = np.zeros((n_envs, 4, 4)) + for env_id in range(n_envs): + base_to_env = np.eye(4) + base_to_env[:3, :3] = _quat_to_matrix(poses[env_id, 3:7]) + base_to_env[:3, 3] = poses[env_id, :3] - origins[env_id] + transforms[env_id] = np.linalg.inv(base_to_env) + return transforms + + +def _corners(world, name: str) -> np.ndarray: + """Per-env corners of one body's oriented bounding box, ``[n_envs, 8, 3]`` in env-local coordinates.""" + corners, _ = world.get_bbox(name) + return _numpy(corners) + + +def _long_axis_yaw(corners: np.ndarray) -> float: + """Yaw of an oriented box's longer horizontal side, or NaN when its footprint is round. + + The axis-aligned extents cannot tell which way a lemon at 45 degrees lies, and a grip closed on + its tapered ends squeezes it out of the jaws. + """ + xy = corners[:, :2] - corners[:, :2].mean(axis=0) + _, spread, axes = np.linalg.svd(xy, full_matrices=False) + if spread[0] < _LONG_ASPECT * spread[1]: + return float("nan") + return float(np.arctan2(axes[0][1], axes[0][0])) + + +def privileged_state( + env, goals: Sequence[OracleGoal], achieved: Sequence[Sequence[bool]] +) -> List[Dict[str, Any]]: + """Ground-truth geometry for the scripted completability policy, one payload per env. + + Driven by the goals the task is SCORED on, so it covers every RoboLab task rather than the + composition suite alone, and emitted in the schema RobotArena and SimplerEnv emit, so one + ScriptedPolicy drives all three. A goal whose predicate names no placement is skipped: there is + no destination to drive to. + """ + world = get_world(env) + n_envs = int(env.num_envs) + + # A subtask's condition chain is the STAGES of one placement (the upstream builder emits + # object_grabbed before the placement predicate), not a sequence of destinations. A relay's legs + # arrive as separate goals already, each latched by the factory, so the geometry to drive to is + # the last condition that actually names a placement. + active = [ + next((geom for _, geom in reversed(list(goal.legs)) if geom is not None), None) for goal in goals + ] + + # Every movable body, so a grasped or knocked distractor shows in the trace. + named = {name for goal in goals for _, leg in goal.legs if leg for name in leg[:2]} + names = sorted(named | {name for name in world.objects if name not in _IGNORED}) + if not names: + return [{"conditions": [], "objects": []} for _ in range(n_envs)] + index_of = {name: i for i, name in enumerate(names)} + + transforms = _base_transforms(world, n_envs) + mins, maxs = (np.zeros((len(names), n_envs, 3)) for _ in range(2)) + yaws = np.full((len(names), n_envs), np.nan) + for i, name in enumerate(names): + corners = _corners(world, name) + for env_id in range(n_envs): + # All eight corners go through the change of frame, so the extremes are those of the + # base-frame box rather than of two env-local corners that a rotation can swap. + base = to_base(transforms[env_id], corners[env_id]) + mins[i, env_id], maxs[i, env_id] = base.min(axis=0), base.max(axis=0) + yaws[i, env_id] = _long_axis_yaw(base) + centers = 0.5 * (mins + maxs) + + grasped = np.zeros((len(names), n_envs), dtype=bool) + for i, name in enumerate(names): + grasped[i] = _numpy(world.in_contact(name, _GRIPPER)) + + payload_goals: List[Dict[str, Any]] = [] + for goal_index, leg in enumerate(active): + if leg is None: + continue + source, destination, relation = leg + payload_goals.append( + { + "source": index_of[source], + "destination": index_of[destination], + "relation": relation, + "satisfied": [bool(achieved[env_id][goal_index]) for env_id in range(n_envs)], + } + ) + + payloads = build_payloads( + names=names, + centers=centers, + mins=mins, + maxs=maxs, + grasped=grasped, + goals=payload_goals, + n_envs=n_envs, + long_axis_yaws=yaws, + ) + return payloads diff --git a/ladder/components/urdf/robots/panda/robot.urdf b/ladder/components/urdf/robots/panda/robot.urdf new file mode 100644 index 0000000000000000000000000000000000000000..e8fa83ef7abbc26480f01d931e1ac20fd7f8ba6f --- /dev/null +++ b/ladder/components/urdf/robots/panda/robot.urdf @@ -0,0 +1,440 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/ladder/core/common/logger/__init__.py b/ladder/core/common/logger/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..c350055204e696e12710c8a73d61cc54b712d543 --- /dev/null +++ b/ladder/core/common/logger/__init__.py @@ -0,0 +1,13 @@ +from .colorize import colorize +from .logger import LadderLogger, enter_ipdb_breakpoint_on_signal, get_logger, get_root_logger, is_interactive +from .stdout import no_stdout_within_functions + +__all__ = [ + 'get_logger', + 'get_root_logger', + 'enter_ipdb_breakpoint_on_signal', + 'is_interactive', + 'LadderLogger', + 'no_stdout_within_functions', + 'colorize', +] diff --git a/ladder/core/common/logger/colorize.py b/ladder/core/common/logger/colorize.py new file mode 100644 index 0000000000000000000000000000000000000000..0da9498ec3a7ffc2a84e821716f6cfca73b03b96 --- /dev/null +++ b/ladder/core/common/logger/colorize.py @@ -0,0 +1,27 @@ +COLORS = { + 'gray': 37, + 'red': 31, + 'green': 32, + 'yellow': 93, + 'blue': 94, + 'magenta': 35, + 'cyan': 36, + 'white': 97, +} + + +def colorize(string: str, color: str, bold: bool = False, highlight: bool = False) -> str: + """ + Modify string so it appears in color on the console + """ + attr = [] + code = COLORS[color] + + if highlight: + code += 10 + attr.append(str(code)) + + if bold: + attr.append('1') + + return '\x1b[%sm%s\x1b[0m' % (';'.join(attr), string) diff --git a/ladder/core/common/logger/logger.py b/ladder/core/common/logger/logger.py new file mode 100644 index 0000000000000000000000000000000000000000..6d124d49121d329d629829f707427e1b05a634fc --- /dev/null +++ b/ladder/core/common/logger/logger.py @@ -0,0 +1,223 @@ +import inspect +import logging +import os +import signal +import sys +import threading +from typing import Any, Dict, Set, Union + +from .colorize import colorize + +DEBUG_PATTERNS = [patt for patt in os.environ.get('DEBUG', '').split(',') if patt] + +# Default overrides for modules with spammy logs. Patterns should be comma-separated +DEFAULT_LOG_MODULE_OVERRIDES = 'filelock=ERROR' + + +def _parse_log_overrides(log_override_pattern: str) -> Dict[str, str]: + """ + Parses log override settings: + Args: + log_override_pattern: str with comma-separated values. Each value should be of the format + `path.to.module=LOG_LEVEL`, e.g. `ladder.core.common.logger.logger=INFO` + Returns: + Dict of module names and corresponding log level + """ + overrides = {} + for setting in log_override_pattern.split(','): + parts = setting.strip().split('=') + if len(parts) < 2: + continue + overrides[parts[0]] = parts[1] + return overrides + + +LOG_MODULE_OVERRIDES = _parse_log_overrides( + ','.join([DEFAULT_LOG_MODULE_OVERRIDES, os.environ.get('LOG_MODULE_OVERRIDES', '')]) +) + +ROOT_LOGGER_LEVEL: str = os.environ.get('ROOT_LOGGER_LEVEL', 'INFO') + +_root_logger_set = False +_root_logger_lock = threading.Lock() + +# Loggers whose records must not reach the console. Enforced by a single filter on the console handler +# created in `get_root_logger`, so a logger can be excluded at any point without any caller having to +# find handlers itself - and without risk of catching a FileHandler, which subclasses StreamHandler. +_CONSOLE_EXCLUDED_LOGGERS: Set[str] = set() + + +def _disable_console_logging(logger_name: str) -> None: + """ + Stop `logger_name`'s records reaching the console. Every other handler still receives them. + + For processes that share one terminal but log to separate files - e.g. every rank of a distributed + job streaming to the same stdout - where the duplicate copies add nothing because each process + already writes its own log file. + """ + _CONSOLE_EXCLUDED_LOGGERS.add(logger_name) + + +class LadderLogger: + def __init__(self, logger: logging.Logger) -> None: + self._logger = logger + self._once_call_sites: Set[tuple] = set() + + @property + def name(self): + return self._logger.name + + def is_enabled_for_level(self, level: int): + return self._logger.isEnabledFor(level) + + def add_handler(self, handler: logging.Handler) -> None: + self._logger.addHandler(handler) + + def debug(self, event_type: str, **info) -> None: + self._log(logging.DEBUG, event_type, info) + + def info(self, event_type: str, color: str | None = None, **info) -> None: + self._log(logging.INFO, event_type, info, color=color) + + def info_once(self, event_type: str, **info) -> None: + """Like info, but will only print for the first time and drop the log statement afterwards.""" + + assert type(self) is LadderLogger, "info_once will not work when subclassing the logger" + # It will also not work when delegating the logger etc. + + call_site = inspect.stack()[1] + if (call_site.filename, call_site.lineno) not in self._once_call_sites: + self._once_call_sites.add((call_site.filename, call_site.lineno)) + self._log(logging.INFO, event_type, info) + + def warn(self, event_type: str, **info) -> None: + self._log(logging.WARN, event_type, info, color='yellow') + + def error(self, event_type: str, **info) -> None: + self._log(logging.ERROR, event_type, info, color='red') + + def critical(self, event_type: str, **info) -> None: + self._log(logging.CRITICAL, event_type, info, color='red') + + def exception(self, event_type: str, _: Exception, **info) -> None: + self._log(logging.ERROR, event_type, info, log=self._log_exception) + + def log(self, level: int, event_type: str, **info) -> None: + self._log(level, event_type, info) + + def _log( + self, level: int, event_type: str, info: Dict[str, Any], log=None, color: str | None = None + ) -> None: + if log is None: + log = self._logger.log + + if not self._logger.isEnabledFor(level): + return + + extra = {'event_type': event_type} + + if info: + extra.update(info) + if 'message' in extra: + extra.pop('message') # message is set by logger. + message = '%s %s' % (event_type, ", ".join(f"{k}={v}" for k, v in info.items())) + else: + message = event_type + + if color: + message = colorize(message, color) + + log(level, message, extra=extra) + + def _log_exception(self, _, msg, *args, **kwargs) -> None: + self._logger.exception(msg, *args, **kwargs) + + def close(self) -> None: + self._logger.handlers.clear() + + def set_level(self, level: Union[int, str]) -> None: + self._logger.setLevel(level) + + def disable_console(self) -> None: + """ + Stop this logger's records reaching the console; every other handler still receives them. + + Use when the decision depends on runtime state that isn't known at import time - e.g. the + distributed rank. Otherwise prefer `get_logger(..., console=False)`. + """ + _disable_console_logging(self.name) + + +def is_interactive() -> bool: + """Determines whether the python application has been invoked from an interactive shell or not""" + return sys.stdin.isatty() + + +def get_root_logger() -> logging.Logger: + global _root_logger_set # noqa: PLW0603 (global-statement) + with _root_logger_lock: + root_logger = logging.getLogger() + if not _root_logger_set: + root_logger.handlers.clear() # There could be some default handlers we don't care about. Remove them. + root_logger.setLevel(ROOT_LOGGER_LEVEL) + stream_handler = logging.StreamHandler() + stream_handler.setLevel(ROOT_LOGGER_LEVEL) + + formatter = logging.Formatter( + '%(levelname)-8s %(asctime)-15s \033[1;96m%(name)-12s\033[0m %(message)s' + ) + + stream_handler.setFormatter(formatter) + stream_handler.addFilter(lambda record: record.name not in _CONSOLE_EXCLUDED_LOGGERS) + root_logger.addHandler(stream_handler) + _root_logger_set = True + return root_logger + + +def get_logger(module_name: str, level=logging.INFO, console: bool = True) -> LadderLogger: + """ + Args: + module_name: Logger name, normally `__name__` + level: Level for this logger + console: If False, this logger's records go to every handler except the console - useful when + many processes share one terminal but each also writes its own log file. Evaluated here, so + only use it for a decision known at import time; otherwise call + `LadderLogger.disable_console()` once the decision is made. + """ + logger = logging.getLogger(module_name) + logger.setLevel(level) + get_root_logger() # Ensure root logger exists. + + if not console: + _disable_console_logging(module_name) + + if any(module_name.startswith(patt) for patt in DEBUG_PATTERNS): + logger.setLevel(logging.DEBUG) + + handlers = [h for h in logging.root.handlers if isinstance(h, logging.StreamHandler)] + for handler in handlers: + handler.setLevel(logging.DEBUG) + + # Override manually configured loggers + _set_log_level_overrides() + + return LadderLogger(logger) + + +def enter_ipdb_breakpoint_on_signal(signal_type=signal.SIGUSR1): + def handle_signal(_, frame): + import ipdb # noqa: T100 + + ipdb.set_trace(frame) # noqa: T100 + + signal.signal(signal_type, handle_signal) + print(f'Run `kill -{signal_type} {os.getpid()}` to enter ipdb`') + + +def _set_log_level_overrides() -> None: + """ + Override certain loggers in `LOG_MODULE_OVERRIDES` to their corresponding levels. + Used to set logger levels from command line (via env variable) or to silence modules with spam logs. + """ + for module, level in LOG_MODULE_OVERRIDES.items(): + logging.getLogger(module).setLevel(level) diff --git a/ladder/core/common/logger/stdout.py b/ladder/core/common/logger/stdout.py new file mode 100644 index 0000000000000000000000000000000000000000..514159471a4df8789809bf774c5fd955d7477c25 --- /dev/null +++ b/ladder/core/common/logger/stdout.py @@ -0,0 +1,39 @@ +import contextlib +import os +import sys + + +class DummyFile: + def write(self, x): + pass + + +@contextlib.contextmanager +def no_stdout_within_functions(): + """ + Silences std output (but not error) within this context manager, + even when calling external functions and processes + """ + # open fds + null_fds = os.open(os.devnull, os.O_RDWR) + # save the current file descriptors to a tuple + save = os.dup(1) + # put /dev/null fds on 1 (stout) but not 2 (stderr) + os.dup2(null_fds, 1) + + # *** run the function *** + yield + + # restore file descriptors so I can print the results + os.dup2(save, 1) + # close the temporary fds + os.close(null_fds) + + +@contextlib.contextmanager +def no_stdout(): + """silences output within this context manager""" + save_stdout = sys.stdout + sys.stdout = DummyFile() + yield + sys.stdout = save_stdout diff --git a/ladder/core/common/platform.py b/ladder/core/common/platform.py new file mode 100644 index 0000000000000000000000000000000000000000..bbe743b404f11f35cb4363595039956c9e8f1b8c --- /dev/null +++ b/ladder/core/common/platform.py @@ -0,0 +1,121 @@ +import os +import socket +from typing import List + +import psutil + + +def is_test_env() -> bool: + return 'TEST_UNDECLARED_OUTPUTS_DIR' in os.environ + + +def is_running_in_docker() -> bool: + def docker_in_cgroup(): + with open('/proc/self/cgroup') as f: + return any('docker' in line for line in f) + + return os.path.exists('/.dockerenv') or (os.path.isfile('/proc/self/cgroup') and docker_in_cgroup()) + + +def _get_ladder_repo_root() -> str: + """Absolute path to the root of this repository (ladder)""" + # NOTE: Use os.path.realpath - this also resolves bazel symlinks and directs to the real root path + return os.path.realpath(os.path.dirname(os.path.realpath(__file__)) + "/../../..") + + +# ----------------- +# Job metadata +# ----------------- + +EXPERIMENT_NAME = os.getenv('EXPERIMENT_NAME', None) + +# ----------------- +# SYSTEM +# ----------------- + +USERNAME = os.getenv('LADDER_USERNAME', '') or os.getenv('USER', '') +HOSTNAME: str = socket.gethostname().split('.')[0] + +NODE_VISIBLE_CPUS_COUNT: int = len(os.sched_getaffinity(os.getpid())) # CPUs visible to this process +NODE_TOTAL_CPUS_COUNT: int = psutil.cpu_count() # Total number of CPUs on this node + +# ----------------- +# Repo +# ----------------- + +# Absolute path to the root of this repository (ladder) +LADDER_ROOT = REPO_ROOT = _get_ladder_repo_root() + +# Set environment variable for the repo root +os.environ['LADDER_ROOT'] = LADDER_ROOT + +# TODO: Can also try /var/lock or ~/. Can make a function that tests permissions +LADDER_LOCKFILES_DIR = os.getenv('LADDER_LOCKFILES_DIR', '/tmp') + +# ----------------- +# CUDA +# ----------------- + + +def cuda_visible_devices_ids() -> List[str]: + """ + Return a list of strings of the values in CUDA_VISIBLE_DEVICES. Note these aren't guaranteed + to be numbers, e.g. GPU UUIDs such as 'GPU-00000000-0000-0000-0000-000000000000' + """ + return [gpu_id for gpu_id in os.getenv("CUDA_VISIBLE_DEVICES", "").split(',') if gpu_id != ''] + + +def cuda_visible_devices_indices() -> List[int]: + """ + Return a list of ints corresponding to the values in CUDA_VISIBLE_DEVICES. + - If CUDA_VISIBLE_DEVICES holds GPU UUIDs, it returns consecutive numbers, one per visible device + - Otherwise, the values in CUDA_VISIBLE_DEVICES, which are a subset of the numbers in `nvidia-smi`, + not guaranteed to be ordered or consecutive + """ + gpu_ids = cuda_visible_devices_ids() + + if any(not gpu_id.isdigit() for gpu_id in gpu_ids): + return [i for i in range(len(gpu_ids))] + return [int(gpu_id) for gpu_id in gpu_ids] + + +# ----------------- +# HUGGING FACE +# ----------------- +HF_TOKEN = os.getenv('HF_TOKEN', '') + +# ----------------- +# Torch +# ----------------- +# `torchrun` would define these env variables before launching the script +# https://pytorch.org/docs/stable/elastic/run.html#environment-variables +IS_RUNNING_IN_TORCHRUN = 'TORCHELASTIC_RUN_ID' in os.environ + +# ----------------- +# Docker +# ----------------- +DOCKER_TRAIN_IMAGE = os.getenv('DOCKER_TRAIN_IMAGE', 'no-docker-image-detected') + +# ----------------- +# Cloud +# ----------------- +# TODO: These are just examples. Proper detection needs to be done +IS_RUNNING_IN_CLOUD = os.getenv('IS_RUNNING_IN_CLOUD', '') == 'true' +IS_RUNNING_IN_AWS = os.getenv('IS_RUNNING_IN_AWS', '') == 'true' +IS_RUNNING_IN_AZURE = os.getenv('IS_RUNNING_IN_AZURE', '') == 'true' +IS_RUNNING_IN_GCP = os.getenv('IS_RUNNING_IN_GCP', '') == 'true' + +if IS_RUNNING_IN_AWS: + TRAINING_PLATFORM = "SageMaker" + NODE_COUNT = int(os.getenv('NODE_COUNT', '-1')) +elif IS_RUNNING_IN_AZURE: + TRAINING_PLATFORM = "AzureML" + NODE_COUNT = int(os.getenv('NODE_COUNT', '-1')) +elif IS_RUNNING_IN_GCP: + TRAINING_PLATFORM = "VertexAI" + NODE_COUNT = int(os.getenv('NODE_COUNT', '-1')) +else: + TRAINING_PLATFORM = "Local Machine" + NODE_COUNT = 1 + +IS_MULTINODE = NODE_COUNT > 1 diff --git a/ladder/core/databib/__init__.py b/ladder/core/databib/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/ladder/core/databib/config/__init__.py b/ladder/core/databib/config/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..255d55f364a6cd8b6412748cabe7faf82059292a --- /dev/null +++ b/ladder/core/databib/config/__init__.py @@ -0,0 +1,16 @@ +from .auto_config import AutoConfig, AutoConfigurable +from .config import Config, ConfigT, Configurable +from .config_parser import ConfigParser, assign_dict_value_by_recursive_access +from .yaml_util import load_config, update_config_dict + +__all__ = [ + 'Config', + 'Configurable', + 'AutoConfig', + 'AutoConfigurable', + 'ConfigT', + 'ConfigParser', + 'assign_dict_value_by_recursive_access', + 'load_config', + 'update_config_dict', +] diff --git a/ladder/core/databib/config/auto_config.py b/ladder/core/databib/config/auto_config.py new file mode 100644 index 0000000000000000000000000000000000000000..224845b319cc454f9e07bfe8734c83cf920907e2 --- /dev/null +++ b/ladder/core/databib/config/auto_config.py @@ -0,0 +1,142 @@ +import functools +import inspect +import pprint +from typing import Any, Type, TypeVar + +from ladder.core.databib.dataclasses.dataclass_utils import parametrize + +from .config import ( + AutoRecordsManager, + Config, + ConfigAttributeContext, + ConfigMeta, + ConfigT, + MissingConfigAttributeError, +) + + +class AutoConfigMeta(ConfigMeta): + @functools.lru_cache(maxsize=None, typed=True) # noqa: B019 + def __getitem__(cls, config_type) -> type: # noqa: N805 + """AutoConfig[ModelConfig]: an AutoConfig that only creates subclasses of ModelConfig.""" + if cls is not AutoConfig: + raise TypeError(f"Only AutoConfig can be subscripted, not {cls.__name__}") + # Prevent things like AutoConfig[TypeVar] or AutoConfig[Any] + if not inspect.isclass(config_type): + raise TypeError(f"{cls.__name__} expects a config class, but got {config_type}") + return parametrize(cls, (config_type,), ConfigT=config_type) + + def __instancecheck__(cls, maybe_config): # noqa: N805 + """ + Override instance checks so that AutoConfig type annotations in classes work properly + isinstance(ConfigA(), AutoConfig[ConfigA]) -> True + isinstance(ConfigB(), AutoConfig[ConfigA]) -> False + isinstance(ConfigA(), AutoConfig) -> True + """ + if not isinstance(cls.ConfigT, TypeVar): + return isinstance(maybe_config, cls.ConfigT) + if isinstance(maybe_config, Config): + return True + return super().__instancecheck__(maybe_config) + + +class AutoConfig(Config, metaclass=AutoConfigMeta): + """ + A generic config class that will be instantiated as a specific config class returned by __new__. + Selecting the correct config class requires setting the correct class path to a corresponding + Configurable class in the autoclass attribute, e.g. databib.config.config.Configurable. The returned + config class is the one to which `ConfigT` attribute points to. Note that only Configurable classes + whose config class is concrete (not a TypeVar) can be specified. + + Example: + class ModelConfig(Config): + ... + + class Model(Configurable[ModelConfig]): + ... + + model_config = AutoConfig(autoclass="full.path.to.module.Model") + assert isinstance(model_config, ModelConfig) # True + assert model_config.autoclass == "full.path.to.module.Model" # True + model = AutoConfiguralbe(model_config) + assert isinstance(model, Model) # True + """ + + ConfigT = ConfigT # a TypeVar until the class is subscripted, e.g. AutoConfig[ModelConfig] + + def __new__(cls, *args, **kwargs): + ConfigType = cls._get_actual_config_class(kwargs) + # Make sure the AutoConfig[...] argument is respected + if not isinstance(cls.ConfigT, TypeVar): + if not issubclass(ConfigType, cls.ConfigT): + raise TypeError( + f"Class {cls.__name__} can only create config instances, which are " + f"subclasses of {cls.ConfigT}, but instance of {ConfigType} requested" + ) + return ConfigType(*args, **kwargs) + + @classmethod + def _get_actual_config_class(cls, data: dict) -> Type[Config]: + key = "autoclass" + + with ConfigAttributeContext(cls, key): + if key not in data: + raise MissingConfigAttributeError(f"Field '{key}' of {cls.__name__} was not provided") + + class_path = data[key] + if class_path == "": + raise ValueError( + f"Attribute {key} missing for {cls.__name__}. Availble data:\n{pprint.pformat(data, indent=2)}" + ) + + ConfigType = AutoRecordsManager.config_cls_from_configurable_path(class_path) + + return ConfigType # type: ignore[return-value] + + @classmethod + def _ignore_redundant_keys(cls, data: dict) -> dict: + # Make sure from_json works with AutoConfig + ConfigType = cls._get_actual_config_class(data) + return ConfigType._ignore_redundant_keys(data) # pylint: disable=protected-access # noqa: SLF001 + + +class AutoConfigurable: + """ + A generic configurable class that will be instantiated as a specific instance of Configurable returned by __new__. + Selecting the correct config class requires passing a config instance with autoclass attribute set to + a full class path, e.g. databib.config.config.Configurable. Note that only Configurable classes whose config + class is concrete (not a TypeVar) can be specified. Note that only Configurable classes will be created + + Example: + class ModelConfig(Config): + ... + + class Model(Configurable[ModelConfig]): + ... + + model_config = AutoConfig(autoclass="full.path.to.module.Model") + assert isinstance(model_config, ModelConfig) # True + assert model_config.autoclass == "full.path.to.module.Model" # True + model = AutoConfiguralbe(model_config) + assert isinstance(model, Model) # True + """ + + def __new__(cls, config, **kwargs): + ConfigurableT = AutoConfigurable.target_class_from_config(config) + return ConfigurableT(config=config, **kwargs) + + @staticmethod + def target_class_from_config(config: Config) -> Type: + if not isinstance(config, Config): + raise TypeError( + f"{AutoConfigurable.__name__} expects an instance of Config, but got {type(config)}" + ) + + return AutoRecordsManager.configurable_cls_from_config(config) # type: ignore[arg-type] + + def __getattr__(self, _name: str) -> Any: + """Define to help pyright""" + + def __call__(self, *_args, **_kwargs) -> Any: + """Define to help pyright""" + return None diff --git a/ladder/core/databib/config/config.py b/ladder/core/databib/config/config.py new file mode 100644 index 0000000000000000000000000000000000000000..0539714a584f79c2d221728b2f286124749c0f7a --- /dev/null +++ b/ladder/core/databib/config/config.py @@ -0,0 +1,780 @@ +import collections +import dataclasses +import enum +import functools +import importlib +import inspect +import os +import pprint +import sys +from abc import ABCMeta +from typing import Any, ClassVar, Dict, Generic, List, Sequence, TextIO, Tuple, Type, TypeVar + +from deepdiff import DeepDiff +from frozendict import FrozenOrderedDict, frozendict + +from ladder.core.databib.config.yaml_util import load_config +from ladder.core.databib.dataclasses import ( + Dataclass, + DataclassDict, + DataclassInterface, + DataclassList, + DataclassMeta, +) +from ladder.core.databib.dataclasses.dataclass_utils import parametrize +from ladder.core.databib.utils.classproperty import cached_classproperty +from ladder.core.databib.utils.typing_utils import ( + annotation_is_dict, + annotation_is_enum, + annotation_is_list, + annotation_is_optional, + annotation_is_tuple, + annotation_is_union, + get_annotation_type_args, + get_dict_annotation_type_args, + get_list_annotation_type_args, + get_maybe_optional_type, + maybe_autocast_native_types, + value_type_is_annotation_type, +) + +_FROZEN_DEFAULTS_TYPE_MAPPING: Dict[str, type] = { + 'dict': frozendict, + 'list': tuple, + 'OrderedDict': FrozenOrderedDict, + 'set': frozenset, +} + +_FROZEN_TYPE_MAPPING: Dict[str, type] = { + 'set': frozenset, +} + + +class ConfigMeta(DataclassMeta): + def __new__(mcs, name, bases, attrs, **kwargs): # noqa: N804 + mcs._assert_config_is_non_generic(name, bases) + mcs._assert_post_init_calls_super(name, attrs) + mcs._assert_class_does_not_overwrite_autoclass_key(name, attrs) + + attrs = mcs._replace_default_values_with_frozen_types(attrs) + + new_cls = super().__new__(mcs, name, bases, attrs, **kwargs) + new_dataclass = dataclasses.dataclass(new_cls, init=False, frozen=True, repr=False) + new_dataclass._prevent_forbidden_type_annotations() # noqa: SLF001 + + # frozen=True causes __setattr__ to get generated (and doesn't allow it to be defined on the class) + # Override the one generated by the dataclass to get around that restriction. + new_dataclass.__setattr__ = new_dataclass.__config_setattr__ + + return new_dataclass + + @staticmethod + def _assert_post_init_calls_super(name, attrs): + """Check the __post_init__ definition of the subclass and make sure it calls super().__post_init__()""" + if name == "Config" or "__post_init__" not in attrs: + return + if "super().__post_init__()" not in inspect.getsource(attrs["__post_init__"]): + raise RuntimeError(f"{name}.__post_init__ method is missing a call to super().__post_init__()") + + @staticmethod + def _assert_config_is_non_generic(name, bases): + """ + Make sure that the Config subclass doesn't inherit from Generic. + This functionality is not supported and causes multiple problems + """ + # AutoConfig[X] is the one parametrized config class + if name == "AutoConfig" or name.startswith("AutoConfig["): + return + for base in bases: + if base is Generic or getattr(base, "__parameters__", False): + raise TypeError(f"Generic config classes are not allowed. Please make {name} non-generic.") + + @staticmethod + def _assert_class_does_not_overwrite_autoclass_key(name, attrs): + """ + Make sure that the special `autoclass` key of AutoConfig is not accidentally overwritten by a Config subclass + """ + if name == "Config": + return + annotations = attrs.get("__annotations__", {}) + autoclass_key = "autoclass" + if autoclass_key in annotations: + raise ValueError( + f"Class {name} defines attribute {autoclass_key}, which is a reserved attribute. Please rename." + ) + + @staticmethod + def _replace_default_values_with_frozen_types(attrs) -> Dict[str, Type]: + """ + Replace the mutable default values in a subclass of Config with their immutable counterparts. + Note that we are replacing only the default values, not the type annotations themselves. + """ + annotations = attrs.get("__annotations__", {}) + return { + key: _FROZEN_DEFAULTS_TYPE_MAPPING[type(value).__name__](value) + if key in annotations and type(value).__name__ in _FROZEN_DEFAULTS_TYPE_MAPPING + else value + for key, value in attrs.items() + } + + def _define_dict_and_list_attribute_classes(cls): # noqa: N805 + """ + Overloading DataclassMeta._define_dict_and_list_attribute_classes + + [IMPORTANT]: Suppose we have a Config + ``` + @dataclass + class MyConfig(Config): + my_field: Dict[str, int] + ``` + The values for my_field will always be cast to `ConfigDict[int]` inside `Config.__init__()` + However, the type `ConfigDict[int]` doesn't exist before we try to create an instance of + `MyConfig`. This is problematic because: + 1. We cannot properly unpickle such classes when using multiprocessing. In particular, the + class `MyConfig` will be known to the subprocess since the class will be imported. + However, `ConfigDict[int]` will be undefined, since the subprocess hasn't created + an instance of `MyConfig` before it tries to unpickle it + 2. If the types inside `Dict` are not allowed in `ConfigDict`, error will be raised only + when we try to create an instance, instead of at class definition time + To resolve these problems, here we define all the equivalent `ConfigDict` and `ConfigList` + classes needed for the new dataclass attributes + """ + import ladder.core.databib.config.config_iterables as config_iterables # pylint: disable=cyclic-import + + # NOTE: At this point, @dataclass has been applied so `__dataclass_fields__` is not the + # absolute source of truth for dataclass attributes. `__annotations__` gives us the fields added in + # this class definition, while `__dataclass_fields__` gives us the fields defined by the parents. + # We check both to also cover any transitive updates on parent class attributes, e.g. if + # called by `__getitem__` + annotations = getattr(cls, '__annotations__', {}) + dataclass_fields = getattr(cls, '__dataclass_fields__', {}) + fields = {**annotations, **{field.name: field.type for field in dataclass_fields.values()}} + + for field_name, field_type in fields.items(): + if annotation_is_dict(field_type): + _, DValueT = get_dict_annotation_type_args(field_type) + try: + config_iterables.ConfigDict[DValueT] + except TypeError as exc: + # Provide more informative error message when DValueT not allowed + message = f"Cannot cast {cls.__name__}.{field_name} to ConfigDict[{DValueT}].\n{exc}" + raise TypeError(message) from exc + if annotation_is_list(field_type): + LValueT: Type = get_list_annotation_type_args(field_type)[0] + try: + config_iterables.ConfigList[LValueT] + except TypeError as exc: + # Provide more informative error message when DValueT not allowed + message = f"Cannot cast {cls.__name__}.{field_name} to ConfigList[{LValueT}].\n{exc}" + raise TypeError(message) from exc + + def _prevent_forbidden_type_annotations(cls): # noqa: N805 + """ + Prevent forbidden type annotations for Config attributes + """ + + for field in getattr(cls, "__dataclass_fields__", {}).values(): + field_type = get_maybe_optional_type(field.type) + + # Forbid Union[Config, ...] and Tuple[Config, ...] + if annotation_is_union(field_type) or annotation_is_tuple(field_type): + type_args: Tuple[Type, ...] = get_annotation_type_args(field_type) + + is_config: List[bool] = [ + inspect.isclass(arg) and isinstance(arg, ConfigMeta) for arg in type_args + ] + if any(is_config): + etype = 'Union' if annotation_is_union(field_type) else 'Tuple' + raise TypeError( + f"Class {cls} has field {field.name}: {field.type}, but {etype} with members " + f"of type Config are not allowed. Find a better abstraction." + ) + + is_enum: List[bool] = [annotation_is_enum(arg) for arg in type_args] + + if sum(is_enum) > 1 and annotation_is_union(field_type): + raise TypeError( + f"Class {cls} has field {field.name}: {field.type}, " + f"but Union with multiple enums is not supported" + ) + + if sum(is_enum) > 0 and annotation_is_tuple(field_type): + raise TypeError( + f"Class {cls} has field {field.name}: {field.type}, " + f"but Tuple with enums is not supported" + ) + continue + + if annotation_is_dict(field_type) or annotation_is_list(field_type): + # Annotation type checks will be run by cls.types below + continue + # Forbid non-class type annotations + if ( + not inspect.isclass(field_type) + and not isinstance(field_type, dataclasses.InitVar) + and field_type is not dataclasses.InitVar + and getattr(field_type, "__origin__", None) is not ClassVar + ): + raise TypeError( + f"Forbidden type annotation in {cls} for field '{field.name}: {field.type}'. " + f"Expected a class, Optional[class], Dict, List, Tuple or Union" + ) + + +class Config(Dataclass, metaclass=ConfigMeta): + autoclass: str = "" # Special attribute for AutoConfig and AutoConfigurable + + def __init__(self, **kwargs): + object.__setattr__(self, "_Config__frozen", False) + + self._set_attributes(**kwargs) + self._init_remaining_subconfigs() + self._cast_dict_and_list_attrs() + self._assert_no_missing_attributes() + self.__post_init__() + self._freeze_mutable_attributes() + self._assert_post_init_is_idempotent() + # Cast dict and list attrs again to ConfigDict and ConfigList in case __post_init__ modified them + self._cast_dict_and_list_attrs() + self._make_type_checks() + + object.__setattr__(self, "_Config__frozen", True) + + def __post_init__(self): + pass + + def _set_attributes(self, **kwargs): + for key, value in kwargs.items(): + with ConfigAttributeContext(self.__class__, key): + # 1. Make sure key is a valid attribute of Config + if key not in self.__dataclass_fields__: + raise UnexpectedConfigAttributeError( + f"{self.__class__.__name__}.__init__() got unexpected keyword argument: " + f"{key}, expects one of: {list(self.__dataclass_fields__.keys())}" + ) + + annotation_type: Type = self.__dataclass_fields__[key].type + default_value: Any = self.__dataclass_fields__[key].default + + # 2. If annotated type is Config, recursively initialize its attributes + if isinstance(annotation_type, ConfigMeta): + # 2.1 `value` is neither a Config, nor a dict -> INVALID + if not isinstance(value, (Config, collections.abc.Mapping)): + raise InvalidConfigAttributeTypeError( + f"{self.__class__.__name__}.{key} expects dict-like object " + f"(nested config), but got string/value: {value}" + ) + + # 2.2. `value` is already a complete Config instance -> default values don't make + # a difference and we can confidently set the attribute to `value` + if isinstance(value, Config): + pass + + # 2.3. A default value is provided as a Config -> then apply `value` on top of the default + elif default_value != dataclasses.MISSING: + assert isinstance(value, collections.abc.Mapping), value + assert isinstance(default_value, Config), default_value + + value = replace_with_auto_config(default_value, value) + assert isinstance(value, annotation_type), value + + # 2.3 If `value` is not a Config, we make it one + if not isinstance(value, Config): + value = annotation_type(**value) + + # 3. Annotated type is Optional[Config] + elif annotation_is_optional(annotation_type) and isinstance( + get_maybe_optional_type(annotation_type), ConfigMeta + ): + # This is not supported yet + if default_value != dataclasses.MISSING and default_value is not None: + raise NotImplementedError( + f"Default values different from None are not supported for Optional[Config], " + f"but got {self.__class__.__name__}.{key}: {annotation_type} = {default_value}" + ) + if (value is not None) and not isinstance(value, Config): + value = get_maybe_optional_type(annotation_type)(**value) + + # 4. Annotated type is Dict[str, Any] + elif annotation_is_dict(annotation_type): + # 4.1 Make sure `value` is dict-like + if not isinstance(value, collections.abc.Mapping): + raise InvalidConfigAttributeTypeError( + f"{self.__class__.__name__}.{key} expects dict-like object " + f", but got value: {value}" + ) + # 4.2. SPECIAL CASE: If `value` is empty dict, then directly overwrite the default. + # Otherwise there is no way to overwrite defaults + if value != {}: + # Merge the current value of the attribute with the new value. + # Note that Dict[str, Config] or nested versions must support + # initialization with dict args for `Config` so that we have proper YAML support + value = _update_dict_of_maybe_configs(getattr(self, key, {}), value) + + # 5. Annotated type is enum.Enum + elif annotation_is_enum(annotation_type) and not isinstance(value, enum.Enum): + assert isinstance(value, str), value + value = annotation_type(value) + + # 6. Annotated type is Union[enum.Enum, ...] + elif isinstance(value, str) and annotation_is_union(annotation_type): + union_types: Tuple[Type, ...] = get_annotation_type_args(annotation_type) + is_enum: List[bool] = [ + annotation_is_enum(union_type) + for union_type in get_annotation_type_args(annotation_type) + ] + if any(is_enum): + enum_type = union_types[is_enum.index(True)] + value = enum_type(value) + + # 7. Autocast native types (e.g. int -> float, or list -> tuple for Tuple fields) + else: + value = maybe_autocast_native_types(value, get_maybe_optional_type(annotation_type)) + + setattr(self, key, value) + + @cached_classproperty + def _cast_fields(cls) -> Dict[str, Any]: # noqa: N805 + """ + Returns a frozendict containing only the fields annotated as Dict and List in this `Config` + and their equivalent `ConfigDict` and `ConfigList` types that values should be cast to. + Note that ConfigMeta already ensures that all Dict and List annotations can be cast. + + Note: we iterate over cls.fields instead of super()._cast_fields(), as the latter will materialize + a config class with DataclassDict / DataclassList children first, which we will throw away anyway. + + Returns: Dict[str, Union[Type[ConfigDict], Type[ConfigList]]] + """ + # pylint: disable=no-self-argument + import ladder.core.databib.config.config_iterables as config_iterables # pylint: disable=cyclic-import + + cast_fields: Dict[str, Type[DataclassInterface]] = {} + for field in cls.fields.values(): + if annotation_is_dict(field.type): + _, value_type = get_dict_annotation_type_args(field.type) + cast_fields[field.name] = config_iterables.ConfigDict[value_type] # type: ignore[name-defined] + elif annotation_is_list(field.type): + (value_type,) = get_list_annotation_type_args(field.type) + cast_fields[field.name] = config_iterables.ConfigList[value_type] # type: ignore[name-defined] + return frozendict(cast_fields) + + def _init_remaining_subconfigs(self): + """Initialize any attributes which were not passed to __init__ (maybe they have default values)""" + for key in self.__dataclass_fields__: + if not hasattr(self, key) and isinstance(self.__dataclass_fields__[key].type, ConfigMeta): + with ConfigAttributeContext(self.__class__, key): + setattr(self, key, self.__dataclass_fields__[key].type()) + + def _cast_dict_and_list_attrs(self): + """ + Cast attributes of sel annotated as Dict or List to their corresponding immutable + config equivalents, i.e. ConfigDict and ConfigList + """ + for field_name, field_type in self._cast_fields.items(): + field_value = getattr(self, field_name) + if type(field_value) is field_type: + # If field_value is already of type DataclassDict or DataclassList, + # then we don't need to worry about ValueT being a dataclass + continue + if inspect.isclass(field_type.ValueT) and issubclass(field_type.ValueT, Config): + # Cast the values inside the Dict or the List to Config (expected Config.__init__ behavior) + if issubclass(field_type, DataclassDict): + if not isinstance(field_value, collections.abc.Mapping): + raise TypeError( + f"Expected dict for field {field_name}, " + f"but got {field_value} of type {type(field_value)}" + ) + field_value = { + key: field_type.ValueT(**value) if not isinstance(value, Config) else value + for key, value in field_value.items() + } + elif issubclass(field_type, DataclassList): + if not isinstance(field_value, Sequence): + raise TypeError( + f"Expected dict for field {field_name}, " + f"but got {field_value} of type {type(field_value)}" + ) + field_value = [ + field_type.ValueT(**value) if not isinstance(value, Config) else value + for value in field_value + ] + setattr(self, field_name, field_type(field_value)) + + def _freeze_mutable_attributes(self): + for key in self.__dataclass_fields__: + value = getattr(self, key) + if type(value).__name__ in _FROZEN_TYPE_MAPPING: + setattr(self, key, _FROZEN_TYPE_MAPPING[type(value).__name__](value)) + + def _assert_no_missing_attributes(self): + all_missing_attributes = [] + for key in self.__dataclass_fields__.keys(): + if not hasattr(self, key): + all_missing_attributes.append(key) + + if all_missing_attributes: + with ConfigAttributeContext(self.__class__, all_missing_attributes[0]): + raise MissingConfigAttributeError( + f"Fields {all_missing_attributes} of {self.__class__.__name__} were not provided" + ) + + def _assert_post_init_is_idempotent(self): + """ + Make sure that when called twice, `self.__post_init__()` keeps references to all basic values + instead of generating new objects. Example: + ``` + config = Config(); + new_config = config.replace(**{}) + ``` + If __post_init__() generates any values dynamically, we want to make sure that config and new_config + have the same dynamically generated values, i.e. config == new_config + """ + post_init_dict = self.as_dict() + self.__post_init__() + self._freeze_mutable_attributes() + twice_post_init_dict = self.as_dict() + if post_init_dict != twice_post_init_dict: + dict_diff = DeepDiff(post_init_dict, twice_post_init_dict) + raise RuntimeError( + f"{self.__class__.__name__}.__post_init__() is not idempotent: " + f"got difference\n{pprint.pformat(dict_diff, indent=2)}" + ) + + def _make_type_checks(self): + """ + Overload Dataclass._make_type_checks to use exact type matching, with the exception of + AutoConfig fields, where we check that the value is an instance of the exact annotation bound + (the resolved Config subclass). This is the same policy ConfigDict / ConfigList apply to their + values via `_auto_config_isinstance`. + """ + import ladder.core.databib.config.config_iterables as config_iterables # pylint: disable=cyclic-import + + for field_name, field_type in self.types.items(): + value = getattr(self, field_name) + with ConfigAttributeContext(self.__class__, field_name): + if not value_type_is_annotation_type( + value, field_type, config_iterables._auto_config_isinstance # noqa: SLF001 + ): + raise InvalidConfigAttributeTypeError( + f"Incorrect type provided for {self.__class__.__name__}.{field_name}: " + f"expected {field_type}, but got value {value} with type {type(value)}" + ) + + @classmethod + def make_empty(cls): + raise NotImplementedError(f"make_empty() is disabled for Config classes ({cls.__name__})") + + @classmethod + def from_yaml(cls, yml_path: str, save_path: bool = True): + if not os.path.exists(yml_path): + raise FileNotFoundError(f'File not found: "{yml_path}"') + + config_dict = load_config(yml_path) + if config_dict is None: + raise ValueError("Config loaded from yaml file cannot be empty.") + + if save_path and 'config_path' in cls.keys(): + return cls(**config_dict, config_path=yml_path) + return cls(**config_dict) + + @classmethod + def from_yaml_file_object(cls, yml_file: TextIO): + config = load_config(yml_file) + if config is None: + raise ValueError("Config loaded from yaml file cannot be empty.") + return cls(**config) + + def __config_setattr__(self, key, value): + with ConfigAttributeContext(self.__class__, key): + if self.__frozen: + raise RuntimeError( + "Cannot set attribute '{}' to '{}' on a frozen object: {}".format(key, value, self) + ) + if key not in self.__dataclass_fields__: + raise UnexpectedConfigAttributeError( + "Cannot set attribute {} on {} as it is not one of its fields".format( + key, self.__class__.__name__ + ) + ) + object.__setattr__(self, key, value) + + def as_json(self, recursive: bool = True) -> dict: + data = super().as_json(recursive=recursive) + # Remove autoclass_key if it's empty to avoid clutter where the field is useless + if self.autoclass == "": + data.pop("autoclass", None) + + # Convert enum.Enum to str + for key, value in data.items(): + if isinstance(value, enum.Enum): + data[key] = str(value.value) + + return data + + +def replace_with_auto_config(default: Config, replacements: Dict[str, Any]) -> Config: + """ + Recursively replace `default` with `replacements` + + NOTE: This is NOT equivalent to `default.replace(**replacements)`. Consider these caveats: + 1. Let default has a field of type AutoConfig and replacements requests a different type via 'autoclass'. + Then `default.replace` will be incorrect as it will return the same type as the current value + of its field instead of the type specified by 'autoclass' in replacements + 2. The above cannot be always resolved by type(default)(**{**default.as_json(), **replacements}) either. + If default has a field of type AutoConfig, but its default value is a specific Config instance, + 'autoclass' will be empty when `default.as_json()` is called and cannot be reconstructed either + since we don't know the target Configurable type. + """ + for key, value in replacements.items(): + if isinstance(value, collections.abc.Mapping): + replacements[key] = replace_with_auto_config(getattr(default, key), value) # type: ignore[arg-type] + + if replacements.get('autoclass', '') != '': + # replacements changes the output type - we cannot call simple default.replace + # NOTE: If the annotation is not AutoConfig, the calling function will complain + import ladder.core.databib.config.auto_config as auto_config # pylint: disable=cyclic-import + + output: Config = auto_config.AutoConfig(**{**default.as_json(recursive=False), **replacements}) + else: + # Calling replace is safe since all nested dicts in replacements have been recursively + # converted to Config instances already and replace will overwrite them directly in `default` + output = default.replace(**replacements) + + return output + + +class ConfigAttributeContext: + attr_tree: List[Tuple[ConfigMeta, str]] = [] + + def __init__(self, config_class: Type[Config], attr_name: str): + self._config_class = config_class + self._attr_name = attr_name + + def __enter__(self): + self.attr_tree.append((self._config_class, self._attr_name)) + + def __exit__(self, exc_type, exc_val, exc_tb): + if exc_type is None: + self.attr_tree.pop() + + @classmethod + def reset(cls): + cls.attr_tree.clear() + + +class ConfigAttributeError(Exception): + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + self._attr_tree = self._get_attr_tree() + ConfigAttributeContext.reset() + + def _get_attr_tree(self): + attr_tree = "" + for i, (config_class, attr_name) in enumerate(ConfigAttributeContext.attr_tree): + attr_tree += config_class.__name__ + "\n" + (i + 1) * "\t" + attr_name + ": " + attr_tree += self.__class__.__name__ + return attr_tree + + def __str__(self): + return super().__str__() + "\n\n" + self._attr_tree + + +class UnexpectedConfigAttributeError(ConfigAttributeError): + pass + + +class MissingConfigAttributeError(ConfigAttributeError): + pass + + +class InvalidConfigAttributeTypeError(ConfigAttributeError): + pass + + +class ConfigAttributeInvalidValueError(ConfigAttributeError): + pass + + +ConfigT = TypeVar("ConfigT", bound=Config) + + +class ConfigurableMeta(ABCMeta): + @functools.lru_cache(maxsize=None, typed=True) # noqa: B019 + def __getitem__(cls, config_type): # noqa: N805 + """Configurable[ModelConfig] (or Model[ModelConfig]): the class with its config class set.""" + if not isinstance(cls.ConfigT, TypeVar): + raise TypeError(f"{cls.__name__} already has a config class: {cls.ConfigT}") + new_cls = parametrize(cls, (config_type,)) + # Set after creation, so only named subclasses (class Model(Configurable[ModelConfig])) register + new_cls.ConfigT = config_type + return new_cls + + +class Configurable(metaclass=ConfigurableMeta): + """A class built from a config, e.g. `class Model(Configurable[ModelConfig])`, which has + `Model.ConfigT is ModelConfig`. A class whose ConfigT is still a TypeVar can be subscripted again.""" + + ConfigT = ConfigT + + def __init__(self, config: ConfigT): + if not config.__class__ == self.ConfigT: + raise TypeError( + f"{self.__class__}.__init__() expects an instance of " + f"{self.ConfigT}, but got {config.__class__}" + ) + self._config = config + + @property + def config(self) -> ConfigT: + return self._config + + @classmethod + def from_kwargs(cls, **kwargs) -> "Configurable": + config = cls.ConfigT(**kwargs) + return cls(config) + + def __init_subclass__(cls, **kwargs) -> None: + super().__init_subclass__(**kwargs) + # Don't add any classes whose config class is still a TypeVar to the registry + if not isinstance(cls.ConfigT, TypeVar) and issubclass(cls.ConfigT, Config): + AutoRecordsManager.add_to_records(cls) + + +class AutoRecordsManager: + CONFIGURABLE_CLS_PATH_TO_CONFIG_CLS: Dict[str, Type[Config]] = {} + CONFIGURABLE_CLS_PATH_TO_CONFIGURABLE_CLS: Dict[str, Type[Configurable]] = {} + + @classmethod + def add_to_records(cls, ConfigurableT: Type[Configurable]): # noqa: N803 + assert issubclass(ConfigurableT, Configurable), ConfigurableT + + ConfigType: Type[Config] = ConfigurableT.ConfigT + + configurable_class_path = f"{ConfigurableT.__module__}.{ConfigurableT.__qualname__}" + + if configurable_class_path in cls.CONFIGURABLE_CLS_PATH_TO_CONFIG_CLS: + # Only print warning if running in interactive shell or notebook, to allow for re-executing cells + if is_interactive_py(): + print( + f"WARNING: Duplicate registration of class {configurable_class_path}. Deleting old records" + ) + else: + raise ValueError(f"Class {configurable_class_path} already registered. System Error.") + + cls.CONFIGURABLE_CLS_PATH_TO_CONFIG_CLS[configurable_class_path] = ConfigType + cls.CONFIGURABLE_CLS_PATH_TO_CONFIGURABLE_CLS[configurable_class_path] = ConfigurableT + + @classmethod + def config_cls_from_configurable_path(cls, configurable_class_path: str) -> Type[Config]: + if configurable_class_path == "" or "." not in configurable_class_path: + raise ValueError( + f"You need to provide the complete path to the target Configurable class, " + f"got '{configurable_class_path}'" + ) + + cls._maybe_import_class_from_module(configurable_class_path) + + klass = cls.CONFIGURABLE_CLS_PATH_TO_CONFIG_CLS.get(configurable_class_path, None) + + if klass is None: + expected_keys = list(cls.CONFIGURABLE_CLS_PATH_TO_CONFIG_CLS.keys()) + raise ValueError( + f"AutoConfig does not recognize the class path: '{configurable_class_path}'. Expected one of the following:" + f"\n{pprint.pformat(expected_keys, indent=2)}\nTo enable AutoConfig initialization, make sure the " + f"target class path points to a class that inherits from Configurable or to a Config class that " + f"is provided as type argument to a Configurable. Also make sure the python module where " + f"the Configurable class is defined is actually imported in your program." + ) + assert klass is not None # Help mypy + return klass + + @classmethod + def configurable_cls_from_configurable_path(cls, configurable_class_path: str) -> Type[Configurable]: + if configurable_class_path == "" or "." not in configurable_class_path: + raise ValueError( + f"You need to provide the complete path to the target Configurable class, " + f"got '{configurable_class_path}'" + ) + + cls._maybe_import_class_from_module(configurable_class_path) + + klass = cls.CONFIGURABLE_CLS_PATH_TO_CONFIGURABLE_CLS.get(configurable_class_path, None) + + if klass is None: + expected_keys = list(cls.CONFIGURABLE_CLS_PATH_TO_CONFIGURABLE_CLS.keys()) + raise ValueError( + f"AutoConfig does not recognize the class path: '{configurable_class_path}'. Expected one of the following:" + f"\n{pprint.pformat(expected_keys, indent=2)}\nTo enable AutoConfig initialization, make sure the " + f"target class path points to a class that inherits from Configurable or to a Config class that " + f"is provided as type argument to a Configurable. Also make sure the python module where " + f"the Configurable class is defined is actually imported in your program." + ) + assert klass is not None # Help mypy + return klass + + @classmethod + def configurable_cls_from_config(cls, config: Config) -> Type[Configurable]: + return cls.configurable_cls_from_configurable_path(config.autoclass) + + @classmethod + def _maybe_import_class_from_module(cls, class_path: str) -> None: + """ + Import a class using its full path. Useful when using YAML configs - allows for automatically + selecting Config / Configurable classes without explicitly importing them in the executable + script. If the Configurable class hasn't been imported, it won't be registered and Auto won't work + """ + if class_path in cls.CONFIGURABLE_CLS_PATH_TO_CONFIGURABLE_CLS: + # Class already known and registered - no need to import + return + + assert "." in class_path, class_path + module_path, class_name = class_path.rsplit(".", maxsplit=1) + try: + module = importlib.import_module(module_path) + except (ImportError, ModuleNotFoundError) as exc: + message = f"Can't import module {module_path}. Error: {exc}" + raise type(exc)(message) from exc + + klass = getattr(module, class_name) + + if not issubclass(klass, Configurable): + raise TypeError(f'Target class {class_path} must point to a subclass of Configurable') + + +def is_interactive_py() -> bool: + return hasattr(sys, 'ps1') + + +def _update_dict_of_maybe_configs(dict_1: Dict[str, Any], dict_2: Dict[str, Any]) -> Dict[str, Any]: + """ + Updates dict_1 with dict_2. Supports Config as value + Wherever there are matching keys, if both values are convertible to dict, then conversion is + done and the dicts are merged. Otherwise, value_2 overwrites value_1 + + Example 1: + dict_1: {'a': 5, 'b': {'c': 6, 'd': 7}} + dict_2: {'b': {'e': 3}} + result: {'a': 5, 'b': {'c': 6, 'd': 7, 'e': 3}} + Example 2: + dict_1: {'a': 5, 'b': {'c': 6, 'd': 7}} + dict_2: {'b': {'d': {'e': 3}}} + result: {'a': 5, 'b': {'c': 6, 'd': {'e': 3}}} + """ + dict_1 = dict(dict_1) + for key, value_2 in dict_2.items(): + value_1: Any = dict_1.get(key, {}) + + # Convert Config -> Dict[str, Any] + if isinstance(value_1, Config): + value_1 = value_1.as_json() + + # Convert Config -> Dict[str, Any] + if isinstance(value_2, Config): + value_2 = value_2.as_json() + + if isinstance(value_1, dict) and isinstance(value_2, dict): + dict_1[key] = _update_dict_of_maybe_configs(value_1, value_2) + # If either of the values isn't a Dict, overwrite value_1 directly + else: + dict_1[key] = value_2 + return dict_1 diff --git a/ladder/core/databib/config/config_iterables.py b/ladder/core/databib/config/config_iterables.py new file mode 100644 index 0000000000000000000000000000000000000000..2e675d0faee4ae837d79292ec7f615a94653deaf --- /dev/null +++ b/ladder/core/databib/config/config_iterables.py @@ -0,0 +1,60 @@ +from typing import Dict, List, Tuple, Type + +# Import as modules to avoid potential circular import problems +import ladder.core.databib.config.auto_config as auto_config +from ladder.core.databib.dataclasses import DataclassDict, DataclassList +from ladder.core.databib.utils.typing_utils import value_type_is_annotation_type + +def _auto_config_isinstance(value, field_type) -> bool: + """ + Run type checks by also accounting for AutoConfig + """ + # Check AutoConfig type + if issubclass(field_type, auto_config.AutoConfig): + return isinstance(value, field_type) + return type(value) is field_type + + +class ConfigDict(DataclassDict): + @classmethod + def _container_classes(cls) -> Tuple[Type, Type]: + return ConfigList, ConfigDict + + def _check_value_types_at_runtime(self): + """Override to allow for AutoConfig checks""" + if value_type_is_annotation_type(self, Dict[str, self.ValueT], _auto_config_isinstance): + return + # Provide more informative error + for key, value in self.items(): + if not value_type_is_annotation_type( + {key: value}, Dict[str, self.ValueT], _auto_config_isinstance + ): + raise TypeError( + f"Incorrect type provided for {self.__class__.__name__}: expected " + f"{self.ValueT}, but got {value} of type {type(value)} at key {key}" + ) + + def replace(self, strict: bool = False, **kwargs): + """ + Overload to change the default value of strict. + In ConfigDict, adding new attributes is common behavior + """ + return super().replace(strict=strict, **kwargs) + + +class ConfigList(DataclassList): + @classmethod + def _container_classes(cls) -> Tuple[Type, Type]: + return ConfigList, ConfigDict + + def _check_value_types_at_runtime(self): + """Override to allow for AutoConfig checks""" + if value_type_is_annotation_type(self, List[self.ValueT], _auto_config_isinstance): + return + # Provide more informative error + for index, value in enumerate(self): + if not value_type_is_annotation_type([value], List[self.ValueT], _auto_config_isinstance): + raise TypeError( + f"Incorrect type provided for {self.__class__.__name__}: expected " + f"{self.ValueT}, but got {value} of type {type(value)} at index {index}" + ) diff --git a/ladder/core/databib/config/config_parser.py b/ladder/core/databib/config/config_parser.py new file mode 100644 index 0000000000000000000000000000000000000000..fb8bd4ba59632ff4aea35729d3e0ba1657f44e4e --- /dev/null +++ b/ladder/core/databib/config/config_parser.py @@ -0,0 +1,148 @@ +import argparse +import ast +from typing import Any, Dict, Iterator, List, Optional, Tuple, Type + +import yaml + +from .config import Config +from .yaml_util import load_config, update_config_dict + + +class ConfigParser(argparse.ArgumentParser): + def __init__(self, config_class: Type[Config], config_path_key: Optional[str] = None): + super().__init__() + self.config_class = config_class + self.config_path_key = config_path_key + self.add_argument("--config", type=str, default=None) + + def _process_known_args(self, known_args: Dict[str, Any]) -> Optional[str]: + config_path = known_args.pop('config', None) + return config_path + + def _load_config(self, config_path: str) -> Dict[str, Any]: + config = load_config(config_path) + + if self.config_path_key is not None: + config = assign_dict_value_by_recursive_access(config, self.config_path_key, config_path) + + return config + + @staticmethod + def _set_recursive_arg_value(args_dict: Dict[str, Any], arg_path: str, arg_value: Any) -> Dict[str, Any]: + """ + Update a nested dictionary with a key and value in place + arg_path: Can be a nested path separated with `.` + """ + # Cast value to the correct type + try: + value: Any = ast.literal_eval(arg_value) + except (ValueError, SyntaxError): + value = arg_value + + name_path: List[str] = arg_path.split('.') + parents_path, base_name = name_path[:-1], name_path[-1] + + base_dict = args_dict + for key in parents_path: + if key not in base_dict: + base_dict[key] = {} + base_dict = base_dict[key] + base_dict[base_name] = value + + return args_dict + + @staticmethod + def _extra_args_name_value_iter(extra_args: List[str]) -> Iterator[Tuple[str, Any]]: + names = set() + i = 0 + while i < len(extra_args): + arg = extra_args[i] + if not arg.startswith('--'): + raise ValueError( + f"Expected argument starting with --, but got only {arg}. " + f"Argument specification should follow the syntax '--name value' or '--name=value'." + ) + + # Arg provided with syntax '--arg_name=value' + if '=' in arg: + data: List[str] = arg[2:].split('=') + if len(data) != 2: + raise ValueError(f"Argument name or value cannot contain '=', but got {arg}") + + name, value = data[0], data[1] + i += 1 + + # Arg provided with syntax '--arg_name value' + else: + if i == len(extra_args) - 1 or extra_args[i + 1].startswith('--'): + raise ValueError( + f"No value provided for argument {arg[2:]}. Expected {arg} or {arg}=." + ) + + name, value = arg[2:], extra_args[i + 1] + i += 2 + + if name in names: + raise ValueError(f"Duplicate arguments provided for {name}") + + names.add(name) + + yield name, yaml.safe_load(value) + + def _parse_cmdline_config_args(self, extra_args: List[str]) -> Dict[str, Any]: + cmdline_args: Dict[str, Any] = {} + + for name, value in self._extra_args_name_value_iter(extra_args): + cmdline_args = self._set_recursive_arg_value(cmdline_args, name, value) + + return cmdline_args + + def _parse_args_to_dict(self, args=None, namespace=None) -> Dict[str, Any]: + parsed_args: Tuple[argparse.Namespace, List[str]] = super().parse_known_args(args, namespace) + known_args: dict = vars(parsed_args[0]) + extra_args: List[str] = parsed_args[1] + + config_path: Optional[str] = self._process_known_args(known_args) + if config_path is not None: + yml_args: Dict[str, Any] = self._load_config(config_path) + else: + yml_args = {} + cmdline_args: Dict[str, Any] = self._parse_cmdline_config_args(extra_args) + config_args: Dict[str, Any] = update_config_dict(yml_args, cmdline_args) + + return config_args + + def _parse_config(self, config_args: Dict[str, Any], config: Optional[Config] = None) -> Config: + if config is None: + return self.config_class(**config_args) + return config.replace(**config_args) + + def _parse_args_to_config(self, args=None, namespace=None, config=None) -> Config: + config_args: Dict[str, Any] = self._parse_args_to_dict(args=args, namespace=namespace) + return self._parse_config(config_args, config) + + def parse_args(self, args=None, namespace=None, config=None): + config = self._parse_args_to_config(args=args, namespace=namespace, config=config) + return config + + def parse_args_to_dict(self, args=None, namespace=None) -> Dict[str, Any]: + """Parse config to a dict instead of a Config object""" + return self._parse_args_to_dict(args=args, namespace=namespace) + + def parse_from_path(self, config_path: str) -> Config: + config_dict = self._load_config(config_path) + return self._parse_config(config_dict) + + +def assign_dict_value_by_recursive_access(data: Dict[str, Any], full_key: str, value: Any) -> Dict[str, Any]: + keys: List[str] = full_key.split('.') + + inner_dict = data + for key in keys[:-1]: + if key not in inner_dict: + inner_dict[key] = {} + inner_dict = inner_dict[key] + + inner_dict[keys[-1]] = value + + return data diff --git a/ladder/core/databib/config/yaml_util.py b/ladder/core/databib/config/yaml_util.py new file mode 100644 index 0000000000000000000000000000000000000000..7d14aab90d4f580984b61e577299929d1f0ed145 --- /dev/null +++ b/ladder/core/databib/config/yaml_util.py @@ -0,0 +1,253 @@ +import os +import pathlib +import re +from typing import Any, Dict, List, Optional, TextIO, Tuple, Union, cast + +import yaml + +# Type alias representing a config "dependency" tree (to trace which configs are imported) +ConfigImportTree = Dict[str, Any] + + +class _OverwriteDict(dict): + """A custom dict class that marks nodes that overwrite existing nodes.""" + + def __repr__(self): + return f"{self.__class__.__name__}({super().__repr__()})" + + +class _OverwriteList(list): + """A custom dict class that marks nodes that overwrite existing nodes.""" + + def __repr__(self): + return f"{self.__class__.__name__}({super().__repr__()})" + + +def load_config(config_path: Union[str, TextIO, pathlib.Path]) -> Dict[str, Any]: + config, _ = _load_raw_yaml_with_import_tree(config_path) + config = _expand_env_vars_recursive(config) # type: ignore[assignment] + config = _strip_overwrite_types(config) # type: ignore[assignment] + return config + + +def _strip_overwrite_types(config: Dict[str, Any] | List[Any] | Any) -> Dict[str, Any] | List[Any] | Any: + if isinstance(config, dict): + return {key: _strip_overwrite_types(value) for key, value in config.items()} + if isinstance(config, list): + return [_strip_overwrite_types(item) for item in config] + return config + + +def _expand_env_vars_recursive(node: Dict[str, Any] | List[Any] | Any) -> Dict[str, Any] | List[Any] | Any: + if isinstance(node, dict): + return type(node)({key: _expand_env_vars_recursive(value) for key, value in node.items()}) + if isinstance(node, list): + return [_expand_env_vars_recursive(item) for item in node] + if isinstance(node, str): + return os.path.expandvars(node) + return node + + +def _merge_or_overwrite_value( + base_value: Dict[str, Any] | List[Any], merge_value: Dict[str, Any] | List[Any] +) -> Dict[str, Any] | List[Any]: + """ + Merge `base_value` and `merge_value` respecing the !overwrite tag, encoded as `_OverwriteDict` or + `_OverwriteList`. If merge_value is of type `_OverwriteDict` or `_OverwriteList`, completely overwrites + `base_value`. Otherwise merges `base_value` and `merge_value` + """ + if isinstance(merge_value, (_OverwriteDict, _OverwriteList)): + return merge_value + + # fmt: off + if ( + isinstance(base_value, dict) != isinstance(merge_value, dict) or + isinstance(base_value, list) != isinstance(merge_value, list) + ): + raise TypeError(f"Can't merge entry {base_value} with entry {merge_value}") + # fmt: on + + if isinstance(base_value, dict): + result = type(base_value)({key: value for key, value in base_value.items()}) + + for key, value in merge_value.items(): # type: ignore[union-attr] + # Overwrite local keys pointing to dict + if isinstance(value, _OverwriteDict): + result[key] = dict(value) # Convert _OverwriteDict to regular dict + # Overwrite local keys pointing to list + elif isinstance(value, _OverwriteList): + result[key] = list(value) # Convert _OverwriteList to regular list + # Merge nested dicts + elif isinstance(value, dict) and key in result: + if not isinstance(result[key], dict): + raise ValueError( + f"Cannot merge values for key {key} of types dict and {type(result[key])}" + ) + result[key] = _merge_or_overwrite_value(result[key], value) + elif isinstance(value, list) and key in result: + if not isinstance(result[key], list): + raise ValueError( + f"Cannot merge values for key {key} of types list and {type(result[key])}" + ) + # NOTE: We merge lists by direct appends + result[key] = [*result[key], *value] + else: + result[key] = value + elif isinstance(base_value, list): + result = type(base_value)([*base_value, *merge_value]) # type: ignore[assignment] + else: + result = merge_value + + return result + + +def _load_raw_yaml_with_import_tree( + config_path: Union[str, TextIO, pathlib.Path] +) -> Tuple[Dict[str, Any], ConfigImportTree]: + if isinstance(config_path, pathlib.Path): + config_path = str(config_path) + + loader = _Loader(config_path) + try: + # Note: get_single_data() returns None for an empty config, we want an empty dict + config = loader.get_single_data() or {} + import_tree = loader.import_tree + return config, import_tree + finally: + loader.dispose() + + +def _ordered_dict_representer(dumper, data): + return yaml.nodes.MappingNode( + yaml.SafeDumper.DEFAULT_MAPPING_TAG, + [(dumper.represent_data(k), dumper.represent_data(v)) for k, v in data.items()], + ) + + +class _Loader(yaml.SafeLoader): + config_path: Optional[str] + + def __init__(self, config_path_or_file: Union[str, TextIO]): + if type(config_path_or_file) is str: + self.config_path = cast(str, config_path_or_file) + config_file = open(self.config_path, 'r') + else: + self.config_path = None + config_file = cast(TextIO, config_path_or_file) # type: ignore + + super().__init__(config_file) + + self.add_implicit_resolver( + 'tag:yaml.org,2002:float', + re.compile( + '''^(?: + [-+]?(?:[0-9][0-9_]*)\\.[0-9_]*(?:[eE][-+]?[0-9]+)? + |[-+]?(?:[0-9][0-9_]*)(?:[eE][-+]?[0-9]+) + |\\.[0-9_]+(?:[eE][-+][0-9]+)? + |[-+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\\.[0-9_]* + |[-+]?\\.(?:inf|Inf|INF) + |\\.(?:nan|NaN|NAN))$''', + re.X, + ), + list('-+0123456789.'), + ) + + self.add_constructor(self.DEFAULT_MAPPING_TAG, self.mapping_constructor) + self.add_constructor('!overwrite', self.overwrite_constructor) + + # Keep track of the !import dependency tree. Note that we only support importing when reading + # from a path on disk. !import doesn't work with file descripors or buffers + self._dep_tree: ConfigImportTree = {} + + def overwrite_constructor(self, loader, node): + if isinstance(node, yaml.MappingNode): + return _OverwriteDict(loader.construct_mapping(node)) + if isinstance(node, yaml.SequenceNode): + return _OverwriteList(loader.construct_sequence(node)) + raise ValueError("Can only use `!overwrite` tag for dicts or lists") + + @property + def import_tree(self) -> ConfigImportTree: + return self._dep_tree + + @staticmethod + def mapping_constructor(loader, node): + # Everything here must go through ``loader``, never through the instance that registered this + # constructor: ``add_constructor`` registers on the Loader CLASS, so that instance is whichever + # loader was built LAST -- for a nested import, the imported file rather than the one being + # parsed. Resolving against it made a relative import land in the wrong directory whenever a + # file's first import pulled in a config from somewhere else, so a path that looked correct + # from its own location failed, and a wrong one could work by accident. + if not isinstance(node, yaml.nodes.MappingNode): + raise yaml.constructor.ConstructorError( + None, None, "expected a mapping node, but found %s" % node.id, node.start_mark + ) + + result = {} + + for key_node, value_node in node.value: + key = loader.construct_object(key_node, deep=True) + value = loader.construct_object(value_node, deep=True) + + if key == "import": + if loader.config_path is None: + raise ValueError('Can only use `!import` tag when loading from a path on disk') + import_path = os.path.join(os.path.dirname(loader.config_path), os.path.expandvars(value)) + if not os.path.isfile(import_path): + raise FileNotFoundError(f'Cannot load referenced yaml at: {import_path}') + + imported_config, imported_dep_tree = _load_raw_yaml_with_import_tree(import_path) + loader.merge_import_tree(import_path, imported_dep_tree) + # Merge the imported config immediately. Order of import and !overwrite matters: + # - `import` after `!overwrite` merges into `!overwrite` + # - `!overwrite` after `import` overwrites whatever is inside the `import` + result = _merge_or_overwrite_value(result, imported_config) + else: + if key not in result: + result[key] = value + else: + result[key] = _merge_or_overwrite_value(result[key], value) + + return result + + def merge_import_tree(self, import_path: str, imported_dep_tree: ConfigImportTree): + # Merge the imported dep tree with the existing one if that config was already imported. + self._dep_tree[import_path] = { + **self._dep_tree.get(import_path, {}), + **imported_dep_tree, + } + + def dispose(self): + self.stream.close() + super().dispose() + + +def update_config_dict(dict_1: Dict[str, Any], dict_2: Dict[str, Any]) -> Dict[str, Any]: + """ + Updates dict_1 with dict_2. Works for nested dictionaries. + If any value in dict_1 is not a dict, it cannot be replaced with a dict and vice-versa. + Example 1: + dict_1: {'a': 5, 'b': {'c': 6, 'd': 7}} + dict_2: {'b': {'e': 3}} + result: {'a': 5, 'b': {'c': 6, 'd': 7, 'e': 3}} + Example 2: + dict_1: {'a': 5, 'b': {'c': 6, 'd': 7}} + dict_2: {'b': {'d': {'e': 3}}} + result: Error: int cannot be replaced by a dict + """ + dict_1 = dict(dict_1) + for key, value in dict_2.items(): + if isinstance(value, dict): + source_value = dict_1.get(key, {}) + if not isinstance(source_value, dict): + raise TypeError( + f'Updating {source_value} of type {type(source_value)} with a dict {value} is not allowed.' + ) + dict_1[key] = update_config_dict(source_value, value) + else: + if isinstance(dict_1.get(key, None), dict): + raise TypeError( + f'Updating {dict_1.get(key)} of type dict with {value} of type {type(value)} is not allowed.' + ) + dict_1[key] = value + return dict_1 diff --git a/ladder/core/databib/dataclasses/__init__.py b/ladder/core/databib/dataclasses/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..25c6489150ce8ba62c3fdcdcdc5feb7013564ae3 --- /dev/null +++ b/ladder/core/databib/dataclasses/__init__.py @@ -0,0 +1,16 @@ +from .dataclass import Dataclass, DataclassMeta, DataclassT, dataclass +from .dataclass_dict import DataclassDict +from .dataclass_interface import DataclassInterface +from .dataclass_list import DataclassList +from .dataclass_utils import zip_dataclass_items + +__all__ = [ + 'dataclass', + 'Dataclass', + 'DataclassT', + 'DataclassDict', + 'DataclassInterface', + 'DataclassList', + 'DataclassMeta', + 'zip_dataclass_items', +] diff --git a/ladder/core/databib/dataclasses/dataclass.py b/ladder/core/databib/dataclasses/dataclass.py new file mode 100644 index 0000000000000000000000000000000000000000..76290f806dec4d1333a6caa17f3adfe1af9c335c --- /dev/null +++ b/ladder/core/databib/dataclasses/dataclass.py @@ -0,0 +1,391 @@ +import collections +import dataclasses +import enum +import inspect +from abc import ABCMeta +from typing import Any, Callable, ClassVar, Dict, List, Type, TypeVar, Union + +import numpy as np +from frozendict import frozendict + +try: + import torch +except ImportError: + # Don't explicitly depend on torch, it's optional + torch = type('', (), {"Tensor": np.ndarray})() + +from ladder.core.databib.dataclasses.dataclass_dict import DataclassDict +from ladder.core.databib.dataclasses.dataclass_interface import DataclassInterface +from ladder.core.databib.dataclasses.dataclass_list import DataclassList +from ladder.core.databib.dataclasses.dataclass_utils import maybe_chained_keys_to_nested_dict +from ladder.core.databib.utils.classproperty import cached_classproperty +from ladder.core.databib.utils.typing_utils import ( + ITERABLE_TYPES, + annotation_is_dict, + annotation_is_list, + annotation_is_tuple, + annotation_is_union, + get_annotation_type_args, + get_dict_annotation_type_args, + get_list_annotation_type_args, + get_maybe_optional_type, + get_origin_type, + value_type_is_annotation_type, +) + +DataclassT = TypeVar("DataclassT", bound="Dataclass") + + +def dataclass(cls: Type[DataclassT], **kwargs) -> Type[DataclassT]: + cls = dataclasses.dataclass(cls, frozen=True, repr=False, kw_only=True, **kwargs) # type: ignore[call-overload] + _prevent_forbidden_type_annotations(cls) + return cls + + +def _prevent_forbidden_type_annotations(cls): + """This cannot be added in a metaclass since __dataclass_fields__ does not exist when the metaclass is called""" + for field in getattr(cls, "__dataclass_fields__", {}).values(): + field_type = get_maybe_optional_type(field.type) + + # Forbid Union or Tuple with any member of type + # [DataclassInterface, Dict, List, Set, Tuple, TypeVar, Union, Optional] + if annotation_is_union(field_type) or annotation_is_tuple(field_type): + type_args = get_annotation_type_args(field_type) + for arg in type_args: + if ( + (inspect.isclass(arg) and issubclass(arg, DataclassInterface)) + or get_origin_type(arg) in ITERABLE_TYPES + (Union,) + or isinstance(arg, TypeVar) + ): + etype = 'Union' if annotation_is_union(field_type) else 'Tuple' + raise TypeError( + f"Class {cls} has field {field.name}: {field.type}, but {etype} with members of type " + f"DataclassInterface, Dict, List, Set, Tuple, Sequence, Union, Optional, TypeVar " + f"are unsafe and not allowed. Remove the Union instead." + ) + continue + if annotation_is_dict(field_type) or annotation_is_list(field_type): + # Annotation type checks will be run by cls.types below + continue + # Forbid non-class type annotations + if ( + not inspect.isclass(field_type) + and not isinstance(field_type, dataclasses.InitVar) + and field_type is not dataclasses.InitVar + and getattr(field_type, "__origin__", None) is not ClassVar + ): + raise TypeError( + f"Forbidden type annotation in {cls} for field '{field.name}: {field.type}'. " + f"Expected a class, Optional[class], Dict, List, Tuple or Union" + ) + + +class DataclassMeta(ABCMeta): + def __new__(mcs, name, bases, attributes, *args, **kwargs): # noqa: N804 + new_cls = super().__new__(mcs, name, bases, attributes, *args, **kwargs) + new_cls._define_dict_and_list_attribute_classes() # noqa: SLF001 + return new_cls + + def _define_dict_and_list_attribute_classes(cls): # noqa: N805 + """ + [IMPORTANT]: Suppose we have a dataclass + ``` + @dataclass + class MyDataclass(Dataclass): + my_field: Dict[str, int] + ``` + The values for my_field will always be cast to `DataclassDict[int]` inside `Dataclass.__init__()` + However, the type `DataclassDict[int]` doesn't exist before we try to create an instance of + `MyDataclass`. This is problematic because: + 1. We cannot properly unpickle such classes when using multiprocessing. In particular, the + class `MyDataclass` will be known to the subprocess since the class will be imported. + However, `DataclassDict[int]` will be undefined, since the subprocess hasn't created + an instance of `MyDataclass` before it tries to unpickle it + 2. If the types inside `Dict` are not allowed in `DataclassDict`, error will be raised only + when we try to create an instance, instead of at class definition time + To resolve these problems, here we define all the equivalent `DataclassDict` and `DataclassList` + classes needed for the new dataclass attributes + """ + + # NOTE: At this point, @dataclass hasn't been applied yet, so `__dataclass_fields__` is not the + # absolute source of truth for dataclass attributes. `__annotations__` gives us the fields added in + # this class definition, while `__dataclass_fields__` gives us the fields defined by the parents. + # We check both to also cover any transitive updates on parent class attributes, e.g. if + # called by `__getitem__` + annotations = getattr(cls, '__annotations__', {}) + dataclass_fields = getattr(cls, '__dataclass_fields__', {}) + fields = {**annotations, **{field.name: field.type for field in dataclass_fields.values()}} + + # Note that we check only direct attribute types and don't recurse, e.g. into something like + # `Dict[str, List[str]]`. For now, we care only about the dataclass field type and cast only that + for field_name, field_type in fields.items(): + if annotation_is_dict(field_type): + _, DValueT = get_dict_annotation_type_args(field_type) + try: + DataclassDict[DValueT] + except TypeError as exc: + # Provide more informative error message when DValueT not allowed + message = f"Cannot cast {cls.__name__}.{field_name} to DataclassDict[{DValueT}].\n{exc}" + raise TypeError(message) from exc + if annotation_is_list(field_type): + LValueT: Type = get_list_annotation_type_args(field_type)[0] + try: + DataclassList[LValueT] + except TypeError as exc: + # Provide more informative error message when DValueT not allowed + message = f"Cannot cast {cls.__name__}.{field_name} to DataclassList[{LValueT}].\n{exc}" + raise TypeError(message) from exc + + +@dataclass +class Dataclass(DataclassInterface, metaclass=DataclassMeta): + def __post_init__(self): + self._cast_dict_and_list_attrs() + self._make_type_checks() + + def _cast_dict_and_list_attrs(self): + """ + Cast attributes of type Dict or List to their corresponding DataclassDict and DataclassList types + """ + for field_name, field_type in self._cast_fields.items(): + value = getattr(self, field_name) + if type(value) is field_type: + continue + try: + object.__setattr__(self, field_name, field_type(value)) + except TypeError: + # Let self._make_type_checks() raise a more informative error + pass + + def _make_type_checks(self): + for field_name, field_type in self.types.items(): + value = getattr(self, field_name) + + if ( + value is None + and inspect.isclass(field_type) + and not issubclass(field_type, DataclassInterface) + ): + # Here we allow None for any field which is not a subclass of DataclassInterface. + # Technically, proper type checks should forbid any None value for non-Optional fields, + # but this is required for Dataclass.make_empty() to work in general. + # TODO: Figure out how to resolve this predicament in general + continue + + if not value_type_is_annotation_type(value, field_type): + raise TypeError( + f"Incorrect type provided for {self.__class__.__name__}.{field_name}: " + f"expected {self.types[field_name]}, but got value {value} with type {type(value)}" + ) + + @cached_classproperty + def fields(cls) -> Dict[str, dataclasses.Field]: # noqa: N805 + # pylint: disable=no-self-argument + """Returns a frozendict of the Field objects, keyed and sorted by field name""" + return frozendict({f.name: f for f in sorted(dataclasses.fields(cls), key=lambda x: x.name)}) + + @cached_classproperty + def annotated_types(cls) -> Dict[str, type]: # noqa: N805 + # pylint: disable=no-self-argument + return frozendict({name: f.type for name, f in cls.fields.items()}) + + @cached_classproperty + def types(cls) -> Dict[str, type]: # noqa: N805 + # pylint: disable=no-self-argument + return frozendict({name: cls._cast_fields.get(name, f.type) for name, f in cls.fields.items()}) + + @cached_classproperty + def _cast_fields(cls) -> Dict[str, Type[DataclassInterface]]: # noqa: N805 + """ + Returns a frozendict containing only the fields annotated as Dict and List in this `Dataclass` + and their equivalent `DataclassDict` and `DataclassList` types that values should be cast to. + Note that DataclassMeta already ensures that all Dict and List annotations can be cast. + """ + # pylint: disable=no-self-argument + cast_fields: Dict[str, Type[DataclassInterface]] = {} + for field in cls.fields.values(): + if annotation_is_dict(field.type): + _, DValueT = get_dict_annotation_type_args(field.type) + cast_fields[field.name] = DataclassDict[DValueT] # type: ignore[valid-type] + if annotation_is_list(field.type): + LValueT: Type = get_list_annotation_type_args(field.type)[0] + cast_fields[field.name] = DataclassList[LValueT] # type: ignore[valid-type] + return frozendict(cast_fields) + + @classmethod + def keys(cls) -> List[str]: + return list(cls.fields) + + @classmethod + def from_subclass(cls: Type[DataclassT], subclass_instance: DataclassT) -> DataclassT: + """Returns an instance of DataclassT, with all subclass properties removed""" + if not isinstance(subclass_instance, cls): + raise TypeError( + f"from_subclass expected an instance of (a subclass of) {cls.__name__}, " + f"but got {subclass_instance.__class__.__name__}" + ) + return cls.from_json( + subclass_instance.as_json(recursive=False), recursive=False, ignore_redundant=True + ) + + @classmethod + def from_json(cls, data: Dict[str, Any], recursive: bool = True, ignore_redundant: bool = False): + """Parses a dict to dataclass object""" + if not isinstance(data, collections.abc.Mapping): + raise TypeError( + f"{cls}.from_json() expects input of type dict, but got input of type {type(data)}" + ) + data = dict(data) + + if ignore_redundant: + data = cls._ignore_redundant_keys(data) + + if recursive: + for key, value in data.items(): + field_type = cls.types.get(key, None) + if field_type is None: + # This key is not part of the dataclass fields. Leave it as is. + # Dataclass.__init__() will raise error + continue + field_type = get_maybe_optional_type(field_type) + # If the field annotation is not a class, let the __new__ method deal with it + if not inspect.isclass(field_type): + continue + # If field is an Enum, convert it to the right type + if issubclass(field_type, enum.Enum): + data[key] = field_type(value) + if issubclass(field_type, DataclassInterface) and value is not None: + data[key] = field_type.from_json( + value, recursive=recursive, ignore_redundant=ignore_redundant + ) + + return cls(**data) + + @classmethod + def _ignore_redundant_keys(cls, data: dict) -> dict: + return {key: value for key, value in data.items() if key in cls.types} + + @classmethod + def from_dict(cls, d: dict, recursive: bool = True, ignore_redundant: bool = False): + return cls.from_json(d, recursive, ignore_redundant) + + @classmethod + def make_empty(cls: Type[DataclassT]) -> DataclassT: + return cls( + **{ + k: v.make_empty() if inspect.isclass(v) and issubclass(v, DataclassInterface) else None + for k, v in cls.types.items() + } + ) + + @property + def is_empty(self) -> bool: + for value in self.values(): + if isinstance(value, DataclassInterface): + if not value.is_empty: + return False + elif value is not None: + return False + return True + + def as_json(self, recursive: bool = True) -> dict: + return { + k: v.as_json() if isinstance(v, DataclassInterface) and recursive else v for k, v in self.items() + } + + def as_dict(self, recursive: bool = True) -> dict: + # cannot use the inbuilt asdict methods as they do a deepcopy of tensors + return self.as_json(recursive=recursive) + + def values(self): + return [getattr(self, name) for name in self.fields] + + def items(self, recursive: bool = False): + for key, value in zip(self.keys(), self.values(), strict=True): + if recursive and isinstance(value, DataclassInterface): + for subkey, subvalue in value.items(recursive=True): + yield f"{key}.{subkey}", subvalue + else: + yield key, value + + def getattr(self, name: str) -> Any: + """ + Recursive counterpart of the builtin getattr(): `name` can address an attribute of a nested + dataclass via dot-separation, e.g. 'nested_field.attr' + """ + value = self + for attr in name.split("."): + # Method names are not in scope inside the method body, so this is the builtin getattr() + value = getattr(value, attr) + return value + + def replace(self, **kwargs): + """ + Return a new instance of Dataclass with the kwargs overwritten. + """ + kwargs = maybe_chained_keys_to_nested_dict(kwargs) + + data = self.as_dict(recursive=False) + + for key, value in kwargs.items(): + if key not in self.types: + raise KeyError(f"Dataclass {self.__class__} does not have a field {key}") + + value_type = self.types[key] + value_type = get_maybe_optional_type(value_type) + + # Recurse if a DataclassInterface field + if inspect.isclass(value_type) and issubclass(value_type, DataclassInterface): + if isinstance(value, dict): + data[key] = data[key].replace(**value) + else: + # If the type is incorrect, delegate to __init__ to handle it + data[key] = value + else: + data[key] = value + + return self.__class__(**data) + + def apply(self, fcn: Callable, recursive: bool = True, skip_nones: bool = False): + def fcn_wrapper(value: Any) -> Any: + if value is None and skip_nones: + return None + if isinstance(value, DataclassInterface) and recursive: + return value.apply(fcn, recursive=True, skip_nones=skip_nones) + return fcn(value) + + return self.__class__(**{key: fcn_wrapper(value) for key, value in self.items()}) + + def __iter__(self): + for name in self.fields: + field_value = getattr(self, name) + if isinstance(field_value, DataclassDict): + for value in field_value.values(): + yield value + elif isinstance(field_value, DataclassInterface): + for value in field_value: + yield value + else: + yield field_value + + def shape_repr(self, indent: int = 0, skip_empty=True) -> str: + DEFAULT_INDENT = 2 + indent_str = " " * indent + field_indent_str = " " * (indent + DEFAULT_INDENT) + shape_strings = [] + + if self.is_empty and skip_empty: + return f"{indent_str}{self.__class__.__name__}.empty()" + + for key, value in self.items(): + if value is None: + shape_strings.append(f"{field_indent_str}{key}: None") + elif isinstance(value, Dataclass): + dataclass_repr = value.shape_repr(indent + DEFAULT_INDENT, skip_empty).strip() + shape_strings.append(f"{field_indent_str}{key}: {dataclass_repr}") + elif isinstance(value, (np.ndarray, torch.Tensor)): + shape_strings.append(f"{field_indent_str}{key}: {list(value.shape)}") + else: + shape_strings.append(f"{field_indent_str}{key}: -X-") + + return f"{indent_str}{self.__class__.__name__}(\n" + ",\n".join(shape_strings) + f",\n{indent_str})" diff --git a/ladder/core/databib/dataclasses/dataclass_dict.py b/ladder/core/databib/dataclasses/dataclass_dict.py new file mode 100644 index 0000000000000000000000000000000000000000..7ba938adb4827931f710e31fd35639044826fecc --- /dev/null +++ b/ladder/core/databib/dataclasses/dataclass_dict.py @@ -0,0 +1,192 @@ +import functools +import inspect +import sys +from abc import ABCMeta +from typing import Any, Callable, Dict, Tuple, Type, TypeVar + +from frozendict import frozendict + +from ladder.core.databib.dataclasses.dataclass_interface import DataclassInterface +from ladder.core.databib.dataclasses.dataclass_utils import parametrize +from ladder.core.databib.utils.typing_utils import ( + cast_container_types, + get_maybe_optional_type, + substitute_annotation_container_types, + type_annotation_contains_type_var, + value_type_is_annotation_type, +) + +ValueT = TypeVar("ValueT") +DataclassDictT = TypeVar("DataclassDictT", bound="DataclassDict") + + +class DataclassDictMeta(ABCMeta): + @functools.lru_cache(maxsize=None, typed=True) # noqa: B019 + def __getitem__(cls, value_type): # noqa: N805 + if not isinstance(cls.ValueT, TypeVar): + raise TypeError(f"{cls.__name__} already has a value type") + # Substitute nested List / Dict in the value type with their DataclassList / DataclassDict + # equivalents (ConfigList / ConfigDict for the Config variants) + ListType, DictType = cls._container_classes() + value_type = substitute_annotation_container_types(value_type, ListType, DictType) + new_cls = parametrize(cls, (value_type,), ValueT=value_type) + new_cls._raise_on_forbidden_value_type() # noqa: SLF001 + return new_cls + + def _raise_on_forbidden_value_type(cls): # noqa: N805 + assert cls.ValueT is not Ellipsis + + if isinstance(get_maybe_optional_type(cls.ValueT), TypeVar): + return + + if type_annotation_contains_type_var(cls.ValueT): + raise TypeError(f"TypeVar annotation allowed only directly or Optional[TypeVar], but got {cls}") + + +class DataclassDict(DataclassInterface, frozendict, metaclass=DataclassDictMeta): + """ + Immutable Dict container compatible with DataclassInterface API. + Assumes key type is always a str. Value type provided at runtime. + + Allowed value type annotations (including nested versions of these): + Any, Dict, List, Optional, Tuple, Union + TypeVar annotation is also allowed, but it can't be nested under any of the above containers + """ + + ValueT = ValueT # a TypeVar until the class is subscripted, e.g. DataclassDict[int] + + + @classmethod + def _container_classes(cls) -> Tuple[Type, Type]: + """The (ListType, DictType) pair nested containers are cast to. Overridden by ConfigDict.""" + from ladder.core.databib.dataclasses.dataclass_list import DataclassList + + return DataclassList, DataclassDict + + def __new__(cls, *args, **kwargs): + values = frozendict(*args, **kwargs) + # Cast value to Dict[cls.ValueT] (note that ValueT already has plain Dict/Dict container types + # substituted with their DataclassList / DataclassDict equivalents). + # NOTE: We cast the outer container to a python list instead of DataclassDict to avoid infinite + # recursion since we are in DataclassDict.__new__. super().__new__() will cast the outer container + values = cast_container_types(values, Dict[str, cls.ValueT]) + return super().__new__(cls, values) + + def __init__(self, *args, **kwargs): + del args, kwargs + # NOTE: frozendict gets auto initialized in __new__. No need to call here + DataclassInterface.__init__(self) + self._check_value_types_at_runtime() + + def _check_value_types_at_runtime(self): + if value_type_is_annotation_type(self, Dict[str, self.ValueT]): + return + # Provide more informative error + for key, value in self.items(): + if not value_type_is_annotation_type({key: value}, Dict[str, self.ValueT]): + raise TypeError( + f"Incorrect type provided for {self.__class__.__name__}: expected " + f"{self.ValueT}, but got {value} of type {type(value)} at key {key}" + ) + + @classmethod + def from_json( + cls, data: Dict[str, Any], recursive: bool = True, ignore_redundant: bool = False + ) -> "DataclassDict[ValueT]": + if not isinstance(data, dict): + raise TypeError( + f"{cls}.from_json() expects input of type dict, but got input of type {type(data)}" + ) + data = dict(data) + + if recursive and inspect.isclass(cls.ValueT) and issubclass(cls.ValueT, DataclassInterface): + data = { + key: cls.ValueT.from_json(value, recursive=recursive, ignore_redundant=ignore_redundant) + for key, value in data.items() + } + + return cls(**data) + + def as_json(self, recursive: bool = True) -> Dict[str, Any]: + return { + key: value.as_json() if isinstance(value, DataclassInterface) and recursive else value + for key, value in self.items() + } + + @classmethod + def make_empty(cls: Type[DataclassDictT]) -> DataclassDictT: + return cls() + + @property + def is_empty(self) -> bool: + return len(self) == 0 + + def items(self, recursive: bool = False): + for key in self.keys(): + value = self[key] + if recursive and isinstance(value, DataclassInterface): + for subkey, subvalue in value.items(recursive=True): + yield f"{key}.{subkey}", subvalue + else: + yield key, value + + def replace(self, strict=False, **kwargs) -> "DataclassDict[ValueT]": + """If strict, no new keys allowed, only existing values can be replaced""" + extra_keys = kwargs.keys() - self.keys() + if strict and len(extra_keys) > 0: + raise KeyError(f"Unexpected extra keys: {extra_keys}") + + data = {**self} + + for key, value in kwargs.items(): + # Recurse if ValueT is a DataclassInterface field + if inspect.isclass(self.ValueT) and issubclass(self.ValueT, DataclassInterface): + # Case 1: value is already the correct type, just assign it + if isinstance(value, self.ValueT): + data[key] = value + # Case 2: value is a plain dict representing the dataclass + elif isinstance(value, dict): + # Case 2.1: new key being added to the dataclass -> cast to ValueT + if key in extra_keys: + data[key] = self.ValueT(**value) + # Case 2.2: existing key being updated -> call replace() on the current instance + else: + data[key] = data[key].replace(**value) + # Case 3: value is some other type - delegate to __init__ to cast or raise error + else: + data[key] = value + else: + data[key] = value + + return self.__class__(**data) + + def apply( + self, fcn: Callable, recursive: bool = True, skip_nones: bool = False + ) -> "DataclassDict[ValueT]": + def fcn_wrapper(v: Any) -> Any: + if v is None and skip_nones: + return None + if isinstance(v, DataclassInterface) and recursive: + return v.apply(fcn, recursive=True, skip_nones=skip_nones) + return fcn(v) + + return self.__class__(**{k: fcn_wrapper(v) for k, v in self.items()}) + + def __reduce__(self) -> Tuple[Callable, Tuple[Any, ...], Dict[str, Any]]: + """ + Needed so we properly support pickling for classes with square brackets in their names. + + A subscripted class such as `DataclassDict[typing.Any]` is created at runtime and registered on + its module under that name (`dataclass_utils.parametrize`). pickle cannot look such a name up + itself, so this returns (callable, args, state): `get_dynamic_class` finds the class by name and + rebuilds the object from its JSON form, and `state` is passed to __setstate__ afterwards. + """ + return ( + get_dynamic_class, + (self.__class__.__module__, self.__class__.__name__, self.as_json()), + self.__dict__, + ) + + +def get_dynamic_class(module_name: str, classname: str, data: Dict[str, Any]): + return getattr(sys.modules[module_name], classname).from_json(data) diff --git a/ladder/core/databib/dataclasses/dataclass_interface.py b/ladder/core/databib/dataclasses/dataclass_interface.py new file mode 100644 index 0000000000000000000000000000000000000000..a8786b6a38b04904caba05ab79827d2aeed49a3d --- /dev/null +++ b/ladder/core/databib/dataclasses/dataclass_interface.py @@ -0,0 +1,46 @@ +import abc +from typing import Any, Callable, Iterator, Tuple, Union + + +class DataclassInterface(abc.ABC): + """ + Interface for dataclasses and the supported methods. + All dataclasses are immutable and all methods return new instances. + """ + + @classmethod + @abc.abstractmethod + def from_json(cls, data, recursive: bool = True, ignore_redundant: bool = False): + pass + + @classmethod + @abc.abstractmethod + def make_empty(cls): + pass + + @abc.abstractmethod + def as_json(self, recursive: bool = True) -> Union[dict, list]: + pass + + @abc.abstractmethod + def replace(self, **kwargs): + """Returns a new instance of the dataclass with the given fields replaced by the given values""" + + @abc.abstractmethod + def apply(self, fcn: Callable, recursive: bool = True, skip_nones: bool = False): + pass + + @abc.abstractmethod + def items(self, recursive: bool = False) -> Iterator[Tuple[str, Any]]: + pass + + def __repr__(self) -> str: + indent = " " * 2 + + fields_reprs = [ + f"{indent}{key}={repr(value)}".replace('\n', f'\n{indent}') + for key, value in self.items(recursive=False) + ] + fields_repr = ("\n" + ",\n".join(fields_reprs) + f",\n") if len(fields_reprs) > 0 else "" + + return f"{self.__class__.__name__}({fields_repr})" diff --git a/ladder/core/databib/dataclasses/dataclass_list.py b/ladder/core/databib/dataclasses/dataclass_list.py new file mode 100644 index 0000000000000000000000000000000000000000..584a672743904d450285a42593244f7f6c3a2964 --- /dev/null +++ b/ladder/core/databib/dataclasses/dataclass_list.py @@ -0,0 +1,171 @@ +import functools +import inspect +import sys +from abc import ABCMeta +from typing import Any, Callable, Dict, List, Sequence, Tuple, Type, TypeVar + +from ladder.core.databib.dataclasses.dataclass_interface import DataclassInterface +from ladder.core.databib.dataclasses.dataclass_utils import parametrize +from ladder.core.databib.utils.typing_utils import ( + cast_container_types, + get_maybe_optional_type, + substitute_annotation_container_types, + type_annotation_contains_type_var, + value_type_is_annotation_type, +) + +ValueT = TypeVar("ValueT") +DataclassListT = TypeVar("DataclassListT", bound="DataclassList") + + +class DataclassListMeta(ABCMeta): + @functools.lru_cache(maxsize=None, typed=True) # noqa: B019 + def __getitem__(cls, value_type): # noqa: N805 + if not isinstance(cls.ValueT, TypeVar): + raise TypeError(f"{cls.__name__} already has a value type") + # Substitute nested List / Dict in the value type with their DataclassList / DataclassDict + # equivalents (ConfigList / ConfigDict for the Config variants) + ListType, DictType = cls._container_classes() + value_type = substitute_annotation_container_types(value_type, ListType, DictType) + new_cls = parametrize(cls, (value_type,), ValueT=value_type) + new_cls._raise_on_forbidden_value_type() # noqa: SLF001 + return new_cls + + def _raise_on_forbidden_value_type(cls): # noqa: N805 + assert cls.ValueT is not Ellipsis + + if isinstance(get_maybe_optional_type(cls.ValueT), TypeVar): + return + + if type_annotation_contains_type_var(cls.ValueT): + raise TypeError(f"TypeVar annotation allowed only directly or Optional[TypeVar], but got {cls}") + + +class DataclassList(DataclassInterface, tuple, metaclass=DataclassListMeta): + """ + Immutable List container compatible with DataclassInterface API. Value type provided at runtime. + + Allowed value type annotations (including nested versions of these): + Any, Dict, List, Optional, Tuple, Union + TypeVar annotation is also allowed, but it can't be nested under any of the above containers + """ + + ValueT = ValueT # a TypeVar until the class is subscripted, e.g. DataclassList[int] + + + @classmethod + def _container_classes(cls) -> Tuple[Type, Type]: + """The (ListType, DictType) pair nested containers are cast to. Overridden by ConfigList.""" + from ladder.core.databib.dataclasses.dataclass_dict import DataclassDict + + return DataclassList, DataclassDict + + def __new__(cls, *args, **kwargs): + value = tuple(*args, **kwargs) + # Cast value to List[cls.ValueT] (note that ValueT already has plain List/Dict container types + # substituted with their DataclassList / DataclassDict equivalents). + # NOTE: We cast the outer container to a python list instead of DataclassList to avoid infinite + # recursion since we are in DataclassList.__new__. super().__new__() will cast the outer container + value = cast_container_types(value, List[cls.ValueT]) + return super().__new__(cls, value) + + def __init__(self, *args, **kwargs): + del args, kwargs + # NOTE: tuple gets auto initialized in __new__. No need to call here + DataclassInterface.__init__(self) + self._check_value_types_at_runtime() + + def _check_value_types_at_runtime(self): + if value_type_is_annotation_type(self, List[self.ValueT]): + return + # Provide more informative error + for index, value in enumerate(self): + if not value_type_is_annotation_type([value], List[self.ValueT]): + raise TypeError( + f"Incorrect type provided for {self.__class__.__name__}: expected " + f"{self.ValueT}, but got {value} of type {type(value)} at index {index}" + ) + + @classmethod + def from_json( + cls, data: Sequence[Any], recursive: bool = True, ignore_redundant: bool = False + ) -> "DataclassList[ValueT]": + if not isinstance(data, (list, tuple)): + raise TypeError( + f"{cls}.from_json() expects input of type list or tuple, but got input of type {type(data)}" + ) + data = list(data) + + if recursive and inspect.isclass(cls.ValueT) and issubclass(cls.ValueT, DataclassInterface): + data = [ + cls.ValueT.from_json(value, recursive=recursive, ignore_redundant=ignore_redundant) + for value in data + ] + + return cls(data) + + def as_json(self, recursive: bool = True) -> List: + return [ + value.as_json() if isinstance(value, DataclassInterface) and recursive else value + for value in self + ] + + @classmethod + def make_empty(cls: Type[DataclassListT]) -> DataclassListT: + return cls([]) + + @property + def is_empty(self) -> bool: + return len(self) == 0 + + def items(self, recursive: bool = False): + for i, value in enumerate(self): + if recursive and isinstance(value, DataclassInterface): + for subkey, subvalue in value.items(recursive=True): + yield f"{i}.{subkey}", subvalue + else: + yield f"{i}", value + + def replace(self, **kwargs) -> "DataclassList[ValueT]": # noqa: ARG002 + raise RuntimeError(f"{self.__class__.__name__}.replace() is not supported") + + def apply( + self, fcn: Callable, recursive: bool = True, skip_nones: bool = False + ) -> "DataclassList[ValueT]": + def fcn_wrapper(v: Any) -> Any: + if v is None and skip_nones: + return None + if isinstance(v, DataclassInterface) and recursive: + return v.apply(fcn, recursive=True, skip_nones=skip_nones) + return fcn(v) + + return self.__class__([fcn_wrapper(value) for value in self]) + + def __repr__(self) -> str: + indent = " " * 2 + + fields_reprs = [ + f"{indent}{repr(value)}".replace('\n', f'\n{indent}') for _, value in self.items(recursive=False) + ] + fields_repr = ("\n" + ",\n".join(fields_reprs) + f",\n") if len(fields_reprs) > 0 else "" + + return f"{self.__class__.__name__}({fields_repr})" + + def __reduce__(self) -> Tuple[Callable, Tuple[Any, ...], Dict[str, Any]]: + """ + Needed so we properly support pickling for classes with square brackets in their names. + + A subscripted class such as `DataclassDict[typing.Any]` is created at runtime and registered on + its module under that name (`dataclass_utils.parametrize`). pickle cannot look such a name up + itself, so this returns (callable, args, state): `get_dynamic_class` finds the class by name and + rebuilds the object from its JSON form, and `state` is passed to __setstate__ afterwards. + """ + return ( + get_dynamic_class, + (self.__class__.__module__, self.__class__.__name__, self.as_json()), + self.__dict__, + ) + + +def get_dynamic_class(module_name: str, classname: str, data: Dict[str, Any]): + return getattr(sys.modules[module_name], classname).from_json(data) diff --git a/ladder/core/databib/dataclasses/dataclass_utils.py b/ladder/core/databib/dataclasses/dataclass_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..4bf4e1c5bb5ed260b4a688c2970224689a9c4910 --- /dev/null +++ b/ladder/core/databib/dataclasses/dataclass_utils.py @@ -0,0 +1,63 @@ +import sys +from typing import Any, Dict, Iterator, Tuple + + +def parametrize(cls: type, args: Tuple[Any, ...], **class_attrs: Any) -> type: + """The subclass `cls[args]`, with `class_attrs` set on it (e.g. DataclassDict[int] with ValueT = int). + It is registered on the module of `cls` under its name, which is how pickling finds it again.""" + names = ", ".join(a.__name__ if isinstance(a, type) else repr(a) for a in args) + name = f"{cls.__name__}[{names}]" + new_cls = type(cls)(name, (cls,), {"__module__": cls.__module__, "__qualname__": name, **class_attrs}) + setattr(sys.modules[cls.__module__], name, new_cls) + return new_cls + + +def zip_dataclass_items(*dataclasses, recursive: bool) -> Iterator[Tuple[str, Tuple[Any, ...]]]: + if len(set(type(dataclass) for dataclass in dataclasses)) > 1: + raise TypeError( + f"Cannot zip dataclasses with non-matching types: {[type(dataclass) for dataclass in dataclasses]}" + ) + + zip_iterator = zip(*[dataclass.items(recursive=recursive) for dataclass in dataclasses], strict=True) + tuples_of_items: Tuple[Tuple[str, Any], ...] + for tuples_of_items in zip_iterator: + items: Tuple[Any, ...] = sum(tuples_of_items, ()) + keys: Tuple[str, ...] = items[0::2] + if len(set(keys)) > 1: + raise ValueError(f"Dataclass keys do NOT match: {keys}") + key: str = keys[0] + values: Tuple[Any, ...] = items[1::2] + yield key, values + + +def maybe_chained_keys_to_nested_dict(data: Dict[str, Any]) -> Dict[str, Any]: + """Converts a dict with keys of the form "key1.key2.key3" to a nested dict""" + unpacked_data: Dict[str, Any] = {} + for key, value in data.items(): + if "." not in key: + unpacked_data = merge_dicts_recursive(unpacked_data, {key: value}) + else: + mainkey, subkey = key.split(".", maxsplit=1) + nested_value = maybe_chained_keys_to_nested_dict({subkey: value}) + unpacked_data = merge_dicts_recursive(unpacked_data, {mainkey: nested_value}) + return unpacked_data + + +def merge_dicts_recursive(dict_1: Dict[str, Any], dict_2: Dict[str, Any]) -> Dict[str, Any]: + """ + Merges dict_1 with dict_2 recursively. + Handles clashing keys: + 1. If both values are dicts, merges them recursively + 2. If any value is not a dict, raises ValueError + """ + merged = dict(dict_1) + + for key, value in dict_2.items(): + if key in merged: + if not type(merged[key]) is type(value) is dict: + raise ValueError(f"Multiple values provided for key {key}: {merged[key]} and {value}") + merged[key] = merge_dicts_recursive(merged[key], value) + else: + merged[key] = value + + return merged diff --git a/ladder/core/databib/utils/classproperty.py b/ladder/core/databib/utils/classproperty.py new file mode 100644 index 0000000000000000000000000000000000000000..71ea482ada31f704694da88775b24ad1ebd20cc8 --- /dev/null +++ b/ladder/core/databib/utils/classproperty.py @@ -0,0 +1,47 @@ +class ClassPropertyDescriptor: + def __init__(self, fget, fset=None): + self.fget = fget + self.fset = fset + + def __get__(self, obj, klass=None): + if klass is None: + klass = type(obj) + return self.fget.__get__(obj, klass)() + + def __set__(self, obj, value): + if not self.fset: + raise AttributeError("can't set attribute") + type_ = type(obj) + return self.fset.__get__(obj, type_)(value) + + def setter(self, func): + if not isinstance(func, (classmethod, staticmethod)): + func = classmethod(func) + self.fset = func + return self + + +def classproperty(func): + if not isinstance(func, (classmethod, staticmethod)): + func = classmethod(func) + + return ClassPropertyDescriptor(func) + + +class CachedClassPropertyDescriptor: + def __init__(self, func): + self.func = func + self._values = {} + + def __get__(self, obj, klass): + if klass not in self._values.keys(): + self._values[klass] = self.func.__get__(obj, klass)() + + return self._values[klass] + + +def cached_classproperty(func): + if not isinstance(func, (classmethod, staticmethod)): + func = classmethod(func) + + return CachedClassPropertyDescriptor(func) diff --git a/ladder/core/databib/utils/typing_utils.py b/ladder/core/databib/utils/typing_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..adf6547a51ab827e28d291f6091f3b8530e64242 --- /dev/null +++ b/ladder/core/databib/utils/typing_utils.py @@ -0,0 +1,407 @@ +import collections +import enum +import inspect +import types +from collections.abc import Sequence +from typing import Any, Callable, Dict, List, Optional, Tuple, Type, TypeVar, Union + +import numpy as np + +ITERABLE_TYPES: Tuple[Type, ...] = (dict, list, set, tuple, Sequence) + + +def annotation_is_union(type_value: Type) -> bool: + return getattr(type_value, '__origin__', None) is Union or type(type_value) is types.UnionType + + +def annotation_is_optional(type_value: Type) -> bool: + if annotation_is_union(type_value): + union_args = set(type_value.__args__) + if len(union_args) == 2 and type(None) in union_args: + return True + return False + + +def get_maybe_optional_type(type_value: Type[Optional[Any]]) -> Type[Any]: + if annotation_is_optional(type_value): + type_args = get_annotation_type_args(type_value) + if type_args[1] is type(None): + return type_args[0] + return type_args[1] + return type_value + + +def annotation_is_dict_list_tuple(type_value: Type) -> bool: + return get_origin_type(type_value) in (dict, tuple, list, Sequence) + + +def annotation_is_dict(type_value: Type) -> bool: + return get_origin_type(type_value) is dict + + +def annotation_is_list(type_value: Type) -> bool: + return get_origin_type(type_value) is list + + +def annotation_is_tuple(type_value: Type) -> bool: + return get_origin_type(type_value) is tuple + + +def annotation_is_enum(type_value: Type) -> bool: + return inspect.isclass(type_value) and issubclass(type_value, enum.Enum) + + +def get_origin_type(type_value: Type) -> Optional[Type]: + return getattr(type_value, '__origin__', type_value) + + +def get_annotation_type_args(type_value: Type) -> Tuple[type, ...]: + return type_value.__args__ + + +def type_annotation_contains_type_var(type_value: Type) -> bool: + if isinstance(type_value, TypeVar): + return True + type_args = getattr(type_value, '__args__', []) + if type_args is None: + return False + return any(type_annotation_contains_type_var(type_arg) for type_arg in type_args) + + +def get_dict_annotation_type_args(type_value: Type) -> Tuple[type, ...]: + type_args = get_annotation_type_args(type_value) + if len(type_args) != 2: + raise TypeError( + f"Dict key and value type annotations required, but got {type_value}. " + f"Scroll up the stack trace to find the offending class definition." + ) + if type_args[0] is not str: + raise TypeError( + f"Only Dict with str key type annotations supported, but got {type_value}" + f"Scroll up the stack trace to find the offending class definition." + ) + if Ellipsis in type_args: + raise TypeError(f"Ellipsis `...` type annotation not allowed in Dict, but got {type_value}") + return type_args + + +def get_list_annotation_type_args(type_value: Type) -> Tuple[type, ...]: + type_args = get_annotation_type_args(type_value) + if len(type_args) != 1: + raise TypeError( + f"List items type annotations required, but got {type_value}. " + f"Scroll up the stack trace to find the offending class definition." + ) + if Ellipsis in type_args: + raise TypeError(f"Ellipsis `...` type annotation not allowed in List, but got {type_value}") + return type_args + + +def maybe_update_type_annotation(type_value: Any, update_map: Dict) -> Any: + """ + Recursively traverse a type annotation and update any type in update_map with its corresponding value + Supported 'recursive' types: Dict, List, Tuple, Optional, Union + """ + if type_value in update_map: + return update_map[type_value] + if annotation_is_dict(type_value): + type_args = get_dict_annotation_type_args(type_value) + return Dict[str, maybe_update_type_annotation(type_args[1], update_map)] # type: ignore[misc] + if annotation_is_list(type_value): + type_args = get_list_annotation_type_args(type_value) + return List[maybe_update_type_annotation(type_args[0], update_map)] # type: ignore[misc] + if annotation_is_tuple(type_value): + type_args = get_annotation_type_args(type_value) + return Tuple[tuple(maybe_update_type_annotation(type_arg, update_map) for type_arg in type_args)] + if annotation_is_union(type_value): # Note that Union case also covers Optional case + type_args = get_annotation_type_args(type_value) + # Rebuild through `Union[...]` rather than `copy_with`, which does not exist on `int | str` unions + return Union[tuple(maybe_update_type_annotation(type_arg, update_map) for type_arg in type_args)] + return type_value + + +def value_type_is_annotation_type( # pylint: disable=too-many-return-statements # noqa: PLR0911 + value: Any, + annotation_type: Type, + check_function: Callable[[Any, Type], bool] = lambda value, annotation_type: type(value) + is annotation_type, +) -> bool: + """ + Check if value complies with annotation_type. Perform exact checks instead of isinstance checks, + but this can be changed by passing check_function + + Supported typing module constructs: Any, Dict, List, Tuple, Optional, Union + + Args: + value: The value to test + annotation_type: The type provided in the annotation, e.g. Any, Dict[...], List[...], etc + check_function: Callable that takes value_type and annotation_type and return True / False if the + types match. Will be called only when annotation_type is an actual class. Will not be called + when annotation_type is a container (e.g. Dict, List, Tuple, Optional, etc). Allows for + overriding checks, e.g. using isinstance instead of exact match + Returns: + True if type(value) is the annotation type, False otherwise + Raises: + TypeError if forbidden annotations, e.g. dict, list, Ellipsis inside Dict or List + """ + + # Forbidden type annotations. Useful for demanding specificity. Forbid ambiguous annotations + # such as `dict` or `list` and require specific type arguments. Note that patterns like + # dict[str, int], Dict[str, int], list[int], List[int] are allowed and required + forbidden_types: Tuple[Type, ...] = (dict, list) + + # Optional + if annotation_is_optional(annotation_type) and value is None: + return True + + annotation_type = get_maybe_optional_type(annotation_type) + value_type = type(value) + + if annotation_type in forbidden_types: + raise TypeError(f"Type annotation {annotation_type} not allowed. Please be more explicit") + + # Any + if annotation_type is Any: + return True + + # annotation_type is an actual type -> we can check for correctness + if inspect.isclass(annotation_type): + # Allow setting float fields with int - python handles this type conversion automatically + if value_type is int and annotation_type is float: + return True + # Allow for np numeric types (e.g. np.float64, np.int64) to be seen as native python types + if isinstance(value, np.number) and type(value.item()) is annotation_type: + return True + + # return value_type is annotation_type + return check_function(value, annotation_type) + + # Union (recursive) + if annotation_is_union(annotation_type): + union_args = get_annotation_type_args(annotation_type) + return any( + value_type_is_annotation_type(value, union_type_arg, check_function) + for union_type_arg in union_args + ) + + # Dict (recursive) + if annotation_is_dict(annotation_type): + # Check for Mapping instead of dict. Many dict-like containers aren't actually instances of dict. + # Ex: frozendict, DataclassDict + if not isinstance(value, collections.abc.Mapping): + return False + + type_args = get_dict_annotation_type_args(annotation_type) + + return all( + value_type_is_annotation_type(k, type_args[0], check_function) for k in value.keys() + ) and all(value_type_is_annotation_type(v, type_args[1], check_function) for v in value.values()) + + # List (recursive) + if annotation_is_list(annotation_type): + # We check for Sequence instead of list. Many list-like containers aren't actually instances of list + # Ex: tuple, DataclassList + if not isinstance(value, collections.abc.Sequence): + return False + + type_args = get_list_annotation_type_args(annotation_type) + + return all(value_type_is_annotation_type(v, type_args[0], check_function) for v in value) + + # Tuple (recursive) + if annotation_is_tuple(annotation_type): + if not isinstance(value, tuple): + return False + + type_args = get_annotation_type_args(annotation_type) + + if Ellipsis in type_args: + if len(type_args) != 2 or type_args[0] is Ellipsis: + raise TypeError( + f"Tuple with Ellipsis `...` type annotation must follow " + f"the pattern Tuple[Type, ...], but got {annotation_type}" + ) + return all(value_type_is_annotation_type(v, type_args[0], check_function) for v in value) + + if len(type_args) != len(value): + return False + + return all( + value_type_is_annotation_type(v, type_arg, check_function) + for v, type_arg in zip(value, type_args, strict=True) + ) + + raise TypeError(f"Unsupported value {value} and annotation type {annotation_type}") + + +def maybe_autocast_native_types(value: Any, annotation_type: Type) -> Any: + """ + Best-effort, lossless coercion of native Python values to match a simple annotation: + - an int provided for a float field is cast to float + - a float provided for an int field is cast to int only when it has no fractional part (e.g. 5.0) + - a list provided for a Tuple field is cast to tuple (YAML / JSON have no tuple literal), recursively + coercing each element to its corresponding tuple element type + Anything else is returned unchanged so that a subsequent type check can still reject real mismatches + (e.g. a non-integral float for an int field, or a wrong-length tuple). Note that bool is intentionally + never cast: although it subclasses int, passing a bool to an int / float field is almost always a mistake. + """ + # list -> Tuple. Only lists (ordered) are cast - casting an unordered set to a tuple makes no sense. + if annotation_is_tuple(annotation_type) and isinstance(value, list): + type_args = get_annotation_type_args(annotation_type) + if Ellipsis in type_args: + # Tuple[T, ...]: every element is coerced to T + return tuple(maybe_autocast_native_types(item, type_args[0]) for item in value) + if len(type_args) == len(value): + return tuple( + maybe_autocast_native_types(item, arg) for item, arg in zip(value, type_args, strict=True) + ) + # Length mismatch: still cast to tuple so the type check reports the real (length) error + return tuple(value) + + if type(value) is bool or not inspect.isclass(annotation_type): + return value + if annotation_type is float and type(value) is int: + return float(value) + if annotation_type is int and type(value) is float and value.is_integer(): + return int(value) + return value + + +def cast_container_types(value: Any, annotation_type: Type) -> Any: + """ + Maybe cast the entries inside `value` to annotation_type. Casting is performed out-of-place + and only for containers of type Dict, List, Tuple. All other types remain unchanged. + + Supported typing annotation constructs: Any, Dict, List, Tuple, Optional, Union + + Args: + value: The actual value + annotation_type: The type provided in the annotation, e.g. Any, Dict[...], List[...], etc + Returns: + value or its casted equivalent + Raises: + TypeError if forbidden annotations, e.g. dict, list, Ellipsis inside Dict or List + """ + + # Forbidden type annotations. Useful for demanding specificity. Forbid ambibuous annotations + # such as `dict` or `list` and require specific type arguments. Note that patterns like + # dict[str, int], Dict[str, int], list[int], List[int] are allowed and required + forbidden_types: Tuple[Type, ...] = (dict, list) + + # Optional + if annotation_is_optional(annotation_type) and value is None: + return value + + annotation_type = get_maybe_optional_type(annotation_type) + + if annotation_type in forbidden_types: + raise TypeError(f"Type annotation {annotation_type} not allowed. Please be more explicit") + + # Any + if annotation_type is Any: + return value + + # annotation_type is an actual type -> we can check for correctness + if inspect.isclass(annotation_type): + from ladder.core.databib.dataclasses.dataclass_dict import DataclassDict + from ladder.core.databib.dataclasses.dataclass_list import DataclassList + + # Cast value to the exact annotation_type so nested containers are wrapped recursively + # NOTE: We check only for DataclassList/DataclassDict (and their Config variants). Checking + # for DataclassInterface also caputes Config, where annotations might be AutoConfig, which + # means value might already have the resolved type, but annotation_type is still AutoConfig + if issubclass(annotation_type, (DataclassList, DataclassDict)) and ( + type(value) is not annotation_type + ): + return annotation_type(value) + return value + + # Union (recursive) + if annotation_is_union(annotation_type): + return value + # if any( + # value_type_is_annotation_type(value, union_type_arg, check_function) + # for union_type_arg in union_args + # ): + # return value + # raise TypeError(f"{value} of type {type(value)} not part of Union args: {annotation_type}") + + # Dict (recursive) + if annotation_is_dict(annotation_type): + # Check for Mapping instead of dict. Many dict-like containers aren't actually instances of dict. + # Ex: frozendict, DataclassDict + if not isinstance(value, collections.abc.Mapping): + raise TypeError(f"Expected a Mapping, but got {value} of type {type(value)}") + + type_args = get_dict_annotation_type_args(annotation_type) + + return { + # NOTE: Don't cast key types - they must be str every time + k: cast_container_types(v, type_args[1]) + for k, v in value.items() + } + + # List (recursive) + if annotation_is_list(annotation_type): + # We check for Sequence instead of list. Many list-like containers aren't actually instances of list + # Ex: tuple, DataclassList + if not isinstance(value, collections.abc.Sequence): + raise TypeError(f"Expected a Sequence, but got {value} of type {type(value)}") + + type_args = get_list_annotation_type_args(annotation_type) + + return [cast_container_types(v, type_args[0]) for v in value] + + # Tuple (recursive) + if annotation_is_tuple(annotation_type): + if not isinstance(value, collections.abc.Sequence): + raise TypeError(f"Expected a Sequence, but got {value} of type {type(value)}") + + type_args = get_annotation_type_args(annotation_type) + + if Ellipsis in type_args: + if len(type_args) != 2 or type_args[0] is Ellipsis: + raise TypeError( + f"Tuple with Ellipsis `...` type annotation must follow " + f"the pattern Tuple[Type, ...], but got {annotation_type}" + ) + return tuple(cast_container_types(v, type_args[0]) for v in value) + + if len(type_args) != len(value): + raise TypeError( + f"Expected a Sequence of len {len(type_args)} for type {annotation_type}, " + f"but got {value} of len {len(value)}" + ) + + return tuple(cast_container_types(v, type_arg) for v, type_arg in zip(value, type_args, strict=True)) + + raise TypeError(f"Unsupported value {value} and annotation type {annotation_type}") + + +def substitute_annotation_container_types( + annotation_type: Type, ListType: Type, DictType: Type # noqa: N803 +) -> Type: + """ + Recursively rewrite a type annotation, replacing every List with `ListType` and every Dict with + `DictType` (e.g. DataclassList / DataclassDict or ConfigList / ConfigDict). Tuple, Optional and + Union are preserved with their members rewritten. Leaves (classes, Any, TypeVar) are unchanged. + + Supported typing constructs: Any, Dict, List, Tuple, Optional, Union + """ + if annotation_is_dict(annotation_type): + _, value_type = get_dict_annotation_type_args(annotation_type) + return DictType[substitute_annotation_container_types(value_type, ListType, DictType)] + if annotation_is_list(annotation_type): + (value_type,) = get_list_annotation_type_args(annotation_type) + return ListType[substitute_annotation_container_types(value_type, ListType, DictType)] + if annotation_is_tuple(annotation_type) or annotation_is_union(annotation_type): + args = get_annotation_type_args(annotation_type) + new_args = tuple( + arg if arg is Ellipsis else substitute_annotation_container_types(arg, ListType, DictType) + for arg in args + ) + # Preserve identity when nothing nested changed, so callers can cheaply detect a no-op via `is`. + if all(new_arg is old_arg for new_arg, old_arg in zip(new_args, args, strict=True)): + return annotation_type + return annotation_type.copy_with(new_args) + return annotation_type diff --git a/packages.txt b/packages.txt new file mode 100644 index 0000000000000000000000000000000000000000..4142a14c51926468eff8dd94257963d7d712a11c --- /dev/null +++ b/packages.txt @@ -0,0 +1,3 @@ +libxt6 +libopengl0 +libglu1-mesa diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000000000000000000000000000000000000..eaf8d5b29284e83711b18e3bfaa6e2dfdc4f4d8f --- /dev/null +++ b/requirements.txt @@ -0,0 +1,1266 @@ +# RoboLab-Long (Isaac Sim 6.0 / Isaac Lab 3.0): LADDER v1.0.0's own lock of its RoboLab environment +# (ladder/components/inference/robots/robolab/requirements.txt.lock), every package pinned to its version, less: +# isaaclab env.py loads the same wheel (3.0.0b2, sha256-pinned) from LADDER's downloads; +# transformers, tokenizers only isaaclab needs them, for an optional image encoder RoboLab never imports; +# huggingface-hub, httpx tokenizers needs huggingface-hub below 1.0, the LeRobot server 1.31 or above; +# the lock's httpx is a pre-release (for jupyterlab) that huggingface-hub 1.x imports; +# coverage, moviepy their locked versions contradict isaacsim-kernel's pins (LADDER's uv overrides them). +# Written by tools/build.py; change the lock in LADDER, not this file. +--extra-index-url https://pypi.nvidia.com +--extra-index-url https://download.pytorch.org/whl/cu128 +absl-py==2.4.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-core + # mujoco + # mujoco-warp + # pytorch-kinematics + # tensorboard +aioboto3==15.5.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-core +aiobotocore==2.25.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # aioboto3 + # isaacsim-kernel +aiodns==3.1.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-kernel +aiofiles==23.2.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # aioboto3 + # isaacsim-kernel +aiohappyeyeballs==2.6.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # aiohttp + # isaacsim-kernel +aiohttp==3.12.14 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # aiobotocore + # black + # isaacsim-kernel +aioitertools==0.11.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # aiobotocore + # isaacsim-kernel +aiosignal==1.4.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # aiohttp + # isaacsim-kernel +annotated-doc==0.0.4 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via fastapi +annotated-types==0.7.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via pydantic +anyio==4.14.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # jupyter-server + # starlette +argon2-cffi==25.1.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via jupyter-server +argon2-cffi-bindings==25.1.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via argon2-cffi +arm-pytorch-utilities==0.5.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via pytorch-kinematics +arrow==1.4.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isoduration +asteval==1.0.6 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-kernel +astroid==4.3.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via build-support-python +asttokens==3.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via stack-data +async-lru==2.3.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via jupyterlab +async-timeout==5.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-kernel +attrs==25.1.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # aiohttp + # isaacsim-kernel + # jsonschema + # referencing + # rerun-sdk +autoflake==2.2.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via build-support-python +av==15.0.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via ladder-core +awscrt==0.23.8 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-core +azure-core==1.38.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # azure-identity + # azure-storage-blob + # isaacsim-core +azure-identity==1.13.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-core +azure-storage-blob==12.17.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-core +babel==2.18.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via jupyterlab-server +beautifulsoup4==4.15.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via nbconvert +black==23.12.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via build-support-python +bleach==6.4.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via nbconvert +boto3==1.40.61 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # aiobotocore + # isaacsim-core + # isaacsim-kernel +botocore==1.40.61 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # aiobotocore + # boto3 + # isaaclab + # isaacsim-core + # s3transfer +cachebox==5.2.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via deepdiff +cbor2==5.9.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-core +certifi==2026.6.17 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # httpx + # requests + # sentry-sdk +cffi==2.0.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # argon2-cffi-bindings + # cryptography + # pycares + # pyzmq +charset-normalizer==3.3.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-kernel + # requests +click==8.1.7 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # black + # isaacsim-kernel + # uvicorn +cloudpickle==3.1.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via gymnasium +cmeel==0.60.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # cmeel-assimp + # cmeel-boost + # cmeel-console-bridge + # cmeel-octomap + # cmeel-qhull + # cmeel-tinyxml2 + # cmeel-urdfdom + # cmeel-zlib + # coal + # eigenpy + # libcoal + # libpinocchio + # pin +cmeel-assimp==6.0.5 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # coal + # libcoal +cmeel-boost==1.90.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # coal + # eigenpy + # libcoal + # libpinocchio + # pin +cmeel-console-bridge==1.0.2.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via cmeel-urdfdom +cmeel-octomap==1.10.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # coal + # libcoal +cmeel-qhull==8.0.2.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # coal + # libcoal +cmeel-tinyxml2==11.0.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via cmeel-urdfdom +cmeel-urdfdom==6.0.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # libpinocchio + # pin +cmeel-zlib==1.3.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via cmeel-assimp +coacd==1.0.7 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-core +coal==3.0.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via pin +comm==0.2.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # ipykernel + # ipywidgets +contourpy==1.3.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-core + # matplotlib +cryptography==46.0.7 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # azure-identity + # azure-storage-blob + # isaacsim-core + # msal + # pyjwt +cuda-bindings==12.9.7 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via torch +cuda-pathfinder==1.5.6 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via cuda-bindings +cuda-toolkit==12.8.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via torch +cycler==0.12.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-core + # matplotlib +debugpy==1.8.21 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via ipykernel +decorator==5.3.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # ipdb + # ipython + # moviepy +deepdiff==9.1.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via ladder-core-databib +defusedxml==0.8.0rc2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via nbconvert +docstring-parser==0.18.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via tyro +eigenpy==3.13.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via coal +einops==0.8.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaaclab +etils==1.13.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-core + # mujoco + # mujoco-warp +executing==2.2.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via stack-data +farama-notifications==0.0.6 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via gymnasium +fastapi==0.120.4 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-kernel +fastjsonschema==2.21.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via nbformat +filelock==3.25.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # huggingface-hub + # isaacsim-core + # torch + # transformers +flaky==3.8.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaaclab +flatdict==4.0.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaaclab +fonttools==4.61.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-core + # matplotlib +fqdn==1.5.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via jsonschema +frozendict==2.4.7 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via ladder-core-databib +frozenlist==1.5.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # aiohttp + # aiosignal + # isaacsim-kernel +fsspec==2026.2.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # etils + # huggingface-hub + # isaacsim-core + # torch +gitdb==4.0.12 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via gitpython +gitpython==3.1.40 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via ladder-core +glfw==2.10.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-core + # mujoco +grpcio==1.82.0rc2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via tensorboard +gymnasium==1.2.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaaclab +h11==0.16.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via uvicorn +h5py==3.16.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaaclab + # ladder-components-inference-robots-robolab +hf-xet==1.5.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via huggingface-hub +hidapi==0.14.0.post2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaaclab +httptools==0.6.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-kernel +hydra-core==1.4.0.dev5 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaaclab +idna==3.10 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # anyio + # idna-ssl + # isaacsim-kernel + # jsonschema + # requests + # yarl +idna-ssl==1.1.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-kernel +imageio==2.37.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-core + # ladder-components-inference-robots-robolab + # moviepy +imageio-ffmpeg==0.6.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # ladder-components-inference-robots-robolab + # moviepy +imgui-bundle==1.92.601 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaaclab + # isaacsim-core +importlib-resources==7.1.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via etils +iniconfig==2.3.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via pytest +ipdb==0.13.13 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via build-support-python +ipykernel==6.27.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # build-support-python + # jupyter + # jupyter-console + # jupyterlab + # qtconsole +ipython==8.18.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # build-support-python + # ipdb + # ipykernel + # ipywidgets + # jupyter-console +ipython-pygments-lexers==1.1.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via qtconsole +ipywidgets==8.1.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via jupyter +isaacsim @ https://pypi.nvidia.com/isaacsim/isaacsim-6.0.0.1-cp312-none-manylinux_2_35_x86_64.whl ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaaclab + # ladder-components-inference-robots-robolab +isaacsim-app==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim +isaacsim-asset==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim +isaacsim-benchmark==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim + # isaacsim-rl +isaacsim-code-editor==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim +isaacsim-core==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim + # isaacsim-gui + # isaacsim-replicator +isaacsim-cortex==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim + # isaacsim-example +isaacsim-example==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim +isaacsim-extscache-kit==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim +isaacsim-extscache-kit-sdk==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim +isaacsim-extscache-physics==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim +isaacsim-gui==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim + # isaacsim-robot-motion + # isaacsim-sensor + # isaacsim-template + # isaacsim-utils +isaacsim-kernel==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim + # isaacsim-app + # isaacsim-code-editor + # isaacsim-core + # isaacsim-storage +isaacsim-replicator==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim +isaacsim-rl==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim +isaacsim-robot==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim + # isaacsim-benchmark + # isaacsim-cortex + # isaacsim-rl + # isaacsim-test +isaacsim-robot-motion==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim + # isaacsim-robot + # isaacsim-robot-setup +isaacsim-robot-setup==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim +isaacsim-ros1==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim + # isaacsim-cortex +isaacsim-ros2==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim +isaacsim-sensor==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim + # isaacsim-robot + # isaacsim-ros1 + # isaacsim-ros2 +isaacsim-storage==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim + # isaacsim-asset + # isaacsim-replicator + # isaacsim-sensor + # isaacsim-template +isaacsim-template==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim +isaacsim-test==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim +isaacsim-utils==6.0.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim + # isaacsim-asset + # isaacsim-sensor +isodate==0.6.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # azure-storage-blob + # isaacsim-core +isoduration==20.11.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via jsonschema +isort==5.13.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via build-support-python +jedi==0.20.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # ipython + # pudb +jinja2==3.1.5 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-kernel + # jupyter-server + # jupyterlab + # jupyterlab-server + # nbconvert + # osqp + # torch +jmespath==1.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # aiobotocore + # boto3 + # botocore + # isaacsim-core +joblib==1.5.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via osqp +json-numpy==2.1.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via ladder-components-inference-robots-robolab +json5==0.9.25 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # build-support-python + # jupyterlab-server +jsonpointer==3.1.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via jsonschema +jsonschema==4.26.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # jupyter-events + # jupyterlab-server + # nbformat +jsonschema-specifications==2025.9.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via jsonschema +junitparser==5.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaaclab +jupyter==1.0.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via build-support-python +jupyter-client==8.6.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # build-support-python + # ipykernel + # jupyter-console + # jupyter-server + # nbclient + # qtconsole +jupyter-console==6.6.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # build-support-python + # jupyter +jupyter-core==5.7.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # build-support-python + # ipykernel + # jupyter-client + # jupyter-console + # jupyter-server + # jupyterlab + # nbclient + # nbconvert + # nbformat + # qtconsole +jupyter-events==0.10.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # build-support-python + # jupyter-server +jupyter-lsp==2.2.5 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # build-support-python + # jupyterlab +jupyter-server==2.14.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # build-support-python + # jupyter-lsp + # jupyterlab + # jupyterlab-server + # notebook + # notebook-shim +jupyter-server-terminals==0.5.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # build-support-python + # jupyter-server +jupyterlab==4.1.8 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # build-support-python + # notebook +jupyterlab-pygments==0.3.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # build-support-python + # nbconvert +jupyterlab-server==2.27.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # build-support-python + # jupyterlab + # notebook +jupyterlab-widgets==3.0.10 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # build-support-python + # ipywidgets +kiwisolver==1.4.9 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-core + # matplotlib +lark==1.3.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via rfc3987-syntax +lazy-loader==0.5 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaaclab +libcoal==3.0.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # coal + # libpinocchio +libpinocchio==4.0.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via pin +llvmlite==0.46.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-core + # numba +loop-rate-limiters==1.2.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via pin-pink +lxml==7.0.0a3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via pytorch-kinematics +markdown==3.10.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via tensorboard +markdown-it-py==4.2.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via rich +markupsafe==3.0.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # jinja2 + # nbconvert + # werkzeug +matplotlib==3.10.8 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # arm-pytorch-utilities + # isaaclab + # isaacsim-core + # ladder-components-inference-robots-robolab + # pytorch-kinematics +matplotlib-inline==0.2.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # ipykernel + # ipython +mdurl==0.1.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via markdown-it-py +mistune==3.3.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via nbconvert +ml-dtypes==0.5.4 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via onnx +mpmath==1.3.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-core + # sympy +msal==1.35.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # azure-identity + # isaacsim-core + # msal-extensions +msal-extensions==1.3.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # azure-identity + # isaacsim-core +msgpack==1.1.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # ladder-components-inference-api-core + # msgpack-numpy +msgpack-numpy==0.4.8 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via ladder-components-inference-api-core +mujoco==3.8.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-core + # mujoco-usd-converter + # mujoco-warp + # newton +mujoco-usd-converter==0.2.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-core +mujoco-warp==3.8.0.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-core + # newton +multidict==6.1.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # aiobotocore + # aiohttp + # isaacsim-kernel + # yarl +mypy-extensions==1.1.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via black +nbclient==0.11.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via nbconvert +nbconvert==7.17.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # jupyter + # jupyter-server +nbformat==5.10.4 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # jupyter-server + # nbclient + # nbconvert +nest-asyncio==1.6.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # ipykernel + # isaacsim-core +networkx==3.6.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-core + # torch +newton==1.2.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-core +newton-actuators==0.1.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-core + # newton +newton-usd-schemas==0.2.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-core + # mujoco-usd-converter + # urdf-usd-converter +notebook==7.0.6 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # build-support-python + # jupyter +notebook-shim==0.2.4 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # jupyterlab + # notebook +numba==0.64.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaaclab +numpy==2.3.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # arm-pytorch-utilities + # cmeel-boost + # coacd + # contourpy + # gymnasium + # h5py + # imageio + # imgui-bundle + # isaaclab + # isaacsim-kernel + # json-numpy + # ladder-core + # ladder-core-databib + # matplotlib + # ml-dtypes + # moviepy + # msgpack-numpy + # mujoco + # mujoco-warp + # numba + # numpy-stl + # onnx + # opencv-python-headless + # osqp + # pandas + # pin-pink + # pycollada + # pytorch-kinematics + # pytorch-seed + # qdldl + # qpsolvers + # rerun-sdk + # scipy + # tensorboard + # torchvision + # transformers + # trimesh + # warp-lang +numpy-stl==4.0.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-core + # mujoco-usd-converter + # urdf-usd-converter +nvidia-cublas-cu12==12.8.4.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # cuda-toolkit + # nvidia-cudnn-cu12 + # nvidia-cusolver-cu12 +nvidia-cuda-cupti-cu12==12.8.90 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via cuda-toolkit +nvidia-cuda-nvrtc-cu12==12.8.93 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via cuda-toolkit +nvidia-cuda-runtime-cu12==12.8.90 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via cuda-toolkit +nvidia-cudnn-cu12==9.19.0.56 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via torch +nvidia-cufft-cu12==11.3.3.83 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via cuda-toolkit +nvidia-cufile-cu12==1.13.1.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via cuda-toolkit +nvidia-curand-cu12==10.3.9.90 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via cuda-toolkit +nvidia-cusolver-cu12==11.7.3.90 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via cuda-toolkit +nvidia-cusparse-cu12==12.5.8.93 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # cuda-toolkit + # nvidia-cusolver-cu12 +nvidia-cusparselt-cu12==0.7.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via torch +nvidia-nccl-cu12==2.28.9 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via torch +nvidia-nvjitlink-cu12==12.8.93 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # cuda-toolkit + # nvidia-cufft-cu12 + # nvidia-cusolver-cu12 + # nvidia-cusparse-cu12 +nvidia-nvshmem-cu12==3.4.5 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via torch +nvidia-nvtx-cu12==12.8.90 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via cuda-toolkit +oauthlib==3.2.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-core + # requests-oauthlib +omegaconf==2.4.0.dev12 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via hydra-core +omniverseclient==2.71.1.7015 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaaclab +onnx==1.21.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaaclab +opencv-python-headless==4.13.0.90 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-core + # ladder-components-inference-api-core +orderly-set==5.5.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via deepdiff +osqp==1.0.5 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-core +overrides==7.7.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via jupyter-server +packaging==26.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # black + # huggingface-hub + # hydra-core + # ipykernel + # isaaclab + # isaacsim-core + # jupyter-server + # jupyterlab + # jupyterlab-server + # lazy-loader + # matplotlib + # nbconvert + # pudb + # pytest + # qtconsole + # qtpy + # tensorboard + # transformers +pandas==3.0.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via ladder-components-inference-robots-robolab +pandocfilters==1.5.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via nbconvert +parso==0.8.7 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via jedi +pathspec==1.1.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via black +pexpect==4.9.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via ipython +pillow==12.1.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # imageio + # isaaclab + # isaacsim-kernel + # ladder-core + # matplotlib + # moviepy + # rerun-sdk + # tensorboard + # torchvision +pin==4.0.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaaclab + # pin-pink +pin-pink==4.2.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaaclab +pip==26.1.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via build-support-python +platformdirs==4.10.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # black + # jupyter-core +pluggy==1.6.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via pytest +portalocker==2.7.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-core +prettytable==3.3.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaaclab +proglog==0.1.12 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via moviepy +prometheus-client==0.25.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via jupyter-server +prompt-toolkit==3.0.52 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # ipython + # jupyter-console +propcache==0.2.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # aiohttp + # isaacsim-kernel + # yarl +protobuf==7.35.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaaclab + # onnx + # tensorboard +psutil==5.9.8 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # ipykernel + # isaacsim-kernel + # ladder-core +ptyprocess==0.7.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # pexpect + # terminado +pudb==2024.1.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via build-support-python +pure-eval==0.2.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via stack-data +pyarrow==22.0.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaaclab + # rerun-sdk +pycares==5.0.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # aiodns + # isaacsim-kernel +pycollada==0.9.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-core + # urdf-usd-converter +pycparser==3.0 ; implementation_name != 'PyPy' and platform_machine == 'x86_64' and sys_platform == 'linux' + # via cffi +pydantic==2.11.10 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # fastapi + # isaaclab +pydantic-core==2.33.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via pydantic +pyflakes==3.4.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via autoflake +pyglet==3.0.dev6 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaaclab +pygments==2.20.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # ipython + # ipython-pygments-lexers + # jupyter-console + # nbconvert + # pudb + # pytest + # qtconsole + # rich +pyjwt==2.13.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via msal +pynvvideocodec==2.1.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-core +pyopengl==3.1.10 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via mujoco +pyopengl-accelerate==3.1.10 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-core +pyparsing==3.3.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-core + # matplotlib +pyperclip==1.11.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-core +pypng==0.20220715.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via qrcode +pytest==9.0.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # build-support-python + # isaaclab + # pytest-cov + # pytest-mock + # pytest-socket + # pytest-timeout +pytest-cov==4.1.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via build-support-python +pytest-mock==3.8.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # build-support-python + # isaaclab +pytest-socket==0.5.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via build-support-python +pytest-timeout==2.1.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via build-support-python +python-dateutil==2.9.0.post0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # aiobotocore + # arrow + # botocore + # isaacsim-core + # jupyter-client + # ladder-core-databib + # matplotlib + # pandas + # pycollada +python-dotenv==1.2.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # ladder-components-inference-robots-robolab + # moviepy +python-json-logger==4.1.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via jupyter-events +python-multipart==0.0.22 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-kernel +python-utils==3.9.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-core + # numpy-stl +pytorch-kinematics==0.7.6 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via ladder-components-inference-robots-robolab +pytorch-seed==0.2.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # arm-pytorch-utilities + # pytorch-kinematics +pytz==2024.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-kernel +pyyaml==6.0.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # huggingface-hub + # isaacsim-core + # jupyter-events + # ladder-core + # ladder-core-databib + # omegaconf + # pytorch-kinematics + # transformers +pyzmq==26.2.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # ipykernel + # jupyter-client + # jupyter-console + # jupyter-server + # ladder-components-inference-api-core +qdldl==0.1.7.post5 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-core +qpsolvers==4.12.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via pin-pink +qrcode==7.4.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-kernel +qtconsole==5.7.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via jupyter +qtpy==2.4.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via qtconsole +referencing==0.37.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # jsonschema + # jsonschema-specifications + # jupyter-events +regex==2026.6.28 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via transformers +requests==2.32.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # azure-core + # build-support-python + # huggingface-hub + # isaacsim-kernel + # jupyterlab-server + # ladder-components-inference-robots-robolab + # msal + # requests-oauthlib + # transformers +requests-oauthlib==1.3.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-core +rerun-sdk==0.31.4 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaaclab +rfc3339-validator==0.1.4 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # jsonschema + # jupyter-events +rfc3986-validator==0.1.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # jsonschema + # jupyter-events +rfc3987-syntax==1.1.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via jsonschema +rich==15.0.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via tyro +rpds-py==2026.6.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # jsonschema + # referencing +rtree==1.4.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-core +s3transfer==0.14.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # boto3 + # isaacsim-core +safetensors==0.8.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via transformers +scipy==1.17.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # arm-pytorch-utilities + # isaacsim-core + # ladder-components-inference-api-core + # osqp + # qdldl + # qpsolvers +send2trash==2.1.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via jupyter-server +sentry-sdk==2.42.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-kernel +setuptools==80.10.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # hidapi + # ladder-components-inference-robots-robolab + # osqp + # tensorboard + # torch +shtab==1.8.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via tyro +six==1.17.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # azure-identity + # isaacsim-core + # isodate + # python-dateutil + # rfc3339-validator +smmap==5.0.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via gitdb +soupsieve==2.8.4 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via beautifulsoup4 +stack-data==0.6.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via ipython +starlette==0.49.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # fastapi + # isaaclab +sympy==1.14.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-core + # torch +tensorboard==2.21.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaaclab +tensorboard-data-server==0.7.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via tensorboard +terminado==0.18.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # jupyter-server + # jupyter-server-terminals +tinycss2==1.5.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via bleach +tinyobjloader==2.0.0rc13 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-core + # mujoco-usd-converter + # urdf-usd-converter +toml==0.10.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaaclab + # isaacsim-kernel + # toml-union +toml-union==1.0.4 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via build-support-python +tomli==2.2.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via build-support-python +torch @ https://download.pytorch.org/whl/cu128/torch-2.11.0%2Bcu128-cp312-cp312-manylinux_2_28_x86_64.whl ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # arm-pytorch-utilities + # isaaclab + # isaacsim-core + # ladder-components-inference-robots-robolab + # pytorch-kinematics + # pytorch-seed + # torchvision +torchaudio==2.11.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-core +torchvision @ https://download.pytorch.org/whl/cu128/torchvision-0.26.0%2Bcu128-cp312-cp312-manylinux_2_28_x86_64.whl ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaaclab + # isaacsim-core + # ladder-components-inference-robots-robolab +tornado==6.5.7 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # ipykernel + # jupyter-client + # jupyter-server + # jupyterlab + # notebook + # terminado +tqdm==4.67.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # huggingface-hub + # isaaclab + # ladder-core + # proglog + # transformers +traitlets==5.15.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # ipykernel + # ipython + # ipywidgets + # jupyter-client + # jupyter-console + # jupyter-core + # jupyter-events + # jupyter-server + # jupyterlab + # matplotlib-inline + # nbclient + # nbconvert + # nbformat + # qtconsole +trimesh==4.11.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaaclab + # isaacsim-core +triton==3.6.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via torch +typeguard==4.4.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via tyro +types-decorator==5.1.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via build-support-python +types-frozendict==2.0.9 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via build-support-python +types-python-dateutil==2.8.19.14 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via build-support-python +types-pyyaml==6.0.5 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via build-support-python +typing-extensions==4.12.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # aiosignal + # anyio + # azure-core + # azure-storage-blob + # beautifulsoup4 + # build-support-python + # etils + # fastapi + # grpcio + # gymnasium + # huggingface-hub + # isaaclab + # isaacsim-kernel + # numpy-stl + # onnx + # pydantic + # pydantic-core + # python-utils + # qrcode + # referencing + # rerun-sdk + # starlette + # torch + # typeguard + # typing-inspection + # tyro + # urwid +typing-inspection==0.4.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via pydantic +tyro==0.9.17 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via ladder-components-inference-robots-robolab +tzdata==2026.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via arrow +urdf-usd-converter==0.1.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-core +uri-template==1.3.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via jsonschema +urllib3==2.6.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # botocore + # isaacsim-kernel + # requests + # sentry-sdk +urwid==2.6.16 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # build-support-python + # pudb + # urwid-readline +urwid-readline==0.15.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via pudb +usd-exchange==2.3.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # mujoco-usd-converter + # urdf-usd-converter +uvicorn==0.29.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-kernel +warp-lang==1.13.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaaclab + # mujoco-warp + # newton + # newton-actuators +watchdog==4.0.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via isaacsim-kernel +wcwidth==0.8.2 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # prettytable + # prompt-toolkit + # urwid +webcolors==25.10.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via jsonschema +webencodings==0.5.1 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # bleach + # tinycss2 +websocket-client==1.9.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via jupyter-server +websockets==12.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # isaacsim-kernel + # ladder-components-inference-robots-robolab +werkzeug==3.1.8 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via tensorboard +widgetsnbextension==4.0.15 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via ipywidgets +wrapt==1.16.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # aiobotocore + # isaacsim-kernel +xxhash==3.5.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via ladder-core +yarl==1.18.3 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via + # aiohttp + # isaacsim-kernel +zipp==4.1.0 ; platform_machine == 'x86_64' and sys_platform == 'linux' + # via etils