edge-inference/cohab-lekiwi

CoHab LeKiwi: pi0.5 floor-sorting skill policy

Two robots with the Cosmos-Reason2-8B planner in the held-out apartment: the first minute of a task, at 2x speed. Left: robot A, right: robot B; the caption shows each robot's current instruction. Each robot picks and puts away two objects (4 of 6 sorted after one minute).

One vision-language-action policy for the four steps of floor sorting on a LeKiwi (SO-101 arm): go to an object, pick it up, bring it to its bin, put it in. It reads the robot's front and wrist cameras, its joint state, base velocity and the goal position in the robot frame, and outputs chunks of 10 actions (6 arm joint targets + base x/y/theta velocity) at 15 Hz. A Cosmos-Reason2-8B planner gives each of two robots its next step as a text instruction.

• Fine-tuned from lerobot/pi05_base (all weights, LeRobot 0.6.1, bf16) on simulator teacher data from a scripted teacher with retries and small arm/base noise. Final checkpoint of the run, chosen in advance.
• Simulation only so far (ManiSkill 3, ReplicaCAD); not yet run on a real LeKiwi.

Results (two robots, six objects and two bins per task, apartment 5 not used for training)

192 tasks: 24 seeds (0-7, 40-55) x 8 parallel tasks, each a different random layout.

	150 s per task	300 s per task
objects sorted per task (of 6)	3.16 [95% CI 2.95, 3.38] (607 of 1,152, 52.7%)	3.71 [3.51, 3.92] (713 of 1,152, 61.9%)
tasks with 0 / 1 / 2 / 3 / 4 / 5 / 6 sorted	4 / 28 / 31 / 48 / 44 / 25 / 12	4 / 10 / 28 / 36 / 48 / 49 / 17
per-step success: go to / pick up / bring / put	69% / 80% / 70% / 92%	59% / 75% / 70% / 87%

CI: bootstrap over the 192 tasks. Per-task results, the raw per-seed records (planner decisions, camera checks, latencies) and the summary are in eval/v12/ (head = 150 s, double_time = 300 s).

How the robots know things in this evaluation (no simulator truth in the loop except where stated):

• Next step: Cosmos-Reason2-8B chooses each robot's next step from its camera image and a text menu.
• Where the goal is: the robot's own odometry, a prior map and its camera (simulated SLAM; median camera error about 6 cm), plus added goal noise (--goal-noise 0.10).
• "Arrived" at an object or bin: Cosmos-Reason2-8B judges from the camera.
• "Holding the object": from the gripper's angle and load.
• Stated assumption: while bringing, the arm is held at the pose the pick ended with (a fixed rule over the policy's arm output; without it the policy drops more objects). The simulated chassis passes through rugs.

Use

from lerobot.policies.pi05.modeling_pi05 import PI05Policy
policy = PI05Policy.from_pretrained("edge-inference/cohab-lekiwi")

Camera inputs are renamed front -> base_0_rgb, wrist -> left_wrist_0_rgb (see train_config.json).

Code and full history: github.com/edge-inference/cohab-lekiwi (docs/PLAN.md). License: BSD 3-Clause.