| --- |
| license: apache-2.0 |
| library_name: pytorch |
| pipeline_tag: robotics |
| tags: |
| - robotics |
| - world-model |
| - model-predictive-control |
| - cable-driven-parallel-robot |
| - stringman |
| - task-cards |
| datasets: |
| - MacroSwan/pilot-tasks |
| - MacroSwan/cablepilot-motion |
| --- |
| |
| # cablepilot |
|
|
| A **28,401-parameter world model** of a cable robot, a **sampling planner** that imagines a few hundred futures with it |
| every control step, and **task cards**: plain YAML files that say what a job is, which anyone (or their agent) can |
| write, improve, score and share. |
|
|
| No demonstrations. No GPU. A new task is a card plus a few numbers, not a dataset and a training run. |
|
|
| > **Agents:** read [`AGENTS.md`](AGENTS.md). It is written for you: install, check the setup, score a card, contribute. |
|
|
| | | | |
| |---|---| |
| | model | 28k parameters, 124 KB, trained on task-free simulated motion in 12,000 random rooms, 11 min on a laptop CPU | |
| | one plan | 128 futures x 2 s imagined in ~25 ms on a laptop CPU (5 plans per second leaves ~85 % of the budget free) | |
| | any room | the room enters only as its 4 calibrated mount points: a room it never saw is just 4 new numbers | |
| | clean the room (sim) | 29/36 toys into the box over 3 episodes in a real bedroom imported from a Stringman recording, 0 furniture contacts, 62 s per toy | |
| | real robot | **not yet run on hardware.** The Stringman adapter is written, not tested: see "Status" | |
|
|
| ## Why a small world model |
|
|
| The usual route to a robot that does chores is to record people driving it, then train a large policy on the |
| recordings. On Stringman the public record of that route is 97 hours of teleoperation and 56 models (52 M to 4.1 B |
| parameters), on rented H100s / H200s, and most of those models cannot decide fast enough on a laptop. |
|
|
| This repository takes the other route. It learns **the machine** (how the gantry, the lines and the swinging pole |
| respond to commands), once, from simulation, without any task in it. Then each task is a **cost** on imagined futures |
| (near the toy, low over it, not swinging, lines not too tight, clear of the bed) plus a few **events** on what the robot |
| measures (the jaw stopped on something, the lines weigh 30 g more). The planner (MPPI, "the vortex": the purple futures |
| in the videos) picks the commands whose futures cost least, five times a second. |
|
|
| So a new task costs a person an afternoon of thinking, not a week of teleoperation, and anyone's improvement to a task |
| is a small text diff that the next person starts from. |
|
|
| ## Try it (3 minutes, no robot) |
|
|
| ```bash |
| git clone https://huggingface.co/MacroSwan/cablepilot && cd cablepilot |
| pip install -e ".[sim]" # numpy, torch, pyyaml + mujoco for the simulator |
| cablepilot doctor # what this machine can do (threads, plan time) |
| cablepilot fetch clean_the_room # the task folder from the hub: card + best tuning + results |
| cablepilot check pilot-tasks/tasks/clean_the_room/card.yaml |
| cablepilot run pilot-tasks/tasks/clean_the_room/card.yaml --world bedroom --video |
| ``` |
|
|
| The run writes `runs/<...>/result.json` (the score, with the hashes of what ran) and a film: the room with the plan in |
| purple | the overhead camera with what it found | the gripper camera. |
|
|
| ## The three files of a task |
|
|
| Tasks live in the dataset [`MacroSwan/pilot-tasks`](https://huggingface.co/datasets/MacroSwan/pilot-tasks), one folder each, |
| shared by every pilot (cable robot, arm, drone, ...): |
|
|
| ``` |
| tasks/clean_the_room/ |
| card.yaml WHAT: stages, skills, fallbacks. Body-free: the same file runs on an arm or a cable robot. |
| tuning/cable.yaml HOW MUCH, for this body: the cost weights and thresholds (every number lives here) |
| results/*.json what each run scored, where, with which card and tuning (by hash), by whom |
| ``` |
|
|
| A card reads like the job: *observe the floor; approach the stray thing; grasp it; bring it over the box, still holding |
| it; let go. If you lose sight of it, look again; if it slips, approach again.* Each stage names a **skill** from a |
| shared vocabulary (`see`, `pre_grasp`, `grasp`, `pre_inside`, `release`, `avoid`, ...). A card may only use skills the |
| body has; `cablepilot check` lists the ones it is missing, and that list is the best place to contribute. |
|
|
| Cards carry their **lineage** (`parent`, `by`, `why`), so a task grows as a tree: *clean the room* -> *clean the room, |
| sorted* -> *put things back where they live*; *pen in cup* -> *pen in box*. Two cards with the same `signature()` are the |
| same task with different names. |
|
|
| ## Your setup |
|
|
| Everything that differs between two cable robots goes in a small `rig.yaml` (the agent fills it with you): |
|
|
| ```yaml |
| rig: living-room |
| robot: stringman # stringman | sim |
| gripper: stringman # picks the world model trained for that gripper |
| mounts: robot # read the calibrated anchor points from the robot, or give (4, 3) metres |
| cameras: {overhead: ortho, gripper: true} |
| compute: {device: auto, threads: auto, plan_hz: 5} |
| places: {toy_box: hamper} # a card's site -> one of the robot's named places |
| ``` |
|
|
| `cablepilot doctor --rig rig.yaml` measures this machine against it and writes `doctor.json` beside it. |
|
|
| **Vision.** cablepilot does not bring its own heavy vision model. It uses the cameras your robot already has, in the |
| cheapest way that works. The overhead view compares the floor with the clean floor and masks out the robot's own |
| pixels (about 9 ms per look). The gripper camera then finds the target exactly as the hand gets close. Both feed one |
| belief per object, which the task reads. On Stringman, the overhead view is the robot's own stitched floor image and |
| the gripper view is its gripper camera. |
|
|
| ## Inside |
|
|
| | | file | |
| |---|---| |
| | world model (JEPA latent + a hand-written cable/pendulum backbone it corrects) | `cablepilot/model/learned.py`, `weights/stringman_g1.pt` | |
| | the vortex (MPPI: correlated noise, temperature by effective sample size, warm start) | `cablepilot/planner/mppi.py` | |
| | task cards: schema, validation, signature, lineage | `cablepilot/core/cards.py` | |
| | the cable body's skills and how a card runs on them | `cablepilot/cardrun.py`, `cablepilot/tasks/pick_place.py` | |
| | perception: overhead differencing, gripper camera, one belief per object | `cablepilot/perception/` | |
| | MuJoCo twin of Stringman (4 lines, spools, pole, gripper), every number with its source | `cablepilot/sim/`, `configs/stringman.yaml` | |
| | the real robot (Neufangled's Python API, as a Maneuver) | `cablepilot/stringman.py` | |
|
|
| State (14): gantry position and velocity, pole swing and its rate, wrist angle and rate, finger closure and rate. |
| Command (5): gantry velocity (what `move_direction_speed` takes), wrist rate, closure target. Model step 0.1 s, |
| 1 s of history. |
|
|
| ## Train it yourself |
|
|
| The model was trained on the public dataset [`MacroSwan/cablepilot-motion`](https://huggingface.co/datasets/MacroSwan/cablepilot-motion): |
| 12,000 trajectories, about 100 hours of task-free simulated motion, each in its own random room. It can be regenerated |
| in about 15 minutes on 4 cores (`scripts/cdpr_datagen.py`) and retrained on a laptop CPU without a GPU |
| (`scripts/train_cdpr_wm.py`). `scripts/import_room.py` turns a Stringman LeRobot recording into a simulated room. |
| AGENTS.md, Step 7, gives the commands and the bar a new model has to beat. |
|
|
| ## Status |
|
|
| - **Simulation:** proven. Model test on 500 held-out windows in 310 unseen rooms: grasp point 19.6 mm median error at |
| 2 s, the same as MuJoCo with nominal physics (21.0 mm). `clean_the_room` in the bedroom imported from a Stringman |
| recording: 29/36 over three episodes (this release, MuJoCo 3.15.0). Scores move by about one toy in twelve between |
| library versions (MuJoCo 4.0 gives 28/36 with the same code), so compare on the reference simulator and several |
| episodes. |
| - **Real robot:** not yet run. `cablepilot/stringman.py` maps the model's state and commands onto Neufangled's |
| published Python API. It starts in `dry_run` mode, which plans and logs but sends nothing. The first real test is |
| `hover()`. The places where it guesses at the API are marked `ASSUME`. This is the first thing to do together with |
| a Stringman owner. |
| - **Gripper:** the released model is for Stringman's stock gripper. An SO-101 pinch model exists and is not released |
| yet. |
|
|
| ## Limits |
|
|
| The twin's masses, line stretch, spool acceleration and swing damping are estimates until measured on a real robot. |
| Furniture, the floor and the objects are not in the model (the task's costs handle them). Payloads up to 0.5 kg. |
|
|
| ## Credits and license |
|
|
| Apache-2.0. The two real rooms are imported from Neufangled's public Stringman recordings |
| [`naavox/bedroom-sep6`](https://huggingface.co/datasets/naavox/bedroom-sep6) and |
| [`naavox/playroom-sep29-2`](https://huggingface.co/datasets/naavox/playroom-sep29-2) (Apache-2.0). Stringman is |
| Neufangled's robot ([neufangled.com](https://neufangled.com)); this project is independent of it. By Edgar Moreau |
| ([huggingface.co/Ethgar](https://huggingface.co/Ethgar)). |
|
|
