--- license: apache-2.0 library_name: pytorch pipeline_tag: robotics tags: - robotics - world-model - model-predictive-control - cable-driven-parallel-robot - stringman - task-cards datasets: - MacroSwan/pilot-tasks - MacroSwan/cablepilot-motion --- # cablepilot A **28,401-parameter world model** of a cable robot, a **sampling planner** that imagines a few hundred futures with it every control step, and **task cards**: plain YAML files that say what a job is, which anyone (or their agent) can write, improve, score and share. No demonstrations. No GPU. A new task is a card plus a few numbers, not a dataset and a training run. > **Agents:** read [`AGENTS.md`](AGENTS.md). It is written for you: install, check the setup, score a card, contribute. | | | |---|---| | model | 28k parameters, 124 KB, trained on task-free simulated motion in 12,000 random rooms, 11 min on a laptop CPU | | one plan | 128 futures x 2 s imagined in ~25 ms on a laptop CPU (5 plans per second leaves ~85 % of the budget free) | | any room | the room enters only as its 4 calibrated mount points: a room it never saw is just 4 new numbers | | clean the room (sim) | 29/36 toys into the box over 3 episodes in a real bedroom imported from a Stringman recording, 0 furniture contacts, 62 s per toy | | real robot | **not yet run on hardware.** The Stringman adapter is written, not tested: see "Status" | ## Why a small world model The usual route to a robot that does chores is to record people driving it, then train a large policy on the recordings. On Stringman the public record of that route is 97 hours of teleoperation and 56 models (52 M to 4.1 B parameters), on rented H100s / H200s, and most of those models cannot decide fast enough on a laptop. This repository takes the other route. It learns **the machine** (how the gantry, the lines and the swinging pole respond to commands), once, from simulation, without any task in it. Then each task is a **cost** on imagined futures (near the toy, low over it, not swinging, lines not too tight, clear of the bed) plus a few **events** on what the robot measures (the jaw stopped on something, the lines weigh 30 g more). The planner (MPPI, "the vortex": the purple futures in the videos) picks the commands whose futures cost least, five times a second. So a new task costs a person an afternoon of thinking, not a week of teleoperation, and anyone's improvement to a task is a small text diff that the next person starts from. ## Try it (3 minutes, no robot) ```bash git clone https://huggingface.co/MacroSwan/cablepilot && cd cablepilot pip install -e ".[sim]" # numpy, torch, pyyaml + mujoco for the simulator cablepilot doctor # what this machine can do (threads, plan time) cablepilot fetch clean_the_room # the task folder from the hub: card + best tuning + results cablepilot check pilot-tasks/tasks/clean_the_room/card.yaml cablepilot run pilot-tasks/tasks/clean_the_room/card.yaml --world bedroom --video ``` The run writes `runs/<...>/result.json` (the score, with the hashes of what ran) and a film: the room with the plan in purple | the overhead camera with what it found | the gripper camera. ## The three files of a task Tasks live in the dataset [`MacroSwan/pilot-tasks`](https://huggingface.co/datasets/MacroSwan/pilot-tasks), one folder each, shared by every pilot (cable robot, arm, drone, ...): ``` tasks/clean_the_room/ card.yaml WHAT: stages, skills, fallbacks. Body-free: the same file runs on an arm or a cable robot. tuning/cable.yaml HOW MUCH, for this body: the cost weights and thresholds (every number lives here) results/*.json what each run scored, where, with which card and tuning (by hash), by whom ``` A card reads like the job: *observe the floor; approach the stray thing; grasp it; bring it over the box, still holding it; let go. If you lose sight of it, look again; if it slips, approach again.* Each stage names a **skill** from a shared vocabulary (`see`, `pre_grasp`, `grasp`, `pre_inside`, `release`, `avoid`, ...). A card may only use skills the body has; `cablepilot check` lists the ones it is missing, and that list is the best place to contribute. Cards carry their **lineage** (`parent`, `by`, `why`), so a task grows as a tree: *clean the room* -> *clean the room, sorted* -> *put things back where they live*; *pen in cup* -> *pen in box*. Two cards with the same `signature()` are the same task with different names. ## Your setup Everything that differs between two cable robots goes in a small `rig.yaml` (the agent fills it with you): ```yaml rig: living-room robot: stringman # stringman | sim gripper: stringman # picks the world model trained for that gripper mounts: robot # read the calibrated anchor points from the robot, or give (4, 3) metres cameras: {overhead: ortho, gripper: true} compute: {device: auto, threads: auto, plan_hz: 5} places: {toy_box: hamper} # a card's site -> one of the robot's named places ``` `cablepilot doctor --rig rig.yaml` measures this machine against it and writes `doctor.json` beside it. **Vision.** cablepilot does not bring its own heavy vision model. It uses the cameras your robot already has, in the cheapest way that works. The overhead view compares the floor with the clean floor and masks out the robot's own pixels (about 9 ms per look). The gripper camera then finds the target exactly as the hand gets close. Both feed one belief per object, which the task reads. On Stringman, the overhead view is the robot's own stitched floor image and the gripper view is its gripper camera. ## Inside | | file | |---|---| | world model (JEPA latent + a hand-written cable/pendulum backbone it corrects) | `cablepilot/model/learned.py`, `weights/stringman_g1.pt` | | the vortex (MPPI: correlated noise, temperature by effective sample size, warm start) | `cablepilot/planner/mppi.py` | | task cards: schema, validation, signature, lineage | `cablepilot/core/cards.py` | | the cable body's skills and how a card runs on them | `cablepilot/cardrun.py`, `cablepilot/tasks/pick_place.py` | | perception: overhead differencing, gripper camera, one belief per object | `cablepilot/perception/` | | MuJoCo twin of Stringman (4 lines, spools, pole, gripper), every number with its source | `cablepilot/sim/`, `configs/stringman.yaml` | | the real robot (Neufangled's Python API, as a Maneuver) | `cablepilot/stringman.py` | State (14): gantry position and velocity, pole swing and its rate, wrist angle and rate, finger closure and rate. Command (5): gantry velocity (what `move_direction_speed` takes), wrist rate, closure target. Model step 0.1 s, 1 s of history. ## Train it yourself The model was trained on the public dataset [`MacroSwan/cablepilot-motion`](https://huggingface.co/datasets/MacroSwan/cablepilot-motion): 12,000 trajectories, about 100 hours of task-free simulated motion, each in its own random room. It can be regenerated in about 15 minutes on 4 cores (`scripts/cdpr_datagen.py`) and retrained on a laptop CPU without a GPU (`scripts/train_cdpr_wm.py`). `scripts/import_room.py` turns a Stringman LeRobot recording into a simulated room. AGENTS.md, Step 7, gives the commands and the bar a new model has to beat. ## Status - **Simulation:** proven. Model test on 500 held-out windows in 310 unseen rooms: grasp point 19.6 mm median error at 2 s, the same as MuJoCo with nominal physics (21.0 mm). `clean_the_room` in the bedroom imported from a Stringman recording: 29/36 over three episodes (this release, MuJoCo 3.15.0). Scores move by about one toy in twelve between library versions (MuJoCo 4.0 gives 28/36 with the same code), so compare on the reference simulator and several episodes. - **Real robot:** not yet run. `cablepilot/stringman.py` maps the model's state and commands onto Neufangled's published Python API. It starts in `dry_run` mode, which plans and logs but sends nothing. The first real test is `hover()`. The places where it guesses at the API are marked `ASSUME`. This is the first thing to do together with a Stringman owner. - **Gripper:** the released model is for Stringman's stock gripper. An SO-101 pinch model exists and is not released yet. ## Limits The twin's masses, line stretch, spool acceleration and swing damping are estimates until measured on a real robot. Furniture, the floor and the objects are not in the model (the task's costs handle them). Payloads up to 0.5 kg. ## Credits and license Apache-2.0. The two real rooms are imported from Neufangled's public Stringman recordings [`naavox/bedroom-sep6`](https://huggingface.co/datasets/naavox/bedroom-sep6) and [`naavox/playroom-sep29-2`](https://huggingface.co/datasets/naavox/playroom-sep29-2) (Apache-2.0). Stringman is Neufangled's robot ([neufangled.com](https://neufangled.com)); this project is independent of it. By Edgar Moreau ([huggingface.co/Ethgar](https://huggingface.co/Ethgar)).