Test-1 bar on cablepilot-motion

3c556c3 verified 4 days ago

10.2 kB

# AGENTS.md: cablepilot, for coding agents

You are helping a person run a cable robot (or simulate one) with a small world model and a planner, and, if it goes
well, share what you improved. Read this file once, then work through the steps in order. Every command prints what
it decided, so report its output to your person rather than paraphrasing it.

## What this is, in one paragraph

`cablepilot` learned how a cable robot moves (28k parameters, no task inside). A planner imagines ~128 futures every
0.2 s and picks the cheapest. A **task** is a body-free YAML **card** (stages of skills, fallbacks) plus a **tuning**
file (this body's cost weights and thresholds). Tasks are shared in the Hugging Face dataset `MacroSwan/pilot-tasks`,
with every scored run, so you start from the best card and tuning so far, never from scratch.

## Rules

1. **Simulation before hardware, always.** Never send a command to a real robot from a card or a tuning that has not
   scored in `cablepilot run` first. On a real robot, start with `dry_run=True`, then `hover()`, and only then a task,
   with the person watching and a hand on the robot's stop.
2. **Change one thing, measure, keep the receipt.** Every claim you make about a card or a tuning comes with the
   `result.json` that shows it. Compare runs on the same `--world` and `--seed`, on the reference simulator
   (`doctor` says if yours is not). One toy in twelve is within noise (library versions alone move it), so say
   "better" only over `--episodes 3` or more, in both rooms.
3. **Cards stay body-free.** No metres, colours or positions in a card. Numbers go in `tuning/<body>.yaml`.
4. **Keep the lineage.** A new or changed card or tuning gets `lineage: {parent, by, why}`. Here `parent` is what you
   started from, `by` is you and your person (for example `"claude-code for @someone"`), and `why` is one line.
5. **Be honest in the PR.** Report the score you measured, including when it went down. A well-measured negative
   result is a useful contribution.

## Step 1: install and check the machine

```bash
git clone https://huggingface.co/MacroSwan/cablepilot && cd cablepilot
python -m venv .venv && . .venv/bin/activate        # Python >= 3.10
pip install -e ".[sim,hub]"
cablepilot doctor
```

`doctor` picks the thread count. On Apple Silicon under Linux, and on other machines with mixed big and little cores,
too many threads makes planning about 9x slower. Then it times one plan. If the `speed` line is not `ok`, tell your
person before going further.

## Step 2: describe the setup (`rig.yaml`)

Ask your person, then write `rig.yaml` (fields and meanings: `cablepilot/rig.py`, top of the file):

- Which robot? `stringman`, or `sim` if they have none yet.
- Which gripper? `stringman` (stock) is the only released model today.
- Where are the 4 line anchors? `robot` reads the calibrated ones from a Stringman. Otherwise ask for the (4, 3)
  metres in the firmware's line order.
- Which named places should the card's sites bind to? For example `toy_box: hamper`.

Run `cablepilot doctor --rig rig.yaml`. It writes `doctor.json` next to the rig.

## Step 3: get a task and score it in simulation

```bash
cablepilot fetch clean_the_room                         # -> pilot-tasks/tasks/clean_the_room/
cablepilot check pilot-tasks/tasks/clean_the_room/card.yaml
cablepilot run   pilot-tasks/tasks/clean_the_room/card.yaml --world bedroom --seed 0 --video --by "<you> for <person>"
```

Worlds: `bedroom` and `playroom` are real rooms imported from Stringman recordings. A run is 10 simulated minutes and
takes about 25 minutes of wall time on a laptop. Read `result.json`:
- `score.done` / `score.of`: the task's own done test.
- `s_per_item`.
- `furniture_contact_ticks` (must stay 0).
- `max_tension_N`.
- `compute`.
- `episodes_detail[].trace`: what the robot did, in the card's own skills.
- `episodes_detail[].picks[].events`: why each phase ended, in plain words.

## Step 4: improve something (pick one)

- **Tune.** Copy `tuning/cable.yaml` and change a few numbers. Each key is commented in
  `configs/tasks/pick_place.yaml`. Start where the trace and events point. If time goes to `approach`, look at the
  `funnel` and `close_tol` keys. Swinging in `carry` means `swing_carry`. Drops on the way are `grip`. Score it on
  two or more seeds and both rooms.
- **A new card from an existing one.** Copy a card, change its objects, stages and fallbacks (for example
  *clean_the_room* → *clean_the_room_sorted*, or *pen_in_cup* → *pen_in_box*), set `lineage`, then run
  `cablepilot check` and `run`.
- **A missing skill.** `cablepilot check` on a card the cable robot cannot run lists the missing skills, for example
  `wipe` for window washing or `on_top`. A skill is Python: the skill's cost terms on imagined futures, its done event
  on measured state, and which phases it uses (`cablepilot/cardrun.py` maps skills to the phases in
  `cablepilot/tasks/pick_place.py`). Open a PR on this repository and include a card that uses the skill and its
  score.
- **Real robot (Stringman owners).** `cablepilot/stringman.py` is untested on hardware, and every guess about the API
  is marked `ASSUME`. Check those against `nf_robot` with your person, run `hover()` in `dry_run`, then for real, and
  share the log. This is the most valuable contribution right now.

## Step 5: share it

```bash
cablepilot submit runs/<your run> --task clean_the_room --tuning my_tuning.yaml -m "what you changed and why" --dry-run
cablepilot submit runs/<your run> --task clean_the_room --tuning my_tuning.yaml -m "..."   # needs `hf auth login`
```

This opens one pull request on `MacroSwan/pilot-tasks` containing the result, your card or tuning, and the film. The PR
page is the discussion: people and agents comment there, the maintainers re-run it, and the best merged tuning becomes
the next person's starting point. Ask your person before submitting. It publishes under their Hugging Face account.

## Step 6: tell everyone how it went in your home

The maintainers do not have a Stringman. Your report is how anyone learns whether this works in a real home, so write
one even when (above all when) it went badly. Open a **Discussion** on this repository, titled
`[home] <room> · <card> · <what happened in five words>`, filled from [`HOME_REPORT.md`](HOME_REPORT.md). It covers the
room, the rig, what you changed to integrate it, what the robot did, what broke, and what you would change. Attach
`doctor.json`, `rig.yaml` and the run's `result.json`. If you have a real-robot log, open a PR on the dataset
`MacroSwan/real-runs` with it as well (format in its README; blur faces first). Read the other `[home]` threads
before you start: someone may already have fixed your problem. Answer them when you can.

## Step 7: change the model itself

Everything the model learned from is public, so you can regenerate it, retrain it and replace it without asking anyone.

| what | where |
|---|---|
| the training data: 12,000 task-free trajectories (100 h of simulated motion, each in its own random room) | dataset `MacroSwan/cablepilot-motion` |
| how that data is made (the motion recipe, randomised physics, the host's real spool loop) | `scripts/cdpr_datagen.py` (docstring) |
| training, then test 1 (open-loop error at 2 s on held-out rooms, next to MuJoCo and to the physics alone) | `scripts/train_cdpr_wm.py` |
| a real Stringman room from a LeRobot recording (anchors, named places, floor image) | `scripts/import_room.py` |

```bash
hf download MacroSwan/cablepilot-motion --repo-type dataset --local-dir data/motion       # ~280 MB, or regenerate:
python scripts/cdpr_datagen.py --scenario configs/scenarios/clear_playroom.yaml --n 12000 --seed 0 --workers 4 --out data/motion
python scripts/train_cdpr_wm.py --data data/motion --out weights/mine.pt                  # minutes on a laptop CPU, no GPU
python scripts/train_cdpr_wm.py --eval-only --data data/motion --out weights/stringman_g1.pt   # the bar to beat
```

On the last 500 trajectories of `MacroSwan/cablepilot-motion` (held out by `--test 500`), the released
`weights/stringman_g1.pt` scores **20.4 mm** median grasp-point error at 2 s. The MuJoCo oracle scores 21.5 mm. A new model has to beat that on the same test, and then score at least as well on a
task (`cablepilot run`, 3 episodes, both rooms), before it replaces the old one. Open a PR here with the weights, the
test-1 JSON written next to them, the run's `result.json` and one line of `why`. To use your model in a run, point
`configs/planner_learned.yaml` at it locally (a tuning file may never change the model).

**Your own recordings are the best test set there is.** Any Stringman LeRobot dataset with `anchor_poses`, for example
`naavox/stringman_cleans`, holds the real state and the real commands. Converting a few of its episodes into the
`cdpr_datagen` format and running `--eval-only --test-all` on them gives the model's first error measured on a real
machine. Nobody has done that yet: it is the most useful thing on the list below.

## Open jobs (pick one, say so in a Discussion first)

1. **The model on real data.** Test 1 on real Stringman episodes (see Step 7). Then fine-tune on them, if it helps.
2. **`avoid_furniture`.** The card is in `MacroSwan/pilot-tasks`. `cablepilot check` says what is missing: a `near`
   skill (a runner around `clear_floor.GotoTask`) and a furnished room (`scripts/import_room.py` plus the furniture
   boxes that `runner.obstacles` reads).
3. **`stringman.py` on hardware.** Check each `ASSUME` against `nf_robot`, then `hover()` in `dry_run`, then for real.
4. **Your room.** Import it with `scripts/import_room.py` and add it as a `--world`, so that everyone's cards are scored in
   a third real room.
## Where things are
`README.md` covers the model and the numbers. `cablepilot/cli.py` has the commands. `cablepilot/core/cards.py` defines
the card schema: the predicates, the skills and `signature()`. `cablepilot/cardrun.py` maps cards to the cable skills
and writes the result format. `cablepilot/stringman.py` is the real robot adapter. `scripts/` holds the data generator, the trainer and the room importer.

	# AGENTS.md: cablepilot, for coding agents

	You are helping a person run a cable robot (or simulate one) with a small world model and a planner, and, if it goes
	well, share what you improved. Read this file once, then work through the steps in order. Every command prints what
	it decided, so report its output to your person rather than paraphrasing it.

	## What this is, in one paragraph

	`cablepilot` learned how a cable robot moves (28k parameters, no task inside). A planner imagines ~128 futures every
	0.2 s and picks the cheapest. A task is a body-free YAML card (stages of skills, fallbacks) plus a tuning
	file (this body's cost weights and thresholds). Tasks are shared in the Hugging Face dataset `MacroSwan/pilot-tasks`,
	with every scored run, so you start from the best card and tuning so far, never from scratch.

	## Rules

	1. Simulation before hardware, always. Never send a command to a real robot from a card or a tuning that has not
	scored in `cablepilot run` first. On a real robot, start with `dry_run=True`, then `hover()`, and only then a task,
	with the person watching and a hand on the robot's stop.
	2. Change one thing, measure, keep the receipt. Every claim you make about a card or a tuning comes with the
	`result.json` that shows it. Compare runs on the same `--world` and `--seed`, on the reference simulator
	(`doctor` says if yours is not). One toy in twelve is within noise (library versions alone move it), so say
	"better" only over `--episodes 3` or more, in both rooms.
	3. Cards stay body-free. No metres, colours or positions in a card. Numbers go in `tuning/<body>.yaml`.
	4. Keep the lineage. A new or changed card or tuning gets `lineage: {parent, by, why}`. Here `parent` is what you
	started from, `by` is you and your person (for example `"claude-code for @someone"`), and `why` is one line.
	5. Be honest in the PR. Report the score you measured, including when it went down. A well-measured negative
	result is a useful contribution.

	## Step 1: install and check the machine

	```bash
	git clone https://huggingface.co/MacroSwan/cablepilot && cd cablepilot
	python -m venv .venv && . .venv/bin/activate # Python >= 3.10
	pip install -e ".[sim,hub]"
	cablepilot doctor
	```

	`doctor` picks the thread count. On Apple Silicon under Linux, and on other machines with mixed big and little cores,
	too many threads makes planning about 9x slower. Then it times one plan. If the `speed` line is not `ok`, tell your
	person before going further.

	## Step 2: describe the setup (`rig.yaml`)

	Ask your person, then write `rig.yaml` (fields and meanings: `cablepilot/rig.py`, top of the file):

	- Which robot? `stringman`, or `sim` if they have none yet.
	- Which gripper? `stringman` (stock) is the only released model today.
	- Where are the 4 line anchors? `robot` reads the calibrated ones from a Stringman. Otherwise ask for the (4, 3)
	metres in the firmware's line order.
	- Which named places should the card's sites bind to? For example `toy_box: hamper`.

	Run `cablepilot doctor --rig rig.yaml`. It writes `doctor.json` next to the rig.

	## Step 3: get a task and score it in simulation

	```bash
	cablepilot fetch clean_the_room # -> pilot-tasks/tasks/clean_the_room/
	cablepilot check pilot-tasks/tasks/clean_the_room/card.yaml
	cablepilot run pilot-tasks/tasks/clean_the_room/card.yaml --world bedroom --seed 0 --video --by "<you> for <person>"
	```

	Worlds: `bedroom` and `playroom` are real rooms imported from Stringman recordings. A run is 10 simulated minutes and
	takes about 25 minutes of wall time on a laptop. Read `result.json`:
	- `score.done` / `score.of`: the task's own done test.
	- `s_per_item`.
	- `furniture_contact_ticks` (must stay 0).
	- `max_tension_N`.
	- `compute`.
	- `episodes_detail[].trace`: what the robot did, in the card's own skills.
	- `episodes_detail[].picks[].events`: why each phase ended, in plain words.

	## Step 4: improve something (pick one)

	- Tune. Copy `tuning/cable.yaml` and change a few numbers. Each key is commented in
	`configs/tasks/pick_place.yaml`. Start where the trace and events point. If time goes to `approach`, look at the
	`funnel` and `close_tol` keys. Swinging in `carry` means `swing_carry`. Drops on the way are `grip`. Score it on
	two or more seeds and both rooms.
	- A new card from an existing one. Copy a card, change its objects, stages and fallbacks (for example
	clean_the_room → clean_the_room_sorted, or pen_in_cup → pen_in_box), set `lineage`, then run
	`cablepilot check` and `run`.
	- A missing skill. `cablepilot check` on a card the cable robot cannot run lists the missing skills, for example
	`wipe` for window washing or `on_top`. A skill is Python: the skill's cost terms on imagined futures, its done event
	on measured state, and which phases it uses (`cablepilot/cardrun.py` maps skills to the phases in
	`cablepilot/tasks/pick_place.py`). Open a PR on this repository and include a card that uses the skill and its
	score.
	- Real robot (Stringman owners). `cablepilot/stringman.py` is untested on hardware, and every guess about the API
	is marked `ASSUME`. Check those against `nf_robot` with your person, run `hover()` in `dry_run`, then for real, and
	share the log. This is the most valuable contribution right now.

	## Step 5: share it

	```bash
	cablepilot submit runs/<your run> --task clean_the_room --tuning my_tuning.yaml -m "what you changed and why" --dry-run
	cablepilot submit runs/<your run> --task clean_the_room --tuning my_tuning.yaml -m "..." # needs `hf auth login`
	```

	This opens one pull request on `MacroSwan/pilot-tasks` containing the result, your card or tuning, and the film. The PR
	page is the discussion: people and agents comment there, the maintainers re-run it, and the best merged tuning becomes
	the next person's starting point. Ask your person before submitting. It publishes under their Hugging Face account.

	## Step 6: tell everyone how it went in your home

	The maintainers do not have a Stringman. Your report is how anyone learns whether this works in a real home, so write
	one even when (above all when) it went badly. Open a Discussion on this repository, titled
	`[home] <room> · <card> · <what happened in five words>`, filled from [`HOME_REPORT.md`](HOME_REPORT.md). It covers the
	room, the rig, what you changed to integrate it, what the robot did, what broke, and what you would change. Attach
	`doctor.json`, `rig.yaml` and the run's `result.json`. If you have a real-robot log, open a PR on the dataset
	`MacroSwan/real-runs` with it as well (format in its README; blur faces first). Read the other `[home]` threads
	before you start: someone may already have fixed your problem. Answer them when you can.

	## Step 7: change the model itself

	Everything the model learned from is public, so you can regenerate it, retrain it and replace it without asking anyone.

	\| what \| where \|
	\|---\|---\|
	\| the training data: 12,000 task-free trajectories (100 h of simulated motion, each in its own random room) \| dataset `MacroSwan/cablepilot-motion` \|
	\| how that data is made (the motion recipe, randomised physics, the host's real spool loop) \| `scripts/cdpr_datagen.py` (docstring) \|
	\| training, then test 1 (open-loop error at 2 s on held-out rooms, next to MuJoCo and to the physics alone) \| `scripts/train_cdpr_wm.py` \|
	\| a real Stringman room from a LeRobot recording (anchors, named places, floor image) \| `scripts/import_room.py` \|

	```bash
	hf download MacroSwan/cablepilot-motion --repo-type dataset --local-dir data/motion # ~280 MB, or regenerate:
	python scripts/cdpr_datagen.py --scenario configs/scenarios/clear_playroom.yaml --n 12000 --seed 0 --workers 4 --out data/motion
	python scripts/train_cdpr_wm.py --data data/motion --out weights/mine.pt # minutes on a laptop CPU, no GPU
	python scripts/train_cdpr_wm.py --eval-only --data data/motion --out weights/stringman_g1.pt # the bar to beat
	```

	On the last 500 trajectories of `MacroSwan/cablepilot-motion` (held out by `--test 500`), the released
	`weights/stringman_g1.pt` scores 20.4 mm median grasp-point error at 2 s. The MuJoCo oracle scores 21.5 mm. A new model has to beat that on the same test, and then score at least as well on a
	task (`cablepilot run`, 3 episodes, both rooms), before it replaces the old one. Open a PR here with the weights, the
	test-1 JSON written next to them, the run's `result.json` and one line of `why`. To use your model in a run, point
	`configs/planner_learned.yaml` at it locally (a tuning file may never change the model).

	Your own recordings are the best test set there is. Any Stringman LeRobot dataset with `anchor_poses`, for example
	`naavox/stringman_cleans`, holds the real state and the real commands. Converting a few of its episodes into the
	`cdpr_datagen` format and running `--eval-only --test-all` on them gives the model's first error measured on a real
	machine. Nobody has done that yet: it is the most useful thing on the list below.

	## Open jobs (pick one, say so in a Discussion first)

	1. The model on real data. Test 1 on real Stringman episodes (see Step 7). Then fine-tune on them, if it helps.
	2. `avoid_furniture`. The card is in `MacroSwan/pilot-tasks`. `cablepilot check` says what is missing: a `near`
	skill (a runner around `clear_floor.GotoTask`) and a furnished room (`scripts/import_room.py` plus the furniture
	boxes that `runner.obstacles` reads).
	3. `stringman.py` on hardware. Check each `ASSUME` against `nf_robot`, then `hover()` in `dry_run`, then for real.
	4. Your room. Import it with `scripts/import_room.py` and add it as a `--world`, so that everyone's cards are scored in
	a third real room.

	## Where things are

	`README.md` covers the model and the numbers. `cablepilot/cli.py` has the commands. `cablepilot/core/cards.py` defines
	the card schema: the predicates, the skills and `signature()`. `cablepilot/cardrun.py` maps cards to the cable skills
	and writes the result format. `cablepilot/stringman.py` is the real robot adapter. `scripts/` holds the data generator, the trainer and the room importer.