Instructions to use PINlab/pi05-so101 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LeRobot
How to use PINlab/pi05-so101 with LeRobot:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
|
Download README.md from PINlab/pi05-so101: direct link, hf CLI and curl.
- Browser
- Download file 6.72 kB
-
https://huggingface.co/PINlab/pi05-so101/resolve/main/README.md
- Command line
-
hf download hf://PINlab/pi05-so101/README.md
-
curl -L -o README.md https://huggingface.co/PINlab/pi05-so101/resolve/main/README.md
6.72 kB
| library_name: openpi | |
| license: apache-2.0 | |
| tags: [robotics, pi0, pi05, openpi, so101, lerobot, tactile] | |
| datasets: [endoard/grab_ball_3cam_skin, endoard/grab_cup_3cam_skin, endoard/grab_red_cube_3cam_skin, endoard/grab_green_cube_3cam_skin, endoard/grab_green_cylinder_3cam_skin, endoard/grab_red_cube_distractor_green_cube_3cam_skin, endoard/grab_green_cube_distractor_red_cube_3cam_skin] | |
| # pi0.5 SO-101 with a tactile skin | |
| pi0.5 policies for a single SO-101 arm with a 5-chip magnetic tactile skin, fine-tuned from | |
| `pi05_base`. One folder per training run, one sub-folder per checkpoint step. | |
| None has been evaluated on the robot yet. | |
| ## What is here | |
| | folder | config | cameras | prompt(s) | data | steps | | |
| |---|---|---|---|---|---| | |
| | `grab_ball_skin/` | `pi05_so101_grab_ball_skin` | 2: base = `fixed` (train: `fixed`/`side` at random), wrist = `handeye` | "Grab the ball" | `endoard/grab_ball_3cam_skin`, first 100 episodes / 46,076 frames | 10000, 20000, 29999 | | |
| | `grab_ball_skin_3cam/` | `pi05_so101_grab_ball_skin_3cam` | 3: `fixed`, `handeye`, `side` | "Grab the ball" | `endoard/grab_ball_3cam_skin`, 150 episodes / 65,989 frames | 10000, 20000, 29999 | | |
| | `grab_cup_skin_3cam/` | `pi05_so101_grab_cup_skin_3cam` | 3: `fixed`, `handeye`, `side` | "Grab the blue cup" | `endoard/grab_cup_3cam_skin`, 166 episodes / 64,812 frames | 10000, 20000, 29999 | | |
| | `grab_3obj_skin_3cam/` | `pi05_so101_grab_3obj_skin_3cam` | 3: `fixed`, `handeye`, `side` | "Grab the red cube" / "Grab the green cube" / "Grab the green cylinder" | `endoard/grab_{red_cube,green_cube,green_cylinder}_3cam_skin` merged, 450 episodes / 161,918 frames, one task per source | 10000, 20000, 28000 | | |
| | `grab_cubes_skin_3cam/` | `pi05_so101_grab_cubes_skin_3cam` | 3: `fixed`, `handeye`, `side` | "Grab the red cube" / "Grab the green cube" (see [Prompts](#prompts)) | `endoard/grab_{red_cube,green_cube,red_cube_distractor_green_cube,green_cube_distractor_red_cube}_3cam_skin` merged, 550 episodes / 191,324 frames | 10000, 20000, 29999 | | |
| | `lift_leave_cubes_skin_3cam/` | `pi05_so101_lift_leave_cubes_skin_3cam` | 3: `fixed`, `handeye`, `side` | "Lift the red cube" / "Lift the green cube" / "Lift the red cube, leave the green cube on the table" / "Lift the green cube, leave the red cube on the table" (see [Prompts](#prompts)) | same 4 cube datasets merged, 550 episodes / 191,324 frames | 10000, 20000, 29999 | | |
| `grab_3obj_skin_3cam` is multi-task: the prompt selects the object. Its run was stopped at | |
| step 28,500 of 30k, so 28000 is its last checkpoint. Its held-out split is stratified per task | |
| (15 episodes each). | |
| `grab_cubes_skin_3cam` and `lift_leave_cubes_skin_3cam` are trained on the same 550 cube episodes | |
| with different prompts; their held-out split is stratified per source (15/15/15/10 episodes). | |
| ## Prompts | |
| Every policy here was trained on a fixed set of text prompts. At inference send one of them | |
| **verbatim** as the observation `prompt` (same wording, case and punctuation); other phrasings | |
| are untested. | |
| | folder | training prompts (exact strings) | | |
| |---|---| | |
| | `grab_ball_skin/`, `grab_ball_skin_3cam/` | `Grab the ball` | | |
| | `grab_cup_skin_3cam/` | `Grab the blue cup` | | |
| | `grab_3obj_skin_3cam/` | `Grab the red cube`, `Grab the green cube`, `Grab the green cylinder` | | |
| | `grab_cubes_skin_3cam/` | `Grab the red cube`, `Grab the green cube` | | |
| | `lift_leave_cubes_skin_3cam/` | `Lift the red cube`, `Lift the green cube`, `Lift the red cube, leave the green cube on the table`, `Lift the green cube, leave the red cube on the table` | | |
| Which prompt each source dataset was trained with in the two cube runs: | |
| | source dataset (episodes) | scene | `grab_cubes_skin_3cam` | `lift_leave_cubes_skin_3cam` | | |
| |---|---|---|---| | |
| | `endoard/grab_red_cube_3cam_skin` (150) | red cube alone | `Grab the red cube` | `Lift the red cube` | | |
| | `endoard/grab_green_cube_3cam_skin` (150) | green cube alone | `Grab the green cube` | `Lift the green cube` | | |
| | `endoard/grab_red_cube_distractor_green_cube_3cam_skin` (150) | red cube + green cube | `Grab the red cube` | `Lift the red cube, leave the green cube on the table` | | |
| | `endoard/grab_green_cube_distractor_red_cube_3cam_skin` (100) | green cube + red cube | `Grab the green cube` | `Lift the green cube, leave the red cube on the table` | | |
| So `grab_cubes_skin_3cam` uses one prompt per target colour whether or not the other cube is | |
| on the table, and `lift_leave_cubes_skin_3cam` also names the distractor when there is one. | |
| Each step folder is an openpi checkpoint without optimizer state: `params/` (EMA weights), | |
| `assets/<asset_id>/norm_stats.json` (quantile stats), `_CHECKPOINT_METADATA`, and a `README.md` | |
| with that checkpoint's details. | |
| ## Download one checkpoint | |
| ```python | |
| from huggingface_hub import snapshot_download | |
| d = snapshot_download("PINlab/pi05-so101", allow_patterns="grab_cup_skin_3cam/29999/*") | |
| # serve with --policy.dir={d}/grab_cup_skin_3cam/29999 | |
| ``` | |
| Or with the CLI, several folders at once (here both cube policies at step 20000, ~12 GB each). | |
| Needs read access to PINlab and `hf auth login` (or `HF_TOKEN`); on an older `huggingface_hub` | |
| use `huggingface-cli download` with the same arguments: | |
| ```bash | |
| hf download PINlab/pi05-so101 \ | |
| --include "grab_cubes_skin_3cam/20000/*" "lift_leave_cubes_skin_3cam/20000/*" \ | |
| --local-dir pi05-so101 | |
| # -> pi05-so101/grab_cubes_skin_3cam/20000/ and pi05-so101/lift_leave_cubes_skin_3cam/20000/ | |
| ``` | |
| Swap the folder and step in `--include` for any other checkpoint in the table. | |
| ## Serving | |
| The checkpoints carry no config. Serve each one under the config in the table, which pins | |
| its `asset_id`: | |
| ```bash | |
| python scripts/serve_policy.py policy:checkpoint \ | |
| --policy.config=pi05_so101_grab_cup_skin_3cam \ | |
| --policy.dir=<download>/grab_cup_skin_3cam/29999 | |
| ``` | |
| The policy returns 16 actions **spaced 2 frames apart (15 Hz)**, absolute joints (the delta is | |
| undone server-side). Replay at 15 Hz or interpolate; replaying at 30 Hz halves the speed. | |
| The 3-camera policies need all three cameras every step. | |
| ## Recipe (all runs) | |
| - State 21-D (padded to 32): 5 joints (deg), gripper (0-100), 5 skin chips x xyz (uT), raw. | |
| - Action 6-D (padded to 32): **delta on the 5 joints, absolute gripper** (`make_bool_mask(5, -1)`). | |
| - Images: `base_0_rgb` = `fixed`, `left_wrist_0_rgb` = `handeye`; `right_wrist_0_rgb` = `side` | |
| (mask on) for 3-camera runs, zeros with mask off for `grab_ball_skin`. | |
| - `action_horizon=16`, `action_stride=2`, prompt from the dataset task string (remapped per | |
| source for the two cube runs, see [Prompts](#prompts)). | |
| - From `pi05_base`, 4 nodes x 1 A100-64GB, `--fsdp-devices=4`, batch 32, AdamW with cosine | |
| decay, EMA 0.99, seed 42, 30k steps, 10% of episodes held out. | |