Instructions to use MikelRubi/act-pick with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LeRobot
How to use MikelRubi/act-pick with LeRobot:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
|
Download README.md from MikelRubi/act-pick: direct link, hf CLI and curl.
- Browser
- Download file 15.1 kB
-
https://huggingface.co/MikelRubi/act-pick/resolve/main/README.md
- Command line
-
hf download hf://MikelRubi/act-pick/README.md
-
curl -L -o README.md https://huggingface.co/MikelRubi/act-pick/resolve/main/README.md
15.1 kB
| library_name: lerobot | |
| license: apache-2.0 | |
| pipeline_tag: robotics | |
| tags: | |
| - lerobot | |
| - act | |
| - robotics | |
| - so101 | |
| - robotekin-hackathon-2026 | |
| - pick | |
| datasets: | |
| - MikelRubi/robotekin-so101-pick | |
| - MikelRubi/so101-pick-cube_20261023_191626 | |
| - MikelRubi/so101-pick-cube_20261023_193855 | |
| - MikelRubi/so101-pick-cube_20261023_195654 | |
| - MikelRubi/so101-pick-cube_20261023_200003 | |
| - MikelRubi/so101-pick-cube_20261023_201205 | |
| - MikelRubi/so101-pick-cube_20261023_203046 | |
| - MikelRubi/so101-pick-cube_20261023_203822 | |
| - MikelRubi/so101-pick-cube_20261023_205608 | |
| - MikelRubi/so101-pick-cube_20261023_205924 | |
| - MikelRubi/so101-pick-cube_20261023_211050 | |
| - MikelRubi/so101-pick-cube_20261023_211739 | |
| - MikelRubi/so101-pick-cube_20261023_212258 | |
| - MikelRubi/so101-pick-cube_20261023_213054 | |
| - MikelRubi/so101-pick-cube_20261023_221518 | |
| - MikelRubi/so101-pick-cube_20261023_221852 | |
| - MikelRubi/so101-pick-cube_20261023_222526 | |
| - MikelRubi/so101-pick-cube_20261023_223103 | |
| - MikelRubi/so101-pick-cube_20261023_223515 | |
| - MikelRubi/so101-pick-cube_20261023_223658 | |
| - MikelRubi/so101-pick-cube_20261023_224034 | |
| - MikelRubi/so101-pick-cube_20261023_224547 | |
| - MikelRubi/so101-pick-cube_20261023_224858 | |
| - MikelRubi/so101-pick-cube_20261023_225412 | |
| - MikelRubi/so101-pick-cube_20261023_225556 | |
| - MikelRubi/so101-pick-cube_20261023_225932 | |
| - MikelRubi/so101-pick-cube_20261024_084532 | |
| - MikelRubi/so101-pick-cube_20261024_085325 | |
| - MikelRubi/so101-pick-cube_20261024_085554 | |
| - MikelRubi/so101-pick-cube_20261024_085829 | |
| - MikelRubi/so101-pick-cube_20261024_090220 | |
| - MikelRubi/so101-pick-cube_20261024_090900 | |
| - MikelRubi/so101-pick-cube_20261024_091415 | |
| - MikelRubi/so101-pick-cube_20261024_092742 | |
| - MikelRubi/so101-pick-cube_20261024_094405 | |
| - MikelRubi/so101-pick-cube_20261024_095901 | |
| - MikelRubi/so101-pick-cube_20261024_102241 | |
| - MikelRubi/so101-pick-cube_20261024_103318 | |
| - MikelRubi/so101-pick-cube_20261024_103634 | |
| - MikelRubi/so101-pick-cube_20261024_104618 | |
| - MikelRubi/so101-pick-cube_20261024_105854 | |
| - MikelRubi/so101-pick-cube_20261024_110840 | |
| - MikelRubi/so101-pick-cube_20261024_111303 | |
| - MikelRubi/so101-pick-cube_20261024_112511 | |
| - MikelRubi/so101-pick-cube_20261024_113237 | |
| - MikelRubi/so101-pick-cube_20261024_114214 | |
| - MikelRubi/so101-pick-cube_20261024_115532 | |
| - MikelRubi/so101-pick-cube_20261024_121209 | |
| - MikelRubi/so101-pick-cube_20261024_122039 | |
| - MikelRubi/so101-pick-cube_20261024_122345 | |
| - MikelRubi/so101-pick-cube_20261024_124413 | |
| - MikelRubi/so101-pick-cube_20261024_125836 | |
| - MikelRubi/so101-pick-cube_20261024_131047 | |
| - MikelRubi/so101-pick-cube_20261024_133639 | |
| - MikelRubi/so101-pick-cube_20261024_140131 | |
| - MikelRubi/so101-pick-cube_20261024_141427 | |
| - MikelRubi/so101-pick-cube_20261024_142915 | |
| - MikelRubi/so101-pick-cube_20261024_144537 | |
| - MikelRubi/so101-pick-cube_20261024_145140 | |
| - MikelRubi/so101-pick-cube_20261024_145921 | |
| - MikelRubi/so101-pick-cube_20261024_165825 | |
| - MikelRubi/so101-pick-cube_20261024_171106 | |
| - MikelRubi/so101-pick-cube_20261024_171946 | |
| - MikelRubi/so101-pick-cube_20261024_173031 | |
| - MikelRubi/so101-pick-cube_20261024_181939 | |
| - MikelRubi/so101-pick-cube_20261024_222410 | |
| - MikelRubi/so101-pick-cube_20261024_223612 | |
| - MikelRubi/up-down_20261023_211221 | |
| - MikelRubi/up-down_20261023_211821 | |
| - MikelRubi/up-down_20261023_212336 | |
| - MikelRubi/up-down_20261023_213122 | |
| - MikelRubi/up-down_20261023_222052 | |
| - MikelRubi/up-down_20261023_222555 | |
| - MikelRubi/up-down_20261023_223150 | |
| - MikelRubi/up-down_20261023_223615 | |
| - MikelRubi/up-down_20261023_223804 | |
| - MikelRubi/up-down_20261023_224139 | |
| - MikelRubi/up-down_20261023_224633 | |
| - MikelRubi/up-down_20261023_224932 | |
| - MikelRubi/up-down_20261023_225507 | |
| - MikelRubi/up-down_20261023_225625 | |
| - MikelRubi/up-down_20261023_230112 | |
| # Pick — ACT policy for picking the part (SO-101) | |
| **Picks one bracket from the crate, grasping it according to its orientation, lifts it and holds it up to the | |
| inspection camera.** Stage 1 of the pipeline below: it leaves the part ready for | |
| [`MikelRubi/pin-classifier`](https://huggingface.co/MikelRubi/pin-classifier) and for one of the sorting policies. | |
| ## Recommended checkpoint | |
| | | | | |
| |---|---| | |
| | **Checkpoint** | `pick_v3_super/checkpoints/006000/` | | |
| | **Why** | Error 15.16 on the held-out episodes, 27 % below "stay still" (20.71); trained on the most data (398 episodes) | | |
| | **Caveat** | Tied with `pick_v3_strict/002400` (15.14), within noise. Every run gets worse after about one pass over its data | | |
| | **Status** | No pick model has been tested on the robot yet | | |
| If there is little robot time, try **v3_super 6K → v3_strict 2.4K → v3_1-2-pieces 18K** in that order. | |
| ## The system | |
| Built at the **Robotekin LeRobot Hackathon 2026** (23–25 October 2026, Robotory, Vitoria-Gasteiz). One SO-101 arm runs | |
| a small quality-control cell for a metal bracket that may or may not carry a pin. **Project page:** | |
| [`MikelRubi/robotekin-so101-pin-sorting`](https://huggingface.co/spaces/MikelRubi/robotekin-so101-pin-sorting). | |
| ```mermaid | |
| flowchart LR | |
| order["Order (voice / text)<br/>intent model → sorting rule"] --> tray["Part lies in the tray"] | |
| tray --> pick["1 · act-pick<br/>grasp the part according to its orientation,<br/>lift it, hold it up to the camera"] | |
| pick --> cls{"2 · pin-classifier<br/>front + wrist images"} | |
| cls -- "rule: usually pin (GOOD)" --> good["3a · act-up-down-good<br/>carry it to the green box, release"] | |
| cls -- "rule: usually no pin (BAD)" --> bad["3b · act-up-down-bad<br/>carry it to the white box, release"] | |
| ``` | |
| | Stage | Repository | Model | | |
| |---|---|---| | |
| | 1. Picking | [`MikelRubi/act-pick`](https://huggingface.co/MikelRubi/act-pick) | ACT policy | | |
| | 2. Inspection | [`MikelRubi/pin-classifier`](https://huggingface.co/MikelRubi/pin-classifier) | EfficientNet-B0 image classifier | | |
| | 3a. Sorting, GOOD | [`MikelRubi/act-up-down-good`](https://huggingface.co/MikelRubi/act-up-down-good) | ACT policy | | |
| | 3b. Sorting, BAD | [`MikelRubi/act-up-down-bad`](https://huggingface.co/MikelRubi/act-up-down-bad) | ACT policy | | |
| Picking is a separate policy because the bracket's geometry makes the right grasp depend on its orientation; the two | |
| sorting policies start where the pick policy ends (part held in front of the camera). | |
| **Orchestration.** The team's integrated demo, [`30_full_demo_app.py`](https://github.com/iker-azur/hackathon-robotekin/tree/main/demos/06_imitation_learning), runs the loop on the real follower: | |
| an order (voice in Spanish, English or Basque, text, or Telegram) is read by the intent model | |
| [`IkerAzurmendi/bin-intent-gliner2`](https://huggingface.co/IkerAzurmendi/bin-intent-gliner2); the arm waits until a | |
| part lies still in the tray; the pick policy lifts it to the camera; the classifier averages 1.5 s of frames; the order | |
| maps PIN / NO PIN to the green or the white box, and that box's policy carries the part there (`policy_routes.json`, | |
| run through `run_policy.py`). In the usual order GOOD parts go to the green box and BAD parts to the white box. During | |
| data collection the operator looked at the part and chose which sorting demonstration to record. The start/end-pose | |
| contract between stages is in [`act-up-down-bad/HANDOFF.md`](https://huggingface.co/MikelRubi/act-up-down-bad/blob/main/HANDOFF.md). | |
| > **Naming note: `good` / `bad` describe the part, not the quality of a demonstration.** | |
| > GOOD = part **with** a pin → green box. BAD = part **without** a pin (defective) → white box. | |
| > `up-down-bad` is a policy trained on *successful* demonstrations of carrying a defective part to the white box. No | |
| > failed or negative demonstrations were intentionally used for training. Other informal names kept for compatibility: | |
| > `so101-pick-cube_*` datasets contain the bracket, not a cube, and `up-down_*` datasets are | |
| > the GOOD-box task (except the 23 October warm-up sessions). | |
| ## Hardware | |
| | Component | What is known | Source | | |
| |---|---|---| | |
| | Arm | SO-101 follower, 6 joints: `shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper` (degrees; gripper 0–100) | dataset `robot_type: so_follower`, recording scripts | | |
| | Teleoperation | SO-101 leader arm driving the follower (leader/follower); `action` = leader position | recording scripts | | |
| | Cameras | Two USB RGB cameras: `front` (fixed overhead camera looking down at the tray; the part is lifted towards it for inspection) and `wrist` (on the gripper). 640×480 at 30 fps; the first sessions of 23 October used 1280×720 at 10 fps | dataset `meta/info.json` | | |
| | Work cell | Orange tray with the parts, inspection position in front of the camera, green (big) box and white (small) box | recordings, `tray_detect.py`, `demo_orders.py` | | |
| | Recording PC | Linux, arms on USB serial ports | team notes | | |
| Not documented yet: camera models and exact mounting positions, workspace dimensions and lighting, servo models, | |
| the part's dimensions, and the calibration procedure beyond LeRobot's standard `lerobot-calibrate`. | |
| ## Repository layout | |
| | Folder | Original repository (`IkerAzurmendi/…`) | Saved steps | | |
| |---|---|---| | |
| | `pick_v0_mixed/` | `act_so101_all_30fps` | 30K | | |
| | `pick_v1/` | `act-pick-*` | 6K · 12K · 18K · 24K · 30K | | |
| | `pick_v2/` | `act-pick-v2-*` | 6K · 12K · 18K · 24K · 30K | | |
| | `pick_v3_super/` | `act-pick-v3-6000`, `act-pick-v3-super-*` | 6K · 12K · 18K · 24K · 30K | | |
| | `pick_v3_strict/` | `act-pick-v3-strict-*` | 2.4K · 4.8K · 7.2K · 9.6K · 12K | | |
| | `pick_v3_1-2-pieces/` | `act-pick-v3-1-2-pieces-*` | 6K · 12K · 18K · 24K · 30K | | |
| | `pick_v3_chunk60/` | `act-pick-v3-chunk60-*` | 2.4K · 4.8K · 7.2K · 9.6K · 12K | | |
| | `pick_v3_1piece/` | `act-pick-v3-1piece-*` | 2.4K · 4.8K · 7.2K · 9.6K · 12K | | |
| | `pick_v3_recent/` | `act-pick-v3-recent-*` | 2.4K · 4.8K · 7.2K · 9.6K · 12K | | |
| Each checkpoint holds `pretrained_model/` (weights and normalisers, loaded with `ACTPolicy.from_pretrained`), | |
| `dataset_features.json` (names, order and shapes of inputs and outputs) and `model_card_original.md` (the author's | |
| original card with its metrics). The originals published only the policy, without `training_state/`, so training | |
| cannot be resumed from these checkpoints. | |
| ## Training data | |
| Real recordings `MikelRubi/so101-pick-cube_*` (66 sessions on 23 and 24 October, leader/follower teleoperation). From | |
| 24 October the task is *"Pick a bracket from the crate and hold it up to the camera"* (41 sessions, 350 episodes, 30 | |
| fps, 640×480). The 25 sessions of 23 October were made with the first recorder (stored task *"Pick up the cube and | |
| place it in the bowl"*, 10 fps at 1280×720 or 30 fps at 640×480); most of their episodes end with the part dropped in | |
| the bin. `pick_v0_mixed` used them together with the 23 October `up-down_*` return trips; later versions keep a | |
| filtered subset (12 episodes at 30 fps in v2 and v3; `v3_super` also the 10 fps sessions, up-sampled to 30 fps and | |
| centre-cropped to 4:3). | |
| Each version applies a different filter (table below). All v3 runs are evaluated on the **same 16 held-out episodes**: | |
| 12 from `so101-pick-cube_20261024_181939` (one part, varied positions) and 4 from `so101-pick-cube_20261024_144537` | |
| (two parts). | |
| **Inputs:** `front` and `wrist` cameras (RGB 480×640) and the 6 joint positions. **Output:** chunks of 90 absolute joint | |
| targets at 30 fps (60 for `v3_chunk60`), replanning every 30. | |
| ## Versions | |
| Common base: LeRobot ACT, batch 8, chunk 90, replanning every 30 actions, learning rate 1e-5, image augmentation, seed | |
| 1000. | |
| | Version | Training episodes | What changes | | |
| |---|---|---| | |
| | v0_mixed | 145 | First experiment: pick and return trips mixed in one policy | | |
| | v1 | 165 | First pick-only run | | |
| | v2 | 278 | Only 30 fps recordings with 1–2 parts, 5–15 s episodes, quality filters, bad takes removed | | |
| | v3_super | **398** | Everything that passes the quality filter: 30 fps and 10 fps (up-sampled), any number of parts | | |
| | v3_strict | 205 | Stricter filter: 6–12 s, at most 5° start gap and 1 s idle | | |
| | v3_1-2-pieces | 331 | The v2 recipe plus the two new evening sessions of 24 October | | |
| | v3_chunk60 | 331 | Same data as v3_1-2-pieces with a **chunk of 60** (2 s plan instead of 3 s) | | |
| | v3_1piece | 292 | Only episodes with exactly one part, loose filters | | |
| | v3_recent | 100 | Only the recent sessions, with parts in corners and along the edges | | |
| ## Results (offline validation) | |
| Mean error in degrees over the predicted chunk (↓ better), and in brackets the improvement over staying still. The | |
| stay-still baseline is 20.71 on the 16 v3 episodes, 21.10 on the 15 v2 episodes and 16.23 with chunk 60. | |
| | Version | Best checkpoint | Error at best | All checkpoints | | |
| |---|---|---|---| | |
| | v0_mixed | 30K | 21.80 (1 %) | pick-cube only; up-down 19.41 (32 %) | | |
| | v1 | 6K | 16.87 (19 %) | — | | |
| | v2 | 30K | 15.24 (28 %) | 15.36 · 15.57 · 15.57 · 15.34 · **15.24** | | |
| | **v3_super** | **6K** | **15.16 (27 %)** | **15.16** · 16.08 · 16.04 · 16.36 · 16.27 | | |
| | v3_strict | 2.4K | 15.14 (27 %) | **15.14** · 15.44 · 15.52 · 15.53 · 15.96 | | |
| | v3_1-2-pieces | 18K | 15.35 (26 %) | 15.86 · 15.89 · **15.35** · 15.73 · 16.38 | | |
| | v3_chunk60 | 4.8K | 11.94 (26 %) * | 13.13 · **11.94** · 12.16 · 12.43 · 13.04 | | |
| | v3_1piece | 9.6K | 15.47 (25 %) | 16.00 · 15.82 · 15.70 · **15.47** · 15.86 | | |
| | v3_recent | 12K | 16.87 (19 %) | 17.19 · 17.10 · 17.00 · 17.10 · **16.87** | | |
| \* With chunk 60 the raw error is not comparable, because it looks a shorter time ahead; compare its percentage. | |
| **Reading:** | |
| - **Data:** going from 165 to ≥ 278 filtered episodes (v1 → v2/v3) lowers the error from 16.9 to about 15.2. Beyond | |
| that, the different data mixes are tied within noise. | |
| - **Training length:** the best point comes early (2.4K–6K steps); training longer makes it worse. | |
| - **Joints:** in every model, `wrist_roll` does not beat staying still. | |
| - **Limits of these metrics:** they are offline errors against the demonstrations and do not say whether the arm | |
| grasps the part. The real test is the robot. | |
| ## Usage | |
| ```python | |
| from lerobot.policies.act.modeling_act import ACTPolicy | |
| policy = ACTPolicy.from_pretrained("<folder>/pick_v3_super/checkpoints/006000/pretrained_model") | |
| ``` | |
| Observations with the same names and units as the recordings: `front` and `wrist` cameras at 640×480 and 30 fps, and | |
| the joint state. Call `policy.reset()` before each part. | |
| ## Limitations | |
| - Offline evaluation only; no pick model has been run on the physical robot yet. | |
| - The offline error does not measure grasp success. | |
| ## Credits and acknowledgements | |
| Models trained by Iker Azurmendi (originally published as `IkerAzurmendi/act-pick-*`). Created by **Iker Azurmendi, Mikel Zulueta, Ander Autillo and Diego Suarez** during the **Robotekin LeRobot Hackathon 2026** (23–25 October 2026, Vitoria-Gasteiz). | |
| Thanks to **Bravo Robots** and to **Robotekin** for the hackathon, to Hugging Face for LeRobot, and to everyone who | |
| took part in it. | |