--- library_name: lerobot license: apache-2.0 pipeline_tag: robotics tags: - lerobot - act - robotics - so101 - robotekin-hackathon-2026 - pick datasets: - MikelRubi/robotekin-so101-pick - MikelRubi/so101-pick-cube_20261023_191626 - MikelRubi/so101-pick-cube_20261023_193855 - MikelRubi/so101-pick-cube_20261023_195654 - MikelRubi/so101-pick-cube_20261023_200003 - MikelRubi/so101-pick-cube_20261023_201205 - MikelRubi/so101-pick-cube_20261023_203046 - MikelRubi/so101-pick-cube_20261023_203822 - MikelRubi/so101-pick-cube_20261023_205608 - MikelRubi/so101-pick-cube_20261023_205924 - MikelRubi/so101-pick-cube_20261023_211050 - MikelRubi/so101-pick-cube_20261023_211739 - MikelRubi/so101-pick-cube_20261023_212258 - MikelRubi/so101-pick-cube_20261023_213054 - MikelRubi/so101-pick-cube_20261023_221518 - MikelRubi/so101-pick-cube_20261023_221852 - MikelRubi/so101-pick-cube_20261023_222526 - MikelRubi/so101-pick-cube_20261023_223103 - MikelRubi/so101-pick-cube_20261023_223515 - MikelRubi/so101-pick-cube_20261023_223658 - MikelRubi/so101-pick-cube_20261023_224034 - MikelRubi/so101-pick-cube_20261023_224547 - MikelRubi/so101-pick-cube_20261023_224858 - MikelRubi/so101-pick-cube_20261023_225412 - MikelRubi/so101-pick-cube_20261023_225556 - MikelRubi/so101-pick-cube_20261023_225932 - MikelRubi/so101-pick-cube_20261024_084532 - MikelRubi/so101-pick-cube_20261024_085325 - MikelRubi/so101-pick-cube_20261024_085554 - MikelRubi/so101-pick-cube_20261024_085829 - MikelRubi/so101-pick-cube_20261024_090220 - MikelRubi/so101-pick-cube_20261024_090900 - MikelRubi/so101-pick-cube_20261024_091415 - MikelRubi/so101-pick-cube_20261024_092742 - MikelRubi/so101-pick-cube_20261024_094405 - MikelRubi/so101-pick-cube_20261024_095901 - MikelRubi/so101-pick-cube_20261024_102241 - MikelRubi/so101-pick-cube_20261024_103318 - MikelRubi/so101-pick-cube_20261024_103634 - MikelRubi/so101-pick-cube_20261024_104618 - MikelRubi/so101-pick-cube_20261024_105854 - MikelRubi/so101-pick-cube_20261024_110840 - MikelRubi/so101-pick-cube_20261024_111303 - MikelRubi/so101-pick-cube_20261024_112511 - MikelRubi/so101-pick-cube_20261024_113237 - MikelRubi/so101-pick-cube_20261024_114214 - MikelRubi/so101-pick-cube_20261024_115532 - MikelRubi/so101-pick-cube_20261024_121209 - MikelRubi/so101-pick-cube_20261024_122039 - MikelRubi/so101-pick-cube_20261024_122345 - MikelRubi/so101-pick-cube_20261024_124413 - MikelRubi/so101-pick-cube_20261024_125836 - MikelRubi/so101-pick-cube_20261024_131047 - MikelRubi/so101-pick-cube_20261024_133639 - MikelRubi/so101-pick-cube_20261024_140131 - MikelRubi/so101-pick-cube_20261024_141427 - MikelRubi/so101-pick-cube_20261024_142915 - MikelRubi/so101-pick-cube_20261024_144537 - MikelRubi/so101-pick-cube_20261024_145140 - MikelRubi/so101-pick-cube_20261024_145921 - MikelRubi/so101-pick-cube_20261024_165825 - MikelRubi/so101-pick-cube_20261024_171106 - MikelRubi/so101-pick-cube_20261024_171946 - MikelRubi/so101-pick-cube_20261024_173031 - MikelRubi/so101-pick-cube_20261024_181939 - MikelRubi/so101-pick-cube_20261024_222410 - MikelRubi/so101-pick-cube_20261024_223612 - MikelRubi/up-down_20261023_211221 - MikelRubi/up-down_20261023_211821 - MikelRubi/up-down_20261023_212336 - MikelRubi/up-down_20261023_213122 - MikelRubi/up-down_20261023_222052 - MikelRubi/up-down_20261023_222555 - MikelRubi/up-down_20261023_223150 - MikelRubi/up-down_20261023_223615 - MikelRubi/up-down_20261023_223804 - MikelRubi/up-down_20261023_224139 - MikelRubi/up-down_20261023_224633 - MikelRubi/up-down_20261023_224932 - MikelRubi/up-down_20261023_225507 - MikelRubi/up-down_20261023_225625 - MikelRubi/up-down_20261023_230112 --- # Pick — ACT policy for picking the part (SO-101) **Picks one bracket from the crate, grasping it according to its orientation, lifts it and holds it up to the inspection camera.** Stage 1 of the pipeline below: it leaves the part ready for [`MikelRubi/pin-classifier`](https://huggingface.co/MikelRubi/pin-classifier) and for one of the sorting policies. ## Recommended checkpoint | | | |---|---| | **Checkpoint** | `pick_v3_super/checkpoints/006000/` | | **Why** | Error 15.16 on the held-out episodes, 27 % below "stay still" (20.71); trained on the most data (398 episodes) | | **Caveat** | Tied with `pick_v3_strict/002400` (15.14), within noise. Every run gets worse after about one pass over its data | | **Status** | No pick model has been tested on the robot yet | If there is little robot time, try **v3_super 6K → v3_strict 2.4K → v3_1-2-pieces 18K** in that order. ## The system Built at the **Robotekin LeRobot Hackathon 2026** (23–25 October 2026, Robotory, Vitoria-Gasteiz). One SO-101 arm runs a small quality-control cell for a metal bracket that may or may not carry a pin. **Project page:** [`MikelRubi/robotekin-so101-pin-sorting`](https://huggingface.co/spaces/MikelRubi/robotekin-so101-pin-sorting). ```mermaid flowchart LR order["Order (voice / text)
intent model → sorting rule"] --> tray["Part lies in the tray"] tray --> pick["1 · act-pick
grasp the part according to its orientation,
lift it, hold it up to the camera"] pick --> cls{"2 · pin-classifier
front + wrist images"} cls -- "rule: usually pin (GOOD)" --> good["3a · act-up-down-good
carry it to the green box, release"] cls -- "rule: usually no pin (BAD)" --> bad["3b · act-up-down-bad
carry it to the white box, release"] ``` | Stage | Repository | Model | |---|---|---| | 1. Picking | [`MikelRubi/act-pick`](https://huggingface.co/MikelRubi/act-pick) | ACT policy | | 2. Inspection | [`MikelRubi/pin-classifier`](https://huggingface.co/MikelRubi/pin-classifier) | EfficientNet-B0 image classifier | | 3a. Sorting, GOOD | [`MikelRubi/act-up-down-good`](https://huggingface.co/MikelRubi/act-up-down-good) | ACT policy | | 3b. Sorting, BAD | [`MikelRubi/act-up-down-bad`](https://huggingface.co/MikelRubi/act-up-down-bad) | ACT policy | Picking is a separate policy because the bracket's geometry makes the right grasp depend on its orientation; the two sorting policies start where the pick policy ends (part held in front of the camera). **Orchestration.** The team's integrated demo, [`30_full_demo_app.py`](https://github.com/iker-azur/hackathon-robotekin/tree/main/demos/06_imitation_learning), runs the loop on the real follower: an order (voice in Spanish, English or Basque, text, or Telegram) is read by the intent model [`IkerAzurmendi/bin-intent-gliner2`](https://huggingface.co/IkerAzurmendi/bin-intent-gliner2); the arm waits until a part lies still in the tray; the pick policy lifts it to the camera; the classifier averages 1.5 s of frames; the order maps PIN / NO PIN to the green or the white box, and that box's policy carries the part there (`policy_routes.json`, run through `run_policy.py`). In the usual order GOOD parts go to the green box and BAD parts to the white box. During data collection the operator looked at the part and chose which sorting demonstration to record. The start/end-pose contract between stages is in [`act-up-down-bad/HANDOFF.md`](https://huggingface.co/MikelRubi/act-up-down-bad/blob/main/HANDOFF.md). > **Naming note: `good` / `bad` describe the part, not the quality of a demonstration.** > GOOD = part **with** a pin → green box. BAD = part **without** a pin (defective) → white box. > `up-down-bad` is a policy trained on *successful* demonstrations of carrying a defective part to the white box. No > failed or negative demonstrations were intentionally used for training. Other informal names kept for compatibility: > `so101-pick-cube_*` datasets contain the bracket, not a cube, and `up-down_*` datasets are > the GOOD-box task (except the 23 October warm-up sessions). ## Hardware | Component | What is known | Source | |---|---|---| | Arm | SO-101 follower, 6 joints: `shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper` (degrees; gripper 0–100) | dataset `robot_type: so_follower`, recording scripts | | Teleoperation | SO-101 leader arm driving the follower (leader/follower); `action` = leader position | recording scripts | | Cameras | Two USB RGB cameras: `front` (fixed overhead camera looking down at the tray; the part is lifted towards it for inspection) and `wrist` (on the gripper). 640×480 at 30 fps; the first sessions of 23 October used 1280×720 at 10 fps | dataset `meta/info.json` | | Work cell | Orange tray with the parts, inspection position in front of the camera, green (big) box and white (small) box | recordings, `tray_detect.py`, `demo_orders.py` | | Recording PC | Linux, arms on USB serial ports | team notes | Not documented yet: camera models and exact mounting positions, workspace dimensions and lighting, servo models, the part's dimensions, and the calibration procedure beyond LeRobot's standard `lerobot-calibrate`. ## Repository layout | Folder | Original repository (`IkerAzurmendi/…`) | Saved steps | |---|---|---| | `pick_v0_mixed/` | `act_so101_all_30fps` | 30K | | `pick_v1/` | `act-pick-*` | 6K · 12K · 18K · 24K · 30K | | `pick_v2/` | `act-pick-v2-*` | 6K · 12K · 18K · 24K · 30K | | `pick_v3_super/` | `act-pick-v3-6000`, `act-pick-v3-super-*` | 6K · 12K · 18K · 24K · 30K | | `pick_v3_strict/` | `act-pick-v3-strict-*` | 2.4K · 4.8K · 7.2K · 9.6K · 12K | | `pick_v3_1-2-pieces/` | `act-pick-v3-1-2-pieces-*` | 6K · 12K · 18K · 24K · 30K | | `pick_v3_chunk60/` | `act-pick-v3-chunk60-*` | 2.4K · 4.8K · 7.2K · 9.6K · 12K | | `pick_v3_1piece/` | `act-pick-v3-1piece-*` | 2.4K · 4.8K · 7.2K · 9.6K · 12K | | `pick_v3_recent/` | `act-pick-v3-recent-*` | 2.4K · 4.8K · 7.2K · 9.6K · 12K | Each checkpoint holds `pretrained_model/` (weights and normalisers, loaded with `ACTPolicy.from_pretrained`), `dataset_features.json` (names, order and shapes of inputs and outputs) and `model_card_original.md` (the author's original card with its metrics). The originals published only the policy, without `training_state/`, so training cannot be resumed from these checkpoints. ## Training data Real recordings `MikelRubi/so101-pick-cube_*` (66 sessions on 23 and 24 October, leader/follower teleoperation). From 24 October the task is *"Pick a bracket from the crate and hold it up to the camera"* (41 sessions, 350 episodes, 30 fps, 640×480). The 25 sessions of 23 October were made with the first recorder (stored task *"Pick up the cube and place it in the bowl"*, 10 fps at 1280×720 or 30 fps at 640×480); most of their episodes end with the part dropped in the bin. `pick_v0_mixed` used them together with the 23 October `up-down_*` return trips; later versions keep a filtered subset (12 episodes at 30 fps in v2 and v3; `v3_super` also the 10 fps sessions, up-sampled to 30 fps and centre-cropped to 4:3). Each version applies a different filter (table below). All v3 runs are evaluated on the **same 16 held-out episodes**: 12 from `so101-pick-cube_20261024_181939` (one part, varied positions) and 4 from `so101-pick-cube_20261024_144537` (two parts). **Inputs:** `front` and `wrist` cameras (RGB 480×640) and the 6 joint positions. **Output:** chunks of 90 absolute joint targets at 30 fps (60 for `v3_chunk60`), replanning every 30. ## Versions Common base: LeRobot ACT, batch 8, chunk 90, replanning every 30 actions, learning rate 1e-5, image augmentation, seed 1000. | Version | Training episodes | What changes | |---|---|---| | v0_mixed | 145 | First experiment: pick and return trips mixed in one policy | | v1 | 165 | First pick-only run | | v2 | 278 | Only 30 fps recordings with 1–2 parts, 5–15 s episodes, quality filters, bad takes removed | | v3_super | **398** | Everything that passes the quality filter: 30 fps and 10 fps (up-sampled), any number of parts | | v3_strict | 205 | Stricter filter: 6–12 s, at most 5° start gap and 1 s idle | | v3_1-2-pieces | 331 | The v2 recipe plus the two new evening sessions of 24 October | | v3_chunk60 | 331 | Same data as v3_1-2-pieces with a **chunk of 60** (2 s plan instead of 3 s) | | v3_1piece | 292 | Only episodes with exactly one part, loose filters | | v3_recent | 100 | Only the recent sessions, with parts in corners and along the edges | ## Results (offline validation) Mean error in degrees over the predicted chunk (↓ better), and in brackets the improvement over staying still. The stay-still baseline is 20.71 on the 16 v3 episodes, 21.10 on the 15 v2 episodes and 16.23 with chunk 60. | Version | Best checkpoint | Error at best | All checkpoints | |---|---|---|---| | v0_mixed | 30K | 21.80 (1 %) | pick-cube only; up-down 19.41 (32 %) | | v1 | 6K | 16.87 (19 %) | — | | v2 | 30K | 15.24 (28 %) | 15.36 · 15.57 · 15.57 · 15.34 · **15.24** | | **v3_super** | **6K** | **15.16 (27 %)** | **15.16** · 16.08 · 16.04 · 16.36 · 16.27 | | v3_strict | 2.4K | 15.14 (27 %) | **15.14** · 15.44 · 15.52 · 15.53 · 15.96 | | v3_1-2-pieces | 18K | 15.35 (26 %) | 15.86 · 15.89 · **15.35** · 15.73 · 16.38 | | v3_chunk60 | 4.8K | 11.94 (26 %) * | 13.13 · **11.94** · 12.16 · 12.43 · 13.04 | | v3_1piece | 9.6K | 15.47 (25 %) | 16.00 · 15.82 · 15.70 · **15.47** · 15.86 | | v3_recent | 12K | 16.87 (19 %) | 17.19 · 17.10 · 17.00 · 17.10 · **16.87** | \* With chunk 60 the raw error is not comparable, because it looks a shorter time ahead; compare its percentage. **Reading:** - **Data:** going from 165 to ≥ 278 filtered episodes (v1 → v2/v3) lowers the error from 16.9 to about 15.2. Beyond that, the different data mixes are tied within noise. - **Training length:** the best point comes early (2.4K–6K steps); training longer makes it worse. - **Joints:** in every model, `wrist_roll` does not beat staying still. - **Limits of these metrics:** they are offline errors against the demonstrations and do not say whether the arm grasps the part. The real test is the robot. ## Usage ```python from lerobot.policies.act.modeling_act import ACTPolicy policy = ACTPolicy.from_pretrained("/pick_v3_super/checkpoints/006000/pretrained_model") ``` Observations with the same names and units as the recordings: `front` and `wrist` cameras at 640×480 and 30 fps, and the joint state. Call `policy.reset()` before each part. ## Limitations - Offline evaluation only; no pick model has been run on the physical robot yet. - The offline error does not measure grasp success. ## Credits and acknowledgements Models trained by Iker Azurmendi (originally published as `IkerAzurmendi/act-pick-*`). Created by **Iker Azurmendi, Mikel Zulueta, Ander Autillo and Diego Suarez** during the **Robotekin LeRobot Hackathon 2026** (23–25 October 2026, Vitoria-Gasteiz). Thanks to **Bravo Robots** and to **Robotekin** for the hackathon, to Hugging Face for LeRobot, and to everyone who took part in it.