File size: 15,053 Bytes
89a898c
 
 
 
 
 
 
 
 
 
 
 
bb06bbd
89a898c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
bb06bbd
 
 
89a898c
 
 
bb06bbd
 
89a898c
bb06bbd
 
89a898c
 
 
 
 
 
 
 
 
 
 
 
bb06bbd
 
 
 
 
 
 
 
89a898c
 
 
 
 
 
 
 
 
 
 
 
 
 
bb06bbd
 
89a898c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1311343
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
---

library_name: lerobot
license: apache-2.0
pipeline_tag: robotics
tags:
- lerobot
- act
- robotics
- so101
- robotekin-hackathon-2026
- pick
datasets:
- MikelRubi/robotekin-so101-pick
- MikelRubi/so101-pick-cube_20261023_191626
- MikelRubi/so101-pick-cube_20261023_193855
- MikelRubi/so101-pick-cube_20261023_195654
- MikelRubi/so101-pick-cube_20261023_200003
- MikelRubi/so101-pick-cube_20261023_201205
- MikelRubi/so101-pick-cube_20261023_203046
- MikelRubi/so101-pick-cube_20261023_203822
- MikelRubi/so101-pick-cube_20261023_205608
- MikelRubi/so101-pick-cube_20261023_205924
- MikelRubi/so101-pick-cube_20261023_211050
- MikelRubi/so101-pick-cube_20261023_211739
- MikelRubi/so101-pick-cube_20261023_212258
- MikelRubi/so101-pick-cube_20261023_213054
- MikelRubi/so101-pick-cube_20261023_221518
- MikelRubi/so101-pick-cube_20261023_221852
- MikelRubi/so101-pick-cube_20261023_222526
- MikelRubi/so101-pick-cube_20261023_223103
- MikelRubi/so101-pick-cube_20261023_223515
- MikelRubi/so101-pick-cube_20261023_223658
- MikelRubi/so101-pick-cube_20261023_224034
- MikelRubi/so101-pick-cube_20261023_224547
- MikelRubi/so101-pick-cube_20261023_224858
- MikelRubi/so101-pick-cube_20261023_225412
- MikelRubi/so101-pick-cube_20261023_225556
- MikelRubi/so101-pick-cube_20261023_225932
- MikelRubi/so101-pick-cube_20261024_084532
- MikelRubi/so101-pick-cube_20261024_085325
- MikelRubi/so101-pick-cube_20261024_085554
- MikelRubi/so101-pick-cube_20261024_085829
- MikelRubi/so101-pick-cube_20261024_090220
- MikelRubi/so101-pick-cube_20261024_090900
- MikelRubi/so101-pick-cube_20261024_091415
- MikelRubi/so101-pick-cube_20261024_092742
- MikelRubi/so101-pick-cube_20261024_094405
- MikelRubi/so101-pick-cube_20261024_095901
- MikelRubi/so101-pick-cube_20261024_102241
- MikelRubi/so101-pick-cube_20261024_103318
- MikelRubi/so101-pick-cube_20261024_103634
- MikelRubi/so101-pick-cube_20261024_104618
- MikelRubi/so101-pick-cube_20261024_105854
- MikelRubi/so101-pick-cube_20261024_110840
- MikelRubi/so101-pick-cube_20261024_111303
- MikelRubi/so101-pick-cube_20261024_112511
- MikelRubi/so101-pick-cube_20261024_113237
- MikelRubi/so101-pick-cube_20261024_114214
- MikelRubi/so101-pick-cube_20261024_115532
- MikelRubi/so101-pick-cube_20261024_121209
- MikelRubi/so101-pick-cube_20261024_122039
- MikelRubi/so101-pick-cube_20261024_122345
- MikelRubi/so101-pick-cube_20261024_124413
- MikelRubi/so101-pick-cube_20261024_125836
- MikelRubi/so101-pick-cube_20261024_131047
- MikelRubi/so101-pick-cube_20261024_133639
- MikelRubi/so101-pick-cube_20261024_140131
- MikelRubi/so101-pick-cube_20261024_141427
- MikelRubi/so101-pick-cube_20261024_142915
- MikelRubi/so101-pick-cube_20261024_144537
- MikelRubi/so101-pick-cube_20261024_145140
- MikelRubi/so101-pick-cube_20261024_145921
- MikelRubi/so101-pick-cube_20261024_165825
- MikelRubi/so101-pick-cube_20261024_171106
- MikelRubi/so101-pick-cube_20261024_171946
- MikelRubi/so101-pick-cube_20261024_173031
- MikelRubi/so101-pick-cube_20261024_181939
- MikelRubi/so101-pick-cube_20261024_222410
- MikelRubi/so101-pick-cube_20261024_223612
- MikelRubi/up-down_20261023_211221
- MikelRubi/up-down_20261023_211821
- MikelRubi/up-down_20261023_212336
- MikelRubi/up-down_20261023_213122
- MikelRubi/up-down_20261023_222052
- MikelRubi/up-down_20261023_222555
- MikelRubi/up-down_20261023_223150
- MikelRubi/up-down_20261023_223615
- MikelRubi/up-down_20261023_223804
- MikelRubi/up-down_20261023_224139
- MikelRubi/up-down_20261023_224633
- MikelRubi/up-down_20261023_224932
- MikelRubi/up-down_20261023_225507
- MikelRubi/up-down_20261023_225625
- MikelRubi/up-down_20261023_230112
---


# Pick — ACT policy for picking the part (SO-101)

**Picks one bracket from the crate, grasping it according to its orientation, lifts it and holds it up to the

inspection camera.** Stage 1 of the pipeline below: it leaves the part ready for
[`MikelRubi/pin-classifier`](https://huggingface.co/MikelRubi/pin-classifier) and for one of the sorting policies.

## Recommended checkpoint

| | |
|---|---|
| **Checkpoint** | `pick_v3_super/checkpoints/006000/` |
| **Why** | Error 15.16 on the held-out episodes, 27 % below "stay still" (20.71); trained on the most data (398 episodes) |
| **Caveat** | Tied with `pick_v3_strict/002400` (15.14), within noise. Every run gets worse after about one pass over its data |
| **Status** | No pick model has been tested on the robot yet |

If there is little robot time, try **v3_super 6K → v3_strict 2.4K → v3_1-2-pieces 18K** in that order.



## The system



Built at the **Robotekin LeRobot Hackathon 2026** (23–25 October 2026, Robotory, Vitoria-Gasteiz). One SO-101 arm runs

a small quality-control cell for a metal bracket that may or may not carry a pin. **Project page:**

[`MikelRubi/robotekin-so101-pin-sorting`](https://huggingface.co/spaces/MikelRubi/robotekin-so101-pin-sorting).





```mermaid

flowchart LR

    order["Order (voice / text)<br/>intent model → sorting rule"] --> tray["Part lies in the tray"]

    tray --> pick["1 · act-pick<br/>grasp the part according to its orientation,<br/>lift it, hold it up to the camera"]

    pick --> cls{"2 · pin-classifier<br/>front + wrist images"}

    cls -- "rule: usually pin (GOOD)" --> good["3a · act-up-down-good<br/>carry it to the green box, release"]

    cls -- "rule: usually no pin (BAD)" --> bad["3b · act-up-down-bad<br/>carry it to the white box, release"]

```



| Stage | Repository | Model |

|---|---|---|

| 1. Picking | [`MikelRubi/act-pick`](https://huggingface.co/MikelRubi/act-pick) | ACT policy |

| 2. Inspection | [`MikelRubi/pin-classifier`](https://huggingface.co/MikelRubi/pin-classifier) | EfficientNet-B0 image classifier |

| 3a. Sorting, GOOD | [`MikelRubi/act-up-down-good`](https://huggingface.co/MikelRubi/act-up-down-good) | ACT policy |

| 3b. Sorting, BAD | [`MikelRubi/act-up-down-bad`](https://huggingface.co/MikelRubi/act-up-down-bad) | ACT policy |



Picking is a separate policy because the bracket's geometry makes the right grasp depend on its orientation; the two

sorting policies start where the pick policy ends (part held in front of the camera).



**Orchestration.** The team's integrated demo, [`30_full_demo_app.py`](https://github.com/iker-azur/hackathon-robotekin/tree/main/demos/06_imitation_learning), runs the loop on the real follower:

an order (voice in Spanish, English or Basque, text, or Telegram) is read by the intent model

[`IkerAzurmendi/bin-intent-gliner2`](https://huggingface.co/IkerAzurmendi/bin-intent-gliner2); the arm waits until a

part lies still in the tray; the pick policy lifts it to the camera; the classifier averages 1.5 s of frames; the order

maps PIN / NO PIN to the green or the white box, and that box's policy carries the part there (`policy_routes.json`,

run through `run_policy.py`). In the usual order GOOD parts go to the green box and BAD parts to the white box. During

data collection the operator looked at the part and chose which sorting demonstration to record. The start/end-pose

contract between stages is in [`act-up-down-bad/HANDOFF.md`](https://huggingface.co/MikelRubi/act-up-down-bad/blob/main/HANDOFF.md).



> **Naming note: `good` / `bad` describe the part, not the quality of a demonstration.**

> GOOD = part **with** a pin → green box. BAD = part **without** a pin (defective) → white box.

> `up-down-bad` is a policy trained on *successful* demonstrations of carrying a defective part to the white box. No

> failed or negative demonstrations were intentionally used for training. Other informal names kept for compatibility:

> `so101-pick-cube_*` datasets contain the bracket, not a cube, and `up-down_*` datasets are

> the GOOD-box task (except the 23 October warm-up sessions).



## Hardware



| Component | What is known | Source |

|---|---|---|

| Arm | SO-101 follower, 6 joints: `shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper` (degrees; gripper 0–100) | dataset `robot_type: so_follower`, recording scripts |

| Teleoperation | SO-101 leader arm driving the follower (leader/follower); `action` = leader position | recording scripts |

| Cameras | Two USB RGB cameras: `front` (fixed overhead camera looking down at the tray; the part is lifted towards it for inspection) and `wrist` (on the gripper). 640×480 at 30 fps; the first sessions of 23 October used 1280×720 at 10 fps | dataset `meta/info.json` |

| Work cell | Orange tray with the parts, inspection position in front of the camera, green (big) box and white (small) box | recordings, `tray_detect.py`, `demo_orders.py` |

| Recording PC | Linux, arms on USB serial ports | team notes |



Not documented yet: camera models and exact mounting positions, workspace dimensions and lighting, servo models,

the part's dimensions, and the calibration procedure beyond LeRobot's standard `lerobot-calibrate`.



## Repository layout



| Folder | Original repository (`IkerAzurmendi/…`) | Saved steps |

|---|---|---|

| `pick_v0_mixed/` | `act_so101_all_30fps` | 30K |

| `pick_v1/` | `act-pick-*` | 6K · 12K · 18K · 24K · 30K |

| `pick_v2/` | `act-pick-v2-*` | 6K · 12K · 18K · 24K · 30K |

| `pick_v3_super/` | `act-pick-v3-6000`, `act-pick-v3-super-*` | 6K · 12K · 18K · 24K · 30K |

| `pick_v3_strict/` | `act-pick-v3-strict-*` | 2.4K · 4.8K · 7.2K · 9.6K · 12K |

| `pick_v3_1-2-pieces/` | `act-pick-v3-1-2-pieces-*` | 6K · 12K · 18K · 24K · 30K |

| `pick_v3_chunk60/` | `act-pick-v3-chunk60-*` | 2.4K · 4.8K · 7.2K · 9.6K · 12K |

| `pick_v3_1piece/` | `act-pick-v3-1piece-*` | 2.4K · 4.8K · 7.2K · 9.6K · 12K |

| `pick_v3_recent/` | `act-pick-v3-recent-*` | 2.4K · 4.8K · 7.2K · 9.6K · 12K |



Each checkpoint holds `pretrained_model/` (weights and normalisers, loaded with `ACTPolicy.from_pretrained`),

`dataset_features.json` (names, order and shapes of inputs and outputs) and `model_card_original.md` (the author's

original card with its metrics). The originals published only the policy, without `training_state/`, so training

cannot be resumed from these checkpoints.



## Training data



Real recordings `MikelRubi/so101-pick-cube_*` (66 sessions on 23 and 24 October, leader/follower teleoperation). From

24 October the task is *"Pick a bracket from the crate and hold it up to the camera"* (41 sessions, 350 episodes, 30

fps, 640×480). The 25 sessions of 23 October were made with the first recorder (stored task *"Pick up the cube and

place it in the bowl"*, 10 fps at 1280×720 or 30 fps at 640×480); most of their episodes end with the part dropped in

the bin. `pick_v0_mixed` used them together with the 23 October `up-down_*` return trips; later versions keep a

filtered subset (12 episodes at 30 fps in v2 and v3; `v3_super` also the 10 fps sessions, up-sampled to 30 fps and

centre-cropped to 4:3).



Each version applies a different filter (table below). All v3 runs are evaluated on the **same 16 held-out episodes**:

12 from `so101-pick-cube_20261024_181939` (one part, varied positions) and 4 from `so101-pick-cube_20261024_144537`

(two parts).



**Inputs:** `front` and `wrist` cameras (RGB 480×640) and the 6 joint positions. **Output:** chunks of 90 absolute joint
targets at 30 fps (60 for `v3_chunk60`), replanning every 30.

## Versions

Common base: LeRobot ACT, batch 8, chunk 90, replanning every 30 actions, learning rate 1e-5, image augmentation, seed
1000.

| Version | Training episodes | What changes |
|---|---|---|
| v0_mixed | 145 | First experiment: pick and return trips mixed in one policy |

| v1 | 165 | First pick-only run |

| v2 | 278 | Only 30 fps recordings with 1–2 parts, 5–15 s episodes, quality filters, bad takes removed |

| v3_super | **398** | Everything that passes the quality filter: 30 fps and 10 fps (up-sampled), any number of parts |
| v3_strict | 205 | Stricter filter: 6–12 s, at most 5° start gap and 1 s idle |

| v3_1-2-pieces | 331 | The v2 recipe plus the two new evening sessions of 24 October |
| v3_chunk60 | 331 | Same data as v3_1-2-pieces with a **chunk of 60** (2 s plan instead of 3 s) |
| v3_1piece | 292 | Only episodes with exactly one part, loose filters |

| v3_recent | 100 | Only the recent sessions, with parts in corners and along the edges |

## Results (offline validation)

Mean error in degrees over the predicted chunk (↓ better), and in brackets the improvement over staying still. The
stay-still baseline is 20.71 on the 16 v3 episodes, 21.10 on the 15 v2 episodes and 16.23 with chunk 60.

| Version | Best checkpoint | Error at best | All checkpoints |
|---|---|---|---|
| v0_mixed | 30K | 21.80 (1 %) | pick-cube only; up-down 19.41 (32 %) |

| v1 | 6K | 16.87 (19 %) | — |

| v2 | 30K | 15.24 (28 %) | 15.36 · 15.57 · 15.57 · 15.34 · **15.24** |

| **v3_super** | **6K** | **15.16 (27 %)** | **15.16** · 16.08 · 16.04 · 16.36 · 16.27 |

| v3_strict | 2.4K | 15.14 (27 %) | **15.14** · 15.44 · 15.52 · 15.53 · 15.96 |
| v3_1-2-pieces | 18K | 15.35 (26 %) | 15.86 · 15.89 · **15.35** · 15.73 · 16.38 |

| v3_chunk60 | 4.8K | 11.94 (26 %) * | 13.13 · **11.94** · 12.16 · 12.43 · 13.04 |
| v3_1piece | 9.6K | 15.47 (25 %) | 16.00 · 15.82 · 15.70 · **15.47** · 15.86 |

| v3_recent | 12K | 16.87 (19 %) | 17.19 · 17.10 · 17.00 · 17.10 · **16.87** |

\* With chunk 60 the raw error is not comparable, because it looks a shorter time ahead; compare its percentage.

**Reading:**
- **Data:** going from 165 to ≥ 278 filtered episodes (v1 → v2/v3) lowers the error from 16.9 to about 15.2. Beyond
  that, the different data mixes are tied within noise.
- **Training length:** the best point comes early (2.4K–6K steps); training longer makes it worse.
- **Joints:** in every model, `wrist_roll` does not beat staying still.
- **Limits of these metrics:** they are offline errors against the demonstrations and do not say whether the arm
  grasps the part. The real test is the robot.

## Usage

```python

from lerobot.policies.act.modeling_act import ACTPolicy

policy = ACTPolicy.from_pretrained("<folder>/pick_v3_super/checkpoints/006000/pretrained_model")

```

Observations with the same names and units as the recordings: `front` and `wrist` cameras at 640×480 and 30 fps, and
the joint state. Call `policy.reset()` before each part.

## Limitations

- Offline evaluation only; no pick model has been run on the physical robot yet.
- The offline error does not measure grasp success.

## Credits and acknowledgements

Models trained by Iker Azurmendi (originally published as `IkerAzurmendi/act-pick-*`). Created by **Iker Azurmendi, Mikel Zulueta, Ander Autillo and Diego Suarez** during the **Robotekin LeRobot Hackathon 2026** (23–25 October 2026, Vitoria-Gasteiz).
Thanks to **Bravo Robots** and to **Robotekin** for the hackathon, to Hugging Face for LeRobot, and to everyone who
took part in it.