File size: 10,915 Bytes
969aa15
 
 
 
 
 
 
 
 
 
 
b6b21a5
969aa15
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
b6b21a5
 
 
969aa15
 
 
b6b21a5
 
969aa15
b6b21a5
 
969aa15
 
 
 
 
 
 
 
 
 
 
 
b6b21a5
 
 
 
 
 
 
 
969aa15
 
 
 
 
 
 
 
 
 
 
 
 
 
b6b21a5
 
969aa15
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8811ccf
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
---

library_name: pytorch
license: apache-2.0
pipeline_tag: image-classification
tags:
- image-classification
- robotics
- so101
- robotekin-hackathon-2026
- pin-inspection
datasets:
- MikelRubi/robotekin-so101-pin-images
- MikelRubi/so101-pick-cube_20261023_193855
- MikelRubi/so101-pick-cube_20261023_195654
- MikelRubi/so101-pick-cube_20261023_200003
- MikelRubi/so101-pick-cube_20261023_201205
- MikelRubi/so101-pick-cube_20261023_203046
- MikelRubi/so101-pick-cube_20261023_203822
- MikelRubi/so101-pick-cube_20261023_205608
- MikelRubi/so101-pick-cube_20261023_205924
- MikelRubi/so101-pick-cube_20261023_211739
- MikelRubi/so101-pick-cube_20261023_212258
- MikelRubi/so101-pick-cube_20261023_213054
- MikelRubi/so101-pick-cube_20261023_221852
- MikelRubi/so101-pick-cube_20261023_222526
- MikelRubi/so101-pick-cube_20261023_223103
- MikelRubi/so101-pick-cube_20261023_223658
- MikelRubi/so101-pick-cube_20261023_224034
- MikelRubi/so101-pick-cube_20261023_224547
- MikelRubi/so101-pick-cube_20261023_224858
- MikelRubi/so101-pick-cube_20261023_225412
- MikelRubi/so101-pick-cube_20261023_225556
- MikelRubi/so101-pick-cube_20261023_225932
- MikelRubi/so101-pick-cube_20261024_084532
- MikelRubi/so101-pick-cube_20261024_085325
- MikelRubi/so101-pick-cube_20261024_085554
- MikelRubi/so101-pick-cube_20261024_085829
- MikelRubi/so101-pick-cube_20261024_090220
- MikelRubi/so101-pick-cube_20261024_090900
- MikelRubi/so101-pick-cube_20261024_091415
- MikelRubi/so101-pick-cube_20261024_092742
- MikelRubi/so101-pick-cube_20261024_094405
- MikelRubi/so101-pick-cube_20261024_095901
- MikelRubi/so101-pick-cube_20261024_102241
- MikelRubi/so101-pick-cube_20261024_103318
- MikelRubi/so101-pick-cube_20261024_103634
- MikelRubi/so101-pick-cube_20261024_104618
- MikelRubi/so101-pick-cube_20261024_105854
- MikelRubi/so101-pick-cube_20261024_110840
- MikelRubi/so101-pick-cube_20261024_111303
- MikelRubi/so101-pick-cube_20261024_112511
- MikelRubi/so101-pick-cube_20261024_113237
- MikelRubi/so101-pick-cube_20261024_114214
- MikelRubi/so101-pick-cube_20261024_115532
- MikelRubi/so101-pick-cube_20261024_223612
---


# Pin classifier — part with pin / without pin (SO-101)

**Looks at the part held by the arm, through the front and wrist cameras, and decides whether it has a pin (`pin`,

GOOD → green box) or not (`nopin`, BAD → white box).** Stage 2 of the pipeline below: its verdict selects the sorting
policy.

## Model

| | |
|---|---|
| **File** | `model.pt` |
| **Architecture** | ImageNet-pretrained EfficientNet-B0 (4.0 M parameters), **shared by both cameras**. For each camera the feature map is mean- and max-pooled; the two cameras are concatenated (front, then wrist; 5120 values) and one linear layer gives the 2 classes |
| **Input** | Two RGB images, `front` and `wrist`, centre-cropped to 4:3 (never stretched) and resized to 448×336. ImageNet normalisation is inside the model |
| **Output** | `nopin` / `pin` and the probability of `pin` |
| **Accuracy** | **≈ 85 %** on unseen recording sessions (cross-validation grouped by session). Majority-class baseline: 52 % |
| **Caveat** | Measured in the same room, with the same operator and lighting as the training data. Not tested on another day, under other lighting or with another batch of parts |

## The system

Built at the **Robotekin LeRobot Hackathon 2026** (23–25 October 2026, Robotory, Vitoria-Gasteiz). One SO-101 arm runs
a small quality-control cell for a metal bracket that may or may not carry a pin. **Project page:**
[`MikelRubi/robotekin-so101-pin-sorting`](https://huggingface.co/spaces/MikelRubi/robotekin-so101-pin-sorting).


```mermaid

flowchart LR

    order["Order (voice / text)<br/>intent model → sorting rule"] --> tray["Part lies in the tray"]

    tray --> pick["1 · act-pick<br/>grasp the part according to its orientation,<br/>lift it, hold it up to the camera"]

    pick --> cls{"2 · pin-classifier<br/>front + wrist images"}

    cls -- "rule: usually pin (GOOD)" --> good["3a · act-up-down-good<br/>carry it to the green box, release"]

    cls -- "rule: usually no pin (BAD)" --> bad["3b · act-up-down-bad<br/>carry it to the white box, release"]

```

| Stage | Repository | Model |
|---|---|---|
| 1. Picking | [`MikelRubi/act-pick`](https://huggingface.co/MikelRubi/act-pick) | ACT policy |
| 2. Inspection | [`MikelRubi/pin-classifier`](https://huggingface.co/MikelRubi/pin-classifier) | EfficientNet-B0 image classifier |
| 3a. Sorting, GOOD | [`MikelRubi/act-up-down-good`](https://huggingface.co/MikelRubi/act-up-down-good) | ACT policy |
| 3b. Sorting, BAD | [`MikelRubi/act-up-down-bad`](https://huggingface.co/MikelRubi/act-up-down-bad) | ACT policy |

Picking is a separate policy because the bracket's geometry makes the right grasp depend on its orientation; the two
sorting policies start where the pick policy ends (part held in front of the camera).

**Orchestration.** The team's integrated demo, [`30_full_demo_app.py`](https://github.com/iker-azur/hackathon-robotekin/tree/main/demos/06_imitation_learning), runs the loop on the real follower:
an order (voice in Spanish, English or Basque, text, or Telegram) is read by the intent model
[`IkerAzurmendi/bin-intent-gliner2`](https://huggingface.co/IkerAzurmendi/bin-intent-gliner2); the arm waits until a
part lies still in the tray; the pick policy lifts it to the camera; the classifier averages 1.5 s of frames; the order
maps PIN / NO PIN to the green or the white box, and that box's policy carries the part there (`policy_routes.json`,
run through `run_policy.py`). In the usual order GOOD parts go to the green box and BAD parts to the white box. During
data collection the operator looked at the part and chose which sorting demonstration to record. The start/end-pose
contract between stages is in [`act-up-down-bad/HANDOFF.md`](https://huggingface.co/MikelRubi/act-up-down-bad/blob/main/HANDOFF.md).

> **Naming note: `good` / `bad` describe the part, not the quality of a demonstration.**
> GOOD = part **with** a pin → green box. BAD = part **without** a pin (defective) → white box.
> `up-down-bad` is a policy trained on *successful* demonstrations of carrying a defective part to the white box. No
> failed or negative demonstrations were intentionally used for training. Other informal names kept for compatibility:
> `so101-pick-cube_*` datasets contain the bracket, not a cube, and `up-down_*` datasets are

> the GOOD-box task (except the 23 October warm-up sessions).



## Hardware



| Component | What is known | Source |

|---|---|---|

| Arm | SO-101 follower, 6 joints: `shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper` (degrees; gripper 0–100) | dataset `robot_type: so_follower`, recording scripts |

| Teleoperation | SO-101 leader arm driving the follower (leader/follower); `action` = leader position | recording scripts |

| Cameras | Two USB RGB cameras: `front` (fixed overhead camera looking down at the tray; the part is lifted towards it for inspection) and `wrist` (on the gripper). 640×480 at 30 fps; the first sessions of 23 October used 1280×720 at 10 fps | dataset `meta/info.json` |

| Work cell | Orange tray with the parts, inspection position in front of the camera, green (big) box and white (small) box | recordings, `tray_detect.py`, `demo_orders.py` |

| Recording PC | Linux, arms on USB serial ports | team notes |



Not documented yet: camera models and exact mounting positions, workspace dimensions and lighting, servo models,

the part's dimensions, and the calibration procedure beyond LeRobot's standard `lerobot-calibrate`.



## Usage



```python

from pin_classifier import load_model, predict      # pin_classifier.py is in this repository

model = load_model("model.pt")                      # needs torch, torchvision, numpy and Pillow

label, p_pin = predict(model, front_rgb, wrist_rgb) # HxWx3 uint8 RGB arrays, any size

```



`pin_classifier.py` rebuilds the architecture and loads the weights in strict mode. It was checked against the 110

labelled photo pairs of `so101-pick-cube_20261024_115532/pin_snapshots`: it classifies all 110 correctly, as expected,

because those photos are part of the training data. With the cameras swapped the accuracy drops to 56–66 %: the order

matters.



## Training data



1004 front + wrist pairs from the `MikelRubi/so101-pick-cube_*` recordings (23 and 24 October 2026):
- **654 pairs** from the last 3 frames of 218 pick episodes (112 `pin`, 106 `nopin`), labelled by eye. 16 of the `pin`
  and 72 of the `nopin` are "moderate" (blurred or partial view). 39 ambiguous episodes and 5 without a part were
  discarded.
- **350 pairs** of photos taken on purpose (186 `pin`, 164 `nopin`), stored in the `pin_snapshots/` folder of each
  dataset.

**Training:** 25 epochs, class-weighted cross-entropy, AdamW and OneCycle (backbone at a tenth of the head's learning
rate), and strong augmentation: rotation, zoom, shift, flip, saturation and white balance, blur and noise.

## Results (5-fold cross-validation; each fold holds out whole recording sessions)

| Subset | n | Accuracy | Majority class |
|---|---|---|---|
| All pairs | 1004 | **0.85** | 0.52 |
| Episode end frames | 654 | 0.87 | 0.51 |
| Photos taken on purpose | 350 | 0.83 | 0.53 |
| Per episode | 568 | 0.84 | 0.52 |
| Per episode, clear labels only | 480 | 0.84 | 0.59 |

Per fold: 0.76 · 0.85 · 1.00 · 0.88 · 0.88. The `model.pt` in this repository is trained on all the data, so these
figures come from the five cross-validation models, not from it (`metrics.json`: training accuracy 0.998). The labels
are a visual judgement, not ground truth.

**Other architectures tried:**

| Architecture | Accuracy |
|---|---|
| ResNet18 at 448×336 | 0.85 on all pairs, 0.82 per episode |
| Small CNN trained from scratch | 0.55 (chance) |
| ResNet18 with less data | 0.66 (627 pairs) and 0.81 (905 pairs) |

More data, pretrained weights and higher resolution made the biggest difference.

## Files

- `model.pt`: dictionary with `state_dict`, `classes`, `height`, `width`, `backbone`, `cameras` and `channels`.
- `pin_classifier.py`: loading and inference.
- `metrics.json`: report of the final training run.
- `model_card_original.md`: the author's original card.

## Credits and acknowledgements

Model trained by Iker Azurmendi (originally published as `IkerAzurmendi/pin_classifier_so101`; `IkerAzurmendi/pin-classifier`
holds the same `model.pt`). Created by **Iker Azurmendi, Mikel Zulueta, Ander Autillo and Diego Suarez** during the **Robotekin LeRobot Hackathon 2026** (23–25 October 2026, Vitoria-Gasteiz).
Thanks to **Bravo Robots** and to **Robotekin** for the hackathon, to Hugging Face for LeRobot, and to everyone who
took part in it.