pin-classifier / README.md
MikelRubi's picture
Acknowledgements: project team
8811ccf verified
|
Raw History Blame Contribute Delete
10.9 kB
---
library_name: pytorch
license: apache-2.0
pipeline_tag: image-classification
tags:
- image-classification
- robotics
- so101
- robotekin-hackathon-2026
- pin-inspection
datasets:
- MikelRubi/robotekin-so101-pin-images
- MikelRubi/so101-pick-cube_20261023_193855
- MikelRubi/so101-pick-cube_20261023_195654
- MikelRubi/so101-pick-cube_20261023_200003
- MikelRubi/so101-pick-cube_20261023_201205
- MikelRubi/so101-pick-cube_20261023_203046
- MikelRubi/so101-pick-cube_20261023_203822
- MikelRubi/so101-pick-cube_20261023_205608
- MikelRubi/so101-pick-cube_20261023_205924
- MikelRubi/so101-pick-cube_20261023_211739
- MikelRubi/so101-pick-cube_20261023_212258
- MikelRubi/so101-pick-cube_20261023_213054
- MikelRubi/so101-pick-cube_20261023_221852
- MikelRubi/so101-pick-cube_20261023_222526
- MikelRubi/so101-pick-cube_20261023_223103
- MikelRubi/so101-pick-cube_20261023_223658
- MikelRubi/so101-pick-cube_20261023_224034
- MikelRubi/so101-pick-cube_20261023_224547
- MikelRubi/so101-pick-cube_20261023_224858
- MikelRubi/so101-pick-cube_20261023_225412
- MikelRubi/so101-pick-cube_20261023_225556
- MikelRubi/so101-pick-cube_20261023_225932
- MikelRubi/so101-pick-cube_20261024_084532
- MikelRubi/so101-pick-cube_20261024_085325
- MikelRubi/so101-pick-cube_20261024_085554
- MikelRubi/so101-pick-cube_20261024_085829
- MikelRubi/so101-pick-cube_20261024_090220
- MikelRubi/so101-pick-cube_20261024_090900
- MikelRubi/so101-pick-cube_20261024_091415
- MikelRubi/so101-pick-cube_20261024_092742
- MikelRubi/so101-pick-cube_20261024_094405
- MikelRubi/so101-pick-cube_20261024_095901
- MikelRubi/so101-pick-cube_20261024_102241
- MikelRubi/so101-pick-cube_20261024_103318
- MikelRubi/so101-pick-cube_20261024_103634
- MikelRubi/so101-pick-cube_20261024_104618
- MikelRubi/so101-pick-cube_20261024_105854
- MikelRubi/so101-pick-cube_20261024_110840
- MikelRubi/so101-pick-cube_20261024_111303
- MikelRubi/so101-pick-cube_20261024_112511
- MikelRubi/so101-pick-cube_20261024_113237
- MikelRubi/so101-pick-cube_20261024_114214
- MikelRubi/so101-pick-cube_20261024_115532
- MikelRubi/so101-pick-cube_20261024_223612
---
# Pin classifier — part with pin / without pin (SO-101)
**Looks at the part held by the arm, through the front and wrist cameras, and decides whether it has a pin (`pin`,
GOOD → green box) or not (`nopin`, BAD → white box).** Stage 2 of the pipeline below: its verdict selects the sorting
policy.
## Model
| | |
|---|---|
| **File** | `model.pt` |
| **Architecture** | ImageNet-pretrained EfficientNet-B0 (4.0 M parameters), **shared by both cameras**. For each camera the feature map is mean- and max-pooled; the two cameras are concatenated (front, then wrist; 5120 values) and one linear layer gives the 2 classes |
| **Input** | Two RGB images, `front` and `wrist`, centre-cropped to 4:3 (never stretched) and resized to 448×336. ImageNet normalisation is inside the model |
| **Output** | `nopin` / `pin` and the probability of `pin` |
| **Accuracy** | **≈ 85 %** on unseen recording sessions (cross-validation grouped by session). Majority-class baseline: 52 % |
| **Caveat** | Measured in the same room, with the same operator and lighting as the training data. Not tested on another day, under other lighting or with another batch of parts |
## The system
Built at the **Robotekin LeRobot Hackathon 2026** (23–25 October 2026, Robotory, Vitoria-Gasteiz). One SO-101 arm runs
a small quality-control cell for a metal bracket that may or may not carry a pin. **Project page:**
[`MikelRubi/robotekin-so101-pin-sorting`](https://huggingface.co/spaces/MikelRubi/robotekin-so101-pin-sorting).
```mermaid
flowchart LR
order["Order (voice / text)<br/>intent model → sorting rule"] --> tray["Part lies in the tray"]
tray --> pick["1 · act-pick<br/>grasp the part according to its orientation,<br/>lift it, hold it up to the camera"]
pick --> cls{"2 · pin-classifier<br/>front + wrist images"}
cls -- "rule: usually pin (GOOD)" --> good["3a · act-up-down-good<br/>carry it to the green box, release"]
cls -- "rule: usually no pin (BAD)" --> bad["3b · act-up-down-bad<br/>carry it to the white box, release"]
```
| Stage | Repository | Model |
|---|---|---|
| 1. Picking | [`MikelRubi/act-pick`](https://huggingface.co/MikelRubi/act-pick) | ACT policy |
| 2. Inspection | [`MikelRubi/pin-classifier`](https://huggingface.co/MikelRubi/pin-classifier) | EfficientNet-B0 image classifier |
| 3a. Sorting, GOOD | [`MikelRubi/act-up-down-good`](https://huggingface.co/MikelRubi/act-up-down-good) | ACT policy |
| 3b. Sorting, BAD | [`MikelRubi/act-up-down-bad`](https://huggingface.co/MikelRubi/act-up-down-bad) | ACT policy |
Picking is a separate policy because the bracket's geometry makes the right grasp depend on its orientation; the two
sorting policies start where the pick policy ends (part held in front of the camera).
**Orchestration.** The team's integrated demo, [`30_full_demo_app.py`](https://github.com/iker-azur/hackathon-robotekin/tree/main/demos/06_imitation_learning), runs the loop on the real follower:
an order (voice in Spanish, English or Basque, text, or Telegram) is read by the intent model
[`IkerAzurmendi/bin-intent-gliner2`](https://huggingface.co/IkerAzurmendi/bin-intent-gliner2); the arm waits until a
part lies still in the tray; the pick policy lifts it to the camera; the classifier averages 1.5 s of frames; the order
maps PIN / NO PIN to the green or the white box, and that box's policy carries the part there (`policy_routes.json`,
run through `run_policy.py`). In the usual order GOOD parts go to the green box and BAD parts to the white box. During
data collection the operator looked at the part and chose which sorting demonstration to record. The start/end-pose
contract between stages is in [`act-up-down-bad/HANDOFF.md`](https://huggingface.co/MikelRubi/act-up-down-bad/blob/main/HANDOFF.md).
> **Naming note: `good` / `bad` describe the part, not the quality of a demonstration.**
> GOOD = part **with** a pin → green box. BAD = part **without** a pin (defective) → white box.
> `up-down-bad` is a policy trained on *successful* demonstrations of carrying a defective part to the white box. No
> failed or negative demonstrations were intentionally used for training. Other informal names kept for compatibility:
> `so101-pick-cube_*` datasets contain the bracket, not a cube, and `up-down_*` datasets are
> the GOOD-box task (except the 23 October warm-up sessions).
## Hardware
| Component | What is known | Source |
|---|---|---|
| Arm | SO-101 follower, 6 joints: `shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper` (degrees; gripper 0–100) | dataset `robot_type: so_follower`, recording scripts |
| Teleoperation | SO-101 leader arm driving the follower (leader/follower); `action` = leader position | recording scripts |
| Cameras | Two USB RGB cameras: `front` (fixed overhead camera looking down at the tray; the part is lifted towards it for inspection) and `wrist` (on the gripper). 640×480 at 30 fps; the first sessions of 23 October used 1280×720 at 10 fps | dataset `meta/info.json` |
| Work cell | Orange tray with the parts, inspection position in front of the camera, green (big) box and white (small) box | recordings, `tray_detect.py`, `demo_orders.py` |
| Recording PC | Linux, arms on USB serial ports | team notes |
Not documented yet: camera models and exact mounting positions, workspace dimensions and lighting, servo models,
the part's dimensions, and the calibration procedure beyond LeRobot's standard `lerobot-calibrate`.
## Usage
```python
from pin_classifier import load_model, predict # pin_classifier.py is in this repository
model = load_model("model.pt") # needs torch, torchvision, numpy and Pillow
label, p_pin = predict(model, front_rgb, wrist_rgb) # HxWx3 uint8 RGB arrays, any size
```
`pin_classifier.py` rebuilds the architecture and loads the weights in strict mode. It was checked against the 110
labelled photo pairs of `so101-pick-cube_20261024_115532/pin_snapshots`: it classifies all 110 correctly, as expected,
because those photos are part of the training data. With the cameras swapped the accuracy drops to 56–66 %: the order
matters.
## Training data
1004 front + wrist pairs from the `MikelRubi/so101-pick-cube_*` recordings (23 and 24 October 2026):
- **654 pairs** from the last 3 frames of 218 pick episodes (112 `pin`, 106 `nopin`), labelled by eye. 16 of the `pin`
and 72 of the `nopin` are "moderate" (blurred or partial view). 39 ambiguous episodes and 5 without a part were
discarded.
- **350 pairs** of photos taken on purpose (186 `pin`, 164 `nopin`), stored in the `pin_snapshots/` folder of each
dataset.
**Training:** 25 epochs, class-weighted cross-entropy, AdamW and OneCycle (backbone at a tenth of the head's learning
rate), and strong augmentation: rotation, zoom, shift, flip, saturation and white balance, blur and noise.
## Results (5-fold cross-validation; each fold holds out whole recording sessions)
| Subset | n | Accuracy | Majority class |
|---|---|---|---|
| All pairs | 1004 | **0.85** | 0.52 |
| Episode end frames | 654 | 0.87 | 0.51 |
| Photos taken on purpose | 350 | 0.83 | 0.53 |
| Per episode | 568 | 0.84 | 0.52 |
| Per episode, clear labels only | 480 | 0.84 | 0.59 |
Per fold: 0.76 · 0.85 · 1.00 · 0.88 · 0.88. The `model.pt` in this repository is trained on all the data, so these
figures come from the five cross-validation models, not from it (`metrics.json`: training accuracy 0.998). The labels
are a visual judgement, not ground truth.
**Other architectures tried:**
| Architecture | Accuracy |
|---|---|
| ResNet18 at 448×336 | 0.85 on all pairs, 0.82 per episode |
| Small CNN trained from scratch | 0.55 (chance) |
| ResNet18 with less data | 0.66 (627 pairs) and 0.81 (905 pairs) |
More data, pretrained weights and higher resolution made the biggest difference.
## Files
- `model.pt`: dictionary with `state_dict`, `classes`, `height`, `width`, `backbone`, `cameras` and `channels`.
- `pin_classifier.py`: loading and inference.
- `metrics.json`: report of the final training run.
- `model_card_original.md`: the author's original card.
## Credits and acknowledgements
Model trained by Iker Azurmendi (originally published as `IkerAzurmendi/pin_classifier_so101`; `IkerAzurmendi/pin-classifier`
holds the same `model.pt`). Created by **Iker Azurmendi, Mikel Zulueta, Ander Autillo and Diego Suarez** during the **Robotekin LeRobot Hackathon 2026** (23–25 October 2026, Vitoria-Gasteiz).
Thanks to **Bravo Robots** and to **Robotekin** for the hackathon, to Hugging Face for LeRobot, and to everyone who
took part in it.