|
Download README.md from MikelRubi/pin-classifier: direct link, hf CLI and curl.
- Browser
- Download file 10.9 kB
-
https://huggingface.co/MikelRubi/pin-classifier/resolve/main/README.md
- Command line
-
hf download hf://MikelRubi/pin-classifier/README.md
-
curl -L -o README.md https://huggingface.co/MikelRubi/pin-classifier/resolve/main/README.md
10.9 kB
| library_name: pytorch | |
| license: apache-2.0 | |
| pipeline_tag: image-classification | |
| tags: | |
| - image-classification | |
| - robotics | |
| - so101 | |
| - robotekin-hackathon-2026 | |
| - pin-inspection | |
| datasets: | |
| - MikelRubi/robotekin-so101-pin-images | |
| - MikelRubi/so101-pick-cube_20261023_193855 | |
| - MikelRubi/so101-pick-cube_20261023_195654 | |
| - MikelRubi/so101-pick-cube_20261023_200003 | |
| - MikelRubi/so101-pick-cube_20261023_201205 | |
| - MikelRubi/so101-pick-cube_20261023_203046 | |
| - MikelRubi/so101-pick-cube_20261023_203822 | |
| - MikelRubi/so101-pick-cube_20261023_205608 | |
| - MikelRubi/so101-pick-cube_20261023_205924 | |
| - MikelRubi/so101-pick-cube_20261023_211739 | |
| - MikelRubi/so101-pick-cube_20261023_212258 | |
| - MikelRubi/so101-pick-cube_20261023_213054 | |
| - MikelRubi/so101-pick-cube_20261023_221852 | |
| - MikelRubi/so101-pick-cube_20261023_222526 | |
| - MikelRubi/so101-pick-cube_20261023_223103 | |
| - MikelRubi/so101-pick-cube_20261023_223658 | |
| - MikelRubi/so101-pick-cube_20261023_224034 | |
| - MikelRubi/so101-pick-cube_20261023_224547 | |
| - MikelRubi/so101-pick-cube_20261023_224858 | |
| - MikelRubi/so101-pick-cube_20261023_225412 | |
| - MikelRubi/so101-pick-cube_20261023_225556 | |
| - MikelRubi/so101-pick-cube_20261023_225932 | |
| - MikelRubi/so101-pick-cube_20261024_084532 | |
| - MikelRubi/so101-pick-cube_20261024_085325 | |
| - MikelRubi/so101-pick-cube_20261024_085554 | |
| - MikelRubi/so101-pick-cube_20261024_085829 | |
| - MikelRubi/so101-pick-cube_20261024_090220 | |
| - MikelRubi/so101-pick-cube_20261024_090900 | |
| - MikelRubi/so101-pick-cube_20261024_091415 | |
| - MikelRubi/so101-pick-cube_20261024_092742 | |
| - MikelRubi/so101-pick-cube_20261024_094405 | |
| - MikelRubi/so101-pick-cube_20261024_095901 | |
| - MikelRubi/so101-pick-cube_20261024_102241 | |
| - MikelRubi/so101-pick-cube_20261024_103318 | |
| - MikelRubi/so101-pick-cube_20261024_103634 | |
| - MikelRubi/so101-pick-cube_20261024_104618 | |
| - MikelRubi/so101-pick-cube_20261024_105854 | |
| - MikelRubi/so101-pick-cube_20261024_110840 | |
| - MikelRubi/so101-pick-cube_20261024_111303 | |
| - MikelRubi/so101-pick-cube_20261024_112511 | |
| - MikelRubi/so101-pick-cube_20261024_113237 | |
| - MikelRubi/so101-pick-cube_20261024_114214 | |
| - MikelRubi/so101-pick-cube_20261024_115532 | |
| - MikelRubi/so101-pick-cube_20261024_223612 | |
| # Pin classifier — part with pin / without pin (SO-101) | |
| **Looks at the part held by the arm, through the front and wrist cameras, and decides whether it has a pin (`pin`, | |
| GOOD → green box) or not (`nopin`, BAD → white box).** Stage 2 of the pipeline below: its verdict selects the sorting | |
| policy. | |
| ## Model | |
| | | | | |
| |---|---| | |
| | **File** | `model.pt` | | |
| | **Architecture** | ImageNet-pretrained EfficientNet-B0 (4.0 M parameters), **shared by both cameras**. For each camera the feature map is mean- and max-pooled; the two cameras are concatenated (front, then wrist; 5120 values) and one linear layer gives the 2 classes | | |
| | **Input** | Two RGB images, `front` and `wrist`, centre-cropped to 4:3 (never stretched) and resized to 448×336. ImageNet normalisation is inside the model | | |
| | **Output** | `nopin` / `pin` and the probability of `pin` | | |
| | **Accuracy** | **≈ 85 %** on unseen recording sessions (cross-validation grouped by session). Majority-class baseline: 52 % | | |
| | **Caveat** | Measured in the same room, with the same operator and lighting as the training data. Not tested on another day, under other lighting or with another batch of parts | | |
| ## The system | |
| Built at the **Robotekin LeRobot Hackathon 2026** (23–25 October 2026, Robotory, Vitoria-Gasteiz). One SO-101 arm runs | |
| a small quality-control cell for a metal bracket that may or may not carry a pin. **Project page:** | |
| [`MikelRubi/robotekin-so101-pin-sorting`](https://huggingface.co/spaces/MikelRubi/robotekin-so101-pin-sorting). | |
| ```mermaid | |
| flowchart LR | |
| order["Order (voice / text)<br/>intent model → sorting rule"] --> tray["Part lies in the tray"] | |
| tray --> pick["1 · act-pick<br/>grasp the part according to its orientation,<br/>lift it, hold it up to the camera"] | |
| pick --> cls{"2 · pin-classifier<br/>front + wrist images"} | |
| cls -- "rule: usually pin (GOOD)" --> good["3a · act-up-down-good<br/>carry it to the green box, release"] | |
| cls -- "rule: usually no pin (BAD)" --> bad["3b · act-up-down-bad<br/>carry it to the white box, release"] | |
| ``` | |
| | Stage | Repository | Model | | |
| |---|---|---| | |
| | 1. Picking | [`MikelRubi/act-pick`](https://huggingface.co/MikelRubi/act-pick) | ACT policy | | |
| | 2. Inspection | [`MikelRubi/pin-classifier`](https://huggingface.co/MikelRubi/pin-classifier) | EfficientNet-B0 image classifier | | |
| | 3a. Sorting, GOOD | [`MikelRubi/act-up-down-good`](https://huggingface.co/MikelRubi/act-up-down-good) | ACT policy | | |
| | 3b. Sorting, BAD | [`MikelRubi/act-up-down-bad`](https://huggingface.co/MikelRubi/act-up-down-bad) | ACT policy | | |
| Picking is a separate policy because the bracket's geometry makes the right grasp depend on its orientation; the two | |
| sorting policies start where the pick policy ends (part held in front of the camera). | |
| **Orchestration.** The team's integrated demo, [`30_full_demo_app.py`](https://github.com/iker-azur/hackathon-robotekin/tree/main/demos/06_imitation_learning), runs the loop on the real follower: | |
| an order (voice in Spanish, English or Basque, text, or Telegram) is read by the intent model | |
| [`IkerAzurmendi/bin-intent-gliner2`](https://huggingface.co/IkerAzurmendi/bin-intent-gliner2); the arm waits until a | |
| part lies still in the tray; the pick policy lifts it to the camera; the classifier averages 1.5 s of frames; the order | |
| maps PIN / NO PIN to the green or the white box, and that box's policy carries the part there (`policy_routes.json`, | |
| run through `run_policy.py`). In the usual order GOOD parts go to the green box and BAD parts to the white box. During | |
| data collection the operator looked at the part and chose which sorting demonstration to record. The start/end-pose | |
| contract between stages is in [`act-up-down-bad/HANDOFF.md`](https://huggingface.co/MikelRubi/act-up-down-bad/blob/main/HANDOFF.md). | |
| > **Naming note: `good` / `bad` describe the part, not the quality of a demonstration.** | |
| > GOOD = part **with** a pin → green box. BAD = part **without** a pin (defective) → white box. | |
| > `up-down-bad` is a policy trained on *successful* demonstrations of carrying a defective part to the white box. No | |
| > failed or negative demonstrations were intentionally used for training. Other informal names kept for compatibility: | |
| > `so101-pick-cube_*` datasets contain the bracket, not a cube, and `up-down_*` datasets are | |
| > the GOOD-box task (except the 23 October warm-up sessions). | |
| ## Hardware | |
| | Component | What is known | Source | | |
| |---|---|---| | |
| | Arm | SO-101 follower, 6 joints: `shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper` (degrees; gripper 0–100) | dataset `robot_type: so_follower`, recording scripts | | |
| | Teleoperation | SO-101 leader arm driving the follower (leader/follower); `action` = leader position | recording scripts | | |
| | Cameras | Two USB RGB cameras: `front` (fixed overhead camera looking down at the tray; the part is lifted towards it for inspection) and `wrist` (on the gripper). 640×480 at 30 fps; the first sessions of 23 October used 1280×720 at 10 fps | dataset `meta/info.json` | | |
| | Work cell | Orange tray with the parts, inspection position in front of the camera, green (big) box and white (small) box | recordings, `tray_detect.py`, `demo_orders.py` | | |
| | Recording PC | Linux, arms on USB serial ports | team notes | | |
| Not documented yet: camera models and exact mounting positions, workspace dimensions and lighting, servo models, | |
| the part's dimensions, and the calibration procedure beyond LeRobot's standard `lerobot-calibrate`. | |
| ## Usage | |
| ```python | |
| from pin_classifier import load_model, predict # pin_classifier.py is in this repository | |
| model = load_model("model.pt") # needs torch, torchvision, numpy and Pillow | |
| label, p_pin = predict(model, front_rgb, wrist_rgb) # HxWx3 uint8 RGB arrays, any size | |
| ``` | |
| `pin_classifier.py` rebuilds the architecture and loads the weights in strict mode. It was checked against the 110 | |
| labelled photo pairs of `so101-pick-cube_20261024_115532/pin_snapshots`: it classifies all 110 correctly, as expected, | |
| because those photos are part of the training data. With the cameras swapped the accuracy drops to 56–66 %: the order | |
| matters. | |
| ## Training data | |
| 1004 front + wrist pairs from the `MikelRubi/so101-pick-cube_*` recordings (23 and 24 October 2026): | |
| - **654 pairs** from the last 3 frames of 218 pick episodes (112 `pin`, 106 `nopin`), labelled by eye. 16 of the `pin` | |
| and 72 of the `nopin` are "moderate" (blurred or partial view). 39 ambiguous episodes and 5 without a part were | |
| discarded. | |
| - **350 pairs** of photos taken on purpose (186 `pin`, 164 `nopin`), stored in the `pin_snapshots/` folder of each | |
| dataset. | |
| **Training:** 25 epochs, class-weighted cross-entropy, AdamW and OneCycle (backbone at a tenth of the head's learning | |
| rate), and strong augmentation: rotation, zoom, shift, flip, saturation and white balance, blur and noise. | |
| ## Results (5-fold cross-validation; each fold holds out whole recording sessions) | |
| | Subset | n | Accuracy | Majority class | | |
| |---|---|---|---| | |
| | All pairs | 1004 | **0.85** | 0.52 | | |
| | Episode end frames | 654 | 0.87 | 0.51 | | |
| | Photos taken on purpose | 350 | 0.83 | 0.53 | | |
| | Per episode | 568 | 0.84 | 0.52 | | |
| | Per episode, clear labels only | 480 | 0.84 | 0.59 | | |
| Per fold: 0.76 · 0.85 · 1.00 · 0.88 · 0.88. The `model.pt` in this repository is trained on all the data, so these | |
| figures come from the five cross-validation models, not from it (`metrics.json`: training accuracy 0.998). The labels | |
| are a visual judgement, not ground truth. | |
| **Other architectures tried:** | |
| | Architecture | Accuracy | | |
| |---|---| | |
| | ResNet18 at 448×336 | 0.85 on all pairs, 0.82 per episode | | |
| | Small CNN trained from scratch | 0.55 (chance) | | |
| | ResNet18 with less data | 0.66 (627 pairs) and 0.81 (905 pairs) | | |
| More data, pretrained weights and higher resolution made the biggest difference. | |
| ## Files | |
| - `model.pt`: dictionary with `state_dict`, `classes`, `height`, `width`, `backbone`, `cameras` and `channels`. | |
| - `pin_classifier.py`: loading and inference. | |
| - `metrics.json`: report of the final training run. | |
| - `model_card_original.md`: the author's original card. | |
| ## Credits and acknowledgements | |
| Model trained by Iker Azurmendi (originally published as `IkerAzurmendi/pin_classifier_so101`; `IkerAzurmendi/pin-classifier` | |
| holds the same `model.pt`). Created by **Iker Azurmendi, Mikel Zulueta, Ander Autillo and Diego Suarez** during the **Robotekin LeRobot Hackathon 2026** (23–25 October 2026, Vitoria-Gasteiz). | |
| Thanks to **Bravo Robots** and to **Robotekin** for the hackathon, to Hugging Face for LeRobot, and to everyone who | |
| took part in it. | |