Robotics
LeRobot
Safetensors
Piper
act
imitation-learning
RoboticsCollectiveAachen commited on
Commit
f3f226b
·
verified ·
1 Parent(s): 34eedc0

Add model card

Browse files
Files changed (1) hide show
  1. README.md +179 -0
README.md ADDED
@@ -0,0 +1,179 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: lerobot
3
+ pipeline_tag: robotics
4
+ tags:
5
+ - robotics
6
+ - lerobot
7
+ - act
8
+ - piper
9
+ - imitation-learning
10
+ datasets:
11
+ - Samisaliveagain/piper-hexagonal-bolt-insertion-100-demos-v1
12
+ ---
13
+
14
+ # ACT: Piper Hexagonal Bolt Insertion (100 demos, 45k steps)
15
+
16
+ An ACT policy for an AgileX Piper arm that picks up a hexagonal bolt and places it in the matching opening, using two RGB cameras and joint positions. This is the 45k-step checkpoint trained on the first, 100-demonstration dataset.
17
+
18
+ > **Status:** superseded. A newer policy trained on a cleaner re-recording of the task, with every episode starting from the zero home pose, is available at [RoboticsCollectiveAachen/act_piper_hexagonal_bolt_insertion_zero_home](https://huggingface.co/RoboticsCollectiveAachen/act_piper_hexagonal_bolt_insertion_zero_home). Prefer that model. This one has not been evaluated on the robot.
19
+
20
+ ---
21
+
22
+ ## Model Details
23
+
24
+ - **Method:** [ACT (Action Chunking with Transformers)](https://huggingface.co/papers/2304.13705), trained from scratch with [LeRobot](https://github.com/huggingface/lerobot)
25
+ - **Robot:** AgileX Piper, single arm (`piper_follower`), joint-space control
26
+ - **Task:** "Pick up the hexagonal bolt and place it in the matching opening."
27
+ - **Control rate:** 30 Hz
28
+ - **Developed by:** RoboticsCollectiveAachen
29
+
30
+ ### Architecture
31
+
32
+ | Setting | Value |
33
+ | --- | --- |
34
+ | Vision backbone | ResNet-18, initialized from ImageNet (`IMAGENET1K_V1`) |
35
+ | Transformer | 4 encoder layers, 1 decoder layer, `dim_model` 512, 8 heads, FFN 3200 |
36
+ | CVAE | enabled, latent dim 32, 4 encoder layers, KL weight 10 |
37
+ | Chunk size / actions executed per chunk | 100 / 100 |
38
+ | Dropout | 0.1 |
39
+ | Parameters | 51.6 M |
40
+
41
+ ## Inputs & Outputs
42
+
43
+ | Feature | Shape | Description |
44
+ | --- | --- | --- |
45
+ | `observation.images.overhead` | 3 × 480 × 640 | Overhead RGB camera |
46
+ | `observation.images.angled` | 3 × 480 × 640 | Angled RGB camera |
47
+ | `observation.state` | 7 | Measured positions: joints 1–6 (degrees), then gripper opening (mm) |
48
+ | `action` (output) | 7 | **Absolute** follower targets in the same order and units as the state |
49
+
50
+ The policy predicts a chunk of 100 future actions (3.3 s at 30 FPS) per inference call.
51
+ All inputs and outputs are normalized with mean/std statistics from the training dataset;
52
+ the bundled pre/post-processors apply this automatically.
53
+
54
+ ## Training Data
55
+
56
+ [Samisaliveagain/piper-hexagonal-bolt-insertion-100-demos-v1](https://huggingface.co/datasets/Samisaliveagain/piper-hexagonal-bolt-insertion-100-demos-v1)
57
+
58
+ - **Episodes:** 100 teleoperated demonstrations (Piper follower driven by a 7-servo Piper leader)
59
+ - **Frames:** 104,172 at 30 FPS (≈ 58 min). Episodes are 26–53 s long (mean 35 s)
60
+ - **Recording limits:** firmware speed 40 %, max target offset 4 °/mm, joint 4 locked at 0 °
61
+ - **Start pose:** not standardized across episodes
62
+
63
+ <a class="flex" href="https://huggingface.co/spaces/lerobot/visualize_dataset?path=Samisaliveagain/piper-hexagonal-bolt-insertion-100-demos-v1">
64
+ <img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/badges/resolve/main/visualize-this-dataset-xl.svg"/>
65
+ <img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/badges/resolve/main/visualize-this-dataset-xl-dark.svg"/>
66
+ </a>
67
+
68
+ ## Training Procedure
69
+
70
+ | Setting | Steps 0 – 25k | Steps 25k – 45k |
71
+ | --- | --- | --- |
72
+ | Hardware | 2 × NVIDIA TITAN Xp (12 GB), DDP | 3 × NVIDIA TITAN Xp (12 GB), DDP |
73
+ | Batch size | 16 per GPU (32 effective) | 16 per GPU (48 effective) |
74
+ | Optimizer | AdamW, lr 1.5e-5 (backbone 1.5e-5), weight decay 1e-4, constant schedule | same |
75
+ | Gradient clipping | 10.0 | same |
76
+ | Precision | fp32 | same |
77
+
78
+ - **Total:** 45,000 steps, ≈ 1.76 M samples, ≈ 17 epochs (resumed from the 25k checkpoint; the run was
79
+ stopped at 45k when the dataset was superseded)
80
+ - **Seed:** 1000
81
+
82
+ **Image augmentation** (applied on the GPU, up to 3 per frame, randomly chosen): brightness
83
+ 0.8–1.2, contrast 0.8–1.2, saturation 0.5–1.5, hue ±0.05, sharpness 0.5–1.5, and a random
84
+ affine transform (±5° rotation, ±5 % translation).
85
+
86
+ **Training loss** (action L1 in normalized units; the KL term falls to ≈0 within the first
87
+ 10k steps, which is typical for ACT):
88
+
89
+ | Step | 5k | 10k | 15k | 20k | 25k | 30k | 35k | 40k | 45k |
90
+ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
91
+ | L1 loss | 0.140 | 0.107 | 0.089 | 0.078 | 0.071 | 0.062 | 0.056 | 0.053 | 0.050 |
92
+
93
+ These are training-set losses. The dataset has no held-out split, so they do not measure
94
+ overfitting or task success.
95
+
96
+ ## How to Use
97
+
98
+ LeRobot has no built-in Piper robot driver, so `lerobot-rollout` cannot drive this
99
+ arm directly. Load the policy inside your own 30 Hz control loop:
100
+
101
+ ```python
102
+ import torch
103
+ from lerobot.policies import make_pre_post_processors
104
+ from lerobot.policies.act import ACTPolicy
105
+
106
+ repo = "RoboticsCollectiveAachen/act_piper_hexagonal_bolt_insertion"
107
+ policy = ACTPolicy.from_pretrained(repo).to("cuda").eval()
108
+ preprocessor, postprocessor = make_pre_post_processors(
109
+ policy.config, pretrained_path=repo
110
+ )
111
+
112
+ policy.reset() # at the start of every episode
113
+ while running: # 30 Hz
114
+ obs = {
115
+ "observation.state": state, # torch.float32, shape (7,): joints 1-6 [deg], gripper [mm]
116
+ "observation.images.overhead": img_ovh, # torch.float32, shape (3, 480, 640), RGB in [0, 1]
117
+ "observation.images.angled": img_ang, # same format
118
+ "task": "Pick up the hexagonal bolt and place it in the matching opening.",
119
+ }
120
+ with torch.inference_mode():
121
+ action = postprocessor(policy.select_action(preprocessor(obs)))
122
+ send_absolute_joint_targets(action.squeeze(0)) # 7 values, same order/units as the state
123
+ ```
124
+
125
+ **Deployment notes**
126
+
127
+ - Episodes in the training data do not share a common start pose. Start each rollout from a pose similar to those in the dataset.
128
+ - Apply the same safety limits used during recording: firmware speed 40 % and a maximum
129
+ target offset of 4 °/mm from the measured position. The training actions are the targets
130
+ *after* this limiting, so the policy has never seen large jumps.
131
+ - Joint 4 was locked to a 0 ° target in the training data. Keep it locked, or force its target to 0.
132
+ - Camera placement, resolution (640 × 480), and the `overhead`/`angled` key names must match
133
+ the recording setup.
134
+ - By default the policy executes each 100-step chunk open-loop. For smoother, more reactive
135
+ motion near the insertion, enable temporal ensembling:
136
+ `policy.config.temporal_ensemble_coeff = 0.01` and `policy.config.n_action_steps = 1`
137
+ (one inference per control step; costs more GPU time).
138
+
139
+ ## Evaluation
140
+
141
+ _No real-robot evaluation has been run on this checkpoint yet._
142
+
143
+ <!-- Fill in after testing on the robot:
144
+
145
+ | Setup | Trials | Successes | Success rate |
146
+ | --- | --- | --- | --- |
147
+ | Training configuration | 10 | ? | ?% |
148
+ -->
149
+
150
+ ## Limitations
151
+
152
+ - Trained on a single robot, cell, camera setup, and object. Expect degraded behavior under
153
+ different lighting, camera poses, bolt or fixture positions outside the demonstrated range,
154
+ or a different Piper unit.
155
+ - The training demonstrations carry no success labels and were not exhaustively reviewed,
156
+ so the policy imitates whatever the operator did, including imperfect attempts.
157
+ - Single-task policy. The task string is not used by ACT, so the policy cannot be steered with language.
158
+ - Run it with a hand on the emergency stop. Nothing in the model enforces joint limits,
159
+ collision avoidance, or force limits.
160
+
161
+ ## Citation
162
+
163
+ ```bibtex
164
+ @inproceedings{zhao2023learning,
165
+ title = {Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware},
166
+ author = {Zhao, Tony Z. and Kumar, Vikash and Levine, Sergey and Finn, Chelsea},
167
+ booktitle = {Robotics: Science and Systems (RSS)},
168
+ year = {2023},
169
+ eprint = {2304.13705},
170
+ archivePrefix = {arXiv}
171
+ }
172
+
173
+ @misc{cadene2024lerobot,
174
+ author = {Cadene, Remi and Alibert, Simon and Soare, Alexander and Gallouedec, Quentin and Zouitine, Adil and Palma, Steven and Kooijmans, Pepijn and Aractingi, Michel and Shukor, Mustafa and Aubakirova, Dana and Russi, Martino and Capuano, Francesco and Pascal, Caroline and Choghari, Jade and Moss, Jess and Wolf, Thomas},
175
+ title = {LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch},
176
+ howpublished = "\url{https://github.com/huggingface/lerobot}",
177
+ year = {2024}
178
+ }
179
+ ```