Blame for README.md

---
license: apache-2.0
library_name: lerobot
pipeline_tag: robotics
tags:
  - robotics
  - lerobot
  - act
  - piper
  - imitation-learning
datasets:
  - Samisaliveagain/piper-hexagonal-bolt-insertion-80-demos-zero-home-v1
---

# ACT: Piper Hexagonal Bolt Insertion (zero-home, 80 demos)

An ACT policy for an AgileX Piper arm that picks up a hexagonal bolt and places it in the matching opening, using two RGB cameras and joint positions. It is trained on 80 demonstrations that all start from the zero home pose. The `main` branch holds the final 100k-step checkpoint. The 80k-step checkpoint is kept on the `step-80000` branch (`revision="step-80000"`).

> **Status:** final 100k-step checkpoint, not yet evaluated on the robot. It has seen each training frame about 94 times. Compare it on the robot with the 80k checkpoint on the `step-80000` branch before relying on it.

---

## Model Details

- **Method:** [ACT (Action Chunking with Transformers)](https://huggingface.co/papers/2304.13705), trained from scratch with [LeRobot](https://github.com/huggingface/lerobot)
- **Robot:** AgileX Piper, single arm (`piper_follower`), joint-space control
- **Task:** "Pick up the hexagonal bolt and place it in the matching opening."
- **Control rate:** 30 Hz
- **Developed by:** RoboticsCollectiveAachen
- **License:** Apache 2.0

### Architecture

| Setting | Value |
| --- | --- |
| Vision backbone | ResNet-18, initialized from ImageNet (`IMAGENET1K_V1`) |
| Transformer | 4 encoder layers, 1 decoder layer, `dim_model` 512, 8 heads, FFN 3200 |
| CVAE | enabled, latent dim 32, 4 encoder layers, KL weight 10 |
| Chunk size / actions executed per chunk | 100 / 100 |
| Dropout | 0.1 |
| Parameters | 51.6 M |

## Inputs & Outputs

| Feature | Shape | Description |
| --- | --- | --- |
| `observation.images.overhead` | 3 × 480 × 640 | Overhead RGB camera |
| `observation.images.angled` | 3 × 480 × 640 | Angled RGB camera |
| `observation.state` | 7 | Measured positions: joints 1–6 (degrees), then gripper opening (mm) |
| `action` (output) | 7 | **Absolute** follower targets in the same order and units as the state |

The policy predicts a chunk of 100 future actions (3.3 s at 30 FPS) per inference call.
All inputs and outputs are normalized with mean/std statistics from the training dataset;
the bundled pre/post-processors apply this automatically.

## Training Data

[Samisaliveagain/piper-hexagonal-bolt-insertion-80-demos-zero-home-v1](https://huggingface.co/datasets/Samisaliveagain/piper-hexagonal-bolt-insertion-80-demos-zero-home-v1)

- **Episodes:** 80 teleoperated demonstrations (Piper follower driven by a 6-joint leader with a gripper trigger)
- **Frames:** 50,921 at 30 FPS (≈ 28 min). Episodes are 14–34 s long (mean 21 s)
- **Recording limits:** firmware speed 40 %, max target offset 4 °/mm, joint 4 locked at 0 °
- **Start pose:** every episode starts within 1 °/mm of all-zero joints with the gripper closed

<a class="flex" href="https://huggingface.co/spaces/lerobot/visualize_dataset?path=Samisaliveagain/piper-hexagonal-bolt-insertion-80-demos-zero-home-v1">
<img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/badges/resolve/main/visualize-this-dataset-xl.svg"/>
<img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/badges/resolve/main/visualize-this-dataset-xl-dark.svg"/>
</a>

## Training Procedure

| Setting | Value |
| --- | --- |
| Hardware | 3 × NVIDIA TITAN Xp (12 GB), DDP |
| Batch size | 16 per GPU (48 effective) |
| Optimizer | AdamW, lr 2e-5 (backbone 2e-5), weight decay 1e-4, constant schedule |
| Gradient clipping | 10.0 |
| Precision | fp32 |
| Steps (this checkpoint) | 100,000 (≈ 4.8 M samples, ≈ 94 epochs) |
| Seed | 1000 |

**Image augmentation** (applied on the GPU, up to 3 per frame, randomly chosen): brightness
0.8–1.2, contrast 0.8–1.2, saturation 0.5–1.5, hue ±0.05, sharpness 0.5–1.5, and a random
affine transform (±5° rotation, ±5 % translation).

**Training loss** (action L1 in normalized units; the KL term falls to ≈0 within the first
10k steps, which is typical for ACT):

| Step | 5k | 10k | 20k | 30k | 40k | 50k | 60k | 70k | 80k | 90k | 100k |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| L1 loss | 0.103 | 0.074 | 0.055 | 0.046 | 0.038 | 0.035 | 0.032 | 0.030 | 0.028 | 0.026 | 0.025 |

These are training-set losses. The dataset has no held-out split, so they do not measure
overfitting or task success.

## How to Use

LeRobot has no built-in Piper robot driver, so `lerobot-rollout` cannot drive this
arm directly. Load the policy inside your own 30 Hz control loop:

```python
import torch
from lerobot.policies import make_pre_post_processors
from lerobot.policies.act import ACTPolicy

repo = "RoboticsCollectiveAachen/act_piper_hexagonal_bolt_insertion_zero_home"
policy = ACTPolicy.from_pretrained(repo).to("cuda").eval()
preprocessor, postprocessor = make_pre_post_processors(
    policy.config, pretrained_path=repo
)

policy.reset()  # at the start of every episode
while running:  # 30 Hz
    obs = {
        "observation.state": state,             # torch.float32, shape (7,): joints 1-6 [deg], gripper [mm]
        "observation.images.overhead": img_ovh, # torch.float32, shape (3, 480, 640), RGB in [0, 1]
        "observation.images.angled": img_ang,   # same format
        "task": "Pick up the hexagonal bolt and place it in the matching opening.",
    }
    with torch.inference_mode():
        action = postprocessor(policy.select_action(preprocessor(obs)))
    send_absolute_joint_targets(action.squeeze(0))  # 7 values, same order/units as the state
```

**Deployment notes**

- **Start every rollout from the zero home pose**: all joint targets at 0 ° and the gripper closed. Every training episode begins there, so other start poses are out of distribution.
- Apply the same safety limits used during recording: firmware speed 40 % and a maximum
  target offset of 4 °/mm from the measured position. The training actions are the targets
  *after* this limiting, so the policy has never seen large jumps.
- Joint 4 was locked to a 0 ° target in the training data. Keep it locked, or force its target to 0.
- Camera placement, resolution (640 × 480), and the `overhead`/`angled` key names must match
  the recording setup.
- By default the policy executes each 100-step chunk open-loop. For smoother, more reactive
  motion near the insertion, enable temporal ensembling:
  `policy.config.temporal_ensemble_coeff = 0.01` and `policy.config.n_action_steps = 1`
  (one inference per control step; costs more GPU time).

## Evaluation

_No real-robot evaluation has been run on this checkpoint yet._

<!-- Fill in after testing on the robot:

| Setup | Trials | Successes | Success rate |
| --- | --- | --- | --- |
| Training configuration | 10 | ? | ?% |
-->

## Limitations

- Trained on a single robot, cell, camera setup, and object. Expect degraded behavior under
  different lighting, camera poses, bolt or fixture positions outside the demonstrated range,
  or a different Piper unit.
- The training demonstrations carry no success labels and were not exhaustively reviewed,
  so the policy imitates whatever the operator did, including imperfect attempts.
- Single-task policy. The task string is not used by ACT, so the policy cannot be steered with language.
- Run it with a hand on the emergency stop. Nothing in the model enforces joint limits,
  collision avoidance, or force limits.
- 94 epochs over 80 demonstrations is a lot of training. The policy may reproduce the demonstrated
  trajectories closely but generalize less to new bolt or fixture placements than an earlier checkpoint would.

## Citation

```bibtex
@inproceedings{zhao2023learning,
    title     = {Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware},
    author    = {Zhao, Tony Z. and Kumar, Vikash and Levine, Sergey and Finn, Chelsea},
    booktitle = {Robotics: Science and Systems (RSS)},
    year      = {2023},
    eprint    = {2304.13705},
    archivePrefix = {arXiv}
}

@misc{cadene2024lerobot,
    author = {Cadene, Remi and Alibert, Simon and Soare, Alexander and Gallouedec, Quentin and Zouitine, Adil and Palma, Steven and Kooijmans, Pepijn and Aractingi, Michel and Shukor, Mustafa and Aubakirova, Dana and Russi, Martino and Capuano, Francesco and Pascal, Caroline and Choghari, Jade and Moss, Jess and Wolf, Thomas},
    title = {LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch},
    howpublished = "\url{https://github.com/huggingface/lerobot}",
    year = {2024}
}
```