Blame for README.md

---
license: apache-2.0
library_name: lerobot
pipeline_tag: robotics
tags:
  - robotics
  - lerobot
  - diffusion-policy
  - piper
  - imitation-learning
datasets:
  - Samisaliveagain/piper-hexagonal-bolt-insertion-100-demos-v1
---

# Diffusion Policy: Piper Hexagonal Bolt Insertion (100 demos, 100k steps)

A Diffusion Policy for an AgileX Piper arm that picks up a hexagonal bolt and places it in the matching opening, using two RGB cameras and joint positions. This is the final 100k-step checkpoint, trained on the first, 100-demonstration dataset.

> **Status:** superseded. Newer policies were trained on a cleaner re-recording of the task, in which every episode starts from the zero home pose: [RoboticsCollectiveAachen/act_piper_hexagonal_bolt_insertion_zero_home](https://huggingface.co/RoboticsCollectiveAachen/act_piper_hexagonal_bolt_insertion_zero_home) and [RoboticsCollectiveAachen/smolvla_piper_hex_bolt_insertion_80](https://huggingface.co/RoboticsCollectiveAachen/smolvla_piper_hex_bolt_insertion_80). Prefer those. This model has not been evaluated on the robot.

---

## Model Details

- **Method:** [Diffusion Policy](https://huggingface.co/papers/2303.04137), trained from scratch with [LeRobot](https://github.com/huggingface/lerobot) (vision backbone initialized from ImageNet)
- **Robot:** AgileX Piper, single arm (`piper_follower`), joint-space control
- **Task:** "Pick up the hexagonal bolt and place it in the matching opening."
- **Control rate:** 30 Hz
- **Developed by:** RoboticsCollectiveAachen
- **License:** Apache 2.0

### Architecture

| Setting | Value |
| --- | --- |
| Vision backbone | ResNet-18, initialized from ImageNet (`IMAGENET1K_V1`), separate encoder per camera, spatial softmax (32 keypoints) |
| Image preprocessing | 480 × 640 resized to 240 × 320, no crop |
| Denoising network | 1-D conditional U-Net, channels 512 / 1024 / 2048, kernel 5, 8 groups, FiLM conditioning |
| Noise schedule (training) | 100 DDPM timesteps, `squaredcos_cap_v2`, ε-prediction, samples clipped to ±1 |
| Sampler (inference) | DDIM, 10 steps |
| Observation history / horizon / actions executed | 2 / 64 / 32 |
| Parameters | 277.9 M |

## Inputs & Outputs

| Feature | Shape | Description |
| --- | --- | --- |
| `observation.images.overhead` | 3 × 480 × 640 | Overhead RGB camera |
| `observation.images.angled` | 3 × 480 × 640 | Angled RGB camera |
| `observation.state` | 7 | Measured positions: joints 1–6 (degrees), then gripper opening (mm) |
| `action` (output) | 7 | **Absolute** follower targets in the same order and units as the state |

From the last 2 observations, the policy plans a 64-step action trajectory and executes the first 32 (1.07 s at 30 FPS) before replanning.
Images are normalized with mean/std statistics. State and action are min/max-scaled to [−1, 1] using the training-dataset range.
The bundled pre/post-processors apply this automatically.
Because samples are clipped to ±1, the policy cannot command targets outside the range covered by the training data.

## Training Data

[Samisaliveagain/piper-hexagonal-bolt-insertion-100-demos-v1](https://huggingface.co/datasets/Samisaliveagain/piper-hexagonal-bolt-insertion-100-demos-v1)

- **Episodes:** 100 teleoperated demonstrations (Piper follower driven by a 7-servo Piper leader)
- **Frames:** 104,172 at 30 FPS (≈ 58 min). Episodes are 26–53 s long (mean 35 s)
- **Recording limits:** firmware speed 40 %, max target offset 4 °/mm, joint 4 locked at 0 °
- **Start pose:** not standardized across episodes

<a class="flex" href="https://huggingface.co/spaces/lerobot/visualize_dataset?path=Samisaliveagain/piper-hexagonal-bolt-insertion-100-demos-v1">
<img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/badges/resolve/main/visualize-this-dataset-xl.svg"/>
<img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/badges/resolve/main/visualize-this-dataset-xl-dark.svg"/>
</a>

## Training Procedure

| Setting | Value |
| --- | --- |
| Hardware | 1 × NVIDIA Quadro RTX 6000 (24 GB) |
| Batch size | 32 |
| Optimizer | Adam, lr 1e-4, betas (0.95, 0.999), weight decay 1e-6 |
| LR schedule | cosine, 500 warmup steps |
| Gradient clipping | 10.0 |
| Precision | fp32 |
| Steps (this checkpoint) | 100,000 (≈ 3.2 M samples, ≈ 31 epochs), 10 h 54 min |
| Image augmentation | none |
| Seed | 1000 |

**Training loss** (MSE of the predicted noise, in normalized units):

| Step | 10k | 20k | 30k | 40k | 50k | 60k | 70k | 80k | 90k | 100k |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Loss | 0.010 | 0.007 | 0.007 | 0.005 | 0.005 | 0.004 | 0.003 | 0.003 | 0.003 | 0.002 |

These are training-set losses. The dataset has no held-out split, so they do not measure
overfitting or task success. The loss was still falling when training stopped.
Intermediate checkpoints (every 10k steps) are not published.

## How to Use

LeRobot has no built-in Piper robot driver, so `lerobot-rollout` cannot drive this
arm directly. Load the policy inside your own 30 Hz control loop:

```python
import torch
from lerobot.policies import make_pre_post_processors
from lerobot.policies.diffusion.modeling_diffusion import DiffusionPolicy

repo = "RoboticsCollectiveAachen/diffusion_piper_hex_bolt_insertion"
policy = DiffusionPolicy.from_pretrained(repo).to("cuda").eval()
preprocessor, postprocessor = make_pre_post_processors(
    policy.config, pretrained_path=repo
)

policy.reset()  # at the start of every episode (clears the observation history and action queue)
while running:  # 30 Hz
    obs = {
        "observation.state": state,             # torch.float32, shape (7,): joints 1-6 [deg], gripper [mm]
        "observation.images.overhead": img_ovh, # torch.float32, shape (3, 480, 640), RGB in [0, 1]
        "observation.images.angled": img_ang,   # same format
    }
    with torch.inference_mode():
        action = postprocessor(policy.select_action(preprocessor(obs)))
    send_absolute_joint_targets(action.squeeze(0))  # 7 values, same order/units as the state
```

**Inference cost** (measured on a Quadro RTX 6000): about 1.2 GB of VRAM and about 110 ms per replanning step.
Replanning happens once every 32 control steps; the other `select_action` calls only pop the next queued action.
Laptop GPUs with 8 GB are enough.

**Deployment notes**

- Episodes in the training data do not share a common start pose. Start each rollout from a pose similar to those in the dataset.
- Apply the same safety limits used during recording: firmware speed 40 % and a maximum
  target offset of 4 °/mm from the measured position. The training actions are the targets
  *after* this limiting, so the policy has never seen large jumps.
- Joint 4 was locked to a 0 ° target in the training data. Keep it locked, or force its target to 0.
- Camera placement, resolution (640 × 480), and the `overhead`/`angled` key names must match
  the recording setup. Pass full-resolution frames; the policy resizes them itself.
- The ~110 ms replanning step happens inside one control tick. Either let that tick run long,
  or run inference in a separate thread so the 30 Hz loop keeps sending the queued actions.

## Evaluation

_No real-robot evaluation has been run on this checkpoint yet._

<!-- Fill in after testing on the robot:

| Setup | Trials | Successes | Success rate |
| --- | --- | --- | --- |
| Training configuration | 10 | ? | ?% |
-->

## Limitations

- Trained on a single robot, cell, camera setup, and object. Expect degraded behavior under
  different lighting, camera poses, bolt or fixture positions outside the demonstrated range,
  or a different Piper unit.
- The training demonstrations carry no success labels and were not exhaustively reviewed,
  so the policy imitates whatever the operator did, including imperfect attempts.
- Single-task policy. It takes no language input.
- Trained without image augmentation, so it may be more sensitive to lighting and camera changes than the ACT models.
- Run it with a hand on the emergency stop. Nothing in the model enforces joint limits,
  collision avoidance, or force limits.

## Citation

```bibtex
@article{chi2024diffusionpolicy,
    author  = {Cheng Chi and Zhenjia Xu and Siyuan Feng and Eric Cousineau and Yilun Du and Benjamin Burchfiel and Russ Tedrake and Shuran Song},
    title   = {Diffusion Policy: Visuomotor Policy Learning via Action Diffusion},
    journal = {The International Journal of Robotics Research},
    year    = {2024}
}

@misc{cadene2024lerobot,
    author = {Cadene, Remi and Alibert, Simon and Soare, Alexander and Gallouedec, Quentin and Zouitine, Adil and Palma, Steven and Kooijmans, Pepijn and Aractingi, Michel and Shukor, Mustafa and Aubakirova, Dana and Russi, Martino and Capuano, Francesco and Pascal, Caroline and Choghari, Jade and Moss, Jess and Wolf, Thomas},
    title = {LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch},
    howpublished = "\url{https://github.com/huggingface/lerobot}",
    year = {2024}
}
```