Model card for final 100k checkpoint

README.md

@@ -1,162 +1,181 @@
 ---
-datasets: Samisaliveagain/piper-hexagonal-bolt-insertion-80-demos-zero-home-v1
+license: apache-2.0
 library_name: lerobot
-model_name: act
 pipeline_tag: robotics
 tags:
-- robotics
-- act
-- lerobot
+  - robotics
+  - lerobot
+  - act
+  - piper
+  - imitation-learning
+datasets:
+  - Samisaliveagain/piper-hexagonal-bolt-insertion-80-demos-zero-home-v1
 ---
 
-# Model Card for act
+# ACT: Piper Hexagonal Bolt Insertion (zero-home, 80 demos)
 
-<!-- Provide a quick summary of what the model is/does. -->
-
-
-[Action Chunking with Transformers (ACT)](https://huggingface.co/papers/2304.13705) is an imitation-learning method that predicts short action chunks instead of single steps. It learns from teleoperated data and often achieves high success rates.
-
-
-
-
-<!-- A short demo is worth more than any description! Record a GIF/video of the policy
-running on your robot, upload it to this repo, and embed it here:
-<p align="center">
-  <img src="https://huggingface.co/<hf_user>/<policy_repo_id>/resolve/main/demo.gif" width="60%"/>
-</p>
--->
-
-This policy has been trained and pushed to the Hub using [LeRobot](https://github.com/huggingface/lerobot).
-
-Learn how to train and run it in the [LeRobot act guide](https://huggingface.co/docs/lerobot/main/en/act), or browse the [full documentation](https://huggingface.co/docs/lerobot/index).
+An ACT policy for an AgileX Piper arm that picks up a hexagonal bolt and places it in the matching opening, using two RGB cameras and joint positions. It is trained on 80 demonstrations that all start from the zero home pose. The `main` branch holds the final 100k-step checkpoint. The 80k-step checkpoint is kept on the `step-80000` branch (`revision="step-80000"`).
 
+> **Status:** final 100k-step checkpoint, not yet evaluated on the robot. It has seen each training frame about 94 times. Compare it on the robot with the 80k checkpoint on the `step-80000` branch before relying on it.
 
 ---
 
 ## Model Details
 
-- **License:** \[More Information Needed]
-- **Robot type:** `piper_follower`
-- **Cameras:** `overhead`, `angled`
+- **Method:** [ACT (Action Chunking with Transformers)](https://huggingface.co/papers/2304.13705), trained from scratch with [LeRobot](https://github.com/huggingface/lerobot)
+- **Robot:** AgileX Piper, single arm (`piper_follower`), joint-space control
+- **Task:** "Pick up the hexagonal bolt and place it in the matching opening."
+- **Control rate:** 30 Hz
+- **Developed by:** RoboticsCollectiveAachen
+- **License:** Apache 2.0
 
+### Architecture
 
-## Inputs & Outputs
-
-The policy consumes these observation features and produces these action features.
+| Setting | Value |
+| --- | --- |
+| Vision backbone | ResNet-18, initialized from ImageNet (`IMAGENET1K_V1`) |
+| Transformer | 4 encoder layers, 1 decoder layer, `dim_model` 512, 8 heads, FFN 3200 |
+| CVAE | enabled, latent dim 32, 4 encoder layers, KL weight 10 |
+| Chunk size / actions executed per chunk | 100 / 100 |
+| Dropout | 0.1 |
+| Parameters | 51.6 M |
 
-**Inputs**
+## Inputs & Outputs
 
-| Feature | Type | Shape |
+| Feature | Shape | Description |
 | --- | --- | --- |
-| `observation.state` | STATE | `(7,)` |
-| `observation.images.overhead` | VISUAL | `(3, 480, 640)` |
-| `observation.images.angled` | VISUAL | `(3, 480, 640)` |
+| `observation.images.overhead` | 3 × 480 × 640 | Overhead RGB camera |
+| `observation.images.angled` | 3 × 480 × 640 | Angled RGB camera |
+| `observation.state` | 7 | Measured positions: joints 1–6 (degrees), then gripper opening (mm) |
+| `action` (output) | 7 | **Absolute** follower targets in the same order and units as the state |
 
-**Outputs**
-
-| Feature | Type | Shape |
-| --- | --- | --- |
-| `action` | ACTION | `(7,)` |
+The policy predicts a chunk of 100 future actions (3.3 s at 30 FPS) per inference call.
+All inputs and outputs are normalized with mean/std statistics from the training dataset;
+the bundled pre/post-processors apply this automatically.
 
+## Training Data
 
-## Training Dataset
+[Samisaliveagain/piper-hexagonal-bolt-insertion-80-demos-zero-home-v1](https://huggingface.co/datasets/Samisaliveagain/piper-hexagonal-bolt-insertion-80-demos-zero-home-v1)
 
-- **Repository:** [Samisaliveagain/piper-hexagonal-bolt-insertion-80-demos-zero-home-v1](https://huggingface.co/datasets/Samisaliveagain/piper-hexagonal-bolt-insertion-80-demos-zero-home-v1)
-- **Episodes:** 80
-- **Frames:** 50921
-- **Frame rate:** 30 FPS
-- **Task(s):** "Pick up the hexagonal bolt and place it in the matching opening"
+- **Episodes:** 80 teleoperated demonstrations (Piper follower driven by a 6-joint leader with a gripper trigger)
+- **Frames:** 50,921 at 30 FPS (≈ 28 min). Episodes are 14–34 s long (mean 21 s)
+- **Recording limits:** firmware speed 40 %, max target offset 4 °/mm, joint 4 locked at 0 °
+- **Start pose:** every episode starts within 1 °/mm of all-zero joints with the gripper closed
 
 <a class="flex" href="https://huggingface.co/spaces/lerobot/visualize_dataset?path=Samisaliveagain/piper-hexagonal-bolt-insertion-80-demos-zero-home-v1">
 <img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/badges/resolve/main/visualize-this-dataset-xl.svg"/>
 <img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/badges/resolve/main/visualize-this-dataset-xl-dark.svg"/>
 </a>
 
-
-## Training Configuration
+## Training Procedure
 
 | Setting | Value |
 | --- | --- |
-| Training steps | 100000 |
-| Batch size | 16 |
-| Optimizer | adamw |
-| Learning rate | 2e-05 |
+| Hardware | 3 × NVIDIA TITAN Xp (12 GB), DDP |
+| Batch size | 16 per GPU (48 effective) |
+| Optimizer | AdamW, lr 2e-5 (backbone 2e-5), weight decay 1e-4, constant schedule |
+| Gradient clipping | 10.0 |
+| Precision | fp32 |
+| Steps (this checkpoint) | 100,000 (≈ 4.8 M samples, ≈ 94 epochs) |
 | Seed | 1000 |
-| LeRobot version | 0.6.2 |
 
----
-
-## How to Get Started with the Model
-
-New to LeRobot? These guides cover the full workflow:
-
-- **[Install LeRobot](https://huggingface.co/docs/lerobot/main/en/installation)** — set up the `lerobot` package.
-- **[Hardware setup](https://huggingface.co/docs/lerobot/main/en/hardware_guide)** — assemble, wire, and calibrate your robot and cameras.
-- **[Record data & train a policy](https://huggingface.co/docs/lerobot/en/il_robots)** — the end-to-end imitation-learning walkthrough.
-- **[CLI cheat-sheet](https://huggingface.co/docs/lerobot/main/en/cheat-sheet)** — quick reference for the `lerobot-*` commands.
-
-The short version to run and train this policy:
-
-### Run the policy on your robot
-
-```bash
-lerobot-rollout \
-  --strategy.type=base \
-  --robot.type=piper_follower \
-  --robot.port=<your_robot_port> \
-  --robot.cameras="{ <camera_1>: {type: opencv, index_or_path: <index_or_path>, width: 640, height: 480, fps: 30}, <camera_2>: {type: opencv, index_or_path: <index_or_path>, width: 640, height: 480, fps: 30}}" \
-  --policy.path=RoboticsCollectiveAachen/act_piper_hexagonal_bolt_insertion_zero_home \
-  --task="Pick up the hexagonal bolt and place it in the matching opening" \
-  --duration=60
+**Image augmentation** (applied on the GPU, up to 3 per frame, randomly chosen): brightness
+0.8–1.2, contrast 0.8–1.2, saturation 0.5–1.5, hue ±0.05, sharpness 0.5–1.5, and a random
+affine transform (±5° rotation, ±5 % translation).
+
+**Training loss** (action L1 in normalized units; the KL term falls to ≈0 within the first
+10k steps, which is typical for ACT):
+
+| Step | 5k | 10k | 20k | 30k | 40k | 50k | 60k | 70k | 80k | 90k | 100k |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| L1 loss | 0.103 | 0.074 | 0.055 | 0.046 | 0.038 | 0.035 | 0.032 | 0.030 | 0.028 | 0.026 | 0.025 |
+
+These are training-set losses. The dataset has no held-out split, so they do not measure
+overfitting or task success.
+
+## How to Use
+
+LeRobot has no built-in Piper robot driver, so `lerobot-rollout` cannot drive this
+arm directly. Load the policy inside your own 30 Hz control loop:
+
+```python
+import torch
+from lerobot.policies import make_pre_post_processors
+from lerobot.policies.act import ACTPolicy
+
+repo = "RoboticsCollectiveAachen/act_piper_hexagonal_bolt_insertion_zero_home"
+policy = ACTPolicy.from_pretrained(repo).to("cuda").eval()
+preprocessor, postprocessor = make_pre_post_processors(
+    policy.config, pretrained_path=repo
+)
+
+policy.reset()  # at the start of every episode
+while running:  # 30 Hz
+    obs = {
+        "observation.state": state,             # torch.float32, shape (7,): joints 1-6 [deg], gripper [mm]
+        "observation.images.overhead": img_ovh, # torch.float32, shape (3, 480, 640), RGB in [0, 1]
+        "observation.images.angled": img_ang,   # same format
+        "task": "Pick up the hexagonal bolt and place it in the matching opening.",
+    }
+    with torch.inference_mode():
+        action = postprocessor(policy.select_action(preprocessor(obs)))
+    send_absolute_joint_targets(action.squeeze(0))  # 7 values, same order/units as the state
 ```
 
-Replace the remaining `<...>` placeholders with your own values: `--robot.port` and the camera names/indices are specific to your machine, and the camera names must match the observation keys this policy was trained on.
-
-When `--strategy.type=base` is used the script doesn't record the episodes. Skipping duration will make the policy run indefinitely. For more information look at [rollout documentation](https://huggingface.co/docs/lerobot/main/en/inference).
+**Deployment notes**
 
-### Train your own policy
-
-```bash
-lerobot-train \
-  --dataset.repo_id=${HF_USER}/<dataset> \
-  --policy.type=act \
-  --output_dir=outputs/train/<policy_repo_id> \
-  --job_name=lerobot_training \
-  --policy.device=cuda \
-  --policy.repo_id=${HF_USER}/<policy_repo_id> \
-  --wandb.enable=true
-```
-
-_Writes checkpoints to `outputs/train/<policy_repo_id>/checkpoints/`._
-
----
+- **Start every rollout from the zero home pose**: all joint targets at 0 ° and the gripper closed. Every training episode begins there, so other start poses are out of distribution.
+- Apply the same safety limits used during recording: firmware speed 40 % and a maximum
+  target offset of 4 °/mm from the measured position. The training actions are the targets
+  *after* this limiting, so the policy has never seen large jumps.
+- Joint 4 was locked to a 0 ° target in the training data. Keep it locked, or force its target to 0.
+- Camera placement, resolution (640 × 480), and the `overhead`/`angled` key names must match
+  the recording setup.
+- By default the policy executes each 100-step chunk open-loop. For smoother, more reactive
+  motion near the insertion, enable temporal ensembling:
+  `policy.config.temporal_ensemble_coeff = 0.01` and `policy.config.n_action_steps = 1`
+  (one inference per control step; costs more GPU time).
 
 ## Evaluation
 
-<!-- Report real-robot results here: run the policy several times per task and count the
-successes. Delete the "No evaluation results" line and fill in this table instead:
+_No real-robot evaluation has been run on this checkpoint yet._
 
-| Task | Trials | Successes | Success rate |
-| ---- | ------ | --------- | ------------ |
-| pick the lego brick | 10 | 8 | 80% |
+<!-- Fill in after testing on the robot:
 
-Also worth noting: anything that affects difficulty (new object positions, lighting,
-distractors, a different robot of the same type, ...).
+| Setup | Trials | Successes | Success rate |
+| --- | --- | --- | --- |
+| Training configuration | 10 | ? | ?% |
 -->
 
-_No evaluation results have been provided for this policy yet._
+## Limitations
 
----
+- Trained on a single robot, cell, camera setup, and object. Expect degraded behavior under
+  different lighting, camera poses, bolt or fixture positions outside the demonstrated range,
+  or a different Piper unit.
+- The training demonstrations carry no success labels and were not exhaustively reviewed,
+  so the policy imitates whatever the operator did, including imperfect attempts.
+- Single-task policy. The task string is not used by ACT, so the policy cannot be steered with language.
+- Run it with a hand on the emergency stop. Nothing in the model enforces joint limits,
+  collision avoidance, or force limits.
+- 94 epochs over 80 demonstrations is a lot of training. The policy may reproduce the demonstrated
+  trajectories closely but generalize less to new bolt or fixture placements than an earlier checkpoint would.
 
 ## Citation
 
-If you use this policy, please cite the method linked in the description above, along with LeRobot:
-
 ```bibtex
+@inproceedings{zhao2023learning,
+    title     = {Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware},
+    author    = {Zhao, Tony Z. and Kumar, Vikash and Levine, Sergey and Finn, Chelsea},
+    booktitle = {Robotics: Science and Systems (RSS)},
+    year      = {2023},
+    eprint    = {2304.13705},
+    archivePrefix = {arXiv}
+}
+
 @misc{cadene2024lerobot,
     author = {Cadene, Remi and Alibert, Simon and Soare, Alexander and Gallouedec, Quentin and Zouitine, Adil and Palma, Steven and Kooijmans, Pepijn and Aractingi, Michel and Shukor, Mustafa and Aubakirova, Dana and Russi, Martino and Capuano, Francesco and Pascal, Caroline and Choghari, Jade and Moss, Jess and Wolf, Thomas},
     title = {LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch},
     howpublished = "\url{https://github.com/huggingface/lerobot}",
     year = {2024}
 }
-```
+```