RoboticsCollectiveAachen/smolvla_piper_hex_bolt_insertion_80

SmolVLA: Piper Hexagonal Bolt Insertion (zero-home, 80 demos)

A SmolVLA vision-language-action policy for an AgileX Piper arm that picks up a hexagonal bolt and places it in the matching opening, using two RGB cameras, joint positions and a language prompt. It is fine-tuned from lerobot/smolvla_base on 80 demonstrations that all start from the zero home pose. The main branch holds the 20k-step checkpoint. A continuation to 40k total steps is on the step-40k branch (revision="step-40k").

Status: 20k-step checkpoint, not yet evaluated on the robot. The 40k checkpoint on the step-40k branch reached a lower training loss (0.010 vs 0.016). Compare both on the robot before relying on either.

---

Model Details

• Method: SmolVLA, fine-tuned from lerobot/smolvla_base with LeRobot. Only the action expert was trained; the vision encoder and language model stayed frozen.
• Robot: AgileX Piper, single arm (piper_follower), joint-space control
• Task: "Pick up the hexagonal bolt and place it in the matching opening."
• Control rate: 30 Hz
• Developed by: RoboticsCollectiveAachen
• License: Apache 2.0

Architecture

Setting	Value
Vision-language backbone	SmolVLM2-500M-Video-Instruct, first 16 layers (frozen)
Action expert	Transformer, width 0.75 × the VLM's; cross-attention to the VLM, interleaved with self-attention every 2 layers
Action head	Flow matching, 10 integration steps at inference
Image preprocessing	Resized with padding to 512 × 512
Language	Task prompt, up to 48 tokens
Chunk size / actions executed per chunk	50 / 50
Parameters	450 M total, 100 M trained (action expert + state projection)

Inputs & Outputs

Feature	Shape	Description
observation.images.overhead	3 × 480 × 640	Overhead RGB camera
observation.images.angled	3 × 480 × 640	Angled RGB camera
observation.state	7	Measured positions: joints 1–6 (degrees), then gripper opening (mm)
task	string	The language prompt above. Use the exact wording
action (output)	7	Absolute follower targets in the same order and units as the state

The policy predicts a chunk of 50 future actions (1.67 s at 30 FPS) per inference call. State and action are normalized with mean/std statistics from the training dataset; images go to the VLM unnormalized. The bundled pre/post-processors apply this automatically, and the preprocessor also renames the cameras to the names the base model uses (overhead → camera1, angled → camera2).

The stored config lists observation.state with shape 6 and a third camera, camera3. Both are inherited from smolvla_base. Pass all 7 state values (all 7 reach the model) and only the two cameras above (a missing third camera is skipped).

Training Data

Samisaliveagain/piper-hexagonal-bolt-insertion-80-demos-zero-home-v1

• Episodes: 80 teleoperated demonstrations (Piper follower driven by a 6-joint leader with a gripper trigger)
• Frames: 50,921 at 30 FPS (≈ 28 min). Episodes are 14–34 s long (mean 21 s)
• Recording limits: firmware speed 40 %, max target offset 4 °/mm, joint 4 locked at 0 °
• Start pose: every episode starts within 1 °/mm of all-zero joints with the gripper closed

Training Procedure

Setting	Value
Hardware	1 × NVIDIA Quadro RTX 6000 (24 GB)
Batch size	32
Optimizer	AdamW, lr 1e-4, betas (0.9, 0.95), weight decay 1e-10
LR schedule	1,000 warmup steps, cosine decay to 2.5e-6 over 20k steps
Gradient clipping	10.0
Precision	fp32 weights, fp16 autocast (the GPU has no bf16 support)
Steps (this checkpoint)	20,000 (≈ 640 k samples, ≈ 12.6 epochs), 4 h 05 min
Image augmentation	none
Seed	1000

Training loss (flow-matching MSE, in normalized units):

Step	2k	4k	6k	8k	10k	12k	14k	16k	18k	20k
Loss	0.101	0.066	0.051	0.041	0.033	0.028	0.022	0.020	0.018	0.016

These are training-set losses. The dataset has no held-out split, so they do not measure overfitting or task success. The loss was still falling at 20k steps; it flattened only because the learning rate had decayed.

How to Use

LeRobot has no built-in Piper robot driver, so lerobot-rollout cannot drive this arm directly. Load the policy inside your own 30 Hz control loop:

import torch
from lerobot.policies import make_pre_post_processors
from lerobot.policies.smolvla import SmolVLAPolicy

repo = "RoboticsCollectiveAachen/smolvla_piper_hex_bolt_insertion_80"
policy = SmolVLAPolicy.from_pretrained(repo).to("cuda").eval()
preprocessor, postprocessor = make_pre_post_processors(
    policy.config, pretrained_path=repo
)

policy.reset()  # at the start of every episode
while running:  # 30 Hz
    obs = {
        "observation.state": state,             # torch.float32, shape (7,): joints 1-6 [deg], gripper [mm]
        "observation.images.overhead": img_ovh, # torch.float32, shape (3, 480, 640), RGB in [0, 1]
        "observation.images.angled": img_ang,   # same format
        "task": "Pick up the hexagonal bolt and place it in the matching opening.",
    }
    with torch.inference_mode():
        action = postprocessor(policy.select_action(preprocessor(obs)))
    send_absolute_joint_targets(action.squeeze(0))  # 7 values, same order/units as the state

Inference cost (measured on a Quadro RTX 6000 with fp16 autocast): about 1.8 GB of VRAM and about 290 ms per 50-action chunk. from_pretrained loads the VLM in bf16, which is fast on RTX 30-series and newer GPUs. On older GPUs without bf16 support (RTX 20-series / Turing), wrap inference in torch.autocast("cuda", dtype=torch.float16) after calling policy.float().

Deployment notes

• Start every rollout from the zero home pose: all joint targets at 0 ° and the gripper closed. Every training episode begins there, so other start poses are out of distribution.
• Apply the same safety limits used during recording: firmware speed 40 % and a maximum target offset of 4 °/mm from the measured position. The training actions are the targets after this limiting, so the policy has never seen large jumps.
• Joint 4 was locked to a 0 ° target in the training data. Keep it locked, or force its target to 0.
• Camera placement, resolution (640 × 480), and the overhead/angled key names must match the recording setup.
• The prompt conditions the policy. Use the exact training prompt; other wording is untested.
• By default each 50-step chunk executes open-loop, with a ~0.3 s pause for inference between chunks. To avoid the pause, run inference in a separate thread, or use LeRobot's real-time chunking (policy.config.rtc_config). For more reactive motion near the insertion, lower policy.config.n_action_steps (for example to 25) at the cost of more inference calls.

Evaluation

No real-robot evaluation has been run on this checkpoint yet.

Limitations

• Trained on a single robot, cell, camera setup, and object. Expect degraded behavior under different lighting, camera poses, bolt or fixture positions outside the demonstrated range, or a different Piper unit.
• The training demonstrations carry no success labels and were not exhaustively reviewed, so the policy imitates whatever the operator did, including imperfect attempts.
• Only the action expert was fine-tuned. The vision encoder was never adapted to this cell; a full fine-tune may perceive the bolt and opening more precisely.
• Trained without image augmentation.
• Run it with a hand on the emergency stop. Nothing in the model enforces joint limits, collision avoidance, or force limits.

Citation

@article{shukor2025smolvla,
    title   = {SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics},
    author  = {Shukor, Mustafa and Aubakirova, Dana and Capuano, Francesco and Kooijmans, Pepijn and Palma, Steven and Zouitine, Adil and Aractingi, Michel and Pascal, Caroline and Russi, Martino and Marafioti, Andres and Alibert, Simon and Cord, Matthieu and Wolf, Thomas and Cadene, Remi},
    journal = {arXiv preprint arXiv:2506.01844},
    year    = {2025}
}

@misc{cadene2024lerobot,
    author = {Cadene, Remi and Alibert, Simon and Soare, Alexander and Gallouedec, Quentin and Zouitine, Adil and Palma, Steven and Kooijmans, Pepijn and Aractingi, Michel and Shukor, Mustafa and Aubakirova, Dana and Russi, Martino and Capuano, Francesco and Pascal, Caroline and Choghari, Jade and Moss, Jess and Wolf, Thomas},
    title = {LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch},
    howpublished = "\url{https://github.com/huggingface/lerobot}",
    year = {2024}
}