SmolVLA multi-object pick-and-place, LeRobot 20k

This is the final 20,000-step LeRobot baseline checkpoint, fine-tuned from lerobot/smolvla_base on Daankrol/pick-and-place-multi-obj. It is a native LeRobot model, not the Studio checkpoint or an OpenVINO export.

The dataset's task text is:

pick up the brown cube and put it in the black box

The checkpoint predicts six joint-position targets for one follower arm. Autonomous inference does not need a leader arm. The instructions below use an SO-101 follower and, optionally, an SO-101 leader for calibration/manual testing. The dataset metadata calls the recording robot PhysicalAIRobot; it does not contain the original serial ports or calibration files.

Checkpoint details

Setting Value
Training steps 20,000
Dataset 200 episodes; 180 training, 20 held out; seed 42
Batch size 64
Training precision BF16 mixed precision on an NVIDIA H100
Trainable components Action expert and state projection; vision encoder frozen
Learning rate 1e-4; 100 warmup steps; cosine decay to 2.5e-6
Action chunk / executed steps 50 / 50
Denoising steps 10
Dataset frame rate 30 FPS
Final logged training loss 0.034
Final offline evaluation loss 0.4978, with a 256-sample evaluation limit
Training run Weights & Biases

These losses are not robot success rates. This upload passed a strict weights load and synthetic CPU inference with the saved preprocessing and normalization files. It has not been evaluated on a physical robot after training.

1. Install LeRobot

Run these commands on the Linux computer connected to the robot and cameras. Use Python 3.12 and a CUDA-capable NVIDIA GPU for the rollout example.

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install 'lerobot[smolvla,feetech]==0.6.0'
python -c 'import torch; print(torch.__version__); assert torch.cuda.is_available(), "CUDA is not available"'

Install a PyTorch build compatible with your NVIDIA driver if the CUDA check fails. The training/load-check environment used LeRobot 0.6.0, PyTorch 2.11.0+cu128, torchvision 0.26.0+cu128, and Transformers 5.15.1. A clean installation of the command above has not been tested; LeRobot's dependency constraints may select a different Transformers version.

The first inference also downloads the SmolVLM configuration and tokenizer from HuggingFaceTB/SmolVLM2-500M-Video-Instruct. Allow internet access for that first run. --policy.load_vlm_weights=false below avoids downloading the original VLM weights because this repository already contains the full trained policy weights.

2. Find the follower and leader arm ports

Do this while no program is controlling the arms. Secure the follower so it cannot fall when torque is off.

lerobot-find-port

The tool asks you to disconnect a USB cable. Unplug only the follower's USB cable, press Enter, note the reported port, then reconnect it. Repeat for the leader if you have one. Label the cables so you do not swap the two arms.

On Linux, also check for stable device names:

ls -l /dev/serial/by-id/

Set these to the ports you found. These values are examples, not the recording setup's port assignments:

export FOLLOWER_PORT=/dev/ttyACM0
export LEADER_PORT=/dev/ttyACM1  # Only needed if you have a leader.
export FOLLOWER_ID=multiobj_follower
export LEADER_ID=multiobj_leader

You can use a /dev/serial/by-id/... path instead of /dev/ttyACM0. If serial access is denied, check the device's owning group. On distributions that use dialout, add your user with sudo usermod -aG dialout "$USER", then log out and back in. Do not run the policy as root to bypass permissions.

3. Reuse or calibrate the arms

Use the follower calibration and joint ranges from data collection when available. Normalized positions depend on those ranges: a different calibration can turn the same model output into a different physical pose. Reuse the corresponding robot ID and calibration directory rather than overwriting a known-good calibration.

For an uncalibrated setup, follow LeRobot's interactive calibration procedure:

lerobot-calibrate \
  --robot.type=so101_follower \
  --robot.port="$FOLLOWER_PORT" \
  --robot.id="$FOLLOWER_ID" \
  --robot.use_degrees=false

If you have a leader, calibrate it separately:

lerobot-calibrate \
  --teleop.type=so101_leader \
  --teleop.port="$LEADER_PORT" \
  --teleop.id="$LEADER_ID" \
  --teleop.use_degrees=false

Keep the same follower ID in calibration and rollout commands. You can pass --robot.calibration_dir=/path/to/calibration to both commands to reuse an existing LeRobot calibration directory. A newly calibrated arm still needs a manual check against the recording setup before autonomous use. Do not run motor setup on an already configured arm just to load this policy.

Keep --robot.use_degrees=false. This model uses normalized body joint positions in approximately [-100, 100] and gripper positions in [0, 100]. LeRobot 0.6 defaults to degrees, which is not the expected convention here. The state and action order is:

shoulder_pan.pos, shoulder_lift.pos, elbow_flex.pos,
wrist_flex.pos, wrist_roll.pos, gripper.pos

4. Find and identify the cameras

Close other programs using the cameras, then capture preview images:

lerobot-find-cameras opencv \
  --output-dir=outputs/camera-check \
  --record-time-s=3

The command prints the camera indices and saves images under outputs/camera-check. Open those images to identify each view. Cover one lens at a time and repeat if the views are unclear. On Linux, ls -l /dev/v4l/by-id/ can help match devices to physical cameras. Check indices again after reconnecting cameras or rebooting.

Use these two views in this exact mapping:

Physical view / robot camera name Model input
overhead: fixed camera looking down at the workspace observation.images.camera1
gripper: wrist/gripper camera observation.images.camera2

The saved config also lists camera3, but training used only the two views above. With empty_cameras=0, this SmolVLA implementation skips the absent third camera. Do not add a third camera or duplicate a view to fill that slot.

The recorded images were RGB, 640x480 at 30 FPS. Match the training camera placement, orientation, and workspace as closely as possible. The model handles its internal resizing; do not swap overhead and gripper.

Set the actual numeric indices reported by camera discovery. The following are examples:

export OVERHEAD_CAMERA=0
export GRIPPER_CAMERA=2

If your hardware uses a different camera backend, configure that backend in --robot.cameras while keeping the names overhead and gripper.

5. Run a short autonomous trial

Before running, verify the calibration and joint directions manually, check both camera previews, and place the arm in a safe starting pose similar to the demonstrations. Keep people clear of the workspace and have a physical power cutoff within reach. A leader arm is not an emergency stop for this autonomous command.

The following command moves the follower. It runs for 15 seconds without recording or uploading a dataset:

lerobot-rollout \
  --strategy.type=base \
  --inference.type=sync \
  --policy.path=Daankrol/smolvla-multiobj-lerobot-20k \
  --policy.device=cuda \
  --policy.load_vlm_weights=false \
  --device=cuda \
  --robot.type=so101_follower \
  --robot.port="$FOLLOWER_PORT" \
  --robot.id="$FOLLOWER_ID" \
  --robot.use_degrees=false \
  --robot.max_relative_target=5 \
  --robot.cameras="{overhead: {type: opencv, index_or_path: ${OVERHEAD_CAMERA}, width: 640, height: 480, fps: 30}, gripper: {type: opencv, index_or_path: ${GRIPPER_CAMERA}, width: 640, height: 480, fps: 30}}" \
  --rename_map='{"observation.images.overhead":"observation.images.camera1","observation.images.gripper":"observation.images.camera2"}' \
  --task="pick up the brown cube and put it in the black box" \
  --fps=30 \
  --duration=15 \
  --return_to_initial_position=false

Pass the rename map even though the saved preprocessor contains it: the rollout CLI overrides that processor from its own --rename_map argument.

max_relative_target=5 limits each joint's target change per command in normalized units. It is not collision avoidance, a speed guarantee, or a substitute for checking calibration. It can change the learned behavior. Only adjust it after supervised testing.

Ctrl+C requests shutdown; use the physical cutoff if motion is unsafe. This example disables the automatic return to the initial pose. LeRobot normally disables motor torque on disconnect, so support the arm safely before shutdown. Do not rely on software shutdown to hold its pose.

At 30 FPS, 50 queued actions cover about 1.7 seconds between policy predictions. Synchronous inference may miss the target rate on slower hardware. Increase --duration only after a short trial behaves as expected. Add --display_data=true if you want LeRobot's visualization.

Files

The repository contains the full model.safetensors, policy configuration, saved pre/postprocessing pipelines and normalization statistics, and the original training configuration. Keep those files together when downloading for offline use. The upload omits optimizer state, intermediate checkpoints, and training logs.

The base model revision was 5e8d12a6e2975b0e5e5fce7c8caf47c371d257b6; the dataset revision was abf1f62b51dd3441b827478829e0d559fa4a96ac.

Downloads last month
17
Safetensors
Model size
0.5B params
Tensor type
F32
·
BF16
·
Video Preview
loading

Model tree for Daankrol/smolvla-multiobj-lerobot-20k

Dataset used to train Daankrol/smolvla-multiobj-lerobot-20k