Instructions to use Daankrol/smolvla-multiobj-lerobot-20k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LeRobot
How to use Daankrol/smolvla-multiobj-lerobot-20k with LeRobot:
# See https://github.com/huggingface/lerobot?tab=readme-ov-file#installation for more details git clone https://github.com/huggingface/lerobot.git cd lerobot pip install -e .[smolvla]
# Launch finetuning on your dataset python lerobot/scripts/train.py \ --policy.path=Daankrol/smolvla-multiobj-lerobot-20k \ --dataset.repo_id=lerobot/svla_so101_pickplace \ --batch_size=64 \ --steps=20000 \ --output_dir=outputs/train/my_smolvla \ --job_name=my_smolvla_training \ --policy.device=cuda \ --wandb.enable=true
# Run the policy using the record function python -m lerobot.record \ --robot.type=so101_follower \ --robot.port=/dev/ttyACM0 \ # <- Use your port --robot.id=my_blue_follower_arm \ # <- Use your robot id --robot.cameras="{ front: {type: opencv, index_or_path: 8, width: 640, height: 480, fps: 30}}" \ # <- Use your cameras --dataset.single_task="Grasp a lego block and put it in the bin." \ # <- Use the same task description you used in your dataset recording --dataset.repo_id=HF_USER/dataset_name \ # <- This will be the dataset name on HF Hub --dataset.episode_time_s=50 \ --dataset.num_episodes=10 \ --policy.path=Daankrol/smolvla-multiobj-lerobot-20k - Notebooks
- Google Colab
- Kaggle
SmolVLA multi-object pick-and-place, LeRobot 20k
This is the final 20,000-step LeRobot baseline checkpoint, fine-tuned from lerobot/smolvla_base on Daankrol/pick-and-place-multi-obj. It is a native LeRobot model, not the Studio checkpoint or an OpenVINO export.
The dataset's task text is:
pick up the brown cube and put it in the black box
The checkpoint predicts six joint-position targets for one follower arm. Autonomous inference does not need a leader arm. The instructions below use an SO-101 follower and, optionally, an SO-101 leader for calibration/manual testing. The dataset metadata calls the recording robot PhysicalAIRobot; it does not contain the original serial ports or calibration files.
Checkpoint details
| Setting | Value |
|---|---|
| Training steps | 20,000 |
| Dataset | 200 episodes; 180 training, 20 held out; seed 42 |
| Batch size | 64 |
| Training precision | BF16 mixed precision on an NVIDIA H100 |
| Trainable components | Action expert and state projection; vision encoder frozen |
| Learning rate | 1e-4; 100 warmup steps; cosine decay to 2.5e-6 |
| Action chunk / executed steps | 50 / 50 |
| Denoising steps | 10 |
| Dataset frame rate | 30 FPS |
| Final logged training loss | 0.034 |
| Final offline evaluation loss | 0.4978, with a 256-sample evaluation limit |
| Training run | Weights & Biases |
These losses are not robot success rates. This upload passed a strict weights load and synthetic CPU inference with the saved preprocessing and normalization files. It has not been evaluated on a physical robot after training.
1. Install LeRobot
Run these commands on the Linux computer connected to the robot and cameras. Use Python 3.12 and a CUDA-capable NVIDIA GPU for the rollout example.
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install 'lerobot[smolvla,feetech]==0.6.0'
python -c 'import torch; print(torch.__version__); assert torch.cuda.is_available(), "CUDA is not available"'
Install a PyTorch build compatible with your NVIDIA driver if the CUDA check fails. The training/load-check environment used LeRobot 0.6.0, PyTorch 2.11.0+cu128, torchvision 0.26.0+cu128, and Transformers 5.15.1. A clean installation of the command above has not been tested; LeRobot's dependency constraints may select a different Transformers version.
The first inference also downloads the SmolVLM configuration and tokenizer from HuggingFaceTB/SmolVLM2-500M-Video-Instruct. Allow internet access for that first run. --policy.load_vlm_weights=false below avoids downloading the original VLM weights because this repository already contains the full trained policy weights.
2. Find the follower and leader arm ports
Do this while no program is controlling the arms. Secure the follower so it cannot fall when torque is off.
lerobot-find-port
The tool asks you to disconnect a USB cable. Unplug only the follower's USB cable, press Enter, note the reported port, then reconnect it. Repeat for the leader if you have one. Label the cables so you do not swap the two arms.
On Linux, also check for stable device names:
ls -l /dev/serial/by-id/
Set these to the ports you found. These values are examples, not the recording setup's port assignments:
export FOLLOWER_PORT=/dev/ttyACM0
export LEADER_PORT=/dev/ttyACM1 # Only needed if you have a leader.
export FOLLOWER_ID=multiobj_follower
export LEADER_ID=multiobj_leader
You can use a /dev/serial/by-id/... path instead of /dev/ttyACM0. If serial access is denied, check the device's owning group. On distributions that use dialout, add your user with sudo usermod -aG dialout "$USER", then log out and back in. Do not run the policy as root to bypass permissions.
3. Reuse or calibrate the arms
Use the follower calibration and joint ranges from data collection when available. Normalized positions depend on those ranges: a different calibration can turn the same model output into a different physical pose. Reuse the corresponding robot ID and calibration directory rather than overwriting a known-good calibration.
For an uncalibrated setup, follow LeRobot's interactive calibration procedure:
lerobot-calibrate \
--robot.type=so101_follower \
--robot.port="$FOLLOWER_PORT" \
--robot.id="$FOLLOWER_ID" \
--robot.use_degrees=false
If you have a leader, calibrate it separately:
lerobot-calibrate \
--teleop.type=so101_leader \
--teleop.port="$LEADER_PORT" \
--teleop.id="$LEADER_ID" \
--teleop.use_degrees=false
Keep the same follower ID in calibration and rollout commands. You can pass --robot.calibration_dir=/path/to/calibration to both commands to reuse an existing LeRobot calibration directory. A newly calibrated arm still needs a manual check against the recording setup before autonomous use. Do not run motor setup on an already configured arm just to load this policy.
Keep --robot.use_degrees=false. This model uses normalized body joint positions in approximately [-100, 100] and gripper positions in [0, 100]. LeRobot 0.6 defaults to degrees, which is not the expected convention here. The state and action order is:
shoulder_pan.pos, shoulder_lift.pos, elbow_flex.pos,
wrist_flex.pos, wrist_roll.pos, gripper.pos
4. Find and identify the cameras
Close other programs using the cameras, then capture preview images:
lerobot-find-cameras opencv \
--output-dir=outputs/camera-check \
--record-time-s=3
The command prints the camera indices and saves images under outputs/camera-check. Open those images to identify each view. Cover one lens at a time and repeat if the views are unclear. On Linux, ls -l /dev/v4l/by-id/ can help match devices to physical cameras. Check indices again after reconnecting cameras or rebooting.
Use these two views in this exact mapping:
| Physical view / robot camera name | Model input |
|---|---|
overhead: fixed camera looking down at the workspace |
observation.images.camera1 |
gripper: wrist/gripper camera |
observation.images.camera2 |
The saved config also lists camera3, but training used only the two views above. With empty_cameras=0, this SmolVLA implementation skips the absent third camera. Do not add a third camera or duplicate a view to fill that slot.
The recorded images were RGB, 640x480 at 30 FPS. Match the training camera placement, orientation, and workspace as closely as possible. The model handles its internal resizing; do not swap overhead and gripper.
Set the actual numeric indices reported by camera discovery. The following are examples:
export OVERHEAD_CAMERA=0
export GRIPPER_CAMERA=2
If your hardware uses a different camera backend, configure that backend in --robot.cameras while keeping the names overhead and gripper.
5. Run a short autonomous trial
Before running, verify the calibration and joint directions manually, check both camera previews, and place the arm in a safe starting pose similar to the demonstrations. Keep people clear of the workspace and have a physical power cutoff within reach. A leader arm is not an emergency stop for this autonomous command.
The following command moves the follower. It runs for 15 seconds without recording or uploading a dataset:
lerobot-rollout \
--strategy.type=base \
--inference.type=sync \
--policy.path=Daankrol/smolvla-multiobj-lerobot-20k \
--policy.device=cuda \
--policy.load_vlm_weights=false \
--device=cuda \
--robot.type=so101_follower \
--robot.port="$FOLLOWER_PORT" \
--robot.id="$FOLLOWER_ID" \
--robot.use_degrees=false \
--robot.max_relative_target=5 \
--robot.cameras="{overhead: {type: opencv, index_or_path: ${OVERHEAD_CAMERA}, width: 640, height: 480, fps: 30}, gripper: {type: opencv, index_or_path: ${GRIPPER_CAMERA}, width: 640, height: 480, fps: 30}}" \
--rename_map='{"observation.images.overhead":"observation.images.camera1","observation.images.gripper":"observation.images.camera2"}' \
--task="pick up the brown cube and put it in the black box" \
--fps=30 \
--duration=15 \
--return_to_initial_position=false
Pass the rename map even though the saved preprocessor contains it: the rollout CLI overrides that processor from its own --rename_map argument.
max_relative_target=5 limits each joint's target change per command in normalized units. It is not collision avoidance, a speed guarantee, or a substitute for checking calibration. It can change the learned behavior. Only adjust it after supervised testing.
Ctrl+C requests shutdown; use the physical cutoff if motion is unsafe. This example disables the automatic return to the initial pose. LeRobot normally disables motor torque on disconnect, so support the arm safely before shutdown. Do not rely on software shutdown to hold its pose.
At 30 FPS, 50 queued actions cover about 1.7 seconds between policy predictions. Synchronous inference may miss the target rate on slower hardware. Increase --duration only after a short trial behaves as expected. Add --display_data=true if you want LeRobot's visualization.
Files
The repository contains the full model.safetensors, policy configuration, saved pre/postprocessing pipelines and normalization statistics, and the original training configuration. Keep those files together when downloading for offline use. The upload omits optimizer state, intermediate checkpoints, and training logs.
The base model revision was 5e8d12a6e2975b0e5e5fce7c8caf47c371d257b6; the dataset revision was abf1f62b51dd3441b827478829e0d559fa4a96ac.
- Downloads last month
- 17
Model tree for Daankrol/smolvla-multiobj-lerobot-20k
Base model
HuggingFaceTB/SmolLM2-360M