Finetuning Models for the SO100/SO101 Robot

This guide shows how to finetune a dataset collected from the SO101 robot, and evaluate the model on the real robot.

Setup

Set an environment variable pointing at your local clone of this repository, then cd into it. Every step below references $GR00T_REPO, so export it in each terminal you open (or add it to your shell profile).

export GR00T_REPO=~/Isaac-GR00T # update to match your path
cd "$GR00T_REPO"

Dataset

To collect the dataset via teleoperation, calibrate the robot, and determine camera indices, please refer to the official documentation in lerobot: https://huggingface.co/docs/lerobot/il_robots

If you do not have a dataset, you can use this one as a basic test of the workflow.

Dataset Path: izuluaga/finish_sandwich

Visualize it with this link

Converting the Dataset

From the repository root, run the following command to convert the dataset to the LeRobot v2 format necessary for finetuning.

cd "$GR00T_REPO"
uv run --project scripts/lerobot_conversion \
  python scripts/lerobot_conversion/convert_v3_to_v2.py \
  --repo-id izuluaga/finish_sandwich \
  --root examples/SO100/finish_sandwich_lerobot

Copy the modality.json file for the SO100 to the root of the dataset.

cp examples/SO100/modality.json examples/SO100/finish_sandwich_lerobot/izuluaga/finish_sandwich/meta/modality.json

Finetuning the Model

Run the shared finetune launcher directly, this will use relative actions by default for all axes except the gripper.

CUDA_VISIBLE_DEVICES=0 NUM_GPUS=1 uv run bash examples/finetune.sh \
  --base-model-path nvidia/GR00T-N1.7-3B \
  --dataset-path examples/SO100/finish_sandwich_lerobot/izuluaga/finish_sandwich \
  --modality-config-path examples/SO100/so100_config.py \
  --embodiment-tag NEW_EMBODIMENT \
  --output-dir /tmp/so100_finetune

Open-Loop Evaluation

Evaluate the finetuned model with the following command:

uv run python gr00t/eval/open_loop_eval.py \
  --dataset-path examples/SO100/finish_sandwich_lerobot/izuluaga/finish_sandwich/ \
  --embodiment-tag NEW_EMBODIMENT \
  --model-path /tmp/so100_finetune/checkpoint-10000 \
  --traj-ids 0 \
  --execution-horizon 16 \
  --steps 400

The script saves one plot per trajectory to /tmp/open_loop_eval/traj_<id>.jpeg by default (pass --save-plot-path to override). Open the output directory to view the results:

xdg-open /tmp/open_loop_eval # macOS: open /tmp/open_loop_eval

Evaluation Results

The evaluation produces visualizations comparing predicted actions against ground truth trajectories:

To read these numbers and decide whether your fine-tune is working, see Interpreting the Result: Is My Fine-tune Working?.

Closed-Loop Evaluation

Please refer to eval_so100.py for how to write SO100 deployment code using Policy API.

From the gr00t/eval/real_robot/SO100 directory, set up client side dependencies:

cd "$GR00T_REPO/gr00t/eval/real_robot/SO100"
uv sync
uv pip install --no-deps -e ../../../../

Start the policy server:

cd "$GR00T_REPO"
uv run python gr00t/eval/run_gr00t_server.py \
  --model-path /tmp/so100_finetune/checkpoint-10000 \
  --embodiment-tag NEW_EMBODIMENT

In a second terminal, navigate to the gr00t/eval/real_robot/SO100 directory.

cd "$GR00T_REPO/gr00t/eval/real_robot/SO100"

Run the eval script as the client from the gr00t/eval/real_robot/SO100 environment created above:

# Update these to match the address and indices assigned by your OS
ROBOT_PORT=/dev/ttyACM2
ROBOT_ID=orange_follower
WRIST_CAM_IDX=2
FRONT_CAM_IDX=6
# Task specific prompt
PROMPT="finish the ham cheese olives sandwich"

uv run --no-sync python eval_so100.py \
  --robot.type=so101_follower \
  --robot.port="$ROBOT_PORT" \
  --robot.id="$ROBOT_ID" \
  --robot.cameras="{ front: {type: opencv, index_or_path: $FRONT_CAM_IDX, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: $WRIST_CAM_IDX, width: 640, height: 480, fps: 30}}" \
  --policy_host=localhost \
  --policy_port=5555 \
  --lang_instruction="$PROMPT"