# RoboCasa Evaluation Benchmark [RoboCasa](https://robocasa.ai/) is a large-scale simulation framework for training generally capable robots to perform everyday tasks, featuring realistic kitchen environments with over 2,500 3D assets and 100 diverse manipulation tasks. This evaluation benchmark uses RoboCasa with the Panda robot equipped with an Omron gripper to test household manipulation tasks including operating kitchen appliances, pick-and-place operations, and interacting with doors, drawers, and various objects. To run it on N1.7, finetune from the base model (`nvidia/GR00T-N1.7-3B`) using the instructions below — RoboCasa is not in the N1.7 pretrained embodiment set, so finetuning is required before evaluation. --- ## Checkpoint Results - GR00T 1.6: [nvidia/GR00T-N1.6-3B](https://huggingface.co/nvidia/GR00T-N1.6-3B) - GR00T 1.7: finetuned checkpoint | Task | GR00T 1.6 | GR00T 1.7 | | ---- | --------- | --------- | | `robocasa_panda_omron/CoffeeSetupMug_PandaOmron_Env` | 31.0% | 30.0% | | `robocasa_panda_omron/CoffeeServeMug_PandaOmron_Env` | 63.5% | 85.0% | | `robocasa_panda_omron/CoffeePressButton_PandaOmron_Env` | 98.5% | 100.0% | | `robocasa_panda_omron/OpenSingleDoor_PandaOmron_Env` | 81.5% | 90.0% | | `robocasa_panda_omron/OpenDoubleDoor_PandaOmron_Env` | 39.0% | 25.0% | | `robocasa_panda_omron/CloseSingleDoor_PandaOmron_Env` | 96.0% | 95.0% | | `robocasa_panda_omron/CloseDoubleDoor_PandaOmron_Env` | 88.5% | 80.0% | | `robocasa_panda_omron/OpenDrawer_PandaOmron_Env` | 81.1% | 95.0% | | `robocasa_panda_omron/CloseDrawer_PandaOmron_Env` | 100.0% | 100.0% | | `robocasa_panda_omron/TurnOnMicrowave_PandaOmron_Env` | 91.5% | 95.0% | | `robocasa_panda_omron/TurnOffMicrowave_PandaOmron_Env` | 96.0% | 95.0% | | `robocasa_panda_omron/PnPCounterToCab_PandaOmron_Env` | 47.5% | 60.0% | | `robocasa_panda_omron/PnPCabToCounter_PandaOmron_Env` | 41.0% | 65.0% | | `robocasa_panda_omron/PnPCounterToSink_PandaOmron_Env` | 46.0% | 60.0% | | `robocasa_panda_omron/PnPSinkToCounter_PandaOmron_Env` | 50.0% | 65.0% | | `robocasa_panda_omron/PnPCounterToMicrowave_PandaOmron_Env` | 19.0% | 30.0% | | `robocasa_panda_omron/PnPMicrowaveToCounter_PandaOmron_Env` | 24.5% | 19.0% | | `robocasa_panda_omron/PnPCounterToStove_PandaOmron_Env` | 63.2% | 60.0% | | `robocasa_panda_omron/PnPStoveToCounter_PandaOmron_Env` | 54.5% | 65.0% | | `robocasa_panda_omron/TurnOnSinkFaucet_PandaOmron_Env` | 89.0% | 95.0% | | `robocasa_panda_omron/TurnOffSinkFaucet_PandaOmron_Env` | 93.5% | 100.0% | | `robocasa_panda_omron/TurnSinkSpout_PandaOmron_Env` | 87.0% | 80.0% | | `robocasa_panda_omron/TurnOnStove_PandaOmron_Env` | 76.5% | 85.0% | | `robocasa_panda_omron/TurnOffStove_PandaOmron_Env` | 31.0% | 25.0% | | **Average** | 66.22% | 70.8% | --- # Finetune Use the RoboCasa Panda Omron LeRobot datasets from Hugging Face, then launch the shared finetune script. These datasets already include the LeRobot metadata and statistics expected by GR00T and match the built-in N1.7 `ROBOCASA_PANDA_OMRON` finetuning tag. For a small local smoke-test dataset, download one task: ```bash uv run hf download nvidia/PhysicalAI-Robotics-GR00T-X-Embodiment-Sim \ --repo-type dataset \ --include "single_panda_gripper.OpenDrawer/**" \ --local-dir /root/.cache/g00t/datasets/PhysicalAI-Robotics-GR00T-X-Embodiment-Sim DATASET_PATH=/root/.cache/g00t/datasets/PhysicalAI-Robotics-GR00T-X-Embodiment-Sim/single_panda_gripper.OpenDrawer uv run python scripts/repair_lerobot_metadata.py "$DATASET_PATH" \ --embodiment-tag ROBOCASA_PANDA_OMRON ``` For the full benchmark training set, download the `single_panda_gripper.*` task directories and pass an `os.pathsep`-separated list of dataset directories to `--dataset-path`: ```bash uv run hf download nvidia/PhysicalAI-Robotics-GR00T-X-Embodiment-Sim \ --repo-type dataset \ --include "single_panda_gripper.*/**" \ --local-dir /root/.cache/g00t/datasets/PhysicalAI-Robotics-GR00T-X-Embodiment-Sim DATASET_PATH=$(find /root/.cache/g00t/datasets/PhysicalAI-Robotics-GR00T-X-Embodiment-Sim \ -maxdepth 1 -type d -name "single_panda_gripper.*" | sort | paste -sd: -) uv run python scripts/repair_lerobot_metadata.py "$DATASET_PATH" \ --embodiment-tag ROBOCASA_PANDA_OMRON ``` ```bash NUM_GPUS=8 MAX_STEPS=60000 GLOBAL_BATCH_SIZE=512 SAVE_STEPS=2000 uv run bash examples/finetune.sh \ --base-model-path nvidia/GR00T-N1.7-3B \ --dataset-path "$DATASET_PATH" \ --embodiment-tag ROBOCASA_PANDA_OMRON \ --output-dir /tmp/robocasa_finetune ``` RoboCasa Panda Omron is not present in the base model checkpoint; use a checkpoint finetuned with `ROBOCASA_PANDA_OMRON` for evaluation. # Evaluate checkpoint First, complete the [one-time simulation environment setup](../../README.md#one-time-simulation-environment-setup), then run this benchmark's setup script (only needed once per benchmark): ```bash bash gr00t/eval/sim/robocasa/setup_RoboCasa.sh ``` #### Downloading RoboCasa Datasets (Optional) To download RoboCasa demonstration datasets, you **must** use the robocasa venv created by the setup script above (the main project venv does not have `robosuite` installed, which `robocasa` requires at import time): ```bash # Download human demonstration datasets gr00t/eval/sim/robocasa/robocasa_uv/.venv/bin/python \ external_dependencies/robocasa/robocasa/scripts/download_datasets.py --ds_types human_im # Download machine-generated datasets gr00t/eval/sim/robocasa/robocasa_uv/.venv/bin/python \ external_dependencies/robocasa/robocasa/scripts/download_datasets.py --ds_types mg ``` > **Note:** Running `python -m robocasa.scripts.download_datasets` from the main project environment will fail because `robocasa` depends on `robosuite`, which is only installed in the robocasa venv. Then, run client server evaluation under the project root directory in separate terminals: **Terminal 1 - Server:** ```bash uv run python gr00t/eval/run_gr00t_server.py \ --model-path \ --embodiment-tag ROBOCASA_PANDA_OMRON \ --use-sim-policy-wrapper ``` **Terminal 2 - Client:** ```bash gr00t/eval/sim/robocasa/robocasa_uv/.venv/bin/python gr00t/eval/rollout_policy.py \ --n-episodes 10 \ --policy-client-host 127.0.0.1 \ --policy-client-port 5555 \ --max-episode-steps 720 \ --env-name robocasa_panda_omron/OpenDrawer_PandaOmron_Env \ --n-action-steps 8 \ --n-envs 5 ```