File size: 4,456 Bytes
3aeb818 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 | # GR00T DROID
The N1.7 base model supports DROID inference out of the box via the `OXE_DROID_RELATIVE_EEF_RELATIVE_JOINT` pretrain tag. A finetuned checkpoint is also available at [`nvidia/GR00T-N1.7-DROID`](https://huggingface.co/nvidia/GR00T-N1.7-DROID).
> **Note:** The DROID dataset contains multiple language instruction paraphrases per episode (`language_instruction`, `language_instruction_2`, `language_instruction_3`). These are used for language augmentation during training. At inference time, only the first language key is used.
## Data Format
The DROID embodiment expects the following modality structure:
| Modality | Keys | Dimensions |
|----------|------|------------|
| Video | `exterior_image_1_left`, `wrist_image_left` | 2 cameras |
| State | `eef_9d`, `gripper_position`, `joint_position` | 9D + 1D + 7D = 17D |
| Action | `eef_9d`, `gripper_position`, `joint_position` | 9D + 1D + 7D = 17D |
| Language | `annotation.language.language_instruction` | text |
Action representations:
- `eef_9d`: relative end-effector (XYZ + rotation 6D)
- `gripper_position`: absolute (1D)
- `joint_position`: relative joint positions (7D)
### Preparing DROID Demo Data
The full DROID dataset ([lerobot/droid_1.0.1](https://huggingface.co/datasets/lerobot/droid_1.0.1)) is ~358 GB with 95k+ episodes in LeRobot v3.0 format. To create a small sample for testing:
```bash
uv pip install jsonlines # one-time dependency
python scripts/download_droid_sample.py
```
This downloads the first data/video chunk (~170 MB) and extracts 3 episodes into `demo_data/droid_sample/` in GR00T LeRobot v2.0 format.
**Key conversion notes:**
- Source is LeRobot v3.0 (consolidated parquet + concatenated videos) — the script converts to v2.0 (per-episode parquet + per-episode mp4).
- Video keys in the raw dataset (`exterior_1_left`, `wrist_left`) differ from the model config keys (`exterior_image_1_left`, `wrist_image_left`). The data loader auto-maps by position — no manual renaming needed.
- Language instructions are loaded via the `task_index` column mapped through `tasks.jsonl`.
## 1. Standalone Inference (with demo data)
After preparing demo data, run inference directly (no server needed):
```bash
uv run python scripts/deployment/standalone_inference_script.py \
--model-path nvidia/GR00T-N1.7-3B \
--dataset-path demo_data/droid_sample \
--embodiment-tag OXE_DROID_RELATIVE_EEF_RELATIVE_JOINT \
--traj-ids 0 1 \
--inference-mode pytorch \
--execution-horizon 8
```
> **Note:** Episode 0 may have an empty language instruction. If inference fails on episode 0, try `--traj-ids 1 2`.
Expected zero-shot performance on the base model (not finetuned):
| Metric | Value |
|--------|-------|
| Average MSE | ~0.0149 |
| Average MAE | ~0.0753 |
| Inference per step (base) | ~262 ms (H100) |
| Inference per step (finetuned) | ~253 ms (H100) |
## 2. Inference Server (for real-world deployment)
### Using the base model (zero-shot):
```bash
uv run python gr00t/eval/run_gr00t_server.py \
--model-path nvidia/GR00T-N1.7-3B \
--embodiment-tag OXE_DROID_RELATIVE_EEF_RELATIVE_JOINT
```
### Using the finetuned model:
```bash
uv run python gr00t/eval/run_gr00t_server.py \
--model-path nvidia/GR00T-N1.7-DROID \
--embodiment-tag OXE_DROID_RELATIVE_EEF_RELATIVE_JOINT
```
## 3. Fine-tuning
Fine-tune the base model on DROID data using the shared launcher:
```bash
NUM_GPUS=8 MAX_STEPS=20000 GLOBAL_BATCH_SIZE=640 SAVE_STEPS=1000 uv run bash examples/finetune.sh \
--base-model-path nvidia/GR00T-N1.7-3B \
--dataset-path demo_data/droid_sample \
--embodiment-tag OXE_DROID_RELATIVE_EEF_RELATIVE_JOINT \
--output-dir /tmp/droid_finetune
```
> **Note:** The above uses the small `demo_data/droid_sample` (3 episodes) for quick validation. For production training, replace `--dataset-path` with the full DROID dataset.
## 4. Robot Control Script
1. Install the DROID package on the robot control laptop/workstation — [instructions](https://droid-dataset.github.io/droid/software-setup/host-installation.html#configuring-the-laptopworkstation)
2. Install dependencies for the GR00T control script in the environment from step 1:
```bash
pip install tyro pydantic numpy==1.26.4
```
3. Enter the camera IDs for your ZED cameras in `examples/DROID/main_gr00t.py`.
4. Start the control script:
```bash
python examples/DROID/main_gr00t.py --external-camera="left" # or "right"
```
|