# GR00T DROID

The N1.7 base model supports DROID inference out of the box via the `OXE_DROID_RELATIVE_EEF_RELATIVE_JOINT` pretrain tag. A finetuned checkpoint is also available at [`nvidia/GR00T-N1.7-DROID`](https://huggingface.co/nvidia/GR00T-N1.7-DROID).

> **Note:** The DROID dataset contains multiple language instruction paraphrases per episode (`language_instruction`, `language_instruction_2`, `language_instruction_3`). These are used for language augmentation during training. At inference time, only the first language key is used.

## Data Format

The DROID embodiment expects the following modality structure:

| Modality | Keys | Dimensions |
|----------|------|------------|
| Video | `exterior_image_1_left`, `wrist_image_left` | 2 cameras |
| State | `eef_9d`, `gripper_position`, `joint_position` | 9D + 1D + 7D = 17D |
| Action | `eef_9d`, `gripper_position`, `joint_position` | 9D + 1D + 7D = 17D |
| Language | `annotation.language.language_instruction` | text |

Action representations:
- `eef_9d`: relative end-effector (XYZ + rotation 6D)
- `gripper_position`: absolute (1D)
- `joint_position`: relative joint positions (7D)

### Preparing DROID Demo Data

The full DROID dataset ([lerobot/droid_1.0.1](https://huggingface.co/datasets/lerobot/droid_1.0.1)) is ~358 GB with 95k+ episodes in LeRobot v3.0 format. To create a small sample for testing:

```bash
uv pip install jsonlines   # one-time dependency
python scripts/download_droid_sample.py
```

This downloads the first data/video chunk (~170 MB) and extracts 3 episodes into `demo_data/droid_sample/` in GR00T LeRobot v2.0 format.

**Key conversion notes:**
- Source is LeRobot v3.0 (consolidated parquet + concatenated videos) — the script converts to v2.0 (per-episode parquet + per-episode mp4).
- Video keys in the raw dataset (`exterior_1_left`, `wrist_left`) differ from the model config keys (`exterior_image_1_left`, `wrist_image_left`). The data loader auto-maps by position — no manual renaming needed.
- Language instructions are loaded via the `task_index` column mapped through `tasks.jsonl`.

## 1. Standalone Inference (with demo data)

After preparing demo data, run inference directly (no server needed):

```bash
uv run python scripts/deployment/standalone_inference_script.py \
    --model-path nvidia/GR00T-N1.7-3B \
    --dataset-path demo_data/droid_sample \
    --embodiment-tag OXE_DROID_RELATIVE_EEF_RELATIVE_JOINT \
    --traj-ids 0 1 \
    --inference-mode pytorch \
    --execution-horizon 8
```

> **Note:** Episode 0 may have an empty language instruction. If inference fails on episode 0, try `--traj-ids 1 2`.

Expected zero-shot performance on the base model (not finetuned):

| Metric | Value |
|--------|-------|
| Average MSE | ~0.0149 |
| Average MAE | ~0.0753 |
| Inference per step (base) | ~262 ms (H100) |
| Inference per step (finetuned) | ~253 ms (H100) |

## 2. Inference Server (for real-world deployment)

### Using the base model (zero-shot):

```bash
uv run python gr00t/eval/run_gr00t_server.py \
    --model-path nvidia/GR00T-N1.7-3B \
    --embodiment-tag OXE_DROID_RELATIVE_EEF_RELATIVE_JOINT
```

### Using the finetuned model:

```bash
uv run python gr00t/eval/run_gr00t_server.py \
    --model-path nvidia/GR00T-N1.7-DROID \
    --embodiment-tag OXE_DROID_RELATIVE_EEF_RELATIVE_JOINT
```

## 3. Fine-tuning

Fine-tune the base model on DROID data using the shared launcher:

```bash
NUM_GPUS=8 MAX_STEPS=20000 GLOBAL_BATCH_SIZE=640 SAVE_STEPS=1000 uv run bash examples/finetune.sh \
    --base-model-path nvidia/GR00T-N1.7-3B \
    --dataset-path demo_data/droid_sample \
    --embodiment-tag OXE_DROID_RELATIVE_EEF_RELATIVE_JOINT \
    --output-dir /tmp/droid_finetune
```

> **Note:** The above uses the small `demo_data/droid_sample` (3 episodes) for quick validation. For production training, replace `--dataset-path` with the full DROID dataset.

## 4. Robot Control Script

1. Install the DROID package on the robot control laptop/workstation — [instructions](https://droid-dataset.github.io/droid/software-setup/host-installation.html#configuring-the-laptopworkstation)

2. Install dependencies for the GR00T control script in the environment from step 1:
```bash
pip install tyro pydantic numpy==1.26.4
```

3. Enter the camera IDs for your ZED cameras in `examples/DROID/main_gr00t.py`.

4. Start the control script:
```bash
python examples/DROID/main_gr00t.py --external-camera="left" # or "right"
```