File size: 6,401 Bytes
c4f8e4a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 | # H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulation
### 📝[Paper](https://arxiv.org/abs/2507.23523) | 🌍[Project Page](https://embodiedfoundation.github.io/hrdt) | 🤗[Model](https://huggingface.co/embodiedfoundation/H-RDT) | 💬[WeChat Contact](#-contact-us)

## 📰 News
• **[2025.8.12]** Updated RoboTwin2 inference code
H-RDT (**H**uman to **R**obotics **D**iffusion **T**ransformer) is a novel approach that leverages **large-scale egocentric human manipulation data** to enhance robot manipulation capabilities. Our key insight is that large-scale egocentric human manipulation videos with paired 3D hand pose annotations provide rich behavioral priors that capture natural manipulation strategies and can benefit robotic policy learning.
## 🚀 Installation
1. **Create conda environment:**
```bash
conda create -n hrdt python=3.10
conda activate hrdt
```
2. **Install dependencies:**
```bash
pip install -r requirements.txt
```
3. **Download pre-trained models:**
```bash
export HF_ENDPOINT=https://hf-mirror.com
huggingface-cli download --resume-download embodiedfoundation/H-RDT --local-dir ./
```
## 🔧 Usage
### Stage 1: Human Data Pre-training (EgoDx)
#### Data Preprocessing
Before training, preprocess the EgoDx dataset:
1. **Configure paths:**
```bash
# Edit datasets/pretrain/setup_pretrain.sh with your paths
nano datasets/pretrain/setup_pretrain.sh
# Set your EgoDx dataset and T5 model paths:
export EGODEX_DATA_ROOT="/path/to/your/egodx/dataset"
export T5_MODEL_PATH="/path/to/your/t5-v1_1-xxl"
```
2. **Setup environment:**
```bash
source datasets/pretrain/setup_pretrain.sh
```
3. **Run data processing pipeline:**
```bash
# Automatically runs: precompute_48d_actions.py → calc_stat.py → encode_lang_batch.py
./datasets/pretrain/run_pretrain_pipeline.sh
```
#### Start Pre-training
After data preprocessing is complete:
**1. EgoDx Pretrain (fresh start):**
1. Configure dataset:
```python
# Edit datasets/dataset.py line ~45
self.dataset_name = "egodx"
```
2. Run training:
```bash
bash pretrain.sh
```
**2. Pretrain Resume:**
Edit `pretrain.sh`, add this line:
```bash
--resume_from_checkpoint="checkpoint-450000" \
```
### Stage 2: Cross-Embodiment Fine-tuning
#### Data Preprocessing (for RobotWin2)
**Pre-computed language embeddings are already provided - no preprocessing needed!**
1. **Setup environment:**
```bash
# Edit datasets/robotwin2/setup_robotwin2.sh if needed (only for regenerating files)
source datasets/robotwin2/setup_robotwin2.sh
```
2. **Data processing pipeline (Not Required):**
```bash
# Not needed - lang_embeddings/ already provided in repository
# Only run if you want to regenerate files:
# ./datasets/robotwin2/run_robotwin2_pipeline.sh
```
#### Robot Fine-tuning (load human pre-trained backbone):
1. Configure dataset:
```python
# Edit datasets/dataset.py line ~45
self.dataset_name = "robotwin_agilex" # or your robot name
# Add your dataset initialization if not exists:
elif self.dataset_name == "your_robot":
self.hdf5_dataset = YourRobotDataset(config=config)
```
2. Run training:
```bash
bash finetune.sh # Already configured with pretrained_backbone_path
```
#### Finetune Resume:
Edit your current finetune script, make these changes:
```bash
# Change this line:
--mode="finetune" \
# To:
--mode="pretrain" \
# And add:
--resume_from_checkpoint="checkpoint-5000" \
```
## 🎯 Training Modes
| Training Scenario | Base Script | Required Shell Script Modifications | Mode & Key Parameters |
|-------------------|-------------|-------------------------------------|----------------------|
| **Human Pretrain (Fresh)** | `pretrain.sh` | `--mode="pretrain"` | Start pretraining on EgoDx human data |
| **Human Pretrain Resume** | `pretrain.sh` | Add: `--resume_from_checkpoint="checkpoint-450000" \` | `--mode="pretrain"` |
| **Robot Fine-tuning** | `finetune.sh` | Change: `--mode="finetune" \`<br>Add: `--pretrained_backbone_path="./checkpoints/pretrain-0618/checkpoint-500000/pytorch_model.bin" \`<br>Change: `--config_path="configs/hrdt_finetune.yaml" \` | Load human pre-trained backbone, fresh action layers |
| **Robot Finetune Resume** | Your finetune script | Change: `--mode="finetune"` → `--mode="pretrain"`<br>Add: `--resume_from_checkpoint="checkpoint-5000" \` | Continue robot fine-tuning |
### Dataset Configuration
Before training, you need to configure the dataset in `datasets/dataset.py`:
#### For Human Pre-training (EgoDx):
```python
# In datasets/dataset.py, line ~45
self.dataset_name = "egodx"
# The EgoDxDataset will be automatically initialized
```
#### For Robot Fine-tuning:
```python
# In datasets/dataset.py, line ~45
self.dataset_name = "your_robot_name" # e.g., "robotwin_agilex"
# Add your dataset to the initialization logic:
elif self.dataset_name == "your_robot_name":
self.hdf5_dataset = YourRobotDataset(
config=config,
# your dataset parameters
)
```
#### Adding New Robot Datasets:
1. Create your dataset folder: `datasets/your_robot/`
2. Implement your dataset class (see `datasets/robotwin2/` as example)
3. Create data processing scripts (see `datasets/pretrain/` or `datasets/robotwin2/` as examples)
4. Import in `datasets/dataset.py`
5. Add initialization logic in `VLAConsumerDataset.__init__`
### Key Configuration Files
- `configs/hrdt_pretrain.yaml`: Human pre-training configuration
- `configs/hrdt_finetune.yaml`: Robot fine-tuning configuration
- `datasets/dataset.py`: Dataset selection and initialization
- Modify `state_dim`, `action_dim`, `output_size` for your robot
## 📞 Contact Us
### WeChat Discussion Group
Join our WeChat group to discuss H-RDT related technical issues:
<div align="center">
<img src="assets/wechat_group_qr.jpg" width="200" alt="WeChat Group QR Code">
<p><em>WeChat Group QR Code</em></p>
</div>
### Personal WeChat
For other questions or collaboration opportunities, please add personal WeChat:
<div align="center">
<img src="assets/personal_wechat_qr.jpg" width="200" alt="Personal WeChat QR Code">
<p><em>Personal WeChat QR Code</em></p>
</div>
---
*Note: If the QR code expires, please contact us through project Issues for the latest contact information.*
|