File size: 7,584 Bytes
211ae03 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 | # Introduction
This module contains the real-world deployment and data collection pipeline for Ψ₀.
# Environment Setup
## 1. Create Conda Environment (Host only)
Create the conda environment from the provided YAML file. This installs all required dependencies, including inverse kinematics libraries and CUDA bindings.
```bash
conda env create -f psi_deploy_env.yaml
```
Activate the environment
```bash
conda activate psi_deploy
```
## 2. Install Unitree SDK2 (Python)
Clone and install our modified version of the Unitree SDK2 Python package:
```bash
git clone git@github.com:physical-superintelligence-lab/unitree_sdk2_python.git
cd unitree_sdk2_python
pip install -e .
cd ..
```
## 3. Install Local Package
```bash
pip install -e .
```
# Ψ₀ Real-World Deployment
## Image Server (Robot only)
On G1's PC, install the image server's requirements
```bash
conda create -n vision python=3.8
conda activate vision
pip install pyrealsense2 opencv-python zmq numpy
```
Copy `realsense_server.py` in the `real/teleop/image_server` directory to the PC of Unitree G_1, and execute the following command **in the PC**:
```bash
conda activate vision
python realsense_server.py
```
## Start Client (Host only)
```bash
conda activate psi_deploy
cd real
bash ./scripts/deploy_psi0-rtc.sh
```
# Teleoperation System Setup
## Apple Vision Pro (AVP) Setup Local streaming (Host and AVP)
**Apple** does not allow WebXR on non-https connections. To test the application locally, we need to create a self-signed certificate and install it on the client. You need a ubuntu machine and a router. Connect the VisionPro and the ubuntu machine to the same router.
1. install mkcert: https://github.com/FiloSottile/mkcert
2. check local ip address:
```bash
ifconfig | grep inet
```
Suppose the local ip address of the ubuntu machine is `192.168.123.2`
3. create certificate:
```bash
mkcert -install && mkcert -cert-file cert.pem -key-file key.pem 192.168.123.2 localhost 127.0.0.1
```
ps. place the generated cert.pem and key.pem files in `teleop`.
```bash
cp cert.pem key.pem teleop/
```
4. open firewall on server:
```bash
sudo ufw allow 8012
```
5. install ca-certificates on VisionPro:
```
mkcert -CAROOT
```
Copy the `rootCA.pem` via AirDrop to VisionPro and install it.
Settings > General > About > Certificate Trust Settings. Under "Enable full trust for root certificates", turn on trust for the certificate.
settings > Apps > Safari > Advanced > Feature Flags > Enable WebXR Related Features
6. open the browser on Safari on VisionPro and go to https://192.168.123.2:8012?ws=wss://192.168.123.2:8012
7. Click `Enter VR` and `Allow` to start the VR session.
## PICO Only Setup Local streaming
1. install PICO streaming python sdk on your host machine (Please follow [TWIST2](https://github.com/amazon-far/TWIST2) related commands as follows)
```bash
conda activate psi_deploy
git clone https://github.com/YanjieZe/XRoboToolkit-PC-Service-Pybind.git
cd XRoboToolkit-PC-Service-Pybind
mkdir -p tmp
cd tmp
git clone https://github.com/XR-Robotics/XRoboToolkit-PC-Service.git
cd XRoboToolkit-PC-Service/RoboticsService/PXREARobotSDK
bash build.sh
cd ../../../..
mkdir -p lib
mkdir -p include
cp tmp/XRoboToolkit-PC-Service/RoboticsService/PXREARobotSDK/PXREARobotSDK.h include/
cp -r tmp/XRoboToolkit-PC-Service/RoboticsService/PXREARobotSDK/nlohmann include/nlohmann/
cp tmp/XRoboToolkit-PC-Service/RoboticsService/PXREARobotSDK/build/libPXREARobotSDK.so lib/
# rm -rf tmp
# Build the project
conda install -c conda-forge pybind11
pip uninstall -y xrobotoolkit_sdk
python setup.py install
```
2. follow the “Install XRoboToolkit-PC-Service” instructions from [XR Robotics](https://github.com/XR-Robotics) to install the PICO service on your workstation.
3. install [PICO SDK](https://github.com/XR-Robotics/XRoboToolkit-Unity-Client/releases/) on your PICO.
4. connect your PICO and workstation using the same WIFI. Suppose PICO IP is `192.168.0.128`, Host IP is `192.168.0.110`
<p align="center">
<img src="../assets/media/pico.png" alt="Psi0 teaser image" />
</p>
5. select `Head`, `Controller` and `Hand` under the Tracking Session, and then enable the `Send` Switch.
6. select `ZEDMINI` under the Remote Vision Session and click the `Listen` button, then input your PC IP inside.
7. put down pico controllers and wait until PICO activates its hand tracking mode automatically (You can see virtual hands around your own hands).
# Start Teleoperation and Data Collection
## Image Server (Robot only)
```bash
conda activate vision
python realsense_server.py
```
## Data collection metadata preparation (Host only)
1. Write down the task information in a json file inside `teleop/task_defs` directory
2. In the `teleop/` directory, run `python taskcreator.py` to generate the task metadata. Take a look at `teleop/task_defs/example.json` for an example task's json format.
## Teleoperating Data Collection Instruction
> Warning : All persons must maintain an adequate safety distance from the robot to avoid danger!
1. Connect your host computer to G1. Then, set up your local IP address to 192.168.123.123 with netmask 255.255.255.0 on G1's network interface.
2. Connect both your computer and AVP to the same local router wifi in which you set up your cert with.
3. Open robot and set to development mode using remote control by pressing L2 + B, then L2 + R2 (For some robots with lower software versions might need to press L1 + A and the L2 + R2).
4. Hang the robot so that G1 is gently touching the ground.
5. On G1 PC: start the image server as above.
6. On host computer: run
```bash
export CYCLONEDDS_URI="<CycloneDDS><Domain><General><NetworkInterfaceAddress>192.168.123.123</NetworkInterfaceAddress></General></Domain></CycloneDDS>"
```
7. (**Vision Pro Option**) On host computer: run `python main.py --robot g1` under the `teleop/` directory and wait until the robot is standing in ready state. The terminal should signal both "master" and "worker" processes are waiting for starting signal.
8. On AVP, connect to robot using https://<host_ip_address_on_your_local_router>:8012/?ws=wss://<host_ip_address_on_your_local_router>:8012. Then, press `Enter VR` and then `Allow` to enter the web interface for teleoperating the G1.
9. (**PICO Option**) On host computer: run `python main.py --robot g1 --pico_streamer --pico_ip 192.168.0.128 (YOUR_PICO_IP)` under the `teleop/` directory and wait until the robot is standing in ready state. The terminal should signal both "master" and "worker" processes are waiting for starting signal.
10. Back on host computer, enter `s` to start recording an episode.
11. Type `q` and enter if the episode is successful, otherwise `d` and enter to discard the last session.
12. Repeat by pressing `s` to start recording the next episode. Record 40 episodes for each task.
13. Type `exit` to shut down the program.
# Acknowledgement
This code builds upon following open-source code-bases. Please visit the URLs to see the respective LICENSES:
1. https://github.com/unitreerobotics/avp_teleoperate
2. https://github.com/OpenTeleVision/TeleVision
3. https://github.com/dexsuite/dex-retargeting
4. https://github.com/vuer-ai/vuer
5. https://github.com/stack-of-tasks/pinocchio
6. https://github.com/casadi/casadi
7. https://github.com/meshcat-dev/meshcat-python
8. https://github.com/zeromq/pyzmq
9. https://github.com/unitreerobotics/unitree_dds_wrapper
10. https://github.com/tonyzhaozh/act
11. https://github.com/facebookresearch/detr
12. https://github.com/OpenTeleVision/AMO
|