File size: 5,570 Bytes
977b06e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 | This document provides instructions for reproducing our **experimental results** with SimplerEnv.
The evaluation process consists of two main parts:
1. Setting up the `simpler_env` environment and dependencies.
2. Running the evaluation by launching services in both `starVLA` and `simpler_env` environments.
We have verified that this workflow runs successfully on both **NVIDIA A100** and **RTX 4090** GPUs.
## ๐ฆ 1. Environment Setup
To set up the environment, please first follow the official [SimplerEnv repository](https://github.com/simpler-env/SimplerEnv) to install the base `simpler_env` environment.
Afterwards, inside the `simpler_env` environment, install the following dependencies:
```bash
conda activate simpler_env
pip install tyro matplotlib mediapy websockets msgpack
pip install numpy==1.24.4
```
โ ๏ธ **Common Issues**
When testing SimplerEnv on NVIDIA A100, you may encounter the following error:
`libvulkan.so.1: cannot open shared object file: No such file or directory`
You can refer to this link to fix: [Installation Guide โ Vulkan Section](https://maniskill.readthedocs.io/en/latest/user_guide/getting_started/installation.html#vulkan)
## ๐ง Verification Your SimplerEnv
We provide a minimal environment verification script:
```bash
python examples/SimplerEnv/test_your_simplerEnv.py
```
If you see the "โ
Env built successfully" message, it means SimplerEnv is installed correctly and ready to use.
---
## ๐ 2. Eval SimplerEnv
The evaluation should be run **from the repository root** using **two separate terminals/machines**, one for each environment:
- **starVLA environment**: runs the policy inference server.
- **simpler_env environment**: runs the simulation eval code.
### Step 0. Download offical checkpoint
Available SimplerEnv WidowX checkpoints (see [docs/model_zoo.md](../../docs/model_zoo.md) for the full list):
- [Qwen3VL-GR00T-Bridge-RT-1](https://huggingface.co/StarVLA/Qwen3VL-GR00T-Bridge-RT-1) โ 65.3 avg success
- [Qwen3VL-PI_v3-Bridge-RT-1](https://huggingface.co/StarVLA/Qwen3VL-PI_v3-Bridge-RT_1) โ **69.8 avg success** (layer-wise cross-DiT flow-matching head, ฯโ.โ
-style)
### Step 1. Start the server (starVLA environment)
In the first terminal, activate the `starVLA` conda environment and run:
```bash
bash examples/SimplerEnv/eval_files/run_policy_server.sh
```
โ ๏ธ **Note:** Please ensure that you specify the correct checkpoint path in
`run_policy_server.sh`
---
### Step 2. Start the simulation (simpler_env environment)
In the second terminal, activate the `simpler_env` conda environment and run:
```bash
export MODEL_PATH=.../checkpoints/steps_50000_pytorch_model.pt # for read normonization json and get name to save video under ckpt dir
bash examples/SimplerEnv/start_simpler_env.sh ${MODEL_PATH}
```
This script will automatically launch the WidowX Robot evaluation tasks, reproducing the benchmark results reported above.
โ ๏ธ **Note:** Please ensure that you specify the correct `SimplerEnv_PATH`in
`start_simpler_env.sh`
โ ๏ธ **Common Issues**
When run policy server but `NotImplementedError:Framework QwenGR00T is not implemented`, you may need to `python QwenGR00T.py` to check your env.
# ๐ Training on OXE
## Data Preparation
Steps:
1) Download a LeRobot-format OXE dataset
- [bridge_orig_lerobot](https://huggingface.co/datasets/IPEC-COMMUNITY/bridge_orig_lerobot)
- [fractal20220817_data_lerobot](https://huggingface.co/datasets/IPEC-COMMUNITY/fractal20220817_data_lerobot)
2) Including `modality.json` in each `*lerobot/meta/modality.json`
- [bridge modality](./train_files/modality.json). Rename as modality.json and put it as `bridge_orig_lerobot/meta/modality.json`
- [fractal modality](./train_files/fractal_modality.json). Rename as `modality.json` and put it as `fractal20220817_data_lerobot/meta/modality.json`
3) Add your dataset path to `config.yaml`:
```yaml
datasets:
vla_data:
dataset_py: lerobot_datasets
data_root_dir: playground/Datasets/OXE_LEROBOT_DATASET # path to your dataset
data_mix: bridge_rt_1
```
### Check Your Dataoader
We provide a simple way to check your dataloader. Make sure you can load batched data:
```bash
python starVLA/dataloader/lerobot_datasets.py --config_yaml examples/SimplerEnv/train_files/starvla_cotrain_oxe.yaml
```
## Framework Preparation
Before running, you need to ensure that your framework can `forward` and `predict_action` using a fake data example.
Try the following command:
```bash
python starVLA/model/framework/VLM4A/QwenGR00T.py --config_yaml examples/SimplerEnv/train_files/starvla_cotrain_oxe.yaml
```
Note: You can modify the following code snippet to align with your dataset:
```python
# Generate a fake sample
image = Image.fromarray(np.random.randint(0, 255, (224, 224, 3), dtype=np.uint8))
# Create a sample
sample = {
"action": np.random.uniform(-1, 1, size=(16, 7)).astype(np.float16), # action_chunk, action_dim
"image": [image, image], # two views
"lang": "This is a fake for testing.",
"state": np.random.uniform(-1, 1, size=(1, 7)).astype(np.float16), # chunk, state_dim
}
```
## Training
Once everything is ready, use our provided script to start training:
```bash
bash ./examples/SimplerEnv/train_files/run_oxe_train.sh
```
โ ๏ธ **Note:** Ensure that the script explicitly uses the validated config path in `run_lerobot_datasets.sh`. If not already passed, add the `--config_yaml` argument.
|