Add Florence-Bridge code and handoff documentation

977b06e verified 7 days ago

5.57 kB

This document provides instructions for reproducing our **experimental results** with SimplerEnv.  


The evaluation process consists of two main parts:  

1. Setting up the `simpler_env` environment and dependencies.  
2. Running the evaluation by launching services in both `starVLA` and `simpler_env` environments.  

We have verified that this workflow runs successfully on both **NVIDIA A100** and **RTX 4090** GPUs.  

## 📦 1. Environment Setup

To set up the environment, please first follow the official [SimplerEnv repository](https://github.com/simpler-env/SimplerEnv) to install the base `simpler_env` environment. 

Afterwards, inside the `simpler_env` environment, install the following dependencies:  

```bash
conda activate simpler_env
pip install tyro matplotlib mediapy websockets msgpack
pip install numpy==1.24.4
```

⚠️ **Common Issues**
When testing SimplerEnv on NVIDIA A100, you may encounter the following error:
`libvulkan.so.1: cannot open shared object file: No such file or directory`
You can refer to this link to fix: [Installation Guide – Vulkan Section](https://maniskill.readthedocs.io/en/latest/user_guide/getting_started/installation.html#vulkan)


## 🔧 Verification Your SimplerEnv
We provide a minimal environment verification script:

```bash
python examples/SimplerEnv/test_your_simplerEnv.py
```

If you see the "✅ Env built successfully" message, it means SimplerEnv is installed correctly and ready to use.


---


## 🚀 2. Eval SimplerEnv

The evaluation should be run **from the repository root** using **two separate terminals/machines**, one for each environment:  

- **starVLA environment**: runs the policy inference server.  
- **simpler_env environment**: runs the simulation eval code.  
### Step 0. Download offical checkpoint
Available SimplerEnv WidowX checkpoints (see [docs/model_zoo.md](../../docs/model_zoo.md) for the full list):
- [Qwen3VL-GR00T-Bridge-RT-1](https://huggingface.co/StarVLA/Qwen3VL-GR00T-Bridge-RT-1) — 65.3 avg success
- [Qwen3VL-PI_v3-Bridge-RT-1](https://huggingface.co/StarVLA/Qwen3VL-PI_v3-Bridge-RT_1) — **69.8 avg success** (layer-wise cross-DiT flow-matching head, π₀.₅-style)
### Step 1. Start the server (starVLA environment)
In the first terminal, activate the `starVLA` conda environment and run:  
```bash
bash examples/SimplerEnv/eval_files/run_policy_server.sh
```
⚠️ **Note:** Please ensure that you specify the correct checkpoint path in  
`run_policy_server.sh`  

---


### Step 2. Start the simulation (simpler_env environment)
In the second terminal, activate the `simpler_env` conda environment and run:  

```bash
export MODEL_PATH=.../checkpoints/steps_50000_pytorch_model.pt # for read normonization json and get name to save video under ckpt dir
bash examples/SimplerEnv/start_simpler_env.sh ${MODEL_PATH} 
```
This script will automatically launch the WidowX Robot evaluation tasks, reproducing the benchmark results reported above.

⚠️ **Note:** Please ensure that you specify the correct `SimplerEnv_PATH`in 
`start_simpler_env.sh`  


⚠️ **Common Issues**

When run policy server but `NotImplementedError:Framework QwenGR00T is not implemented`, you may need to `python QwenGR00T.py` to check your env.



# 🚀 Training on OXE

## Data Preparation


Steps:
1) Download a LeRobot-format OXE dataset 
- [bridge_orig_lerobot](https://huggingface.co/datasets/IPEC-COMMUNITY/bridge_orig_lerobot)
- [fractal20220817_data_lerobot](https://huggingface.co/datasets/IPEC-COMMUNITY/fractal20220817_data_lerobot)

2) Including `modality.json` in each `*lerobot/meta/modality.json`
- [bridge modality](./train_files/modality.json). Rename as modality.json and put it as `bridge_orig_lerobot/meta/modality.json`
- [fractal modality](./train_files/fractal_modality.json). Rename as `modality.json` and put it as `fractal20220817_data_lerobot/meta/modality.json`

3) Add your dataset path to `config.yaml`:
    ```yaml
    datasets:
      vla_data:
        dataset_py: lerobot_datasets
        data_root_dir: playground/Datasets/OXE_LEROBOT_DATASET  # path to your dataset
        data_mix: bridge_rt_1
    ```

### Check Your Dataoader
We provide a simple way to check your dataloader. Make sure you can load batched data:

```bash
python starVLA/dataloader/lerobot_datasets.py --config_yaml examples/SimplerEnv/train_files/starvla_cotrain_oxe.yaml
```

## Framework Preparation

Before running, you need to ensure that your framework can `forward` and `predict_action` using a fake data example.

Try the following command:

```bash
python starVLA/model/framework/VLM4A/QwenGR00T.py --config_yaml examples/SimplerEnv/train_files/starvla_cotrain_oxe.yaml
```

Note: You can modify the following code snippet to align with your dataset:

```python
    # Generate a fake sample
    image = Image.fromarray(np.random.randint(0, 255, (224, 224, 3), dtype=np.uint8))
    # Create a sample
    sample = {
        "action": np.random.uniform(-1, 1, size=(16, 7)).astype(np.float16),  # action_chunk, action_dim
        "image": [image, image],  # two views
        "lang": "This is a fake for testing.",
        "state": np.random.uniform(-1, 1, size=(1, 7)).astype(np.float16),  # chunk, state_dim
    }
```

## Training

Once everything is ready, use our provided script to start training:

```bash
bash ./examples/SimplerEnv/train_files/run_oxe_train.sh
```

⚠️ **Note:** Ensure that the script explicitly uses the validated config path in `run_lerobot_datasets.sh`. If not already passed, add the `--config_yaml` argument.

	This document provides instructions for reproducing our experimental results with SimplerEnv.


	The evaluation process consists of two main parts:

	1. Setting up the `simpler_env` environment and dependencies.
	2. Running the evaluation by launching services in both `starVLA` and `simpler_env` environments.

	We have verified that this workflow runs successfully on both NVIDIA A100 and RTX 4090 GPUs.

	## 📦 1. Environment Setup

	To set up the environment, please first follow the official [SimplerEnv repository](https://github.com/simpler-env/SimplerEnv) to install the base `simpler_env` environment.

	Afterwards, inside the `simpler_env` environment, install the following dependencies:

	```bash
	conda activate simpler_env
	pip install tyro matplotlib mediapy websockets msgpack
	pip install numpy==1.24.4
	```

	⚠️ Common Issues
	When testing SimplerEnv on NVIDIA A100, you may encounter the following error:
	`libvulkan.so.1: cannot open shared object file: No such file or directory`
	You can refer to this link to fix: [Installation Guide – Vulkan Section](https://maniskill.readthedocs.io/en/latest/user_guide/getting_started/installation.html#vulkan)


	## 🔧 Verification Your SimplerEnv
	We provide a minimal environment verification script:

	```bash
	python examples/SimplerEnv/test_your_simplerEnv.py
	```

	If you see the "✅ Env built successfully" message, it means SimplerEnv is installed correctly and ready to use.


	---


	## 🚀 2. Eval SimplerEnv

	The evaluation should be run from the repository root using two separate terminals/machines, one for each environment:

	- starVLA environment: runs the policy inference server.
	- simpler_env environment: runs the simulation eval code.


	### Step 0. Download offical checkpoint

	Available SimplerEnv WidowX checkpoints (see [docs/model_zoo.md](../../docs/model_zoo.md) for the full list):

	- [Qwen3VL-GR00T-Bridge-RT-1](https://huggingface.co/StarVLA/Qwen3VL-GR00T-Bridge-RT-1) — 65.3 avg success
	- [Qwen3VL-PI_v3-Bridge-RT-1](https://huggingface.co/StarVLA/Qwen3VL-PI_v3-Bridge-RT_1) — 69.8 avg success (layer-wise cross-DiT flow-matching head, π₀.₅-style)



	### Step 1. Start the server (starVLA environment)

	In the first terminal, activate the `starVLA` conda environment and run:

	```bash
	bash examples/SimplerEnv/eval_files/run_policy_server.sh
	```

	⚠️ Note: Please ensure that you specify the correct checkpoint path in
	`run_policy_server.sh`

	---


	### Step 2. Start the simulation (simpler_env environment)

	In the second terminal, activate the `simpler_env` conda environment and run:

	```bash
	export MODEL_PATH=.../checkpoints/steps_50000_pytorch_model.pt # for read normonization json and get name to save video under ckpt dir
	bash examples/SimplerEnv/start_simpler_env.sh ${MODEL_PATH}
	```
	This script will automatically launch the WidowX Robot evaluation tasks, reproducing the benchmark results reported above.

	⚠️ Note: Please ensure that you specify the correct `SimplerEnv_PATH`in
	`start_simpler_env.sh`


	⚠️ Common Issues

	When run policy server but `NotImplementedError:Framework QwenGR00T is not implemented`, you may need to `python QwenGR00T.py` to check your env.



	# 🚀 Training on OXE

	## Data Preparation


	Steps:
	1) Download a LeRobot-format OXE dataset
	- [bridge_orig_lerobot](https://huggingface.co/datasets/IPEC-COMMUNITY/bridge_orig_lerobot)
	- [fractal20220817_data_lerobot](https://huggingface.co/datasets/IPEC-COMMUNITY/fractal20220817_data_lerobot)

	2) Including `modality.json` in each `*lerobot/meta/modality.json`
	- [bridge modality](./train_files/modality.json). Rename as modality.json and put it as `bridge_orig_lerobot/meta/modality.json`
	- [fractal modality](./train_files/fractal_modality.json). Rename as `modality.json` and put it as `fractal20220817_data_lerobot/meta/modality.json`

	3) Add your dataset path to `config.yaml`:
	```yaml
	datasets:
	vla_data:
	dataset_py: lerobot_datasets
	data_root_dir: playground/Datasets/OXE_LEROBOT_DATASET # path to your dataset
	data_mix: bridge_rt_1
	```


	### Check Your Dataoader
	We provide a simple way to check your dataloader. Make sure you can load batched data:

	```bash
	python starVLA/dataloader/lerobot_datasets.py --config_yaml examples/SimplerEnv/train_files/starvla_cotrain_oxe.yaml
	```

	## Framework Preparation

	Before running, you need to ensure that your framework can `forward` and `predict_action` using a fake data example.

	Try the following command:

	```bash
	python starVLA/model/framework/VLM4A/QwenGR00T.py --config_yaml examples/SimplerEnv/train_files/starvla_cotrain_oxe.yaml
	```

	Note: You can modify the following code snippet to align with your dataset:

	```python
	# Generate a fake sample
	image = Image.fromarray(np.random.randint(0, 255, (224, 224, 3), dtype=np.uint8))
	# Create a sample
	sample = {
	"action": np.random.uniform(-1, 1, size=(16, 7)).astype(np.float16), # action_chunk, action_dim
	"image": [image, image], # two views
	"lang": "This is a fake for testing.",
	"state": np.random.uniform(-1, 1, size=(1, 7)).astype(np.float16), # chunk, state_dim
	}
	```

	## Training

	Once everything is ready, use our provided script to start training:

	```bash
	bash ./examples/SimplerEnv/train_files/run_oxe_train.sh
	```

	⚠️ Note: Ensure that the script explicitly uses the validated config path in `run_lerobot_datasets.sh`. If not already passed, add the `--config_yaml` argument.