File size: 1,611 Bytes
2d3265e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 |
# Evaluate Policies on SIMPLE
pull `SIMPLE`
```
git submodule update --init --recursive
cd third_party/SIMPLE
```
## Install Dokcer version
Setup environment variables
```
cp .env.sample .env;
# then edit the .env file to set the "DATA_DIR" to current directory (e.g., `DATA_DIR=/home/***/projects/psi/third_party/SIMPLE/data`)
```
Build docker image
```
docker compose build isaac-sim
```
## Install UV version (TODO)
```
git submodule update --init --recursive third_party/SIMPLE
```
## Evaluation
copy the training dataset to SIMPLE `data`, which will be mounted into the docker container
```
cp -r $DATA_HOME/G1WholebodyBendPick-v0-psi0 ./data/
```
### Using nix
Evaluate a PSI checkpoint directly from this repo with the SIMPLE library API:
```
source .venv-psi/bin/activate
python examples/simple/simple_eval.py \
--run-dir /path/to/.runs/finetune/... \
--ckpt-step 40000 \
--data-dir /hfm/data/simple/G1WholebodyBendPick-v0-psi0 \
--env-id simple/G1WholebodyBendPick-v0 \
--num-episodes 1 \
--save-video
```
> After evaluation is done, checkout the videos from `third_party/SIMPLE/data/evals/psi0`
> REMINDER: Start server before starting the eval client
### Using Docker
launch the eval client
```
export task=simple/G1WholebodyBendPick-v0
export policy=psi0
export host=localhost
export port=22085
GPUs=1 docker compose run eval $task $policy \
--host=$host \
--port=$port \
--sim-mode=mujoco_isaac \
--headless \
--max-episode-steps=360 \
--data-format=lerobot \
--data-dir=data/G1WholebodyBendPick-v0-psi0 \
--num-episodes=10
```
|