File size: 1,611 Bytes
2d3265e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
 
# Evaluate Policies on SIMPLE
pull `SIMPLE`
```
git submodule update --init --recursive
cd third_party/SIMPLE
```

## Install Dokcer version

Setup environment variables
```
cp .env.sample .env; 
# then edit the .env file to set the "DATA_DIR" to current directory (e.g., `DATA_DIR=/home/***/projects/psi/third_party/SIMPLE/data`)
```

Build docker image
```
docker compose build isaac-sim
```

## Install UV version (TODO)
```
git submodule update --init --recursive third_party/SIMPLE
```

## Evaluation

copy the training dataset to SIMPLE `data`, which will be mounted into the docker container
```
cp -r $DATA_HOME/G1WholebodyBendPick-v0-psi0 ./data/
```

### Using nix
Evaluate a PSI checkpoint directly from this repo with the SIMPLE library API:
```
source .venv-psi/bin/activate
python examples/simple/simple_eval.py \
    --run-dir /path/to/.runs/finetune/... \
    --ckpt-step 40000 \
    --data-dir /hfm/data/simple/G1WholebodyBendPick-v0-psi0 \
    --env-id simple/G1WholebodyBendPick-v0 \
    --num-episodes 1 \
    --save-video
```
> After evaluation is done, checkout the videos from `third_party/SIMPLE/data/evals/psi0`
> REMINDER: Start server before starting the eval client

### Using Docker

launch the eval client
```
export task=simple/G1WholebodyBendPick-v0
export policy=psi0
export host=localhost
export port=22085
GPUs=1 docker compose run eval $task $policy \
    --host=$host \
    --port=$port  \
    --sim-mode=mujoco_isaac \
    --headless \
    --max-episode-steps=360 \
    --data-format=lerobot \
    --data-dir=data/G1WholebodyBendPick-v0-psi0 \
    --num-episodes=10
```