File size: 1,265 Bytes
23e1f91 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 | # Offline Policy Training and Value Model
Run all below commands from `policy_and_value/policy_offline_and_value`.
## Available Release Configs
The release registers these config names in `src/openpi_value/training/config.py`:
- `Policy_offline_release`
- `value_release`
- `Compute_norm`
## Compute Normalization Stats
```bash
python scripts/compute_norm_stats_fast.py --config-name Compute_norm
```
## Train Policy or Value Models
```bash
# usage: bash train.sh <CONFIG_NAME> <NUM_GPUS> [extra args]
bash train.sh Policy_offline_release 8
bash train.sh value_release 8
```
Resume example:
```bash
bash train.sh Policy_offline_release 8 --resume
```
## Label Per-frame Value Predictions with the Value Model
* Labeling value and advantage for lerobot datasets using learned value model.
* This step is required for both offline and online policy improvement with advantage conditioning.
```bash
# usage: bash label_value.sh <CONFIG_NAME> <CKPT_DIR>
bash label_value.sh vis_value_release_joint_T /path/to/checkpoints/value_release_joint/<exp>/<step>
```
## Visualize Value Predictions
```bash
# usage: bash vis_value.sh <CONFIG_NAME> <CKPT_DIR>
bash vis_value.sh vis_value_release_joint_T /path/to/checkpoints/value_release_joint/<exp>/<step>
```
|