# Offline Policy Training and Value Model

Run all below commands from `policy_and_value/policy_offline_and_value`.

## Available Release Configs

The release registers these config names in `src/openpi_value/training/config.py`:

- `Policy_offline_release`
- `value_release`
- `Compute_norm`

## Compute Normalization Stats

```bash
python scripts/compute_norm_stats_fast.py --config-name Compute_norm
```

## Train Policy or Value Models

```bash
# usage: bash train.sh <CONFIG_NAME> <NUM_GPUS> [extra args]
bash train.sh Policy_offline_release 8
bash train.sh value_release 8
```

Resume example:

```bash
bash train.sh Policy_offline_release 8 --resume
```

## Label Per-frame Value Predictions with the Value Model

* Labeling value and advantage for lerobot datasets using learned value model. 

* This step is required for both offline and online policy improvement with advantage conditioning.

```bash
# usage: bash label_value.sh <CONFIG_NAME> <CKPT_DIR>
bash label_value.sh vis_value_release_joint_T /path/to/checkpoints/value_release_joint/<exp>/<step>
```

## Visualize Value Predictions

```bash
# usage: bash vis_value.sh <CONFIG_NAME> <CKPT_DIR>
bash vis_value.sh vis_value_release_joint_T /path/to/checkpoints/value_release_joint/<exp>/<step>
```