README.md

---
library_name: pytorch
tags: [robotics, rm65, openpi, pi05, imagewam]
---
# RealMan RM65 right-arm policies: pi0.5 (openpi) and ImageWAM (private)

This repo holds two policies finetuned on the same RealMan RM65 right-arm dataset. The dataset has
4 pick-and-hold tasks, 187 episodes and 58,399 frames. Each policy comes with its two latest checkpoints.

| model | training run | checkpoints | size each |
|---|---|---|---|
| pi0.5 (openpi) | `pi05_rm65_4tasks_trim`, 30k steps | `25000`, `29999` (final) | 6.7G (params only, no optimizer state) |
| ImageWAM FLUX.2-klein-4B | `rm65_4tasks_trim_flux2_klein_4b/2026-09-25_07-56-03`, 10 epochs | `step_006000.pt`, `step_006020.pt` (final) | 17.8G |

## Robot interface (both models)

- **State and action:** each is `float32[7]` = `right_q1..right_q6` (joint angles, rad) + `right_gripper`
  (0 = closed, 100 = open). Actions are **absolute** joint targets, not deltas.
- **Cameras:** `top` (head) and `wrist` (right wrist), both 640x480 RGB, recorded at 30 fps.
- **Prompts:** use the exact strings the models were trained on. Each one names the object to pick:
  - `Pick up the rightmost yellow Lay's canister with the right arm and retract while keeping it grasped.`
  - `Pick up the rightmost Nongfu Spring water bottle with the right arm, retract, and hold it without releasing it.`
  - `Pick up the rightmost hand soap bottle, retract the arm, and hold the bottle without releasing it.`
  - `Pick up the rightmost Coca-Cola container, retract the arm, and hold it without releasing it.`

## Training data

The training data merges the 4 per-task deliveries (`rm65_4tasks_187_20260921_delivery`: yellow / nongfu /
handsoap / cocacola) and then trims them:

- **Head:** the static frames at the start of every episode are dropped.
- **Tail:** the static frames at the end are cut to 10.
- **Middle:** short pauses inside an episode are kept.
- **Result:** 66,121 frames down to 58,399.

To rebuild the dataset, run these two scripts in order:
1. `openpi/code/scripts/merge_datasets.py`
2. `openpi/code/scripts/trim_static_frames.py`

## openpi pi0.5 — `openpi/`

- **Checkpoints:** `checkpoints/pi05_rm65_4tasks_trim/<step>/` contains `params/` and
  `assets/rm65/4tasks_v1_trim/norm_stats.json`. The norm stats must stay next to the params they came with.
- **Code** (`code/`):
  - **`rm65_config_excerpt.py`** holds `LeRobotRM65DataConfig` plus the `pi05_rm65_4tasks_trim` TrainConfig.
    Add both to `src/openpi/training/config.py`.
  - **`b601_policy.py`** provides the input/output transforms. RM65 has the same 7-dim, two-camera layout
    as B601, so they are reused unchanged.
- **Model and training setup:**
  - pi0.5 base, `action_horizon=30` (1 s of actions at 30 fps).
  - SigLIP trained fully, the Gemma-2B LLM with LoRA, the action expert fully.
  - Batch 32, 30k steps.
- **Serving:**

      uv run scripts/serve_policy.py policy:checkpoint \
          --policy.config=pi05_rm65_4tasks_trim \
          --policy.dir=<this repo>/openpi/checkpoints/pi05_rm65_4tasks_trim/29999

- **Inference input:** `observation/top`, `observation/wrist` (HxWx3 uint8), `observation/state` (7) and `prompt`.
- **Inference output:** `actions` of shape [30, 7].

## ImageWAM — `imagewam/`

The layout mirrors the ImageWAM repo root, so you can copy this directory over `<IW>/ImageWAM/`:

- **Weights and stats:**
  - `runs/rm65_4tasks_trim_flux2_klein_4b/2026-09-25_07-56-03/checkpoints/weights/step_*.pt` are the
    full model weights, with LoRA merged.
  - **`dataset_stats.json`** in the same run directory holds the normalization stats. It must always be
    paired with the checkpoints from **this run**.
  - `config.yaml` in the same run directory is the resolved training config. Its absolute paths point
    to the original machine.
- **Configs and launch script:**
  - `configs/data/rm65_4tasks_trim.yaml`
  - `configs/task/rm65_4tasks_trim_flux2_klein_4b.yaml`
  - `scripts/rm65/run_train_rm65_4tasks_trim.sh`
- **Text cache:** `dataset/rm65_4tasks_trim/flux2_qwen3_cache_4b/` holds the Qwen3 embeddings of the 4 prompts,
  so you don't need to run Qwen3 again for them.

**How RM65 maps onto the model:** the base checkpoint is the RoboTwin 14-dim joint-space model, with layout
`[arm 6, gripper] x 2`. RM65 occupies the **left-arm slot**:

- **Action and state:** the 7 dims sit at `[0:7]` of the model's 14 dims. `[7:14]` is zero padding and is
  masked out of the loss. Take `action[..., :7]` from the model output.
- **Cameras:** the robotwin 288x256 layout has three slots, filled like this:
  - `cam_high` <- `top`
  - `cam_left` <- `wrist`
  - `cam_right` <- a black (all-zero) frame
- **Action chunk:** 16 steps at 30 fps.

**Training setup:**
- LoRA with rank 16 on the video expert; the action expert is trained fully.
- Learning rate 5e-5.
- Effective batch 96, 10 epochs.
- The loss is upweighted around gripper open/close transitions.

**Serving:** the ImageWAM repo only ships a G1 serving script (`scripts/g1/serve_imagewam_g1.py`,
which is 16-dim and G1-specific). An RM65 server has not been written yet. It has to reproduce the camera
and dimension mapping above.

## Not included

- **FLUX.2 base weights** (needed to build ImageWAM):
  - `flux-2-klein-base-4b.safetensors` from black-forest-labs/FLUX.2-klein-base-4B
  - `ae.safetensors` from black-forest-labs/FLUX.2-dev (gated)
- **pi0.5 base weights:** only needed for retraining, from `gs://openpi-assets/checkpoints/pi05_base`.
- **The dataset itself.**