File size: 5,784 Bytes

4b05bc7

---
license: apache-2.0
pipeline_tag: robotics
library_name: openpi
tags:
  - robotics
  - openpi
  - pi0.5
  - vision-language-action
  - reinforcement-learning
  - bimanual-manipulation
  - roboeval
datasets:
  - helen9975/roboeval_ee_delta_20hz_scaled
---

# RoboEval CoRL 2026 solution: off-policy RL with online inference tuning

This is the policy I submitted to the [RoboEval Challenge at CoRL 2026](https://robo-eval.github.io/challenge.html):
one modified π0.5 for eight bimanual tasks with two Franka arms, in simulation. The weights are an average of two
checkpoints of the same run, 0.65 × step 48250 + 0.35 × step 58000.

- Code: https://github.com/IliaLarchenko/RoboEvalSolution
- Report: https://arxiv.org/abs/2611.11464
- Project page: https://ilialarchenko.com/projects/roboeval2026/

## Results

Scored with the official RoboEval Diagnostic Score (RDS), each task with its final inference settings.

| | Episodes per task | RDS | Success |
|---|---:|---:|---:|
| Official seeds (1000–1049) | 50 | **0.987** | 99.25% |
| Held-out random seeds | 500 | **0.990** | 99.60% |

| Task | Official success | Official RDS | Held-out success | Held-out RDS |
|---|---:|---:|---:|---:|
| LiftPot | 1.000 | 1.000 | 1.000 | 0.999 |
| LiftTray | 0.980 | 0.984 | 0.998 | 0.998 |
| StackTwoBlocks | 1.000 | 0.977 | 0.996 | 0.979 |
| StackSingleBookShelf | 1.000 | 1.000 | 0.998 | 0.998 |
| CubeHandover | 1.000 | 1.000 | 1.000 | 0.998 |
| PackBox | 1.000 | 0.969 | 1.000 | 0.974 |
| PickSingleBookFromTable | 0.960 | 0.967 | 0.988 | 0.988 |
| RotateValve | 1.000 | 1.000 | 0.988 | 0.989 |

A 50-episode run varies by about ±0.005 RDS, so the two rows agree within noise. I made every decision on held-out
seeds and never tuned on the official ones.

## Inputs and outputs

- **Inputs:** five RGB cameras at 256×256 (head, front, external, left wrist, right wrist), proprioception (joint
  positions and velocities, gripper states) and a task id. There is no text prompt. Some tasks hide some of these
  inputs at inference.
- **Actions:** chunks of 30 steps (1.5 s at 20 Hz) of 14-D end-effector actions for both arms: per arm a 3-D position,
  a 3-D rotation vector and a gripper command, relative to the commanded pose at the chunk start. They convert to the
  challenge's end-effector delta actions.
- **Tasks:** LiftPot, LiftTray, StackTwoBlocks, StackSingleBookShelf, CubeHandover, PackBox, PickSingleBookFromTable
  and RotateValve, all on the PositionAndOrientation variation.

## How to run it

The policy runs through the code repository, not through `transformers`. You need a Linux machine with an NVIDIA GPU
(driver >= 580 for the default CUDA 13 build; on older drivers use `bash setup.sh --cuda12`).

```bash
git clone --recurse-submodules https://github.com/IliaLarchenko/RoboEvalSolution.git && cd RoboEvalSolution
bash setup.sh    # .venv (policy) and .venv-sim (simulator)
.venv/bin/hf download IliaLarchenko/RoboEvalSolution --local-dir ../checkpoint

CUDA_VISIBLE_DEVICES=0 .venv/bin/python scripts/serve_policy.py --config-name pi05_roboeval \
    --checkpoint-dir ../checkpoint --num-candidates 6 &
MUJOCO_GL=egl .venv-sim/bin/python scripts/rollout.py --server ws://localhost:8765 --out ../runs/eval_x \
    --episodes-per-task 50 --workers 20 --best configs/settings.json
```

`rollout.py` builds the environments like the official evaluator and writes the official scores to `summary.json`. It
refuses the official seeds 1000–1049. Stop the server afterwards with `kill %1`.

## Inference settings

The settings matter a lot and differ per task: the number of flow steps, noise temperature, CFG scale, best-of-N
ranked by the Q head, how many actions to execute and carry over, execution speed, gripper threshold and hidden
inputs. They are in
[`configs/settings.json`](https://github.com/IliaLarchenko/RoboEvalSolution/blob/main/configs/settings.json).
Without them you won't reproduce the scores above.

## Training

The model is π0.5 (a PaliGemma VLM plus a flow-matching action expert) with a value query, a Q head and advantage
conditioning, so it is also its own value function. I trained it in an off-policy RL loop with advantages of the exact
challenge score (AWR, and advantage conditioning inspired by RECAP). Every rollout was also a Thompson-sampling trial
of about 20 inference parameters. Training used about 600 GPU-hours.

Data:

- the organisers' [RoboEval demonstrations](https://huggingface.co/datasets/helen9975/roboeval_ee_delta_20hz_scaled);
- my own scripted demonstrations with full reward, generated in simulation from privileged state;
- more than 240k of the policy's own rollouts, plus recovery episodes that continue from its failures.

The only external pretraining is the public π0.5 base model.

## Limitations

- It was trained and evaluated in simulation only and has never run on a real robot.
- It is tuned to these eight tasks, this robot, these cameras and this exact score. Don't expect it to work elsewhere
  without retraining.
- Part of the score comes from per-task inference tuning, which is slow and would be hard to justify in a real
  deployment.

## License

Code and weights are released under Apache-2.0.

## Citation

```bibtex
@misc{larchenko2026roboeval,
  title  = {Off-Policy RL with Online Inference Tuning: My Solution to the CoRL 2026 RoboEval Challenge},
  author = {Larchenko, Ilia},
  year   = {2026},
  eprint = {2611.11464},
  archivePrefix = {arXiv},
  primaryClass = {cs.RO},
  url    = {https://arxiv.org/abs/2611.11464}
}
```

The benchmark, simulator and demonstrations are from [RoboEval](https://robo-eval.github.io)
([Wang et al., 2025](https://arxiv.org/abs/2507.00435)). The base model is π0.5 by
[Physical Intelligence](https://github.com/Physical-Intelligence/openpi).