File size: 15,100 Bytes
73bcc98 18489c8 73bcc98 18489c8 73bcc98 18489c8 b25545a 73bcc98 18489c8 73bcc98 18489c8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 | ---
license: other
base_model: physical-intelligence/pi05-base
tags:
- robotics
- vla
- pi0.5
- openpi
- cobotmagic
- embodichain
pipeline_tag: robotics
---
# RoboSynChallenge team checkpoints (Ο0.5)
Ο0.5 checkpoints fine-tuned by our team for the [RoboSynChallenge](http://robosyn-bench.net) CobotMagic simulation tasks, trained with [openpi](https://github.com/Physical-Intelligence/openpi) from `pi05_base` on the official `RoboSynChallenge/cobotmagic_Sim_<task>` datasets (1,000 expert episodes per task, LeRobot v2.1).
The repository has one folder per task (ten tasks); inside, `<run>/<step>/` is one openpi checkpoint step: `params/` (full model parameters, also for LoRA runs), `assets/<repo_id>/norm_stats.json` and, for most folders, `train_state/` (optimizer state) for resuming.
## Checkpoints (`<task>/<run>/<step>`)
| Path | Task | Run | Step | Success / episodes | Mean action steps / limit | Evaluation |
|---|---|---|---|---|---|---|
| `table_rearrangement/pi05_full_v1/5000` | table_rearrangement | full fine-tune | 5000 | **60** / 100 | 246.6 / 361 | random setting, 100 episodes |
| `table_rearrangement/pi05_full_v1/10000` | table_rearrangement | full fine-tune | 10000 | **73** / 100 | 219.1 / 361 | random setting, 100 episodes |
| `table_rearrangement/pi05_full_v1/15000` | table_rearrangement | full fine-tune | 15000 | **78** / 100 | 209.3 / 361 | random setting, 100 episodes |
| `table_rearrangement/pi05_full_v1/19999` | table_rearrangement | full fine-tune | 19999 | **76** / 100 | 229.3 / 361 | random setting, 100 episodes |
| `table_rearrangement/pi05_lora_v1/2500` | table_rearrangement | LoRA | 2500 | **32** / 100 | 299.2 / 361 | random setting, 100 episodes |
| `table_rearrangement/pi05_lora_v1/5000` | table_rearrangement | LoRA | 5000 | **39** / 100 | 294.6 / 361 | random setting, 100 episodes |
| `table_rearrangement/pi05_lora_v1/7500` | table_rearrangement | LoRA | 7500 | **52** / 100 | 259.5 / 361 | random setting, 100 episodes |
| `table_rearrangement/pi05_lora_v1/10000` | table_rearrangement | LoRA | 10000 | **56** / 100 | 265.4 / 361 | random setting, 100 episodes |
| `table_rearrangement/pi05_lora_v1/12500` | table_rearrangement | LoRA | 12500 | **70** / 100 | 233.6 / 361 | random setting, 100 episodes |
| `table_rearrangement/pi05_lora_v1/14999` | table_rearrangement | LoRA | 14999 | **76** / 100 | 223.9 / 361 | random setting, 100 episodes |
| `items_handover/pi05_lora_v1/2500` | items_handover | LoRA | 2500 | **7** / 100 | 348.3 / 350 | random setting, 100 episodes |
| `items_handover/pi05_lora_v1/5000` | items_handover | LoRA | 5000 | **1** / 100 | 349.5 / 350 | random setting, 100 episodes |
| `items_handover/pi05_lora_v1/7500` | items_handover | LoRA | 7500 | **11** / 100 | 349.0 / 350 | random setting, 100 episodes |
| `items_handover/pi05_lora_v1/10000` | items_handover | LoRA | 10000 | **14** / 100 | 347.6 / 350 | random setting, 100 episodes |
| `items_handover/pi05_lora_v1/12500` | items_handover | LoRA | 12500 | **38** / 100 | 342.2 / 350 | random setting, 100 episodes |
| `items_handover/pi05_lora_v1/14999` | items_handover | LoRA | 14999 | **14** / 100 | 348.4 / 350 | random setting, 100 episodes |
| `items_handover/pi05_aug_lora_v1/2500` | items_handover | LoRA (team-generated data) | 2500 | **0** / 100 | 350.0 / 350 | random setting, 100 episodes; trained on the team-generated data (see Notes); measured with the broken success check, not re-run (checkpoint removed) |
| `items_handover/pi05_aug_lora_v1/5000` | items_handover | LoRA (team-generated data) | 5000 | 0 / 25 | β | early-stopped (corrected check, 2026-10-11) |
| `items_handover/pi05_aug_lora_v1/7500` | items_handover | LoRA (team-generated data) | 7500 | 0 / 25 | β | early-stopped (broken check, checkpoint removed) |
| `items_handover/pi05_aug_lora_v1/10000` | items_handover | LoRA (team-generated data) | 10000 | 0 / 25 | β | early-stopped (broken check, checkpoint removed) |
| `items_handover/pi05_aug_lora_v1/12500` | items_handover | LoRA (team-generated data) | 12500 | 0 / 25 | β | early-stopped (corrected check, 2026-10-11) |
| `items_handover/pi05_aug_lora_v1/14999` | items_handover | LoRA (team-generated data) | 14999 | 0 / 25 | β | early-stopped (broken check, checkpoint removed) |
| `items_handover/pi05_mix_lora_v1/2500` | items_handover | LoRA (mixed data) | 2500 | 0 / 25 | β | early-stopped (broken check, checkpoint removed); official + team-generated data (2702 episodes) |
| `items_handover/pi05_mix_lora_v1/5000` | items_handover | LoRA (mixed data) | 5000 | 2 / 40 | β | early-stopped (corrected check, 2026-10-11) |
| `items_handover/pi05_mix_lora_v1/7500` | items_handover | LoRA (mixed data) | 7500 | 0 / 25 | β | early-stopped (broken check, checkpoint removed) |
| `items_handover/pi05_mix_lora_v1/10000` | items_handover | LoRA (mixed data) | 10000 | 1 / 13 | β | cut short, NAS full (broken check, checkpoint removed) |
| `items_handover/pi05_mix_lora_v1/12500` | items_handover | LoRA (mixed data) | 12500 | **16** / 100 | 348.3 / 350 | random setting, 100 episodes, gripper rescaling on (corrected check, 2026-10-11) |
| `items_handover/pi05_mix_lora_v1/14999` | items_handover | LoRA (mixed data) | 14999 | 0 / 25 | β | early-stopped (broken check, checkpoint removed) |
| `items_handover/pi05_full_v1/5000` | items_handover | Full | 5000 | **58** / 100 | 340.7 / 350 | random setting, 100 episodes, gripper rescaling on; full fine-tune on the official data |
| `items_handover/pi05_full_v1/10000` | items_handover | Full | 10000 | **72** / 100 | 337.7 / 350 | random setting, 100 episodes, gripper rescaling on |
| `items_handover/pi05_full_v1/15000` | items_handover | Full | 15000 | **78** / 100 | 338.4 / 350 | random setting, 100 episodes, gripper rescaling on; recommended |
| `items_handover/pi05_full_v1/19999` | items_handover | Full | 19999 | **77** / 100 | 340.2 / 350 | random setting, 100 episodes, gripper rescaling on; final |
| `items_handover/pi05_short_v1/1000` | items_handover | Full (5k, lr 1e-5) | 1000 | **23** / 100 | 345.4 / 350 | random setting, 100 episodes, gripper rescaling on; 5000-step full fine-tune with the schedule scaled to its length |
| `items_handover/pi05_short_v1/2000` | items_handover | Full (5k, lr 1e-5) | 2000 | **34** / 100 | 343.8 / 350 | random setting, 100 episodes, gripper rescaling on |
| `items_handover/pi05_short_v1/3000` | items_handover | Full (5k, lr 1e-5) | 3000 | **44** / 100 | 343.6 / 350 | random setting, 100 episodes, gripper rescaling on |
| `items_handover/pi05_short_v1/4000` | items_handover | Full (5k, lr 1e-5) | 4000 | **59** / 100 | 342.2 / 350 | random setting, 100 episodes, gripper rescaling on |
| `items_handover/pi05_short_v1/4999` | items_handover | Full (5k, lr 1e-5) | 4999 | **65** / 100 | 340.3 / 350 | random setting, 100 episodes, gripper rescaling on; final |
| `handle_basket/pi05_lora_v1/2500` | handle_basket | LoRA | 2500 | **28** / 100 | 472.9 / 500 | random setting, 100 episodes |
| `handle_basket/pi05_lora_v1/5000` | handle_basket | LoRA | 5000 | **35** / 100 | 467.1 / 500 | random setting, 100 episodes |
| `handle_basket/pi05_lora_v1/7500` | handle_basket | LoRA | 7500 | **79** / 100 | 414.0 / 500 | random setting, 100 episodes |
| `handle_basket/pi05_lora_v1/10000` | handle_basket | LoRA | 10000 | **45** / 100 | 447.2 / 500 | random setting, 100 episodes |
| `handle_basket/pi05_lora_v1/12500` | handle_basket | LoRA | 12500 | **25** / 100 | 469.4 / 500 | random setting, 100 episodes |
| `handle_basket/pi05_lora_v1/14999` | handle_basket | LoRA | 14999 | **65** / 100 | 431.6 / 500 | random setting, 100 episodes |
| `item_assembly/pi05_lora_v1/2500` | item_assembly | LoRA | 2500 | **0** / 18 | β / 500 | early-stopped at 0/18 (no realistic chance against the baselines), gripper rescaling on |
| `item_assembly/pi05_lora_v1/5000` | item_assembly | LoRA | 5000 | **0** / 18 | β / 500 | early-stopped at 0/18 (no realistic chance against the baselines), gripper rescaling on |
| `item_assembly/pi05_lora_v1/7500` | item_assembly | LoRA | 7500 | **0** / 25 | β / 500 | early-stopped at 0/25 (no realistic chance against the baselines), gripper rescaling on |
| `item_assembly/pi05_lora_v1/10000` | item_assembly | LoRA | 10000 | **16** / 100 | 469.1 / 500 | random setting, 100 episodes, gripper rescaling on |
| `item_assembly/pi05_lora_v1/12500` | item_assembly | LoRA | 12500 | **2** / 40 | β / 500 | early-stopped at 2/40 (no realistic chance against the baselines), gripper rescaling on |
| `item_assembly/pi05_lora_v1/14999` | item_assembly | LoRA | 14999 | **13** / 100 | 474.6 / 500 | random setting, 100 episodes, gripper rescaling on |
## Recommended checkpoint per task
| Task | Best of ours | Official / community baselines (per 100) |
|---|---|---|
| table_rearrangement | `table_rearrangement/pi05_full_v1/15000 (78/100)` | ACT 96, SmolVLA 90, DP 17 |
| items_handover | `items_handover/pi05_full_v1/15000 (78/100)` | ACT 26, SmolVLA 22, DP 0 |
| handle_basket | `handle_basket/pi05_lora_v1/7500 (79/100)` | ACT 54, SmolVLA 33, DP 12 |
| item_assembly | `item_assembly/pi05_lora_v1/10000 (16/100, below every baseline)` | ACT 65, community Ο0.5 64, SmolVLA 51, DP 38 |
## Training recipes
| Run | Trainable weights | Steps | Batch | LR schedule | EMA | Wall clock | Checkpoint size |
|---|---|---|---|---|---|---|---|
| `table_rearrangement/pi05_full_v1` (full) | all 3.35B parameters | 20,000 | 64 | cosine 2.5e-5 β 2.5e-6 over 30k, warmup 1k | 0.99 | 34 h 58 min, one shared H100 NVL | 42 GB (params 12 GB + train_state 31 GB); the step-5000 folder holds params only |
| `items_handover/pi05_full_v1` (full) | all 3.35B parameters | 20,000 | 64 | cosine 2.5e-5 β 2.5e-6 over 30k, warmup 1k | 0.99 | 17 h 27 min on two H100 NVL (NVLink pair) | 52 GB (params 13 GB + train_state 39 GB) |
| `items_handover/pi05_short_v1` (full, short) | all 3.35B parameters | 5,000 (last save named 4999) | 64 | cosine 1e-5 β 1e-6 over 5k, warmup 250 | 0.99 | 3 h 22 min on two H100 NVL (NVLink pair) | 44 GB (params 12 GB + train_state 32 GB) |
| `<task>/pi05_lora_v1` (LoRA) | LoRA rank 16 (PaliGemma) / 32 (action expert) + SigLIP encoder + projections, 467M of 3.4B | 15,000 (last save named 14999) | 64 | cosine 2.5e-5 β 2.5e-6 over 15k, warmup 1k | off | 8 to 13 h on two H100 NVL | 9.5 GB (params + optimizer state of the trained subset) |
Action horizon 50, images 224Γ224 from three cameras, 14-D state and absolute joint actions with the openpi delta transform for the arm joints (config names `pi05_<task>` and `pi05_<task>_lora` in `policy/pi05/src/openpi/training/config.py` of the team repository).
## Using a checkpoint
From the team repository (`arlo-yang/RoboSynChallenge`), with the Ο0.5 inference worker installed (`policy/pi05_community/setup.sh`):
```bash
python web/scripts/hf_hub.py download checkpoint handle_basket/pi05_lora_v1/7500 # -> checkpoints/handle_basket/pi05_lora_v1/7500
conda activate robosyn && export ROBOSYN_RUNTIME=$CONDA_PREFIX
bash policy/pi05_community/inference.sh eval --task handle_basket --episodes 100 --gpu 0 \
--local-checkpoint checkpoints/handle_basket/pi05_lora_v1/7500 --config-name pi05_handle_basket_lora --label hb_7500 --rescale-gripper
```
Pass the openpi config the checkpoint was trained with (`--config-name pi05_<task>` for the full fine-tune, `pi05_<task>_lora` for LoRA). The worker executes 10 of the 50 predicted actions per call.
**Gripper scaling.** The official datasets store the two gripper targets in [0, 1] for Items hand-over, Handle basket and Item assembly, while the simulator clips actions to [0, 0.05]; pass `--rescale-gripper` for those tasks. Table rearrangement stores physical gripper positions and must not use it. (The Items hand-over `pi05_lora_v1` and Handle basket numbers above were measured without rescaling, which happens to act as an open/close threshold; the other Items hand-over runs and Item assembly used the rescaling.)
## Notes
- Success rates come from our own 100-episode evaluations in the `random` setting with expert feasibility screening, on the same EmbodiChain simulator as the official baselines; they are not official scores. A 100-episode run has roughly a 5-point standard error.
- LoRA success is not monotonic in training steps on these tasks (e.g. Handle basket 79 at step 7500, then 45, 25, 65) while the training loss falls smoothly; pick checkpoints by evaluation, not by loss.
- Replays, training records, loss curves and per-checkpoint analyses are on the team page (task pages, "Policies" and "Our fine-tuning runs").
- Datasets for augmentation experiments are in `2bidoubi/Robo_trajectory`; this repository holds model weights only.
- **Corrected Items hand-over scores (2026-10-11).** Between 2026-10-03 and 2026-10-11 our local copy of the Items hand-over task carried a rewritten pen-holder overlap check that rejected nearly every real success. Every Items hand-over evaluation of `pi05_full_v1`, `pi05_short_v1`, `pi05_mix_lora_v1` and `pi05_aug_lora_v1` from that period was invalid; the official check was restored (team repository commit 401a2f5) and every checkpoint still on disk was re-evaluated. The table above shows the corrected numbers; rows marked "broken check" are old results of checkpoints that were removed before the bug was found. `pi05_lora_v1` was evaluated before the bug and is unaffected.
- `items_handover/pi05_full_v1/*`: full fine-tune of all weights on the official data (20k steps, EMA 0.99, same recipe as `table_rearrangement/pi05_full_v1`); 58, 72, 78 and 77 per 100 at steps 5000, 10000, 15000 and 19999. Success rises steadily and plateaus after step 15000; `items_handover/pi05_full_v1/15000` (78/100) is the recommended Items hand-over checkpoint, twice the LoRA peak (38) and three times the ACT baseline (26).
- `items_handover/pi05_short_v1/*`: the full fine-tune repeated for 5000 steps with the learning-rate schedule scaled to the run (warmup 250, peak 1e-5, cosine to 1e-6): 23, 34, 44, 59 and 65 per 100 at steps 1000 to 4999, still rising at the end. A good policy in 3.4 h instead of 17.5 h, but 13 points below the 20k run.
- `items_handover/pi05_mix_lora_v1/*`: the same LoRA recipe on the official 1000 episodes merged with the 1702 team-generated ones (`web/scripts/merge_lerobot_datasets.py`); 16/100 at step 12500 against 38/100 for the official-only LoRA, because the generated episodes (no appearance randomisation) dominate the mix. Kept for the record.
- `items_handover/pi05_aug_lora_v1/*` was trained with the same LoRA recipe on the team-generated `robotrajectory/RoboSyn-Trajectory` folder `items_handover_augmented_1702` (1702 episodes) instead of the official release. Those episodes have no appearance randomisation (white table, grey holder, black pen, fixed layout, no distractors), while the official data and the evaluation randomise textures, colours, poses and distractors, so no checkpoint scores above 0 to 2 successes; evaluations were stopped early (0 successes after 25 episodes or at most 2 after 40). Kept for the record only.
|