Robo_model / README.md
2bidoubi's picture
Model card: corrected Items hand-over scores, full fine-tune 78/100 at step 15000 recommended
18489c8 verified
|
Raw History Blame Contribute Delete
15.1 kB
metadata
license: other
base_model: physical-intelligence/pi05-base
tags:
  - robotics
  - vla
  - pi0.5
  - openpi
  - cobotmagic
  - embodichain
pipeline_tag: robotics

RoboSynChallenge team checkpoints (Ο€0.5)

Ο€0.5 checkpoints fine-tuned by our team for the RoboSynChallenge CobotMagic simulation tasks, trained with openpi from pi05_base on the official RoboSynChallenge/cobotmagic_Sim_<task> datasets (1,000 expert episodes per task, LeRobot v2.1). The repository has one folder per task (ten tasks); inside, <run>/<step>/ is one openpi checkpoint step: params/ (full model parameters, also for LoRA runs), assets/<repo_id>/norm_stats.json and, for most folders, train_state/ (optimizer state) for resuming.

Checkpoints (<task>/<run>/<step>)

Path Task Run Step Success / episodes Mean action steps / limit Evaluation
table_rearrangement/pi05_full_v1/5000 table_rearrangement full fine-tune 5000 60 / 100 246.6 / 361 random setting, 100 episodes
table_rearrangement/pi05_full_v1/10000 table_rearrangement full fine-tune 10000 73 / 100 219.1 / 361 random setting, 100 episodes
table_rearrangement/pi05_full_v1/15000 table_rearrangement full fine-tune 15000 78 / 100 209.3 / 361 random setting, 100 episodes
table_rearrangement/pi05_full_v1/19999 table_rearrangement full fine-tune 19999 76 / 100 229.3 / 361 random setting, 100 episodes
table_rearrangement/pi05_lora_v1/2500 table_rearrangement LoRA 2500 32 / 100 299.2 / 361 random setting, 100 episodes
table_rearrangement/pi05_lora_v1/5000 table_rearrangement LoRA 5000 39 / 100 294.6 / 361 random setting, 100 episodes
table_rearrangement/pi05_lora_v1/7500 table_rearrangement LoRA 7500 52 / 100 259.5 / 361 random setting, 100 episodes
table_rearrangement/pi05_lora_v1/10000 table_rearrangement LoRA 10000 56 / 100 265.4 / 361 random setting, 100 episodes
table_rearrangement/pi05_lora_v1/12500 table_rearrangement LoRA 12500 70 / 100 233.6 / 361 random setting, 100 episodes
table_rearrangement/pi05_lora_v1/14999 table_rearrangement LoRA 14999 76 / 100 223.9 / 361 random setting, 100 episodes
items_handover/pi05_lora_v1/2500 items_handover LoRA 2500 7 / 100 348.3 / 350 random setting, 100 episodes
items_handover/pi05_lora_v1/5000 items_handover LoRA 5000 1 / 100 349.5 / 350 random setting, 100 episodes
items_handover/pi05_lora_v1/7500 items_handover LoRA 7500 11 / 100 349.0 / 350 random setting, 100 episodes
items_handover/pi05_lora_v1/10000 items_handover LoRA 10000 14 / 100 347.6 / 350 random setting, 100 episodes
items_handover/pi05_lora_v1/12500 items_handover LoRA 12500 38 / 100 342.2 / 350 random setting, 100 episodes
items_handover/pi05_lora_v1/14999 items_handover LoRA 14999 14 / 100 348.4 / 350 random setting, 100 episodes
items_handover/pi05_aug_lora_v1/2500 items_handover LoRA (team-generated data) 2500 0 / 100 350.0 / 350 random setting, 100 episodes; trained on the team-generated data (see Notes); measured with the broken success check, not re-run (checkpoint removed)
items_handover/pi05_aug_lora_v1/5000 items_handover LoRA (team-generated data) 5000 0 / 25 β€” early-stopped (corrected check, 2026-10-11)
items_handover/pi05_aug_lora_v1/7500 items_handover LoRA (team-generated data) 7500 0 / 25 β€” early-stopped (broken check, checkpoint removed)
items_handover/pi05_aug_lora_v1/10000 items_handover LoRA (team-generated data) 10000 0 / 25 β€” early-stopped (broken check, checkpoint removed)
items_handover/pi05_aug_lora_v1/12500 items_handover LoRA (team-generated data) 12500 0 / 25 β€” early-stopped (corrected check, 2026-10-11)
items_handover/pi05_aug_lora_v1/14999 items_handover LoRA (team-generated data) 14999 0 / 25 β€” early-stopped (broken check, checkpoint removed)
items_handover/pi05_mix_lora_v1/2500 items_handover LoRA (mixed data) 2500 0 / 25 β€” early-stopped (broken check, checkpoint removed); official + team-generated data (2702 episodes)
items_handover/pi05_mix_lora_v1/5000 items_handover LoRA (mixed data) 5000 2 / 40 β€” early-stopped (corrected check, 2026-10-11)
items_handover/pi05_mix_lora_v1/7500 items_handover LoRA (mixed data) 7500 0 / 25 β€” early-stopped (broken check, checkpoint removed)
items_handover/pi05_mix_lora_v1/10000 items_handover LoRA (mixed data) 10000 1 / 13 β€” cut short, NAS full (broken check, checkpoint removed)
items_handover/pi05_mix_lora_v1/12500 items_handover LoRA (mixed data) 12500 16 / 100 348.3 / 350 random setting, 100 episodes, gripper rescaling on (corrected check, 2026-10-11)
items_handover/pi05_mix_lora_v1/14999 items_handover LoRA (mixed data) 14999 0 / 25 β€” early-stopped (broken check, checkpoint removed)
items_handover/pi05_full_v1/5000 items_handover Full 5000 58 / 100 340.7 / 350 random setting, 100 episodes, gripper rescaling on; full fine-tune on the official data
items_handover/pi05_full_v1/10000 items_handover Full 10000 72 / 100 337.7 / 350 random setting, 100 episodes, gripper rescaling on
items_handover/pi05_full_v1/15000 items_handover Full 15000 78 / 100 338.4 / 350 random setting, 100 episodes, gripper rescaling on; recommended
items_handover/pi05_full_v1/19999 items_handover Full 19999 77 / 100 340.2 / 350 random setting, 100 episodes, gripper rescaling on; final
items_handover/pi05_short_v1/1000 items_handover Full (5k, lr 1e-5) 1000 23 / 100 345.4 / 350 random setting, 100 episodes, gripper rescaling on; 5000-step full fine-tune with the schedule scaled to its length
items_handover/pi05_short_v1/2000 items_handover Full (5k, lr 1e-5) 2000 34 / 100 343.8 / 350 random setting, 100 episodes, gripper rescaling on
items_handover/pi05_short_v1/3000 items_handover Full (5k, lr 1e-5) 3000 44 / 100 343.6 / 350 random setting, 100 episodes, gripper rescaling on
items_handover/pi05_short_v1/4000 items_handover Full (5k, lr 1e-5) 4000 59 / 100 342.2 / 350 random setting, 100 episodes, gripper rescaling on
items_handover/pi05_short_v1/4999 items_handover Full (5k, lr 1e-5) 4999 65 / 100 340.3 / 350 random setting, 100 episodes, gripper rescaling on; final
handle_basket/pi05_lora_v1/2500 handle_basket LoRA 2500 28 / 100 472.9 / 500 random setting, 100 episodes
handle_basket/pi05_lora_v1/5000 handle_basket LoRA 5000 35 / 100 467.1 / 500 random setting, 100 episodes
handle_basket/pi05_lora_v1/7500 handle_basket LoRA 7500 79 / 100 414.0 / 500 random setting, 100 episodes
handle_basket/pi05_lora_v1/10000 handle_basket LoRA 10000 45 / 100 447.2 / 500 random setting, 100 episodes
handle_basket/pi05_lora_v1/12500 handle_basket LoRA 12500 25 / 100 469.4 / 500 random setting, 100 episodes
handle_basket/pi05_lora_v1/14999 handle_basket LoRA 14999 65 / 100 431.6 / 500 random setting, 100 episodes
item_assembly/pi05_lora_v1/2500 item_assembly LoRA 2500 0 / 18 β€” / 500 early-stopped at 0/18 (no realistic chance against the baselines), gripper rescaling on
item_assembly/pi05_lora_v1/5000 item_assembly LoRA 5000 0 / 18 β€” / 500 early-stopped at 0/18 (no realistic chance against the baselines), gripper rescaling on
item_assembly/pi05_lora_v1/7500 item_assembly LoRA 7500 0 / 25 β€” / 500 early-stopped at 0/25 (no realistic chance against the baselines), gripper rescaling on
item_assembly/pi05_lora_v1/10000 item_assembly LoRA 10000 16 / 100 469.1 / 500 random setting, 100 episodes, gripper rescaling on
item_assembly/pi05_lora_v1/12500 item_assembly LoRA 12500 2 / 40 β€” / 500 early-stopped at 2/40 (no realistic chance against the baselines), gripper rescaling on
item_assembly/pi05_lora_v1/14999 item_assembly LoRA 14999 13 / 100 474.6 / 500 random setting, 100 episodes, gripper rescaling on

Recommended checkpoint per task

Task Best of ours Official / community baselines (per 100)
table_rearrangement table_rearrangement/pi05_full_v1/15000 (78/100) ACT 96, SmolVLA 90, DP 17
items_handover items_handover/pi05_full_v1/15000 (78/100) ACT 26, SmolVLA 22, DP 0
handle_basket handle_basket/pi05_lora_v1/7500 (79/100) ACT 54, SmolVLA 33, DP 12
item_assembly item_assembly/pi05_lora_v1/10000 (16/100, below every baseline) ACT 65, community Ο€0.5 64, SmolVLA 51, DP 38

Training recipes

Run Trainable weights Steps Batch LR schedule EMA Wall clock Checkpoint size
table_rearrangement/pi05_full_v1 (full) all 3.35B parameters 20,000 64 cosine 2.5e-5 β†’ 2.5e-6 over 30k, warmup 1k 0.99 34 h 58 min, one shared H100 NVL 42 GB (params 12 GB + train_state 31 GB); the step-5000 folder holds params only
items_handover/pi05_full_v1 (full) all 3.35B parameters 20,000 64 cosine 2.5e-5 β†’ 2.5e-6 over 30k, warmup 1k 0.99 17 h 27 min on two H100 NVL (NVLink pair) 52 GB (params 13 GB + train_state 39 GB)
items_handover/pi05_short_v1 (full, short) all 3.35B parameters 5,000 (last save named 4999) 64 cosine 1e-5 β†’ 1e-6 over 5k, warmup 250 0.99 3 h 22 min on two H100 NVL (NVLink pair) 44 GB (params 12 GB + train_state 32 GB)
<task>/pi05_lora_v1 (LoRA) LoRA rank 16 (PaliGemma) / 32 (action expert) + SigLIP encoder + projections, 467M of 3.4B 15,000 (last save named 14999) 64 cosine 2.5e-5 β†’ 2.5e-6 over 15k, warmup 1k off 8 to 13 h on two H100 NVL 9.5 GB (params + optimizer state of the trained subset)

Action horizon 50, images 224Γ—224 from three cameras, 14-D state and absolute joint actions with the openpi delta transform for the arm joints (config names pi05_<task> and pi05_<task>_lora in policy/pi05/src/openpi/training/config.py of the team repository).

Using a checkpoint

From the team repository (arlo-yang/RoboSynChallenge), with the Ο€0.5 inference worker installed (policy/pi05_community/setup.sh):

python web/scripts/hf_hub.py download checkpoint handle_basket/pi05_lora_v1/7500      # -> checkpoints/handle_basket/pi05_lora_v1/7500
conda activate robosyn && export ROBOSYN_RUNTIME=$CONDA_PREFIX
bash policy/pi05_community/inference.sh eval --task handle_basket --episodes 100 --gpu 0 \
  --local-checkpoint checkpoints/handle_basket/pi05_lora_v1/7500 --config-name pi05_handle_basket_lora --label hb_7500 --rescale-gripper

Pass the openpi config the checkpoint was trained with (--config-name pi05_<task> for the full fine-tune, pi05_<task>_lora for LoRA). The worker executes 10 of the 50 predicted actions per call.

Gripper scaling. The official datasets store the two gripper targets in [0, 1] for Items hand-over, Handle basket and Item assembly, while the simulator clips actions to [0, 0.05]; pass --rescale-gripper for those tasks. Table rearrangement stores physical gripper positions and must not use it. (The Items hand-over pi05_lora_v1 and Handle basket numbers above were measured without rescaling, which happens to act as an open/close threshold; the other Items hand-over runs and Item assembly used the rescaling.)

Notes

  • Success rates come from our own 100-episode evaluations in the random setting with expert feasibility screening, on the same EmbodiChain simulator as the official baselines; they are not official scores. A 100-episode run has roughly a 5-point standard error.

  • LoRA success is not monotonic in training steps on these tasks (e.g. Handle basket 79 at step 7500, then 45, 25, 65) while the training loss falls smoothly; pick checkpoints by evaluation, not by loss.

  • Replays, training records, loss curves and per-checkpoint analyses are on the team page (task pages, "Policies" and "Our fine-tuning runs").

  • Datasets for augmentation experiments are in 2bidoubi/Robo_trajectory; this repository holds model weights only.

  • Corrected Items hand-over scores (2026-10-11). Between 2026-10-03 and 2026-10-11 our local copy of the Items hand-over task carried a rewritten pen-holder overlap check that rejected nearly every real success. Every Items hand-over evaluation of pi05_full_v1, pi05_short_v1, pi05_mix_lora_v1 and pi05_aug_lora_v1 from that period was invalid; the official check was restored (team repository commit 401a2f5) and every checkpoint still on disk was re-evaluated. The table above shows the corrected numbers; rows marked "broken check" are old results of checkpoints that were removed before the bug was found. pi05_lora_v1 was evaluated before the bug and is unaffected.

  • items_handover/pi05_full_v1/*: full fine-tune of all weights on the official data (20k steps, EMA 0.99, same recipe as table_rearrangement/pi05_full_v1); 58, 72, 78 and 77 per 100 at steps 5000, 10000, 15000 and 19999. Success rises steadily and plateaus after step 15000; items_handover/pi05_full_v1/15000 (78/100) is the recommended Items hand-over checkpoint, twice the LoRA peak (38) and three times the ACT baseline (26).

  • items_handover/pi05_short_v1/*: the full fine-tune repeated for 5000 steps with the learning-rate schedule scaled to the run (warmup 250, peak 1e-5, cosine to 1e-6): 23, 34, 44, 59 and 65 per 100 at steps 1000 to 4999, still rising at the end. A good policy in 3.4 h instead of 17.5 h, but 13 points below the 20k run.

  • items_handover/pi05_mix_lora_v1/*: the same LoRA recipe on the official 1000 episodes merged with the 1702 team-generated ones (web/scripts/merge_lerobot_datasets.py); 16/100 at step 12500 against 38/100 for the official-only LoRA, because the generated episodes (no appearance randomisation) dominate the mix. Kept for the record.

  • items_handover/pi05_aug_lora_v1/* was trained with the same LoRA recipe on the team-generated robotrajectory/RoboSyn-Trajectory folder items_handover_augmented_1702 (1702 episodes) instead of the official release. Those episodes have no appearance randomisation (white table, grey holder, black pen, fixed layout, no distractors), while the official data and the evaluation randomise textures, colours, poses and distractors, so no checkpoint scores above 0 to 2 successes; evaluations were stopped early (0 successes after 25 episodes or at most 2 after 40). Kept for the record only.