Index 22 finished PI0.5 retraining peaks with 50-episode evaluation provenance
Browse files
README.md
CHANGED
|
@@ -6,12 +6,11 @@ license: mit
|
|
| 6 |
|
| 7 |
## single_task
|
| 8 |
|
| 9 |
-
One PI0.5 LoRA checkpoint per ManiLadder task (
|
| 10 |
Checkpoints are selected by the highest **50-episode** success rate; ties use the earliest step.
|
| 11 |
|
| 12 |
-
- The
|
| 13 |
-
- The
|
| 14 |
-
- The 10 tasks from the HF-data retraining campaign `pi05_hf21_20261025` (added 2026-10-29; raw demos freshly downloaded from Linslab/ManiLadder_Dataset) use seeds 1000–1049. All reached 40,000 training steps and have all eight 5k–40k evaluations. See [its manifest](single_task/pi05_hf21_20261025_manifest.json).
|
| 15 |
|
| 16 |
See [retraining provenance and missing steps](single_task/pi05_retrain_eval50_20261022_manifest.json). These scores use different seed sets across the two groups and are not a controlled comparison.
|
| 17 |
|
|
@@ -22,7 +21,7 @@ Each `single_task/<task>/` folder contains:
|
|
| 22 |
- `eval_result.json` — 50-episode evaluation of this checkpoint
|
| 23 |
|
| 24 |
`single_task/results.csv` lists the selected checkpoint per task; `single_task/all_checkpoints_eval.csv` has the
|
| 25 |
-
50-episode result of every evaluated checkpoint (
|
| 26 |
|
| 27 |
Evaluation: `exec_steps=16`, `sample_steps=10`. Tasks whose checkpoints all scored 0/50 are still included;
|
| 28 |
for them the step choice is arbitrary.
|
|
@@ -78,8 +77,10 @@ for them the step choice is arbitrary.
|
|
| 78 |
| ReorientPen-v1 | 2 | 25000 | 0.38 |
|
| 79 |
| RollingPin-v1 | 2 | 25000 | 0.96 |
|
| 80 |
| RopeBridgeDex-v1 | 2 | 30000 | 0.78 |
|
|
|
|
| 81 |
| StraightenTwoRopesSequential-v1 | 2 | 15000 | 0.36 |
|
| 82 |
| ClutterGrasp-v1 | 3 | 15000 | 0.16 |
|
|
|
|
| 83 |
| DualArmOpenFridge-v1 | 3 | 5000 | 0.00 |
|
| 84 |
| DualArmOrganizeCabinet-v1 | 3 | 5000 | 0.00 |
|
| 85 |
| DualArmPingpongBalance-v1 | 3 | 15000 | 0.20 |
|
|
|
|
| 6 |
|
| 7 |
## single_task
|
| 8 |
|
| 9 |
+
One PI0.5 LoRA checkpoint per ManiLadder task (92 tasks), each finetuned on 20 demos.
|
| 10 |
Checkpoints are selected by the highest **50-episode** success rate; ties use the earliest step.
|
| 11 |
|
| 12 |
+
- The 70 previously published tasks use fixed evaluation seeds 1000–1049.
|
| 13 |
+
- The 22 completed retraining runs published through 2026-10-30 use seeds 800000–800049. All reached 40,000 training steps. 15 have all eight 5k–40k evaluations; 7 have seven evaluations because one intermediate checkpoint was not retained. Their selected weights are the best among the available evaluated checkpoints.
|
|
|
|
| 14 |
|
| 15 |
See [retraining provenance and missing steps](single_task/pi05_retrain_eval50_20261022_manifest.json). These scores use different seed sets across the two groups and are not a controlled comparison.
|
| 16 |
|
|
|
|
| 21 |
- `eval_result.json` — 50-episode evaluation of this checkpoint
|
| 22 |
|
| 23 |
`single_task/results.csv` lists the selected checkpoint per task; `single_task/all_checkpoints_eval.csv` has the
|
| 24 |
+
50-episode result of every evaluated checkpoint (726 total).
|
| 25 |
|
| 26 |
Evaluation: `exec_steps=16`, `sample_steps=10`. Tasks whose checkpoints all scored 0/50 are still included;
|
| 27 |
for them the step choice is arbitrary.
|
|
|
|
| 77 |
| ReorientPen-v1 | 2 | 25000 | 0.38 |
|
| 78 |
| RollingPin-v1 | 2 | 25000 | 0.96 |
|
| 79 |
| RopeBridgeDex-v1 | 2 | 30000 | 0.78 |
|
| 80 |
+
| ScoopWater-v1 | 2 | 15000 | 0.44 |
|
| 81 |
| StraightenTwoRopesSequential-v1 | 2 | 15000 | 0.36 |
|
| 82 |
| ClutterGrasp-v1 | 3 | 15000 | 0.16 |
|
| 83 |
+
| DualArmFoldCarton-v1 | 3 | 25000 | 0.14 |
|
| 84 |
| DualArmOpenFridge-v1 | 3 | 5000 | 0.00 |
|
| 85 |
| DualArmOrganizeCabinet-v1 | 3 | 5000 | 0.00 |
|
| 86 |
| DualArmPingpongBalance-v1 | 3 | 15000 | 0.20 |
|
single_task/all_checkpoints_eval.csv
CHANGED
|
@@ -386,6 +386,14 @@ RopeBridgeDex-v1,level_2,25000,0.72,36,261.64974
|
|
| 386 |
RopeBridgeDex-v1,level_2,30000,0.78,39,238.96741999999998
|
| 387 |
RopeBridgeDex-v1,level_2,35000,0.68,34,256.62446
|
| 388 |
RopeBridgeDex-v1,level_2,40000,0.74,37,245.70774
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 389 |
StraightenTwoRopesSequential-v1,level_2,5000,0.32,16,1831.5910737252236
|
| 390 |
StraightenTwoRopesSequential-v1,level_2,10000,0.32,16,1816.7860989999772
|
| 391 |
StraightenTwoRopesSequential-v1,level_2,15000,0.36,18,1811.211635375023
|
|
@@ -402,6 +410,14 @@ ClutterGrasp-v1,level_3,25000,0.12,6,nan
|
|
| 402 |
ClutterGrasp-v1,level_3,30000,0.16,8,nan
|
| 403 |
ClutterGrasp-v1,level_3,35000,0.14,7,nan
|
| 404 |
ClutterGrasp-v1,level_3,40000,0.16,8,nan
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 405 |
DualArmOpenFridge-v1,level_3,5000,0.0,0,1425.2416665172577
|
| 406 |
DualArmOpenFridge-v1,level_3,10000,0.0,0,1425.2351796388625
|
| 407 |
DualArmOpenFridge-v1,level_3,15000,0.0,0,1425.235145483017
|
|
|
|
| 386 |
RopeBridgeDex-v1,level_2,30000,0.78,39,238.96741999999998
|
| 387 |
RopeBridgeDex-v1,level_2,35000,0.68,34,256.62446
|
| 388 |
RopeBridgeDex-v1,level_2,40000,0.74,37,245.70774
|
| 389 |
+
ScoopWater-v1,level_2,5000,0.32,16,-22.15058991558617
|
| 390 |
+
ScoopWater-v1,level_2,10000,0.22,11,-29.78825742039131
|
| 391 |
+
ScoopWater-v1,level_2,15000,0.44,22,-23.43375153155648
|
| 392 |
+
ScoopWater-v1,level_2,20000,0.42,21,-36.24944539662567
|
| 393 |
+
ScoopWater-v1,level_2,25000,0.22,11,-29.69474442129256
|
| 394 |
+
ScoopWater-v1,level_2,30000,0.28,14,-33.832715197753856
|
| 395 |
+
ScoopWater-v1,level_2,35000,0.32,16,-18.228975554266945
|
| 396 |
+
ScoopWater-v1,level_2,40000,0.38,19,-16.857123770004836
|
| 397 |
StraightenTwoRopesSequential-v1,level_2,5000,0.32,16,1831.5910737252236
|
| 398 |
StraightenTwoRopesSequential-v1,level_2,10000,0.32,16,1816.7860989999772
|
| 399 |
StraightenTwoRopesSequential-v1,level_2,15000,0.36,18,1811.211635375023
|
|
|
|
| 410 |
ClutterGrasp-v1,level_3,30000,0.16,8,nan
|
| 411 |
ClutterGrasp-v1,level_3,35000,0.14,7,nan
|
| 412 |
ClutterGrasp-v1,level_3,40000,0.16,8,nan
|
| 413 |
+
DualArmFoldCarton-v1,level_3,5000,0.0,0,6345.062264726162
|
| 414 |
+
DualArmFoldCarton-v1,level_3,10000,0.02,1,8489.919916040897
|
| 415 |
+
DualArmFoldCarton-v1,level_3,15000,0.04,2,9307.387320964337
|
| 416 |
+
DualArmFoldCarton-v1,level_3,20000,0.04,2,9497.66813904047
|
| 417 |
+
DualArmFoldCarton-v1,level_3,25000,0.14,7,10516.428281509876
|
| 418 |
+
DualArmFoldCarton-v1,level_3,30000,0.06,3,9551.720534439088
|
| 419 |
+
DualArmFoldCarton-v1,level_3,35000,0.1,5,10555.199788849353
|
| 420 |
+
DualArmFoldCarton-v1,level_3,40000,0.04,2,9328.819932291508
|
| 421 |
DualArmOpenFridge-v1,level_3,5000,0.0,0,1425.2416665172577
|
| 422 |
DualArmOpenFridge-v1,level_3,10000,0.0,0,1425.2351796388625
|
| 423 |
DualArmOpenFridge-v1,level_3,15000,0.0,0,1425.235145483017
|
single_task/pi05_retrain_eval50_20261022_manifest.json
CHANGED
|
@@ -160,6 +160,21 @@
|
|
| 160 |
"checkpoint": "single_task/DualArmStraightenRope-v1/lora_step_0035000.npz",
|
| 161 |
"checkpoint_sha256": "057ad6ce58a2019cf1762b4ff8284a6f475e4a17875e6a6a56a82c9b621f1482"
|
| 162 |
},
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 163 |
{
|
| 164 |
"task": "DualArmJugglingBall-v1",
|
| 165 |
"level": "level_2",
|
|
@@ -286,6 +301,21 @@
|
|
| 286 |
"checkpoint": "single_task/DualArmPlugCharger-v1/lora_step_0005000.npz",
|
| 287 |
"checkpoint_sha256": "9ec70795a17dbc33ac7fcc21a8eeb343d2a4844813aee8048c2e4dd60be77a2b"
|
| 288 |
},
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 289 |
{
|
| 290 |
"task": "GroceryPrepare-v1",
|
| 291 |
"level": "level_3",
|
|
|
|
| 160 |
"checkpoint": "single_task/DualArmStraightenRope-v1/lora_step_0035000.npz",
|
| 161 |
"checkpoint_sha256": "057ad6ce58a2019cf1762b4ff8284a6f475e4a17875e6a6a56a82c9b621f1482"
|
| 162 |
},
|
| 163 |
+
{
|
| 164 |
+
"task": "ScoopWater-v1",
|
| 165 |
+
"level": "level_2",
|
| 166 |
+
"suite": "deformable/single_arm_gripper",
|
| 167 |
+
"step": 15000,
|
| 168 |
+
"success_rate": 0.44,
|
| 169 |
+
"successes": 22,
|
| 170 |
+
"episodes": 50,
|
| 171 |
+
"episode_seed_base": 800000,
|
| 172 |
+
"ckpts_evaluated": 8,
|
| 173 |
+
"missing_eval_steps": [],
|
| 174 |
+
"campaign": "pi05_retrain_eval50_20261022",
|
| 175 |
+
"checkpoint": "single_task/ScoopWater-v1/lora_step_0015000.npz",
|
| 176 |
+
"checkpoint_sha256": "94b345769052c61796cbc58956f3a56105fadcf98daa7f45d35bc64b599a6325"
|
| 177 |
+
},
|
| 178 |
{
|
| 179 |
"task": "DualArmJugglingBall-v1",
|
| 180 |
"level": "level_2",
|
|
|
|
| 301 |
"checkpoint": "single_task/DualArmPlugCharger-v1/lora_step_0005000.npz",
|
| 302 |
"checkpoint_sha256": "9ec70795a17dbc33ac7fcc21a8eeb343d2a4844813aee8048c2e4dd60be77a2b"
|
| 303 |
},
|
| 304 |
+
{
|
| 305 |
+
"task": "DualArmFoldCarton-v1",
|
| 306 |
+
"level": "level_3",
|
| 307 |
+
"suite": "rigid_body/dual_arm_gripper",
|
| 308 |
+
"step": 25000,
|
| 309 |
+
"success_rate": 0.14,
|
| 310 |
+
"successes": 7,
|
| 311 |
+
"episodes": 50,
|
| 312 |
+
"episode_seed_base": 800000,
|
| 313 |
+
"ckpts_evaluated": 8,
|
| 314 |
+
"missing_eval_steps": [],
|
| 315 |
+
"campaign": "pi05_retrain_eval50_20261022",
|
| 316 |
+
"checkpoint": "single_task/DualArmFoldCarton-v1/lora_step_0025000.npz",
|
| 317 |
+
"checkpoint_sha256": "b9be59d2eaf9ff7e730b635196923c00a4b11df3b1b9395e1be7d73fab771f71"
|
| 318 |
+
},
|
| 319 |
{
|
| 320 |
"task": "GroceryPrepare-v1",
|
| 321 |
"level": "level_3",
|
single_task/results.csv
CHANGED
|
@@ -48,8 +48,10 @@ PivotBook-v1,level_2,15000,0.38,19,8,single_task/PivotBook-v1/lora_step_0015000.
|
|
| 48 |
ReorientPen-v1,level_2,25000,0.38,19,8,single_task/ReorientPen-v1/lora_step_0025000.npz
|
| 49 |
RollingPin-v1,level_2,25000,0.96,48,8,single_task/RollingPin-v1/lora_step_0025000.npz
|
| 50 |
RopeBridgeDex-v1,level_2,30000,0.78,39,8,single_task/RopeBridgeDex-v1/lora_step_0030000.npz
|
|
|
|
| 51 |
StraightenTwoRopesSequential-v1,level_2,15000,0.36,18,8,single_task/StraightenTwoRopesSequential-v1/lora_step_0015000.npz
|
| 52 |
ClutterGrasp-v1,level_3,15000,0.16,8,8,single_task/ClutterGrasp-v1/lora_step_0015000.npz
|
|
|
|
| 53 |
DualArmOpenFridge-v1,level_3,5000,0.0,0,8,single_task/DualArmOpenFridge-v1/lora_step_0005000.npz
|
| 54 |
DualArmOrganizeCabinet-v1,level_3,5000,0.0,0,8,single_task/DualArmOrganizeCabinet-v1/lora_step_0005000.npz
|
| 55 |
DualArmPingpongBalance-v1,level_3,15000,0.2,10,8,single_task/DualArmPingpongBalance-v1/lora_step_0015000.npz
|
|
|
|
| 48 |
ReorientPen-v1,level_2,25000,0.38,19,8,single_task/ReorientPen-v1/lora_step_0025000.npz
|
| 49 |
RollingPin-v1,level_2,25000,0.96,48,8,single_task/RollingPin-v1/lora_step_0025000.npz
|
| 50 |
RopeBridgeDex-v1,level_2,30000,0.78,39,8,single_task/RopeBridgeDex-v1/lora_step_0030000.npz
|
| 51 |
+
ScoopWater-v1,level_2,15000,0.44,22,8,single_task/ScoopWater-v1/lora_step_0015000.npz
|
| 52 |
StraightenTwoRopesSequential-v1,level_2,15000,0.36,18,8,single_task/StraightenTwoRopesSequential-v1/lora_step_0015000.npz
|
| 53 |
ClutterGrasp-v1,level_3,15000,0.16,8,8,single_task/ClutterGrasp-v1/lora_step_0015000.npz
|
| 54 |
+
DualArmFoldCarton-v1,level_3,25000,0.14,7,8,single_task/DualArmFoldCarton-v1/lora_step_0025000.npz
|
| 55 |
DualArmOpenFridge-v1,level_3,5000,0.0,0,8,single_task/DualArmOpenFridge-v1/lora_step_0005000.npz
|
| 56 |
DualArmOrganizeCabinet-v1,level_3,5000,0.0,0,8,single_task/DualArmOrganizeCabinet-v1/lora_step_0005000.npz
|
| 57 |
DualArmPingpongBalance-v1,level_3,15000,0.2,10,8,single_task/DualArmPingpongBalance-v1/lora_step_0015000.npz
|