Linslab commited on
Commit
f9b1c3c
·
verified ·
1 Parent(s): 127775f

Index 22 finished PI0.5 retraining peaks with 50-episode evaluation provenance

Browse files
README.md CHANGED
@@ -6,12 +6,11 @@ license: mit
6
 
7
  ## single_task
8
 
9
- One PI0.5 LoRA checkpoint per ManiLadder task (90 tasks), each finetuned on 20 demos.
10
  Checkpoints are selected by the highest **50-episode** success rate; ties use the earliest step.
11
 
12
- - The 60 previously published tasks use fixed evaluation seeds 1000–1049.
13
- - The 20 completed retraining runs added on 2026-10-28 use seeds 800000–800049. All reached 40,000 training steps. 13 have all eight 5k–40k evaluations; 7 have seven evaluations because one intermediate checkpoint was not retained. Their selected weights are the best among the available evaluated checkpoints.
14
- - The 10 tasks from the HF-data retraining campaign `pi05_hf21_20261025` (added 2026-10-29; raw demos freshly downloaded from Linslab/ManiLadder_Dataset) use seeds 1000–1049. All reached 40,000 training steps and have all eight 5k–40k evaluations. See [its manifest](single_task/pi05_hf21_20261025_manifest.json).
15
 
16
  See [retraining provenance and missing steps](single_task/pi05_retrain_eval50_20261022_manifest.json). These scores use different seed sets across the two groups and are not a controlled comparison.
17
 
@@ -22,7 +21,7 @@ Each `single_task/<task>/` folder contains:
22
  - `eval_result.json` — 50-episode evaluation of this checkpoint
23
 
24
  `single_task/results.csv` lists the selected checkpoint per task; `single_task/all_checkpoints_eval.csv` has the
25
- 50-episode result of every evaluated checkpoint (710 total).
26
 
27
  Evaluation: `exec_steps=16`, `sample_steps=10`. Tasks whose checkpoints all scored 0/50 are still included;
28
  for them the step choice is arbitrary.
@@ -78,8 +77,10 @@ for them the step choice is arbitrary.
78
  | ReorientPen-v1 | 2 | 25000 | 0.38 |
79
  | RollingPin-v1 | 2 | 25000 | 0.96 |
80
  | RopeBridgeDex-v1 | 2 | 30000 | 0.78 |
 
81
  | StraightenTwoRopesSequential-v1 | 2 | 15000 | 0.36 |
82
  | ClutterGrasp-v1 | 3 | 15000 | 0.16 |
 
83
  | DualArmOpenFridge-v1 | 3 | 5000 | 0.00 |
84
  | DualArmOrganizeCabinet-v1 | 3 | 5000 | 0.00 |
85
  | DualArmPingpongBalance-v1 | 3 | 15000 | 0.20 |
 
6
 
7
  ## single_task
8
 
9
+ One PI0.5 LoRA checkpoint per ManiLadder task (92 tasks), each finetuned on 20 demos.
10
  Checkpoints are selected by the highest **50-episode** success rate; ties use the earliest step.
11
 
12
+ - The 70 previously published tasks use fixed evaluation seeds 1000–1049.
13
+ - The 22 completed retraining runs published through 2026-10-30 use seeds 800000–800049. All reached 40,000 training steps. 15 have all eight 5k–40k evaluations; 7 have seven evaluations because one intermediate checkpoint was not retained. Their selected weights are the best among the available evaluated checkpoints.
 
14
 
15
  See [retraining provenance and missing steps](single_task/pi05_retrain_eval50_20261022_manifest.json). These scores use different seed sets across the two groups and are not a controlled comparison.
16
 
 
21
  - `eval_result.json` — 50-episode evaluation of this checkpoint
22
 
23
  `single_task/results.csv` lists the selected checkpoint per task; `single_task/all_checkpoints_eval.csv` has the
24
+ 50-episode result of every evaluated checkpoint (726 total).
25
 
26
  Evaluation: `exec_steps=16`, `sample_steps=10`. Tasks whose checkpoints all scored 0/50 are still included;
27
  for them the step choice is arbitrary.
 
77
  | ReorientPen-v1 | 2 | 25000 | 0.38 |
78
  | RollingPin-v1 | 2 | 25000 | 0.96 |
79
  | RopeBridgeDex-v1 | 2 | 30000 | 0.78 |
80
+ | ScoopWater-v1 | 2 | 15000 | 0.44 |
81
  | StraightenTwoRopesSequential-v1 | 2 | 15000 | 0.36 |
82
  | ClutterGrasp-v1 | 3 | 15000 | 0.16 |
83
+ | DualArmFoldCarton-v1 | 3 | 25000 | 0.14 |
84
  | DualArmOpenFridge-v1 | 3 | 5000 | 0.00 |
85
  | DualArmOrganizeCabinet-v1 | 3 | 5000 | 0.00 |
86
  | DualArmPingpongBalance-v1 | 3 | 15000 | 0.20 |
single_task/all_checkpoints_eval.csv CHANGED
@@ -386,6 +386,14 @@ RopeBridgeDex-v1,level_2,25000,0.72,36,261.64974
386
  RopeBridgeDex-v1,level_2,30000,0.78,39,238.96741999999998
387
  RopeBridgeDex-v1,level_2,35000,0.68,34,256.62446
388
  RopeBridgeDex-v1,level_2,40000,0.74,37,245.70774
 
 
 
 
 
 
 
 
389
  StraightenTwoRopesSequential-v1,level_2,5000,0.32,16,1831.5910737252236
390
  StraightenTwoRopesSequential-v1,level_2,10000,0.32,16,1816.7860989999772
391
  StraightenTwoRopesSequential-v1,level_2,15000,0.36,18,1811.211635375023
@@ -402,6 +410,14 @@ ClutterGrasp-v1,level_3,25000,0.12,6,nan
402
  ClutterGrasp-v1,level_3,30000,0.16,8,nan
403
  ClutterGrasp-v1,level_3,35000,0.14,7,nan
404
  ClutterGrasp-v1,level_3,40000,0.16,8,nan
 
 
 
 
 
 
 
 
405
  DualArmOpenFridge-v1,level_3,5000,0.0,0,1425.2416665172577
406
  DualArmOpenFridge-v1,level_3,10000,0.0,0,1425.2351796388625
407
  DualArmOpenFridge-v1,level_3,15000,0.0,0,1425.235145483017
 
386
  RopeBridgeDex-v1,level_2,30000,0.78,39,238.96741999999998
387
  RopeBridgeDex-v1,level_2,35000,0.68,34,256.62446
388
  RopeBridgeDex-v1,level_2,40000,0.74,37,245.70774
389
+ ScoopWater-v1,level_2,5000,0.32,16,-22.15058991558617
390
+ ScoopWater-v1,level_2,10000,0.22,11,-29.78825742039131
391
+ ScoopWater-v1,level_2,15000,0.44,22,-23.43375153155648
392
+ ScoopWater-v1,level_2,20000,0.42,21,-36.24944539662567
393
+ ScoopWater-v1,level_2,25000,0.22,11,-29.69474442129256
394
+ ScoopWater-v1,level_2,30000,0.28,14,-33.832715197753856
395
+ ScoopWater-v1,level_2,35000,0.32,16,-18.228975554266945
396
+ ScoopWater-v1,level_2,40000,0.38,19,-16.857123770004836
397
  StraightenTwoRopesSequential-v1,level_2,5000,0.32,16,1831.5910737252236
398
  StraightenTwoRopesSequential-v1,level_2,10000,0.32,16,1816.7860989999772
399
  StraightenTwoRopesSequential-v1,level_2,15000,0.36,18,1811.211635375023
 
410
  ClutterGrasp-v1,level_3,30000,0.16,8,nan
411
  ClutterGrasp-v1,level_3,35000,0.14,7,nan
412
  ClutterGrasp-v1,level_3,40000,0.16,8,nan
413
+ DualArmFoldCarton-v1,level_3,5000,0.0,0,6345.062264726162
414
+ DualArmFoldCarton-v1,level_3,10000,0.02,1,8489.919916040897
415
+ DualArmFoldCarton-v1,level_3,15000,0.04,2,9307.387320964337
416
+ DualArmFoldCarton-v1,level_3,20000,0.04,2,9497.66813904047
417
+ DualArmFoldCarton-v1,level_3,25000,0.14,7,10516.428281509876
418
+ DualArmFoldCarton-v1,level_3,30000,0.06,3,9551.720534439088
419
+ DualArmFoldCarton-v1,level_3,35000,0.1,5,10555.199788849353
420
+ DualArmFoldCarton-v1,level_3,40000,0.04,2,9328.819932291508
421
  DualArmOpenFridge-v1,level_3,5000,0.0,0,1425.2416665172577
422
  DualArmOpenFridge-v1,level_3,10000,0.0,0,1425.2351796388625
423
  DualArmOpenFridge-v1,level_3,15000,0.0,0,1425.235145483017
single_task/pi05_retrain_eval50_20261022_manifest.json CHANGED
@@ -160,6 +160,21 @@
160
  "checkpoint": "single_task/DualArmStraightenRope-v1/lora_step_0035000.npz",
161
  "checkpoint_sha256": "057ad6ce58a2019cf1762b4ff8284a6f475e4a17875e6a6a56a82c9b621f1482"
162
  },
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
163
  {
164
  "task": "DualArmJugglingBall-v1",
165
  "level": "level_2",
@@ -286,6 +301,21 @@
286
  "checkpoint": "single_task/DualArmPlugCharger-v1/lora_step_0005000.npz",
287
  "checkpoint_sha256": "9ec70795a17dbc33ac7fcc21a8eeb343d2a4844813aee8048c2e4dd60be77a2b"
288
  },
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
289
  {
290
  "task": "GroceryPrepare-v1",
291
  "level": "level_3",
 
160
  "checkpoint": "single_task/DualArmStraightenRope-v1/lora_step_0035000.npz",
161
  "checkpoint_sha256": "057ad6ce58a2019cf1762b4ff8284a6f475e4a17875e6a6a56a82c9b621f1482"
162
  },
163
+ {
164
+ "task": "ScoopWater-v1",
165
+ "level": "level_2",
166
+ "suite": "deformable/single_arm_gripper",
167
+ "step": 15000,
168
+ "success_rate": 0.44,
169
+ "successes": 22,
170
+ "episodes": 50,
171
+ "episode_seed_base": 800000,
172
+ "ckpts_evaluated": 8,
173
+ "missing_eval_steps": [],
174
+ "campaign": "pi05_retrain_eval50_20261022",
175
+ "checkpoint": "single_task/ScoopWater-v1/lora_step_0015000.npz",
176
+ "checkpoint_sha256": "94b345769052c61796cbc58956f3a56105fadcf98daa7f45d35bc64b599a6325"
177
+ },
178
  {
179
  "task": "DualArmJugglingBall-v1",
180
  "level": "level_2",
 
301
  "checkpoint": "single_task/DualArmPlugCharger-v1/lora_step_0005000.npz",
302
  "checkpoint_sha256": "9ec70795a17dbc33ac7fcc21a8eeb343d2a4844813aee8048c2e4dd60be77a2b"
303
  },
304
+ {
305
+ "task": "DualArmFoldCarton-v1",
306
+ "level": "level_3",
307
+ "suite": "rigid_body/dual_arm_gripper",
308
+ "step": 25000,
309
+ "success_rate": 0.14,
310
+ "successes": 7,
311
+ "episodes": 50,
312
+ "episode_seed_base": 800000,
313
+ "ckpts_evaluated": 8,
314
+ "missing_eval_steps": [],
315
+ "campaign": "pi05_retrain_eval50_20261022",
316
+ "checkpoint": "single_task/DualArmFoldCarton-v1/lora_step_0025000.npz",
317
+ "checkpoint_sha256": "b9be59d2eaf9ff7e730b635196923c00a4b11df3b1b9395e1be7d73fab771f71"
318
+ },
319
  {
320
  "task": "GroceryPrepare-v1",
321
  "level": "level_3",
single_task/results.csv CHANGED
@@ -48,8 +48,10 @@ PivotBook-v1,level_2,15000,0.38,19,8,single_task/PivotBook-v1/lora_step_0015000.
48
  ReorientPen-v1,level_2,25000,0.38,19,8,single_task/ReorientPen-v1/lora_step_0025000.npz
49
  RollingPin-v1,level_2,25000,0.96,48,8,single_task/RollingPin-v1/lora_step_0025000.npz
50
  RopeBridgeDex-v1,level_2,30000,0.78,39,8,single_task/RopeBridgeDex-v1/lora_step_0030000.npz
 
51
  StraightenTwoRopesSequential-v1,level_2,15000,0.36,18,8,single_task/StraightenTwoRopesSequential-v1/lora_step_0015000.npz
52
  ClutterGrasp-v1,level_3,15000,0.16,8,8,single_task/ClutterGrasp-v1/lora_step_0015000.npz
 
53
  DualArmOpenFridge-v1,level_3,5000,0.0,0,8,single_task/DualArmOpenFridge-v1/lora_step_0005000.npz
54
  DualArmOrganizeCabinet-v1,level_3,5000,0.0,0,8,single_task/DualArmOrganizeCabinet-v1/lora_step_0005000.npz
55
  DualArmPingpongBalance-v1,level_3,15000,0.2,10,8,single_task/DualArmPingpongBalance-v1/lora_step_0015000.npz
 
48
  ReorientPen-v1,level_2,25000,0.38,19,8,single_task/ReorientPen-v1/lora_step_0025000.npz
49
  RollingPin-v1,level_2,25000,0.96,48,8,single_task/RollingPin-v1/lora_step_0025000.npz
50
  RopeBridgeDex-v1,level_2,30000,0.78,39,8,single_task/RopeBridgeDex-v1/lora_step_0030000.npz
51
+ ScoopWater-v1,level_2,15000,0.44,22,8,single_task/ScoopWater-v1/lora_step_0015000.npz
52
  StraightenTwoRopesSequential-v1,level_2,15000,0.36,18,8,single_task/StraightenTwoRopesSequential-v1/lora_step_0015000.npz
53
  ClutterGrasp-v1,level_3,15000,0.16,8,8,single_task/ClutterGrasp-v1/lora_step_0015000.npz
54
+ DualArmFoldCarton-v1,level_3,25000,0.14,7,8,single_task/DualArmFoldCarton-v1/lora_step_0025000.npz
55
  DualArmOpenFridge-v1,level_3,5000,0.0,0,8,single_task/DualArmOpenFridge-v1/lora_step_0005000.npz
56
  DualArmOrganizeCabinet-v1,level_3,5000,0.0,0,8,single_task/DualArmOrganizeCabinet-v1/lora_step_0005000.npz
57
  DualArmPingpongBalance-v1,level_3,15000,0.2,10,8,single_task/DualArmPingpongBalance-v1/lora_step_0015000.npz