Linslab commited on
Commit
7b58fe7
·
verified ·
1 Parent(s): 705f0a1

Index 10 HF-data retraining PI0.5 peaks (pi05_hf21_20261025) with 50-episode evaluations

Browse files
README.md CHANGED
@@ -6,12 +6,12 @@ license: mit
6
 
7
  ## single_task
8
 
9
- One PI0.5 LoRA checkpoint per ManiLadder task (87 tasks), each finetuned on 20 demos.
10
  Checkpoints are selected by the highest **50-episode** success rate; ties use the earliest step.
11
 
12
  - The 60 previously published tasks use fixed evaluation seeds 1000–1049.
13
  - The 20 completed retraining runs added on 2026-10-28 use seeds 800000–800049. All reached 40,000 training steps. 13 have all eight 5k–40k evaluations; 7 have seven evaluations because one intermediate checkpoint was not retained. Their selected weights are the best among the available evaluated checkpoints.
14
- - The 7 tasks from the HF-data retraining campaign `pi05_hf21_20261025` (added 2026-10-29; raw demos freshly downloaded from Linslab/ManiLadder_Dataset) use seeds 1000–1049. All reached 40,000 training steps and have all eight 5k–40k evaluations. See [its manifest](single_task/pi05_hf21_20261025_manifest.json).
15
 
16
  See [retraining provenance and missing steps](single_task/pi05_retrain_eval50_20261022_manifest.json). These scores use different seed sets across the two groups and are not a controlled comparison.
17
 
@@ -22,7 +22,7 @@ Each `single_task/<task>/` folder contains:
22
  - `eval_result.json` — 50-episode evaluation of this checkpoint
23
 
24
  `single_task/results.csv` lists the selected checkpoint per task; `single_task/all_checkpoints_eval.csv` has the
25
- 50-episode result of every evaluated checkpoint (686 total).
26
 
27
  Evaluation: `exec_steps=16`, `sample_steps=10`. Tasks whose checkpoints all scored 0/50 are still included;
28
  for them the step choice is arbitrary.
@@ -81,6 +81,7 @@ for them the step choice is arbitrary.
81
  | StraightenTwoRopesSequential-v1 | 2 | 15000 | 0.36 |
82
  | ClutterGrasp-v1 | 3 | 15000 | 0.16 |
83
  | DualArmOpenFridge-v1 | 3 | 5000 | 0.00 |
 
84
  | DualArmPingpongBalance-v1 | 3 | 15000 | 0.20 |
85
  | DualArmPlugCharger-v1 | 3 | 5000 | 0.00 |
86
  | DualArmSetupTable-v1 | 3 | 25000 | 0.04 |
@@ -99,12 +100,14 @@ for them the step choice is arbitrary.
99
  | BuildTent-v1 | 4 | 10000 | 0.06 |
100
  | DualArmBallClassify-v1 | 4 | 10000 | 0.00 |
101
  | DualArmBilliardBridge-v1 | 4 | 5000 | 0.00 |
 
102
  | DualArmClothHanger-v1 | 4 | 5000 | 0.00 |
103
  | DualArmFlipTable-v1 | 4 | 5000 | 0.00 |
104
  | DualArmJumpRope-v1 | 4 | 20000 | 0.02 |
105
  | DualArmKeyboardType-v1 | 4 | 5000 | 0.00 |
106
  | DualArmLiquidMixing-v1 | 4 | 5000 | 0.00 |
107
  | DualArmMahjongSetup-v1 | 4 | 5000 | 0.02 |
 
108
  | DualArmSendPingpong-v1 | 4 | 40000 | 0.08 |
109
  | DualArmShootBasketball-v1 | 4 | 5000 | 0.00 |
110
  | DualArmWallRings-v1 | 4 | 5000 | 0.00 |
 
6
 
7
  ## single_task
8
 
9
+ One PI0.5 LoRA checkpoint per ManiLadder task (90 tasks), each finetuned on 20 demos.
10
  Checkpoints are selected by the highest **50-episode** success rate; ties use the earliest step.
11
 
12
  - The 60 previously published tasks use fixed evaluation seeds 1000–1049.
13
  - The 20 completed retraining runs added on 2026-10-28 use seeds 800000–800049. All reached 40,000 training steps. 13 have all eight 5k–40k evaluations; 7 have seven evaluations because one intermediate checkpoint was not retained. Their selected weights are the best among the available evaluated checkpoints.
14
+ - The 10 tasks from the HF-data retraining campaign `pi05_hf21_20261025` (added 2026-10-29; raw demos freshly downloaded from Linslab/ManiLadder_Dataset) use seeds 1000–1049. All reached 40,000 training steps and have all eight 5k–40k evaluations. See [its manifest](single_task/pi05_hf21_20261025_manifest.json).
15
 
16
  See [retraining provenance and missing steps](single_task/pi05_retrain_eval50_20261022_manifest.json). These scores use different seed sets across the two groups and are not a controlled comparison.
17
 
 
22
  - `eval_result.json` — 50-episode evaluation of this checkpoint
23
 
24
  `single_task/results.csv` lists the selected checkpoint per task; `single_task/all_checkpoints_eval.csv` has the
25
+ 50-episode result of every evaluated checkpoint (710 total).
26
 
27
  Evaluation: `exec_steps=16`, `sample_steps=10`. Tasks whose checkpoints all scored 0/50 are still included;
28
  for them the step choice is arbitrary.
 
81
  | StraightenTwoRopesSequential-v1 | 2 | 15000 | 0.36 |
82
  | ClutterGrasp-v1 | 3 | 15000 | 0.16 |
83
  | DualArmOpenFridge-v1 | 3 | 5000 | 0.00 |
84
+ | DualArmOrganizeCabinet-v1 | 3 | 5000 | 0.00 |
85
  | DualArmPingpongBalance-v1 | 3 | 15000 | 0.20 |
86
  | DualArmPlugCharger-v1 | 3 | 5000 | 0.00 |
87
  | DualArmSetupTable-v1 | 3 | 25000 | 0.04 |
 
100
  | BuildTent-v1 | 4 | 10000 | 0.06 |
101
  | DualArmBallClassify-v1 | 4 | 10000 | 0.00 |
102
  | DualArmBilliardBridge-v1 | 4 | 5000 | 0.00 |
103
+ | DualArmClothCylinderWipe-v1 | 4 | 5000 | 0.02 |
104
  | DualArmClothHanger-v1 | 4 | 5000 | 0.00 |
105
  | DualArmFlipTable-v1 | 4 | 5000 | 0.00 |
106
  | DualArmJumpRope-v1 | 4 | 20000 | 0.02 |
107
  | DualArmKeyboardType-v1 | 4 | 5000 | 0.00 |
108
  | DualArmLiquidMixing-v1 | 4 | 5000 | 0.00 |
109
  | DualArmMahjongSetup-v1 | 4 | 5000 | 0.02 |
110
+ | DualArmSandShovel-v1 | 4 | 15000 | 0.02 |
111
  | DualArmSendPingpong-v1 | 4 | 40000 | 0.08 |
112
  | DualArmShootBasketball-v1 | 4 | 5000 | 0.00 |
113
  | DualArmWallRings-v1 | 4 | 5000 | 0.00 |
single_task/all_checkpoints_eval.csv CHANGED
@@ -410,6 +410,14 @@ DualArmOpenFridge-v1,level_3,25000,0.0,0,1425.2351526212692
410
  DualArmOpenFridge-v1,level_3,30000,0.0,0,1425.235009675026
411
  DualArmOpenFridge-v1,level_3,35000,0.0,0,1425.2351335525514
412
  DualArmOpenFridge-v1,level_3,40000,0.0,0,1425.2403667545318
 
 
 
 
 
 
 
 
413
  DualArmPingpongBalance-v1,level_3,5000,0.02,1,120.94932576864957
414
  DualArmPingpongBalance-v1,level_3,10000,0.02,1,558.1151409831643
415
  DualArmPingpongBalance-v1,level_3,15000,0.2,10,762.3643163096905
@@ -551,6 +559,14 @@ DualArmBilliardBridge-v1,level_4,25000,0.0,0,1189.2712672090531
551
  DualArmBilliardBridge-v1,level_4,30000,0.0,0,1176.501679096222
552
  DualArmBilliardBridge-v1,level_4,35000,0.0,0,1173.2626583218575
553
  DualArmBilliardBridge-v1,level_4,40000,0.0,0,1188.2963494682313
 
 
 
 
 
 
 
 
554
  DualArmClothHanger-v1,level_4,5000,0.0,0,144.01372902154924
555
  DualArmClothHanger-v1,level_4,10000,0.0,0,111.90421493053437
556
  DualArmClothHanger-v1,level_4,15000,0.0,0,126.82437116622924
@@ -598,6 +614,14 @@ DualArmMahjongSetup-v1,level_4,25000,0.0,0,nan
598
  DualArmMahjongSetup-v1,level_4,30000,0.0,0,nan
599
  DualArmMahjongSetup-v1,level_4,35000,0.0,0,nan
600
  DualArmMahjongSetup-v1,level_4,40000,0.0,0,nan
 
 
 
 
 
 
 
 
601
  DualArmSendPingpong-v1,level_4,5000,0.0,0,493.29558154344556
602
  DualArmSendPingpong-v1,level_4,10000,0.02,1,437.45649006843564
603
  DualArmSendPingpong-v1,level_4,15000,0.0,0,556.9836190354824
 
410
  DualArmOpenFridge-v1,level_3,30000,0.0,0,1425.235009675026
411
  DualArmOpenFridge-v1,level_3,35000,0.0,0,1425.2351335525514
412
  DualArmOpenFridge-v1,level_3,40000,0.0,0,1425.2403667545318
413
+ DualArmOrganizeCabinet-v1,level_3,5000,0.0,0,1487.69516
414
+ DualArmOrganizeCabinet-v1,level_3,10000,0.0,0,1186.68734
415
+ DualArmOrganizeCabinet-v1,level_3,15000,0.0,0,1195.1147
416
+ DualArmOrganizeCabinet-v1,level_3,20000,0.0,0,1163.48162
417
+ DualArmOrganizeCabinet-v1,level_3,25000,0.0,0,1310.3965
418
+ DualArmOrganizeCabinet-v1,level_3,30000,0.0,0,1457.5043400000002
419
+ DualArmOrganizeCabinet-v1,level_3,35000,0.0,0,1095.93272
420
+ DualArmOrganizeCabinet-v1,level_3,40000,0.0,0,1589.47264
421
  DualArmPingpongBalance-v1,level_3,5000,0.02,1,120.94932576864957
422
  DualArmPingpongBalance-v1,level_3,10000,0.02,1,558.1151409831643
423
  DualArmPingpongBalance-v1,level_3,15000,0.2,10,762.3643163096905
 
559
  DualArmBilliardBridge-v1,level_4,30000,0.0,0,1176.501679096222
560
  DualArmBilliardBridge-v1,level_4,35000,0.0,0,1173.2626583218575
561
  DualArmBilliardBridge-v1,level_4,40000,0.0,0,1188.2963494682313
562
+ DualArmClothCylinderWipe-v1,level_4,5000,0.02,1,-0.98432
563
+ DualArmClothCylinderWipe-v1,level_4,10000,0.0,0,-1.0
564
+ DualArmClothCylinderWipe-v1,level_4,15000,0.0,0,-1.0
565
+ DualArmClothCylinderWipe-v1,level_4,20000,0.0,0,-0.98952
566
+ DualArmClothCylinderWipe-v1,level_4,25000,0.0,0,-0.9967400000000001
567
+ DualArmClothCylinderWipe-v1,level_4,30000,0.0,0,-0.9983799999999999
568
+ DualArmClothCylinderWipe-v1,level_4,35000,0.0,0,-0.99098
569
+ DualArmClothCylinderWipe-v1,level_4,40000,0.0,0,-0.9784
570
  DualArmClothHanger-v1,level_4,5000,0.0,0,144.01372902154924
571
  DualArmClothHanger-v1,level_4,10000,0.0,0,111.90421493053437
572
  DualArmClothHanger-v1,level_4,15000,0.0,0,126.82437116622924
 
614
  DualArmMahjongSetup-v1,level_4,30000,0.0,0,nan
615
  DualArmMahjongSetup-v1,level_4,35000,0.0,0,nan
616
  DualArmMahjongSetup-v1,level_4,40000,0.0,0,nan
617
+ DualArmSandShovel-v1,level_4,5000,0.0,0,0.0
618
+ DualArmSandShovel-v1,level_4,10000,0.0,0,0.04
619
+ DualArmSandShovel-v1,level_4,15000,0.02,1,0.27399999999999997
620
+ DualArmSandShovel-v1,level_4,20000,0.0,0,0.37199999999999994
621
+ DualArmSandShovel-v1,level_4,25000,0.0,0,0.252
622
+ DualArmSandShovel-v1,level_4,30000,0.0,0,0.27599999999999997
623
+ DualArmSandShovel-v1,level_4,35000,0.0,0,0.33199999999999996
624
+ DualArmSandShovel-v1,level_4,40000,0.0,0,0.172
625
  DualArmSendPingpong-v1,level_4,5000,0.0,0,493.29558154344556
626
  DualArmSendPingpong-v1,level_4,10000,0.02,1,437.45649006843564
627
  DualArmSendPingpong-v1,level_4,15000,0.0,0,556.9836190354824
single_task/pi05_hf21_20261025_manifest.json CHANGED
@@ -65,6 +65,21 @@
65
  "checkpoint": "single_task/RopeBridgeDex-v1/lora_step_0030000.npz",
66
  "checkpoint_sha256": "f4dd0cfb55ec72eb20fe7bc9050d5f092c10db8662cdbcaf218379d423d64a9f"
67
  },
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
68
  {
69
  "task": "ClutterGrasp-v1",
70
  "level": "level_3",
@@ -80,6 +95,21 @@
80
  "checkpoint": "single_task/ClutterGrasp-v1/lora_step_0015000.npz",
81
  "checkpoint_sha256": "7b97fd0147a1f783091bba9e8625f1d04d7a2f7c20f825a417304fffe9f8954f"
82
  },
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
83
  {
84
  "task": "DualArmLiquidMixing-v1",
85
  "level": "level_4",
@@ -95,6 +125,21 @@
95
  "checkpoint": "single_task/DualArmLiquidMixing-v1/lora_step_0005000.npz",
96
  "checkpoint_sha256": "3511ca0c3cfe1e4f54135e6712112db09a16fc93f48a7e4790e3cbd2e5f429b7"
97
  },
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
98
  {
99
  "task": "DualArmWallRings-v1",
100
  "level": "level_4",
 
65
  "checkpoint": "single_task/RopeBridgeDex-v1/lora_step_0030000.npz",
66
  "checkpoint_sha256": "f4dd0cfb55ec72eb20fe7bc9050d5f092c10db8662cdbcaf218379d423d64a9f"
67
  },
68
+ {
69
+ "task": "DualArmOrganizeCabinet-v1",
70
+ "level": "level_3",
71
+ "suite": "rigid_body/dual_arm_gripper",
72
+ "step": 5000,
73
+ "success_rate": 0.0,
74
+ "successes": 0,
75
+ "episodes": 50,
76
+ "episode_seed_base": 1000,
77
+ "ckpts_evaluated": 8,
78
+ "missing_eval_steps": [],
79
+ "campaign": "pi05_hf21_20261025",
80
+ "checkpoint": "single_task/DualArmOrganizeCabinet-v1/lora_step_0005000.npz",
81
+ "checkpoint_sha256": "12cb8ceffd2497fded388aaa932997228f748273e92671218c4966e4963e3ae1"
82
+ },
83
  {
84
  "task": "ClutterGrasp-v1",
85
  "level": "level_3",
 
95
  "checkpoint": "single_task/ClutterGrasp-v1/lora_step_0015000.npz",
96
  "checkpoint_sha256": "7b97fd0147a1f783091bba9e8625f1d04d7a2f7c20f825a417304fffe9f8954f"
97
  },
98
+ {
99
+ "task": "DualArmClothCylinderWipe-v1",
100
+ "level": "level_4",
101
+ "suite": "deformable/dual_arm_dex",
102
+ "step": 5000,
103
+ "success_rate": 0.02,
104
+ "successes": 1,
105
+ "episodes": 50,
106
+ "episode_seed_base": 1000,
107
+ "ckpts_evaluated": 8,
108
+ "missing_eval_steps": [],
109
+ "campaign": "pi05_hf21_20261025",
110
+ "checkpoint": "single_task/DualArmClothCylinderWipe-v1/lora_step_0005000.npz",
111
+ "checkpoint_sha256": "3d1f7cec76825ea44358c4b04a0496b45e139c4ca2645bb1a2cd912599656312"
112
+ },
113
  {
114
  "task": "DualArmLiquidMixing-v1",
115
  "level": "level_4",
 
125
  "checkpoint": "single_task/DualArmLiquidMixing-v1/lora_step_0005000.npz",
126
  "checkpoint_sha256": "3511ca0c3cfe1e4f54135e6712112db09a16fc93f48a7e4790e3cbd2e5f429b7"
127
  },
128
+ {
129
+ "task": "DualArmSandShovel-v1",
130
+ "level": "level_4",
131
+ "suite": "deformable/dual_arm_gripper",
132
+ "step": 15000,
133
+ "success_rate": 0.02,
134
+ "successes": 1,
135
+ "episodes": 50,
136
+ "episode_seed_base": 1000,
137
+ "ckpts_evaluated": 8,
138
+ "missing_eval_steps": [],
139
+ "campaign": "pi05_hf21_20261025",
140
+ "checkpoint": "single_task/DualArmSandShovel-v1/lora_step_0015000.npz",
141
+ "checkpoint_sha256": "243714778d4959bf2a2199315143d153dc5eab52eb2ece59aee327f263abc82f"
142
+ },
143
  {
144
  "task": "DualArmWallRings-v1",
145
  "level": "level_4",
single_task/results.csv CHANGED
@@ -51,6 +51,7 @@ RopeBridgeDex-v1,level_2,30000,0.78,39,8,single_task/RopeBridgeDex-v1/lora_step_
51
  StraightenTwoRopesSequential-v1,level_2,15000,0.36,18,8,single_task/StraightenTwoRopesSequential-v1/lora_step_0015000.npz
52
  ClutterGrasp-v1,level_3,15000,0.16,8,8,single_task/ClutterGrasp-v1/lora_step_0015000.npz
53
  DualArmOpenFridge-v1,level_3,5000,0.0,0,8,single_task/DualArmOpenFridge-v1/lora_step_0005000.npz
 
54
  DualArmPingpongBalance-v1,level_3,15000,0.2,10,8,single_task/DualArmPingpongBalance-v1/lora_step_0015000.npz
55
  DualArmPlugCharger-v1,level_3,5000,0.0,0,8,single_task/DualArmPlugCharger-v1/lora_step_0005000.npz
56
  DualArmSetupTable-v1,level_3,25000,0.04,2,8,single_task/DualArmSetupTable-v1/lora_step_0025000.npz
@@ -69,12 +70,14 @@ TweezersPickPlace-v1,level_3,5000,0.0,0,8,single_task/TweezersPickPlace-v1/lora_
69
  BuildTent-v1,level_4,10000,0.06,3,8,single_task/BuildTent-v1/lora_step_0010000.npz
70
  DualArmBallClassify-v1,level_4,10000,0.0,0,7,single_task/DualArmBallClassify-v1/lora_step_0010000.npz
71
  DualArmBilliardBridge-v1,level_4,5000,0.0,0,8,single_task/DualArmBilliardBridge-v1/lora_step_0005000.npz
 
72
  DualArmClothHanger-v1,level_4,5000,0.0,0,8,single_task/DualArmClothHanger-v1/lora_step_0005000.npz
73
  DualArmFlipTable-v1,level_4,5000,0.0,0,8,single_task/DualArmFlipTable-v1/lora_step_0005000.npz
74
  DualArmJumpRope-v1,level_4,20000,0.02,1,8,single_task/DualArmJumpRope-v1/lora_step_0020000.npz
75
  DualArmKeyboardType-v1,level_4,5000,0.0,0,7,single_task/DualArmKeyboardType-v1/lora_step_0005000.npz
76
  DualArmLiquidMixing-v1,level_4,5000,0.0,0,8,single_task/DualArmLiquidMixing-v1/lora_step_0005000.npz
77
  DualArmMahjongSetup-v1,level_4,5000,0.02,1,8,single_task/DualArmMahjongSetup-v1/lora_step_0005000.npz
 
78
  DualArmSendPingpong-v1,level_4,40000,0.08,4,8,single_task/DualArmSendPingpong-v1/lora_step_0040000.npz
79
  DualArmShootBasketball-v1,level_4,5000,0.0,0,8,single_task/DualArmShootBasketball-v1/lora_step_0005000.npz
80
  DualArmWallRings-v1,level_4,5000,0.0,0,8,single_task/DualArmWallRings-v1/lora_step_0005000.npz
 
51
  StraightenTwoRopesSequential-v1,level_2,15000,0.36,18,8,single_task/StraightenTwoRopesSequential-v1/lora_step_0015000.npz
52
  ClutterGrasp-v1,level_3,15000,0.16,8,8,single_task/ClutterGrasp-v1/lora_step_0015000.npz
53
  DualArmOpenFridge-v1,level_3,5000,0.0,0,8,single_task/DualArmOpenFridge-v1/lora_step_0005000.npz
54
+ DualArmOrganizeCabinet-v1,level_3,5000,0.0,0,8,single_task/DualArmOrganizeCabinet-v1/lora_step_0005000.npz
55
  DualArmPingpongBalance-v1,level_3,15000,0.2,10,8,single_task/DualArmPingpongBalance-v1/lora_step_0015000.npz
56
  DualArmPlugCharger-v1,level_3,5000,0.0,0,8,single_task/DualArmPlugCharger-v1/lora_step_0005000.npz
57
  DualArmSetupTable-v1,level_3,25000,0.04,2,8,single_task/DualArmSetupTable-v1/lora_step_0025000.npz
 
70
  BuildTent-v1,level_4,10000,0.06,3,8,single_task/BuildTent-v1/lora_step_0010000.npz
71
  DualArmBallClassify-v1,level_4,10000,0.0,0,7,single_task/DualArmBallClassify-v1/lora_step_0010000.npz
72
  DualArmBilliardBridge-v1,level_4,5000,0.0,0,8,single_task/DualArmBilliardBridge-v1/lora_step_0005000.npz
73
+ DualArmClothCylinderWipe-v1,level_4,5000,0.02,1,8,single_task/DualArmClothCylinderWipe-v1/lora_step_0005000.npz
74
  DualArmClothHanger-v1,level_4,5000,0.0,0,8,single_task/DualArmClothHanger-v1/lora_step_0005000.npz
75
  DualArmFlipTable-v1,level_4,5000,0.0,0,8,single_task/DualArmFlipTable-v1/lora_step_0005000.npz
76
  DualArmJumpRope-v1,level_4,20000,0.02,1,8,single_task/DualArmJumpRope-v1/lora_step_0020000.npz
77
  DualArmKeyboardType-v1,level_4,5000,0.0,0,7,single_task/DualArmKeyboardType-v1/lora_step_0005000.npz
78
  DualArmLiquidMixing-v1,level_4,5000,0.0,0,8,single_task/DualArmLiquidMixing-v1/lora_step_0005000.npz
79
  DualArmMahjongSetup-v1,level_4,5000,0.02,1,8,single_task/DualArmMahjongSetup-v1/lora_step_0005000.npz
80
+ DualArmSandShovel-v1,level_4,15000,0.02,1,8,single_task/DualArmSandShovel-v1/lora_step_0015000.npz
81
  DualArmSendPingpong-v1,level_4,40000,0.08,4,8,single_task/DualArmSendPingpong-v1/lora_step_0040000.npz
82
  DualArmShootBasketball-v1,level_4,5000,0.0,0,8,single_task/DualArmShootBasketball-v1/lora_step_0005000.npz
83
  DualArmWallRings-v1,level_4,5000,0.0,0,8,single_task/DualArmWallRings-v1/lora_step_0005000.npz