squaredcuber commited on
Commit
5faf4b6
·
verified ·
1 Parent(s): f4d7e22
Files changed (27) hide show
  1. .gitattributes +13 -0
  2. results/closedloop_checkpoint-10000/results.json +1328 -0
  3. results/closedloop_checkpoint-10000/results.md +68 -0
  4. results/closedloop_checkpoint-10000/videos/box_failure_seed1099513724928.mp4 +3 -0
  5. results/closedloop_checkpoint-10000/videos/box_success_seed1099513724930.mp4 +3 -0
  6. results/closedloop_checkpoint-10000/videos/button_failure_seed1099512676355.mp4 +3 -0
  7. results/closedloop_checkpoint-10000/videos/button_success_seed1099512676352.mp4 +0 -0
  8. results/closedloop_checkpoint-10000/videos/push_failure_seed1099511627787.mp4 +3 -0
  9. results/closedloop_checkpoint-10000/videos/push_success_seed1099511627776.mp4 +0 -0
  10. results/closedloop_checkpoint-6000/results.json +1387 -0
  11. results/closedloop_checkpoint-6000/results.md +68 -0
  12. results/closedloop_checkpoint-6000/videos/box_failure_seed1099513724928.mp4 +3 -0
  13. results/closedloop_checkpoint-6000/videos/box_success_seed1099513724934.mp4 +3 -0
  14. results/closedloop_checkpoint-6000/videos/button_failure_seed1099512676352.mp4 +3 -0
  15. results/closedloop_checkpoint-6000/videos/button_success_seed1099512676355.mp4 +0 -0
  16. results/closedloop_checkpoint-6000/videos/push_failure_seed1099511627776.mp4 +3 -0
  17. results/closedloop_checkpoint-6000/videos/push_success_seed1099511627777.mp4 +3 -0
  18. results/openloop_checkpoint-10000/openloop.log +76 -0
  19. results/openloop_checkpoint-10000/summary.txt +11 -0
  20. results/openloop_checkpoint-10000/traj_0.jpeg +3 -0
  21. results/openloop_checkpoint-10000/traj_300.jpeg +3 -0
  22. results/openloop_checkpoint-10000/traj_600.jpeg +3 -0
  23. results/oracle_smoke/results.json +97 -0
  24. results/oracle_smoke/results.md +33 -0
  25. results/oracle_smoke/videos/box_success_seed1099513724928.mp4 +3 -0
  26. results/oracle_smoke/videos/button_success_seed1099512676352.mp4 +0 -0
  27. results/oracle_smoke/videos/push_success_seed1099511627776.mp4 +0 -0
.gitattributes CHANGED
@@ -33,3 +33,16 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ results/closedloop_checkpoint-10000/videos/box_failure_seed1099513724928.mp4 filter=lfs diff=lfs merge=lfs -text
37
+ results/closedloop_checkpoint-10000/videos/box_success_seed1099513724930.mp4 filter=lfs diff=lfs merge=lfs -text
38
+ results/closedloop_checkpoint-10000/videos/button_failure_seed1099512676355.mp4 filter=lfs diff=lfs merge=lfs -text
39
+ results/closedloop_checkpoint-10000/videos/push_failure_seed1099511627787.mp4 filter=lfs diff=lfs merge=lfs -text
40
+ results/closedloop_checkpoint-6000/videos/box_failure_seed1099513724928.mp4 filter=lfs diff=lfs merge=lfs -text
41
+ results/closedloop_checkpoint-6000/videos/box_success_seed1099513724934.mp4 filter=lfs diff=lfs merge=lfs -text
42
+ results/closedloop_checkpoint-6000/videos/button_failure_seed1099512676352.mp4 filter=lfs diff=lfs merge=lfs -text
43
+ results/closedloop_checkpoint-6000/videos/push_failure_seed1099511627776.mp4 filter=lfs diff=lfs merge=lfs -text
44
+ results/closedloop_checkpoint-6000/videos/push_success_seed1099511627777.mp4 filter=lfs diff=lfs merge=lfs -text
45
+ results/openloop_checkpoint-10000/traj_0.jpeg filter=lfs diff=lfs merge=lfs -text
46
+ results/openloop_checkpoint-10000/traj_300.jpeg filter=lfs diff=lfs merge=lfs -text
47
+ results/openloop_checkpoint-10000/traj_600.jpeg filter=lfs diff=lfs merge=lfs -text
48
+ results/oracle_smoke/videos/box_success_seed1099513724928.mp4 filter=lfs diff=lfs merge=lfs -text
results/closedloop_checkpoint-10000/results.json ADDED
@@ -0,0 +1,1328 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "policy": "groot",
3
+ "model_path": "/root/work/runs/asimov_n17/checkpoint-10000",
4
+ "exec_steps": 8,
5
+ "episodes_per_task": 50,
6
+ "hold_steps": 10,
7
+ "gl_backend": "egl",
8
+ "seeds": "checked against 926 dataset seeds: no overlap (max training seed 1200300)",
9
+ "seed_scheme": "2**40 + task_id * 2**20 + k",
10
+ "tasks": {
11
+ "push": {
12
+ "n": 50,
13
+ "success": 42,
14
+ "rate": 0.84,
15
+ "wilson95": [
16
+ 0.715,
17
+ 0.917
18
+ ],
19
+ "mean_time_to_success_s": 3.93,
20
+ "failures": {
21
+ "target_short_of_pad": 5,
22
+ "target_not_moved": 3
23
+ },
24
+ "per_instruction": {
25
+ "push the blue block onto the blue pad": {
26
+ "success": 2,
27
+ "n": 2
28
+ },
29
+ "push the blue block onto the green pad": {
30
+ "success": 2,
31
+ "n": 2
32
+ },
33
+ "push the blue block onto the orange pad": {
34
+ "success": 2,
35
+ "n": 2
36
+ },
37
+ "push the blue block onto the purple pad": {
38
+ "success": 1,
39
+ "n": 1
40
+ },
41
+ "push the blue block onto the red pad": {
42
+ "success": 1,
43
+ "n": 1
44
+ },
45
+ "push the blue block onto the yellow pad": {
46
+ "success": 1,
47
+ "n": 1
48
+ },
49
+ "push the green block onto the blue pad": {
50
+ "success": 1,
51
+ "n": 1
52
+ },
53
+ "push the green block onto the green pad": {
54
+ "success": 2,
55
+ "n": 2
56
+ },
57
+ "push the green block onto the purple pad": {
58
+ "success": 0,
59
+ "n": 1
60
+ },
61
+ "push the green block onto the red pad": {
62
+ "success": 1,
63
+ "n": 1
64
+ },
65
+ "push the orange block onto the blue pad": {
66
+ "success": 2,
67
+ "n": 2
68
+ },
69
+ "push the orange block onto the purple pad": {
70
+ "success": 1,
71
+ "n": 1
72
+ },
73
+ "push the orange block onto the red pad": {
74
+ "success": 2,
75
+ "n": 2
76
+ },
77
+ "push the orange block onto the yellow pad": {
78
+ "success": 2,
79
+ "n": 2
80
+ },
81
+ "push the purple block onto the blue pad": {
82
+ "success": 1,
83
+ "n": 1
84
+ },
85
+ "push the purple block onto the green pad": {
86
+ "success": 1,
87
+ "n": 1
88
+ },
89
+ "push the purple block onto the orange pad": {
90
+ "success": 3,
91
+ "n": 5
92
+ },
93
+ "push the purple block onto the purple pad": {
94
+ "success": 1,
95
+ "n": 1
96
+ },
97
+ "push the purple block onto the red pad": {
98
+ "success": 2,
99
+ "n": 2
100
+ },
101
+ "push the purple block onto the yellow pad": {
102
+ "success": 1,
103
+ "n": 1
104
+ },
105
+ "push the red block onto the orange pad": {
106
+ "success": 1,
107
+ "n": 1
108
+ },
109
+ "push the red block onto the purple pad": {
110
+ "success": 1,
111
+ "n": 2
112
+ },
113
+ "push the red block onto the red pad": {
114
+ "success": 1,
115
+ "n": 2
116
+ },
117
+ "push the red block onto the yellow pad": {
118
+ "success": 2,
119
+ "n": 4
120
+ },
121
+ "push the yellow block onto the blue pad": {
122
+ "success": 3,
123
+ "n": 3
124
+ },
125
+ "push the yellow block onto the green pad": {
126
+ "success": 2,
127
+ "n": 3
128
+ },
129
+ "push the yellow block onto the purple pad": {
130
+ "success": 1,
131
+ "n": 1
132
+ },
133
+ "push the yellow block onto the red pad": {
134
+ "success": 1,
135
+ "n": 1
136
+ },
137
+ "push the yellow block onto the yellow pad": {
138
+ "success": 1,
139
+ "n": 1
140
+ }
141
+ },
142
+ "wall_s": 1201.1,
143
+ "episodes": [
144
+ {
145
+ "seed": 1099511627776,
146
+ "instruction": "push the yellow block onto the green pad",
147
+ "success": true,
148
+ "steps": 78,
149
+ "time_to_success_s": 3.45,
150
+ "video": "videos/push_success_seed1099511627776.mp4"
151
+ },
152
+ {
153
+ "seed": 1099511627777,
154
+ "instruction": "push the yellow block onto the blue pad",
155
+ "success": true,
156
+ "steps": 104,
157
+ "time_to_success_s": 4.75
158
+ },
159
+ {
160
+ "seed": 1099511627778,
161
+ "instruction": "push the orange block onto the red pad",
162
+ "success": true,
163
+ "steps": 117,
164
+ "time_to_success_s": 5.4
165
+ },
166
+ {
167
+ "seed": 1099511627779,
168
+ "instruction": "push the green block onto the blue pad",
169
+ "success": true,
170
+ "steps": 88,
171
+ "time_to_success_s": 3.95
172
+ },
173
+ {
174
+ "seed": 1099511627780,
175
+ "instruction": "push the orange block onto the yellow pad",
176
+ "success": true,
177
+ "steps": 85,
178
+ "time_to_success_s": 3.8
179
+ },
180
+ {
181
+ "seed": 1099511627781,
182
+ "instruction": "push the blue block onto the orange pad",
183
+ "success": true,
184
+ "steps": 55,
185
+ "time_to_success_s": 2.3
186
+ },
187
+ {
188
+ "seed": 1099511627782,
189
+ "instruction": "push the blue block onto the blue pad",
190
+ "success": true,
191
+ "steps": 192,
192
+ "time_to_success_s": 9.15
193
+ },
194
+ {
195
+ "seed": 1099511627783,
196
+ "instruction": "push the yellow block onto the blue pad",
197
+ "success": true,
198
+ "steps": 160,
199
+ "time_to_success_s": 7.55
200
+ },
201
+ {
202
+ "seed": 1099511627784,
203
+ "instruction": "push the purple block onto the red pad",
204
+ "success": true,
205
+ "steps": 75,
206
+ "time_to_success_s": 3.3
207
+ },
208
+ {
209
+ "seed": 1099511627785,
210
+ "instruction": "push the purple block onto the yellow pad",
211
+ "success": true,
212
+ "steps": 88,
213
+ "time_to_success_s": 3.95
214
+ },
215
+ {
216
+ "seed": 1099511627786,
217
+ "instruction": "push the blue block onto the purple pad",
218
+ "success": true,
219
+ "steps": 76,
220
+ "time_to_success_s": 3.35
221
+ },
222
+ {
223
+ "seed": 1099511627787,
224
+ "instruction": "push the red block onto the yellow pad",
225
+ "success": false,
226
+ "steps": 440,
227
+ "time_to_success_s": null,
228
+ "failure": "target_short_of_pad",
229
+ "video": "videos/push_failure_seed1099511627787.mp4"
230
+ },
231
+ {
232
+ "seed": 1099511627788,
233
+ "instruction": "push the green block onto the purple pad",
234
+ "success": false,
235
+ "steps": 440,
236
+ "time_to_success_s": null,
237
+ "failure": "target_short_of_pad"
238
+ },
239
+ {
240
+ "seed": 1099511627789,
241
+ "instruction": "push the yellow block onto the yellow pad",
242
+ "success": true,
243
+ "steps": 75,
244
+ "time_to_success_s": 3.3
245
+ },
246
+ {
247
+ "seed": 1099511627790,
248
+ "instruction": "push the red block onto the yellow pad",
249
+ "success": true,
250
+ "steps": 54,
251
+ "time_to_success_s": 2.25
252
+ },
253
+ {
254
+ "seed": 1099511627791,
255
+ "instruction": "push the red block onto the yellow pad",
256
+ "success": false,
257
+ "steps": 440,
258
+ "time_to_success_s": null,
259
+ "failure": "target_not_moved"
260
+ },
261
+ {
262
+ "seed": 1099511627792,
263
+ "instruction": "push the blue block onto the red pad",
264
+ "success": true,
265
+ "steps": 73,
266
+ "time_to_success_s": 3.2
267
+ },
268
+ {
269
+ "seed": 1099511627793,
270
+ "instruction": "push the red block onto the purple pad",
271
+ "success": false,
272
+ "steps": 440,
273
+ "time_to_success_s": null,
274
+ "failure": "target_short_of_pad"
275
+ },
276
+ {
277
+ "seed": 1099511627794,
278
+ "instruction": "push the red block onto the orange pad",
279
+ "success": true,
280
+ "steps": 82,
281
+ "time_to_success_s": 3.65
282
+ },
283
+ {
284
+ "seed": 1099511627795,
285
+ "instruction": "push the orange block onto the blue pad",
286
+ "success": true,
287
+ "steps": 84,
288
+ "time_to_success_s": 3.75
289
+ },
290
+ {
291
+ "seed": 1099511627796,
292
+ "instruction": "push the red block onto the red pad",
293
+ "success": true,
294
+ "steps": 76,
295
+ "time_to_success_s": 3.35
296
+ },
297
+ {
298
+ "seed": 1099511627797,
299
+ "instruction": "push the green block onto the red pad",
300
+ "success": true,
301
+ "steps": 88,
302
+ "time_to_success_s": 3.95
303
+ },
304
+ {
305
+ "seed": 1099511627798,
306
+ "instruction": "push the purple block onto the orange pad",
307
+ "success": true,
308
+ "steps": 74,
309
+ "time_to_success_s": 3.25
310
+ },
311
+ {
312
+ "seed": 1099511627799,
313
+ "instruction": "push the orange block onto the red pad",
314
+ "success": true,
315
+ "steps": 72,
316
+ "time_to_success_s": 3.15
317
+ },
318
+ {
319
+ "seed": 1099511627800,
320
+ "instruction": "push the orange block onto the blue pad",
321
+ "success": true,
322
+ "steps": 60,
323
+ "time_to_success_s": 2.55
324
+ },
325
+ {
326
+ "seed": 1099511627801,
327
+ "instruction": "push the blue block onto the green pad",
328
+ "success": true,
329
+ "steps": 55,
330
+ "time_to_success_s": 2.3
331
+ },
332
+ {
333
+ "seed": 1099511627802,
334
+ "instruction": "push the purple block onto the orange pad",
335
+ "success": false,
336
+ "steps": 440,
337
+ "time_to_success_s": null,
338
+ "failure": "target_short_of_pad"
339
+ },
340
+ {
341
+ "seed": 1099511627803,
342
+ "instruction": "push the purple block onto the orange pad",
343
+ "success": false,
344
+ "steps": 440,
345
+ "time_to_success_s": null,
346
+ "failure": "target_not_moved"
347
+ },
348
+ {
349
+ "seed": 1099511627804,
350
+ "instruction": "push the orange block onto the yellow pad",
351
+ "success": true,
352
+ "steps": 174,
353
+ "time_to_success_s": 8.25
354
+ },
355
+ {
356
+ "seed": 1099511627805,
357
+ "instruction": "push the red block onto the red pad",
358
+ "success": false,
359
+ "steps": 440,
360
+ "time_to_success_s": null,
361
+ "failure": "target_not_moved"
362
+ },
363
+ {
364
+ "seed": 1099511627806,
365
+ "instruction": "push the yellow block onto the red pad",
366
+ "success": true,
367
+ "steps": 87,
368
+ "time_to_success_s": 3.9
369
+ },
370
+ {
371
+ "seed": 1099511627807,
372
+ "instruction": "push the orange block onto the purple pad",
373
+ "success": true,
374
+ "steps": 85,
375
+ "time_to_success_s": 3.8
376
+ },
377
+ {
378
+ "seed": 1099511627808,
379
+ "instruction": "push the purple block onto the blue pad",
380
+ "success": true,
381
+ "steps": 63,
382
+ "time_to_success_s": 2.7
383
+ },
384
+ {
385
+ "seed": 1099511627809,
386
+ "instruction": "push the purple block onto the orange pad",
387
+ "success": true,
388
+ "steps": 96,
389
+ "time_to_success_s": 4.35
390
+ },
391
+ {
392
+ "seed": 1099511627810,
393
+ "instruction": "push the blue block onto the green pad",
394
+ "success": true,
395
+ "steps": 74,
396
+ "time_to_success_s": 3.25
397
+ },
398
+ {
399
+ "seed": 1099511627811,
400
+ "instruction": "push the green block onto the green pad",
401
+ "success": true,
402
+ "steps": 78,
403
+ "time_to_success_s": 3.45
404
+ },
405
+ {
406
+ "seed": 1099511627812,
407
+ "instruction": "push the yellow block onto the blue pad",
408
+ "success": true,
409
+ "steps": 77,
410
+ "time_to_success_s": 3.4
411
+ },
412
+ {
413
+ "seed": 1099511627813,
414
+ "instruction": "push the yellow block onto the green pad",
415
+ "success": true,
416
+ "steps": 92,
417
+ "time_to_success_s": 4.15
418
+ },
419
+ {
420
+ "seed": 1099511627814,
421
+ "instruction": "push the green block onto the green pad",
422
+ "success": true,
423
+ "steps": 76,
424
+ "time_to_success_s": 3.35
425
+ },
426
+ {
427
+ "seed": 1099511627815,
428
+ "instruction": "push the yellow block onto the green pad",
429
+ "success": false,
430
+ "steps": 440,
431
+ "time_to_success_s": null,
432
+ "failure": "target_short_of_pad"
433
+ },
434
+ {
435
+ "seed": 1099511627816,
436
+ "instruction": "push the purple block onto the orange pad",
437
+ "success": true,
438
+ "steps": 92,
439
+ "time_to_success_s": 4.15
440
+ },
441
+ {
442
+ "seed": 1099511627817,
443
+ "instruction": "push the red block onto the purple pad",
444
+ "success": true,
445
+ "steps": 95,
446
+ "time_to_success_s": 4.3
447
+ },
448
+ {
449
+ "seed": 1099511627818,
450
+ "instruction": "push the purple block onto the green pad",
451
+ "success": true,
452
+ "steps": 72,
453
+ "time_to_success_s": 3.15
454
+ },
455
+ {
456
+ "seed": 1099511627819,
457
+ "instruction": "push the purple block onto the purple pad",
458
+ "success": true,
459
+ "steps": 45,
460
+ "time_to_success_s": 1.8
461
+ },
462
+ {
463
+ "seed": 1099511627820,
464
+ "instruction": "push the yellow block onto the purple pad",
465
+ "success": true,
466
+ "steps": 97,
467
+ "time_to_success_s": 4.4
468
+ },
469
+ {
470
+ "seed": 1099511627821,
471
+ "instruction": "push the red block onto the yellow pad",
472
+ "success": true,
473
+ "steps": 65,
474
+ "time_to_success_s": 2.8
475
+ },
476
+ {
477
+ "seed": 1099511627822,
478
+ "instruction": "push the purple block onto the red pad",
479
+ "success": true,
480
+ "steps": 75,
481
+ "time_to_success_s": 3.3
482
+ },
483
+ {
484
+ "seed": 1099511627823,
485
+ "instruction": "push the blue block onto the orange pad",
486
+ "success": true,
487
+ "steps": 58,
488
+ "time_to_success_s": 2.45
489
+ },
490
+ {
491
+ "seed": 1099511627824,
492
+ "instruction": "push the blue block onto the yellow pad",
493
+ "success": true,
494
+ "steps": 110,
495
+ "time_to_success_s": 5.05
496
+ },
497
+ {
498
+ "seed": 1099511627825,
499
+ "instruction": "push the blue block onto the blue pad",
500
+ "success": true,
501
+ "steps": 156,
502
+ "time_to_success_s": 7.35
503
+ }
504
+ ]
505
+ },
506
+ "button": {
507
+ "n": 50,
508
+ "success": 32,
509
+ "rate": 0.64,
510
+ "wilson95": [
511
+ 0.501,
512
+ 0.759
513
+ ],
514
+ "mean_time_to_success_s": 2.88,
515
+ "failures": {
516
+ "no_press": 10,
517
+ "wrong_button_pressed": 2,
518
+ "partial_press": 6
519
+ },
520
+ "per_instruction": {
521
+ "press the blue button": {
522
+ "success": 4,
523
+ "n": 8
524
+ },
525
+ "press the green button": {
526
+ "success": 2,
527
+ "n": 4
528
+ },
529
+ "press the orange button": {
530
+ "success": 3,
531
+ "n": 5
532
+ },
533
+ "press the purple button": {
534
+ "success": 8,
535
+ "n": 11
536
+ },
537
+ "press the red button": {
538
+ "success": 9,
539
+ "n": 13
540
+ },
541
+ "press the yellow button": {
542
+ "success": 6,
543
+ "n": 9
544
+ }
545
+ },
546
+ "wall_s": 174.9,
547
+ "episodes": [
548
+ {
549
+ "seed": 1099512676352,
550
+ "instruction": "press the blue button",
551
+ "success": true,
552
+ "steps": 68,
553
+ "time_to_success_s": 2.95,
554
+ "video": "videos/button_success_seed1099512676352.mp4"
555
+ },
556
+ {
557
+ "seed": 1099512676353,
558
+ "instruction": "press the orange button",
559
+ "success": true,
560
+ "steps": 48,
561
+ "time_to_success_s": 1.95
562
+ },
563
+ {
564
+ "seed": 1099512676354,
565
+ "instruction": "press the purple button",
566
+ "success": true,
567
+ "steps": 53,
568
+ "time_to_success_s": 2.2
569
+ },
570
+ {
571
+ "seed": 1099512676355,
572
+ "instruction": "press the orange button",
573
+ "success": false,
574
+ "steps": 260,
575
+ "time_to_success_s": null,
576
+ "failure": "no_press",
577
+ "video": "videos/button_failure_seed1099512676355.mp4"
578
+ },
579
+ {
580
+ "seed": 1099512676356,
581
+ "instruction": "press the blue button",
582
+ "success": false,
583
+ "steps": 260,
584
+ "time_to_success_s": null,
585
+ "failure": "no_press"
586
+ },
587
+ {
588
+ "seed": 1099512676357,
589
+ "instruction": "press the purple button",
590
+ "success": false,
591
+ "steps": 260,
592
+ "time_to_success_s": null,
593
+ "failure": "no_press"
594
+ },
595
+ {
596
+ "seed": 1099512676358,
597
+ "instruction": "press the red button",
598
+ "success": true,
599
+ "steps": 112,
600
+ "time_to_success_s": 5.15
601
+ },
602
+ {
603
+ "seed": 1099512676359,
604
+ "instruction": "press the blue button",
605
+ "success": true,
606
+ "steps": 49,
607
+ "time_to_success_s": 2.0
608
+ },
609
+ {
610
+ "seed": 1099512676360,
611
+ "instruction": "press the red button",
612
+ "success": true,
613
+ "steps": 63,
614
+ "time_to_success_s": 2.7
615
+ },
616
+ {
617
+ "seed": 1099512676361,
618
+ "instruction": "press the yellow button",
619
+ "success": true,
620
+ "steps": 47,
621
+ "time_to_success_s": 1.9
622
+ },
623
+ {
624
+ "seed": 1099512676362,
625
+ "instruction": "press the red button",
626
+ "success": false,
627
+ "steps": 260,
628
+ "time_to_success_s": null,
629
+ "failure": "no_press"
630
+ },
631
+ {
632
+ "seed": 1099512676363,
633
+ "instruction": "press the purple button",
634
+ "success": true,
635
+ "steps": 75,
636
+ "time_to_success_s": 3.3
637
+ },
638
+ {
639
+ "seed": 1099512676364,
640
+ "instruction": "press the red button",
641
+ "success": true,
642
+ "steps": 246,
643
+ "time_to_success_s": 11.85
644
+ },
645
+ {
646
+ "seed": 1099512676365,
647
+ "instruction": "press the purple button",
648
+ "success": false,
649
+ "steps": 260,
650
+ "time_to_success_s": null,
651
+ "failure": "wrong_button_pressed"
652
+ },
653
+ {
654
+ "seed": 1099512676366,
655
+ "instruction": "press the green button",
656
+ "success": false,
657
+ "steps": 260,
658
+ "time_to_success_s": null,
659
+ "failure": "partial_press"
660
+ },
661
+ {
662
+ "seed": 1099512676367,
663
+ "instruction": "press the red button",
664
+ "success": true,
665
+ "steps": 57,
666
+ "time_to_success_s": 2.4
667
+ },
668
+ {
669
+ "seed": 1099512676368,
670
+ "instruction": "press the blue button",
671
+ "success": false,
672
+ "steps": 260,
673
+ "time_to_success_s": null,
674
+ "failure": "partial_press"
675
+ },
676
+ {
677
+ "seed": 1099512676369,
678
+ "instruction": "press the purple button",
679
+ "success": true,
680
+ "steps": 63,
681
+ "time_to_success_s": 2.7
682
+ },
683
+ {
684
+ "seed": 1099512676370,
685
+ "instruction": "press the yellow button",
686
+ "success": false,
687
+ "steps": 260,
688
+ "time_to_success_s": null,
689
+ "failure": "wrong_button_pressed"
690
+ },
691
+ {
692
+ "seed": 1099512676371,
693
+ "instruction": "press the red button",
694
+ "success": true,
695
+ "steps": 53,
696
+ "time_to_success_s": 2.2
697
+ },
698
+ {
699
+ "seed": 1099512676372,
700
+ "instruction": "press the red button",
701
+ "success": true,
702
+ "steps": 52,
703
+ "time_to_success_s": 2.15
704
+ },
705
+ {
706
+ "seed": 1099512676373,
707
+ "instruction": "press the purple button",
708
+ "success": true,
709
+ "steps": 94,
710
+ "time_to_success_s": 4.25
711
+ },
712
+ {
713
+ "seed": 1099512676374,
714
+ "instruction": "press the yellow button",
715
+ "success": false,
716
+ "steps": 260,
717
+ "time_to_success_s": null,
718
+ "failure": "partial_press"
719
+ },
720
+ {
721
+ "seed": 1099512676375,
722
+ "instruction": "press the green button",
723
+ "success": true,
724
+ "steps": 53,
725
+ "time_to_success_s": 2.2
726
+ },
727
+ {
728
+ "seed": 1099512676376,
729
+ "instruction": "press the blue button",
730
+ "success": false,
731
+ "steps": 260,
732
+ "time_to_success_s": null,
733
+ "failure": "no_press"
734
+ },
735
+ {
736
+ "seed": 1099512676377,
737
+ "instruction": "press the yellow button",
738
+ "success": true,
739
+ "steps": 53,
740
+ "time_to_success_s": 2.2
741
+ },
742
+ {
743
+ "seed": 1099512676378,
744
+ "instruction": "press the yellow button",
745
+ "success": true,
746
+ "steps": 48,
747
+ "time_to_success_s": 1.95
748
+ },
749
+ {
750
+ "seed": 1099512676379,
751
+ "instruction": "press the green button",
752
+ "success": true,
753
+ "steps": 49,
754
+ "time_to_success_s": 2.0
755
+ },
756
+ {
757
+ "seed": 1099512676380,
758
+ "instruction": "press the yellow button",
759
+ "success": true,
760
+ "steps": 138,
761
+ "time_to_success_s": 6.45
762
+ },
763
+ {
764
+ "seed": 1099512676381,
765
+ "instruction": "press the blue button",
766
+ "success": true,
767
+ "steps": 49,
768
+ "time_to_success_s": 2.0
769
+ },
770
+ {
771
+ "seed": 1099512676382,
772
+ "instruction": "press the red button",
773
+ "success": true,
774
+ "steps": 44,
775
+ "time_to_success_s": 1.75
776
+ },
777
+ {
778
+ "seed": 1099512676383,
779
+ "instruction": "press the red button",
780
+ "success": false,
781
+ "steps": 260,
782
+ "time_to_success_s": null,
783
+ "failure": "partial_press"
784
+ },
785
+ {
786
+ "seed": 1099512676384,
787
+ "instruction": "press the yellow button",
788
+ "success": true,
789
+ "steps": 56,
790
+ "time_to_success_s": 2.35
791
+ },
792
+ {
793
+ "seed": 1099512676385,
794
+ "instruction": "press the purple button",
795
+ "success": false,
796
+ "steps": 260,
797
+ "time_to_success_s": null,
798
+ "failure": "no_press"
799
+ },
800
+ {
801
+ "seed": 1099512676386,
802
+ "instruction": "press the orange button",
803
+ "success": true,
804
+ "steps": 82,
805
+ "time_to_success_s": 3.65
806
+ },
807
+ {
808
+ "seed": 1099512676387,
809
+ "instruction": "press the red button",
810
+ "success": true,
811
+ "steps": 68,
812
+ "time_to_success_s": 2.95
813
+ },
814
+ {
815
+ "seed": 1099512676388,
816
+ "instruction": "press the red button",
817
+ "success": false,
818
+ "steps": 260,
819
+ "time_to_success_s": null,
820
+ "failure": "partial_press"
821
+ },
822
+ {
823
+ "seed": 1099512676389,
824
+ "instruction": "press the purple button",
825
+ "success": true,
826
+ "steps": 70,
827
+ "time_to_success_s": 3.05
828
+ },
829
+ {
830
+ "seed": 1099512676390,
831
+ "instruction": "press the purple button",
832
+ "success": true,
833
+ "steps": 50,
834
+ "time_to_success_s": 2.05
835
+ },
836
+ {
837
+ "seed": 1099512676391,
838
+ "instruction": "press the red button",
839
+ "success": false,
840
+ "steps": 260,
841
+ "time_to_success_s": null,
842
+ "failure": "no_press"
843
+ },
844
+ {
845
+ "seed": 1099512676392,
846
+ "instruction": "press the yellow button",
847
+ "success": true,
848
+ "steps": 53,
849
+ "time_to_success_s": 2.2
850
+ },
851
+ {
852
+ "seed": 1099512676393,
853
+ "instruction": "press the orange button",
854
+ "success": true,
855
+ "steps": 41,
856
+ "time_to_success_s": 1.6
857
+ },
858
+ {
859
+ "seed": 1099512676394,
860
+ "instruction": "press the blue button",
861
+ "success": true,
862
+ "steps": 58,
863
+ "time_to_success_s": 2.45
864
+ },
865
+ {
866
+ "seed": 1099512676395,
867
+ "instruction": "press the yellow button",
868
+ "success": false,
869
+ "steps": 260,
870
+ "time_to_success_s": null,
871
+ "failure": "no_press"
872
+ },
873
+ {
874
+ "seed": 1099512676396,
875
+ "instruction": "press the red button",
876
+ "success": true,
877
+ "steps": 57,
878
+ "time_to_success_s": 2.4
879
+ },
880
+ {
881
+ "seed": 1099512676397,
882
+ "instruction": "press the orange button",
883
+ "success": false,
884
+ "steps": 260,
885
+ "time_to_success_s": null,
886
+ "failure": "no_press"
887
+ },
888
+ {
889
+ "seed": 1099512676398,
890
+ "instruction": "press the purple button",
891
+ "success": true,
892
+ "steps": 40,
893
+ "time_to_success_s": 1.55
894
+ },
895
+ {
896
+ "seed": 1099512676399,
897
+ "instruction": "press the green button",
898
+ "success": false,
899
+ "steps": 260,
900
+ "time_to_success_s": null,
901
+ "failure": "no_press"
902
+ },
903
+ {
904
+ "seed": 1099512676400,
905
+ "instruction": "press the blue button",
906
+ "success": false,
907
+ "steps": 260,
908
+ "time_to_success_s": null,
909
+ "failure": "partial_press"
910
+ },
911
+ {
912
+ "seed": 1099512676401,
913
+ "instruction": "press the purple button",
914
+ "success": true,
915
+ "steps": 40,
916
+ "time_to_success_s": 1.55
917
+ }
918
+ ]
919
+ },
920
+ "box": {
921
+ "n": 50,
922
+ "success": 28,
923
+ "rate": 0.56,
924
+ "wilson95": [
925
+ 0.423,
926
+ 0.688
927
+ ],
928
+ "mean_time_to_success_s": 13.26,
929
+ "failures": {
930
+ "dropped_or_not_placed": 16,
931
+ "tipped": 2,
932
+ "off_shelf_footprint_or_moving": 2,
933
+ "dropped_off_table": 1,
934
+ "not_released": 1
935
+ },
936
+ "per_instruction": {
937
+ "lift the box and put it on the shelf": {
938
+ "success": 28,
939
+ "n": 50
940
+ }
941
+ },
942
+ "wall_s": 268.2,
943
+ "episodes": [
944
+ {
945
+ "seed": 1099513724928,
946
+ "instruction": "lift the box and put it on the shelf",
947
+ "success": false,
948
+ "steps": 440,
949
+ "time_to_success_s": null,
950
+ "failure": "dropped_or_not_placed",
951
+ "video": "videos/box_failure_seed1099513724928.mp4"
952
+ },
953
+ {
954
+ "seed": 1099513724929,
955
+ "instruction": "lift the box and put it on the shelf",
956
+ "success": false,
957
+ "steps": 440,
958
+ "time_to_success_s": null,
959
+ "failure": "tipped"
960
+ },
961
+ {
962
+ "seed": 1099513724930,
963
+ "instruction": "lift the box and put it on the shelf",
964
+ "success": true,
965
+ "steps": 278,
966
+ "time_to_success_s": 13.45,
967
+ "video": "videos/box_success_seed1099513724930.mp4"
968
+ },
969
+ {
970
+ "seed": 1099513724931,
971
+ "instruction": "lift the box and put it on the shelf",
972
+ "success": true,
973
+ "steps": 228,
974
+ "time_to_success_s": 10.95
975
+ },
976
+ {
977
+ "seed": 1099513724932,
978
+ "instruction": "lift the box and put it on the shelf",
979
+ "success": false,
980
+ "steps": 440,
981
+ "time_to_success_s": null,
982
+ "failure": "dropped_or_not_placed"
983
+ },
984
+ {
985
+ "seed": 1099513724933,
986
+ "instruction": "lift the box and put it on the shelf",
987
+ "success": false,
988
+ "steps": 440,
989
+ "time_to_success_s": null,
990
+ "failure": "off_shelf_footprint_or_moving"
991
+ },
992
+ {
993
+ "seed": 1099513724934,
994
+ "instruction": "lift the box and put it on the shelf",
995
+ "success": true,
996
+ "steps": 297,
997
+ "time_to_success_s": 14.4
998
+ },
999
+ {
1000
+ "seed": 1099513724935,
1001
+ "instruction": "lift the box and put it on the shelf",
1002
+ "success": false,
1003
+ "steps": 440,
1004
+ "time_to_success_s": null,
1005
+ "failure": "dropped_or_not_placed"
1006
+ },
1007
+ {
1008
+ "seed": 1099513724936,
1009
+ "instruction": "lift the box and put it on the shelf",
1010
+ "success": false,
1011
+ "steps": 440,
1012
+ "time_to_success_s": null,
1013
+ "failure": "dropped_off_table"
1014
+ },
1015
+ {
1016
+ "seed": 1099513724937,
1017
+ "instruction": "lift the box and put it on the shelf",
1018
+ "success": true,
1019
+ "steps": 229,
1020
+ "time_to_success_s": 11.0
1021
+ },
1022
+ {
1023
+ "seed": 1099513724938,
1024
+ "instruction": "lift the box and put it on the shelf",
1025
+ "success": true,
1026
+ "steps": 236,
1027
+ "time_to_success_s": 11.35
1028
+ },
1029
+ {
1030
+ "seed": 1099513724939,
1031
+ "instruction": "lift the box and put it on the shelf",
1032
+ "success": false,
1033
+ "steps": 440,
1034
+ "time_to_success_s": null,
1035
+ "failure": "dropped_or_not_placed"
1036
+ },
1037
+ {
1038
+ "seed": 1099513724940,
1039
+ "instruction": "lift the box and put it on the shelf",
1040
+ "success": true,
1041
+ "steps": 339,
1042
+ "time_to_success_s": 16.5
1043
+ },
1044
+ {
1045
+ "seed": 1099513724941,
1046
+ "instruction": "lift the box and put it on the shelf",
1047
+ "success": false,
1048
+ "steps": 440,
1049
+ "time_to_success_s": null,
1050
+ "failure": "dropped_or_not_placed"
1051
+ },
1052
+ {
1053
+ "seed": 1099513724942,
1054
+ "instruction": "lift the box and put it on the shelf",
1055
+ "success": false,
1056
+ "steps": 440,
1057
+ "time_to_success_s": null,
1058
+ "failure": "dropped_or_not_placed"
1059
+ },
1060
+ {
1061
+ "seed": 1099513724943,
1062
+ "instruction": "lift the box and put it on the shelf",
1063
+ "success": true,
1064
+ "steps": 440,
1065
+ "time_to_success_s": 22.0
1066
+ },
1067
+ {
1068
+ "seed": 1099513724944,
1069
+ "instruction": "lift the box and put it on the shelf",
1070
+ "success": true,
1071
+ "steps": 244,
1072
+ "time_to_success_s": 11.75
1073
+ },
1074
+ {
1075
+ "seed": 1099513724945,
1076
+ "instruction": "lift the box and put it on the shelf",
1077
+ "success": true,
1078
+ "steps": 243,
1079
+ "time_to_success_s": 11.7
1080
+ },
1081
+ {
1082
+ "seed": 1099513724946,
1083
+ "instruction": "lift the box and put it on the shelf",
1084
+ "success": true,
1085
+ "steps": 186,
1086
+ "time_to_success_s": 8.85
1087
+ },
1088
+ {
1089
+ "seed": 1099513724947,
1090
+ "instruction": "lift the box and put it on the shelf",
1091
+ "success": true,
1092
+ "steps": 190,
1093
+ "time_to_success_s": 9.05
1094
+ },
1095
+ {
1096
+ "seed": 1099513724948,
1097
+ "instruction": "lift the box and put it on the shelf",
1098
+ "success": true,
1099
+ "steps": 220,
1100
+ "time_to_success_s": 10.55
1101
+ },
1102
+ {
1103
+ "seed": 1099513724949,
1104
+ "instruction": "lift the box and put it on the shelf",
1105
+ "success": false,
1106
+ "steps": 440,
1107
+ "time_to_success_s": null,
1108
+ "failure": "dropped_or_not_placed"
1109
+ },
1110
+ {
1111
+ "seed": 1099513724950,
1112
+ "instruction": "lift the box and put it on the shelf",
1113
+ "success": true,
1114
+ "steps": 207,
1115
+ "time_to_success_s": 9.9
1116
+ },
1117
+ {
1118
+ "seed": 1099513724951,
1119
+ "instruction": "lift the box and put it on the shelf",
1120
+ "success": true,
1121
+ "steps": 258,
1122
+ "time_to_success_s": 12.45
1123
+ },
1124
+ {
1125
+ "seed": 1099513724952,
1126
+ "instruction": "lift the box and put it on the shelf",
1127
+ "success": false,
1128
+ "steps": 440,
1129
+ "time_to_success_s": null,
1130
+ "failure": "dropped_or_not_placed"
1131
+ },
1132
+ {
1133
+ "seed": 1099513724953,
1134
+ "instruction": "lift the box and put it on the shelf",
1135
+ "success": false,
1136
+ "steps": 440,
1137
+ "time_to_success_s": null,
1138
+ "failure": "dropped_or_not_placed"
1139
+ },
1140
+ {
1141
+ "seed": 1099513724954,
1142
+ "instruction": "lift the box and put it on the shelf",
1143
+ "success": true,
1144
+ "steps": 326,
1145
+ "time_to_success_s": 15.85
1146
+ },
1147
+ {
1148
+ "seed": 1099513724955,
1149
+ "instruction": "lift the box and put it on the shelf",
1150
+ "success": true,
1151
+ "steps": 197,
1152
+ "time_to_success_s": 9.4
1153
+ },
1154
+ {
1155
+ "seed": 1099513724956,
1156
+ "instruction": "lift the box and put it on the shelf",
1157
+ "success": false,
1158
+ "steps": 440,
1159
+ "time_to_success_s": null,
1160
+ "failure": "dropped_or_not_placed"
1161
+ },
1162
+ {
1163
+ "seed": 1099513724957,
1164
+ "instruction": "lift the box and put it on the shelf",
1165
+ "success": true,
1166
+ "steps": 365,
1167
+ "time_to_success_s": 17.8
1168
+ },
1169
+ {
1170
+ "seed": 1099513724958,
1171
+ "instruction": "lift the box and put it on the shelf",
1172
+ "success": false,
1173
+ "steps": 440,
1174
+ "time_to_success_s": null,
1175
+ "failure": "dropped_or_not_placed"
1176
+ },
1177
+ {
1178
+ "seed": 1099513724959,
1179
+ "instruction": "lift the box and put it on the shelf",
1180
+ "success": false,
1181
+ "steps": 440,
1182
+ "time_to_success_s": null,
1183
+ "failure": "off_shelf_footprint_or_moving"
1184
+ },
1185
+ {
1186
+ "seed": 1099513724960,
1187
+ "instruction": "lift the box and put it on the shelf",
1188
+ "success": false,
1189
+ "steps": 440,
1190
+ "time_to_success_s": null,
1191
+ "failure": "dropped_or_not_placed"
1192
+ },
1193
+ {
1194
+ "seed": 1099513724961,
1195
+ "instruction": "lift the box and put it on the shelf",
1196
+ "success": false,
1197
+ "steps": 440,
1198
+ "time_to_success_s": null,
1199
+ "failure": "dropped_or_not_placed"
1200
+ },
1201
+ {
1202
+ "seed": 1099513724962,
1203
+ "instruction": "lift the box and put it on the shelf",
1204
+ "success": false,
1205
+ "steps": 440,
1206
+ "time_to_success_s": null,
1207
+ "failure": "dropped_or_not_placed"
1208
+ },
1209
+ {
1210
+ "seed": 1099513724963,
1211
+ "instruction": "lift the box and put it on the shelf",
1212
+ "success": false,
1213
+ "steps": 440,
1214
+ "time_to_success_s": null,
1215
+ "failure": "dropped_or_not_placed"
1216
+ },
1217
+ {
1218
+ "seed": 1099513724964,
1219
+ "instruction": "lift the box and put it on the shelf",
1220
+ "success": true,
1221
+ "steps": 277,
1222
+ "time_to_success_s": 13.4
1223
+ },
1224
+ {
1225
+ "seed": 1099513724965,
1226
+ "instruction": "lift the box and put it on the shelf",
1227
+ "success": false,
1228
+ "steps": 440,
1229
+ "time_to_success_s": null,
1230
+ "failure": "not_released"
1231
+ },
1232
+ {
1233
+ "seed": 1099513724966,
1234
+ "instruction": "lift the box and put it on the shelf",
1235
+ "success": true,
1236
+ "steps": 314,
1237
+ "time_to_success_s": 15.25
1238
+ },
1239
+ {
1240
+ "seed": 1099513724967,
1241
+ "instruction": "lift the box and put it on the shelf",
1242
+ "success": true,
1243
+ "steps": 238,
1244
+ "time_to_success_s": 11.45
1245
+ },
1246
+ {
1247
+ "seed": 1099513724968,
1248
+ "instruction": "lift the box and put it on the shelf",
1249
+ "success": true,
1250
+ "steps": 414,
1251
+ "time_to_success_s": 20.25
1252
+ },
1253
+ {
1254
+ "seed": 1099513724969,
1255
+ "instruction": "lift the box and put it on the shelf",
1256
+ "success": true,
1257
+ "steps": 217,
1258
+ "time_to_success_s": 10.4
1259
+ },
1260
+ {
1261
+ "seed": 1099513724970,
1262
+ "instruction": "lift the box and put it on the shelf",
1263
+ "success": true,
1264
+ "steps": 314,
1265
+ "time_to_success_s": 15.25
1266
+ },
1267
+ {
1268
+ "seed": 1099513724971,
1269
+ "instruction": "lift the box and put it on the shelf",
1270
+ "success": false,
1271
+ "steps": 440,
1272
+ "time_to_success_s": null,
1273
+ "failure": "dropped_or_not_placed"
1274
+ },
1275
+ {
1276
+ "seed": 1099513724972,
1277
+ "instruction": "lift the box and put it on the shelf",
1278
+ "success": true,
1279
+ "steps": 195,
1280
+ "time_to_success_s": 9.3
1281
+ },
1282
+ {
1283
+ "seed": 1099513724973,
1284
+ "instruction": "lift the box and put it on the shelf",
1285
+ "success": true,
1286
+ "steps": 325,
1287
+ "time_to_success_s": 15.8
1288
+ },
1289
+ {
1290
+ "seed": 1099513724974,
1291
+ "instruction": "lift the box and put it on the shelf",
1292
+ "success": true,
1293
+ "steps": 254,
1294
+ "time_to_success_s": 12.25
1295
+ },
1296
+ {
1297
+ "seed": 1099513724975,
1298
+ "instruction": "lift the box and put it on the shelf",
1299
+ "success": false,
1300
+ "steps": 440,
1301
+ "time_to_success_s": null,
1302
+ "failure": "tipped"
1303
+ },
1304
+ {
1305
+ "seed": 1099513724976,
1306
+ "instruction": "lift the box and put it on the shelf",
1307
+ "success": true,
1308
+ "steps": 230,
1309
+ "time_to_success_s": 11.05
1310
+ },
1311
+ {
1312
+ "seed": 1099513724977,
1313
+ "instruction": "lift the box and put it on the shelf",
1314
+ "success": true,
1315
+ "steps": 405,
1316
+ "time_to_success_s": 19.8
1317
+ }
1318
+ ]
1319
+ }
1320
+ },
1321
+ "inference_s": {
1322
+ "mean": 0.095,
1323
+ "p50": 0.0943,
1324
+ "p90": 0.096,
1325
+ "calls": 3970
1326
+ },
1327
+ "wall_s": 1647.2
1328
+ }
results/closedloop_checkpoint-10000/results.md ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Closed-loop evaluation: groot (/root/work/runs/asimov_n17/checkpoint-10000)
2
+
3
+ 50 held-out episodes per task (seeds 2**40 + task_id * 2**20 + k; checked against 926 dataset seeds: no overlap (max training seed 1200300)), 8 of 16 actions executed per inference at 20 Hz, success held 10 steps, GL egl.
4
+
5
+ | Task | Success | 95% CI | Mean time to success | Failures |
6
+ | --- | --- | --- | --- | --- |
7
+ | push | 42/50 (84%) | 0.71-0.92 | 3.93 s | target_short_of_pad 5, target_not_moved 3 |
8
+ | button | 32/50 (64%) | 0.50-0.76 | 2.88 s | no_press 10, partial_press 6, wrong_button_pressed 2 |
9
+ | box | 28/50 (56%) | 0.42-0.69 | 13.26 s | dropped_or_not_placed 16, tipped 2, off_shelf_footprint_or_moving 2, dropped_off_table 1, not_released 1 |
10
+
11
+ Policy calls: {'mean': 0.095, 'p50': 0.0943, 'p90': 0.096, 'calls': 3970}
12
+
13
+ ## push: per instruction
14
+
15
+ | Instruction | Success |
16
+ | --- | --- |
17
+ | push the blue block onto the blue pad | 2/2 |
18
+ | push the blue block onto the green pad | 2/2 |
19
+ | push the blue block onto the orange pad | 2/2 |
20
+ | push the blue block onto the purple pad | 1/1 |
21
+ | push the blue block onto the red pad | 1/1 |
22
+ | push the blue block onto the yellow pad | 1/1 |
23
+ | push the green block onto the blue pad | 1/1 |
24
+ | push the green block onto the green pad | 2/2 |
25
+ | push the green block onto the purple pad | 0/1 |
26
+ | push the green block onto the red pad | 1/1 |
27
+ | push the orange block onto the blue pad | 2/2 |
28
+ | push the orange block onto the purple pad | 1/1 |
29
+ | push the orange block onto the red pad | 2/2 |
30
+ | push the orange block onto the yellow pad | 2/2 |
31
+ | push the purple block onto the blue pad | 1/1 |
32
+ | push the purple block onto the green pad | 1/1 |
33
+ | push the purple block onto the orange pad | 3/5 |
34
+ | push the purple block onto the purple pad | 1/1 |
35
+ | push the purple block onto the red pad | 2/2 |
36
+ | push the purple block onto the yellow pad | 1/1 |
37
+ | push the red block onto the orange pad | 1/1 |
38
+ | push the red block onto the purple pad | 1/2 |
39
+ | push the red block onto the red pad | 1/2 |
40
+ | push the red block onto the yellow pad | 2/4 |
41
+ | push the yellow block onto the blue pad | 3/3 |
42
+ | push the yellow block onto the green pad | 2/3 |
43
+ | push the yellow block onto the purple pad | 1/1 |
44
+ | push the yellow block onto the red pad | 1/1 |
45
+ | push the yellow block onto the yellow pad | 1/1 |
46
+
47
+ Videos: videos/push_success_seed1099511627776.mp4, videos/push_failure_seed1099511627787.mp4
48
+
49
+ ## button: per instruction
50
+
51
+ | Instruction | Success |
52
+ | --- | --- |
53
+ | press the blue button | 4/8 |
54
+ | press the green button | 2/4 |
55
+ | press the orange button | 3/5 |
56
+ | press the purple button | 8/11 |
57
+ | press the red button | 9/13 |
58
+ | press the yellow button | 6/9 |
59
+
60
+ Videos: videos/button_success_seed1099512676352.mp4, videos/button_failure_seed1099512676355.mp4
61
+
62
+ ## box: per instruction
63
+
64
+ | Instruction | Success |
65
+ | --- | --- |
66
+ | lift the box and put it on the shelf | 28/50 |
67
+
68
+ Videos: videos/box_failure_seed1099513724928.mp4, videos/box_success_seed1099513724930.mp4
results/closedloop_checkpoint-10000/videos/box_failure_seed1099513724928.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e1e902307eb96f5625a2dfff3ce4d15491f3e108a1909cb5339e2d3a4b13e798
3
+ size 225596
results/closedloop_checkpoint-10000/videos/box_success_seed1099513724930.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3b207703836d281b6ff8236f7a0d06e9da9dd6351d05ea63309d577f06ee9b90
3
+ size 231732
results/closedloop_checkpoint-10000/videos/button_failure_seed1099512676355.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b527afe084c3a8bff4772cd081d8a143f7d203d66d3bd7f3e57ff86bdc7f43e9
3
+ size 232467
results/closedloop_checkpoint-10000/videos/button_success_seed1099512676352.mp4 ADDED
Binary file (71 kB). View file
 
results/closedloop_checkpoint-10000/videos/push_failure_seed1099511627787.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:95085e8b02b9b62fac0e779b0621bd90147dc4b22f06c95acf7d82e9d020ccd5
3
+ size 342636
results/closedloop_checkpoint-10000/videos/push_success_seed1099511627776.mp4 ADDED
Binary file (88.4 kB). View file
 
results/closedloop_checkpoint-6000/results.json ADDED
@@ -0,0 +1,1387 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "policy": "groot",
3
+ "model_path": "/root/work/runs/asimov_n17/checkpoint-6000",
4
+ "exec_steps": 8,
5
+ "episodes_per_task": 50,
6
+ "hold_steps": 10,
7
+ "gl_backend": "egl",
8
+ "seeds": "checked against 926 dataset seeds: no overlap (max training seed 1200300)",
9
+ "seed_scheme": "2**40 + task_id * 2**20 + k",
10
+ "tasks": {
11
+ "push": {
12
+ "n": 50,
13
+ "success": 23,
14
+ "rate": 0.46,
15
+ "wilson95": [
16
+ 0.33,
17
+ 0.596
18
+ ],
19
+ "mean_time_to_success_s": 5.83,
20
+ "failures": {
21
+ "target_not_moved": 17,
22
+ "target_short_of_pad": 9,
23
+ "target_on_wrong_pad": 1
24
+ },
25
+ "per_instruction": {
26
+ "push the blue block onto the blue pad": {
27
+ "success": 1,
28
+ "n": 2
29
+ },
30
+ "push the blue block onto the green pad": {
31
+ "success": 1,
32
+ "n": 2
33
+ },
34
+ "push the blue block onto the orange pad": {
35
+ "success": 2,
36
+ "n": 2
37
+ },
38
+ "push the blue block onto the purple pad": {
39
+ "success": 0,
40
+ "n": 1
41
+ },
42
+ "push the blue block onto the red pad": {
43
+ "success": 1,
44
+ "n": 1
45
+ },
46
+ "push the blue block onto the yellow pad": {
47
+ "success": 0,
48
+ "n": 1
49
+ },
50
+ "push the green block onto the blue pad": {
51
+ "success": 1,
52
+ "n": 1
53
+ },
54
+ "push the green block onto the green pad": {
55
+ "success": 1,
56
+ "n": 2
57
+ },
58
+ "push the green block onto the purple pad": {
59
+ "success": 0,
60
+ "n": 1
61
+ },
62
+ "push the green block onto the red pad": {
63
+ "success": 0,
64
+ "n": 1
65
+ },
66
+ "push the orange block onto the blue pad": {
67
+ "success": 1,
68
+ "n": 2
69
+ },
70
+ "push the orange block onto the purple pad": {
71
+ "success": 0,
72
+ "n": 1
73
+ },
74
+ "push the orange block onto the red pad": {
75
+ "success": 1,
76
+ "n": 2
77
+ },
78
+ "push the orange block onto the yellow pad": {
79
+ "success": 2,
80
+ "n": 2
81
+ },
82
+ "push the purple block onto the blue pad": {
83
+ "success": 1,
84
+ "n": 1
85
+ },
86
+ "push the purple block onto the green pad": {
87
+ "success": 1,
88
+ "n": 1
89
+ },
90
+ "push the purple block onto the orange pad": {
91
+ "success": 4,
92
+ "n": 5
93
+ },
94
+ "push the purple block onto the purple pad": {
95
+ "success": 0,
96
+ "n": 1
97
+ },
98
+ "push the purple block onto the red pad": {
99
+ "success": 0,
100
+ "n": 2
101
+ },
102
+ "push the purple block onto the yellow pad": {
103
+ "success": 0,
104
+ "n": 1
105
+ },
106
+ "push the red block onto the orange pad": {
107
+ "success": 1,
108
+ "n": 1
109
+ },
110
+ "push the red block onto the purple pad": {
111
+ "success": 0,
112
+ "n": 2
113
+ },
114
+ "push the red block onto the red pad": {
115
+ "success": 0,
116
+ "n": 2
117
+ },
118
+ "push the red block onto the yellow pad": {
119
+ "success": 2,
120
+ "n": 4
121
+ },
122
+ "push the yellow block onto the blue pad": {
123
+ "success": 2,
124
+ "n": 3
125
+ },
126
+ "push the yellow block onto the green pad": {
127
+ "success": 1,
128
+ "n": 3
129
+ },
130
+ "push the yellow block onto the purple pad": {
131
+ "success": 0,
132
+ "n": 1
133
+ },
134
+ "push the yellow block onto the red pad": {
135
+ "success": 0,
136
+ "n": 1
137
+ },
138
+ "push the yellow block onto the yellow pad": {
139
+ "success": 0,
140
+ "n": 1
141
+ }
142
+ },
143
+ "wall_s": 1249.7,
144
+ "episodes": [
145
+ {
146
+ "seed": 1099511627776,
147
+ "instruction": "push the yellow block onto the green pad",
148
+ "success": false,
149
+ "steps": 440,
150
+ "time_to_success_s": null,
151
+ "failure": "target_not_moved",
152
+ "video": "videos/push_failure_seed1099511627776.mp4"
153
+ },
154
+ {
155
+ "seed": 1099511627777,
156
+ "instruction": "push the yellow block onto the blue pad",
157
+ "success": true,
158
+ "steps": 136,
159
+ "time_to_success_s": 6.35,
160
+ "video": "videos/push_success_seed1099511627777.mp4"
161
+ },
162
+ {
163
+ "seed": 1099511627778,
164
+ "instruction": "push the orange block onto the red pad",
165
+ "success": false,
166
+ "steps": 440,
167
+ "time_to_success_s": null,
168
+ "failure": "target_not_moved"
169
+ },
170
+ {
171
+ "seed": 1099511627779,
172
+ "instruction": "push the green block onto the blue pad",
173
+ "success": true,
174
+ "steps": 49,
175
+ "time_to_success_s": 2.0
176
+ },
177
+ {
178
+ "seed": 1099511627780,
179
+ "instruction": "push the orange block onto the yellow pad",
180
+ "success": true,
181
+ "steps": 418,
182
+ "time_to_success_s": 20.45
183
+ },
184
+ {
185
+ "seed": 1099511627781,
186
+ "instruction": "push the blue block onto the orange pad",
187
+ "success": true,
188
+ "steps": 273,
189
+ "time_to_success_s": 13.2
190
+ },
191
+ {
192
+ "seed": 1099511627782,
193
+ "instruction": "push the blue block onto the blue pad",
194
+ "success": true,
195
+ "steps": 56,
196
+ "time_to_success_s": 2.35
197
+ },
198
+ {
199
+ "seed": 1099511627783,
200
+ "instruction": "push the yellow block onto the blue pad",
201
+ "success": true,
202
+ "steps": 106,
203
+ "time_to_success_s": 4.85
204
+ },
205
+ {
206
+ "seed": 1099511627784,
207
+ "instruction": "push the purple block onto the red pad",
208
+ "success": false,
209
+ "steps": 440,
210
+ "time_to_success_s": null,
211
+ "failure": "target_short_of_pad"
212
+ },
213
+ {
214
+ "seed": 1099511627785,
215
+ "instruction": "push the purple block onto the yellow pad",
216
+ "success": false,
217
+ "steps": 440,
218
+ "time_to_success_s": null,
219
+ "failure": "target_short_of_pad"
220
+ },
221
+ {
222
+ "seed": 1099511627786,
223
+ "instruction": "push the blue block onto the purple pad",
224
+ "success": false,
225
+ "steps": 440,
226
+ "time_to_success_s": null,
227
+ "failure": "target_not_moved"
228
+ },
229
+ {
230
+ "seed": 1099511627787,
231
+ "instruction": "push the red block onto the yellow pad",
232
+ "success": false,
233
+ "steps": 440,
234
+ "time_to_success_s": null,
235
+ "failure": "target_on_wrong_pad"
236
+ },
237
+ {
238
+ "seed": 1099511627788,
239
+ "instruction": "push the green block onto the purple pad",
240
+ "success": false,
241
+ "steps": 440,
242
+ "time_to_success_s": null,
243
+ "failure": "target_short_of_pad"
244
+ },
245
+ {
246
+ "seed": 1099511627789,
247
+ "instruction": "push the yellow block onto the yellow pad",
248
+ "success": false,
249
+ "steps": 440,
250
+ "time_to_success_s": null,
251
+ "failure": "target_not_moved"
252
+ },
253
+ {
254
+ "seed": 1099511627790,
255
+ "instruction": "push the red block onto the yellow pad",
256
+ "success": true,
257
+ "steps": 66,
258
+ "time_to_success_s": 2.85
259
+ },
260
+ {
261
+ "seed": 1099511627791,
262
+ "instruction": "push the red block onto the yellow pad",
263
+ "success": false,
264
+ "steps": 440,
265
+ "time_to_success_s": null,
266
+ "failure": "target_not_moved"
267
+ },
268
+ {
269
+ "seed": 1099511627792,
270
+ "instruction": "push the blue block onto the red pad",
271
+ "success": true,
272
+ "steps": 127,
273
+ "time_to_success_s": 5.9
274
+ },
275
+ {
276
+ "seed": 1099511627793,
277
+ "instruction": "push the red block onto the purple pad",
278
+ "success": false,
279
+ "steps": 440,
280
+ "time_to_success_s": null,
281
+ "failure": "target_short_of_pad"
282
+ },
283
+ {
284
+ "seed": 1099511627794,
285
+ "instruction": "push the red block onto the orange pad",
286
+ "success": true,
287
+ "steps": 71,
288
+ "time_to_success_s": 3.1
289
+ },
290
+ {
291
+ "seed": 1099511627795,
292
+ "instruction": "push the orange block onto the blue pad",
293
+ "success": true,
294
+ "steps": 134,
295
+ "time_to_success_s": 6.25
296
+ },
297
+ {
298
+ "seed": 1099511627796,
299
+ "instruction": "push the red block onto the red pad",
300
+ "success": false,
301
+ "steps": 440,
302
+ "time_to_success_s": null,
303
+ "failure": "target_not_moved"
304
+ },
305
+ {
306
+ "seed": 1099511627797,
307
+ "instruction": "push the green block onto the red pad",
308
+ "success": false,
309
+ "steps": 440,
310
+ "time_to_success_s": null,
311
+ "failure": "target_not_moved"
312
+ },
313
+ {
314
+ "seed": 1099511627798,
315
+ "instruction": "push the purple block onto the orange pad",
316
+ "success": true,
317
+ "steps": 132,
318
+ "time_to_success_s": 6.15
319
+ },
320
+ {
321
+ "seed": 1099511627799,
322
+ "instruction": "push the orange block onto the red pad",
323
+ "success": true,
324
+ "steps": 131,
325
+ "time_to_success_s": 6.1
326
+ },
327
+ {
328
+ "seed": 1099511627800,
329
+ "instruction": "push the orange block onto the blue pad",
330
+ "success": false,
331
+ "steps": 440,
332
+ "time_to_success_s": null,
333
+ "failure": "target_not_moved"
334
+ },
335
+ {
336
+ "seed": 1099511627801,
337
+ "instruction": "push the blue block onto the green pad",
338
+ "success": true,
339
+ "steps": 77,
340
+ "time_to_success_s": 3.4
341
+ },
342
+ {
343
+ "seed": 1099511627802,
344
+ "instruction": "push the purple block onto the orange pad",
345
+ "success": true,
346
+ "steps": 101,
347
+ "time_to_success_s": 4.6
348
+ },
349
+ {
350
+ "seed": 1099511627803,
351
+ "instruction": "push the purple block onto the orange pad",
352
+ "success": true,
353
+ "steps": 46,
354
+ "time_to_success_s": 1.85
355
+ },
356
+ {
357
+ "seed": 1099511627804,
358
+ "instruction": "push the orange block onto the yellow pad",
359
+ "success": true,
360
+ "steps": 269,
361
+ "time_to_success_s": 13.0
362
+ },
363
+ {
364
+ "seed": 1099511627805,
365
+ "instruction": "push the red block onto the red pad",
366
+ "success": false,
367
+ "steps": 440,
368
+ "time_to_success_s": null,
369
+ "failure": "target_not_moved"
370
+ },
371
+ {
372
+ "seed": 1099511627806,
373
+ "instruction": "push the yellow block onto the red pad",
374
+ "success": false,
375
+ "steps": 440,
376
+ "time_to_success_s": null,
377
+ "failure": "target_not_moved"
378
+ },
379
+ {
380
+ "seed": 1099511627807,
381
+ "instruction": "push the orange block onto the purple pad",
382
+ "success": false,
383
+ "steps": 440,
384
+ "time_to_success_s": null,
385
+ "failure": "target_short_of_pad"
386
+ },
387
+ {
388
+ "seed": 1099511627808,
389
+ "instruction": "push the purple block onto the blue pad",
390
+ "success": true,
391
+ "steps": 137,
392
+ "time_to_success_s": 6.4
393
+ },
394
+ {
395
+ "seed": 1099511627809,
396
+ "instruction": "push the purple block onto the orange pad",
397
+ "success": true,
398
+ "steps": 114,
399
+ "time_to_success_s": 5.25
400
+ },
401
+ {
402
+ "seed": 1099511627810,
403
+ "instruction": "push the blue block onto the green pad",
404
+ "success": false,
405
+ "steps": 440,
406
+ "time_to_success_s": null,
407
+ "failure": "target_not_moved"
408
+ },
409
+ {
410
+ "seed": 1099511627811,
411
+ "instruction": "push the green block onto the green pad",
412
+ "success": false,
413
+ "steps": 440,
414
+ "time_to_success_s": null,
415
+ "failure": "target_not_moved"
416
+ },
417
+ {
418
+ "seed": 1099511627812,
419
+ "instruction": "push the yellow block onto the blue pad",
420
+ "success": false,
421
+ "steps": 440,
422
+ "time_to_success_s": null,
423
+ "failure": "target_short_of_pad"
424
+ },
425
+ {
426
+ "seed": 1099511627813,
427
+ "instruction": "push the yellow block onto the green pad",
428
+ "success": false,
429
+ "steps": 440,
430
+ "time_to_success_s": null,
431
+ "failure": "target_not_moved"
432
+ },
433
+ {
434
+ "seed": 1099511627814,
435
+ "instruction": "push the green block onto the green pad",
436
+ "success": true,
437
+ "steps": 122,
438
+ "time_to_success_s": 5.65
439
+ },
440
+ {
441
+ "seed": 1099511627815,
442
+ "instruction": "push the yellow block onto the green pad",
443
+ "success": true,
444
+ "steps": 104,
445
+ "time_to_success_s": 4.75
446
+ },
447
+ {
448
+ "seed": 1099511627816,
449
+ "instruction": "push the purple block onto the orange pad",
450
+ "success": false,
451
+ "steps": 440,
452
+ "time_to_success_s": null,
453
+ "failure": "target_short_of_pad"
454
+ },
455
+ {
456
+ "seed": 1099511627817,
457
+ "instruction": "push the red block onto the purple pad",
458
+ "success": false,
459
+ "steps": 440,
460
+ "time_to_success_s": null,
461
+ "failure": "target_short_of_pad"
462
+ },
463
+ {
464
+ "seed": 1099511627818,
465
+ "instruction": "push the purple block onto the green pad",
466
+ "success": true,
467
+ "steps": 61,
468
+ "time_to_success_s": 2.6
469
+ },
470
+ {
471
+ "seed": 1099511627819,
472
+ "instruction": "push the purple block onto the purple pad",
473
+ "success": false,
474
+ "steps": 440,
475
+ "time_to_success_s": null,
476
+ "failure": "target_not_moved"
477
+ },
478
+ {
479
+ "seed": 1099511627820,
480
+ "instruction": "push the yellow block onto the purple pad",
481
+ "success": false,
482
+ "steps": 440,
483
+ "time_to_success_s": null,
484
+ "failure": "target_not_moved"
485
+ },
486
+ {
487
+ "seed": 1099511627821,
488
+ "instruction": "push the red block onto the yellow pad",
489
+ "success": true,
490
+ "steps": 80,
491
+ "time_to_success_s": 3.55
492
+ },
493
+ {
494
+ "seed": 1099511627822,
495
+ "instruction": "push the purple block onto the red pad",
496
+ "success": false,
497
+ "steps": 440,
498
+ "time_to_success_s": null,
499
+ "failure": "target_not_moved"
500
+ },
501
+ {
502
+ "seed": 1099511627823,
503
+ "instruction": "push the blue block onto the orange pad",
504
+ "success": true,
505
+ "steps": 78,
506
+ "time_to_success_s": 3.45
507
+ },
508
+ {
509
+ "seed": 1099511627824,
510
+ "instruction": "push the blue block onto the yellow pad",
511
+ "success": false,
512
+ "steps": 440,
513
+ "time_to_success_s": null,
514
+ "failure": "target_short_of_pad"
515
+ },
516
+ {
517
+ "seed": 1099511627825,
518
+ "instruction": "push the blue block onto the blue pad",
519
+ "success": false,
520
+ "steps": 440,
521
+ "time_to_success_s": null,
522
+ "failure": "target_not_moved"
523
+ }
524
+ ]
525
+ },
526
+ "button": {
527
+ "n": 50,
528
+ "success": 18,
529
+ "rate": 0.36,
530
+ "wilson95": [
531
+ 0.241,
532
+ 0.499
533
+ ],
534
+ "mean_time_to_success_s": 2.32,
535
+ "failures": {
536
+ "no_press": 28,
537
+ "wrong_button_pressed": 3,
538
+ "partial_press": 1
539
+ },
540
+ "per_instruction": {
541
+ "press the blue button": {
542
+ "success": 3,
543
+ "n": 8
544
+ },
545
+ "press the green button": {
546
+ "success": 3,
547
+ "n": 4
548
+ },
549
+ "press the orange button": {
550
+ "success": 3,
551
+ "n": 5
552
+ },
553
+ "press the purple button": {
554
+ "success": 3,
555
+ "n": 11
556
+ },
557
+ "press the red button": {
558
+ "success": 4,
559
+ "n": 13
560
+ },
561
+ "press the yellow button": {
562
+ "success": 2,
563
+ "n": 9
564
+ }
565
+ },
566
+ "wall_s": 205.3,
567
+ "episodes": [
568
+ {
569
+ "seed": 1099512676352,
570
+ "instruction": "press the blue button",
571
+ "success": false,
572
+ "steps": 260,
573
+ "time_to_success_s": null,
574
+ "failure": "no_press",
575
+ "video": "videos/button_failure_seed1099512676352.mp4"
576
+ },
577
+ {
578
+ "seed": 1099512676353,
579
+ "instruction": "press the orange button",
580
+ "success": false,
581
+ "steps": 260,
582
+ "time_to_success_s": null,
583
+ "failure": "no_press"
584
+ },
585
+ {
586
+ "seed": 1099512676354,
587
+ "instruction": "press the purple button",
588
+ "success": false,
589
+ "steps": 260,
590
+ "time_to_success_s": null,
591
+ "failure": "no_press"
592
+ },
593
+ {
594
+ "seed": 1099512676355,
595
+ "instruction": "press the orange button",
596
+ "success": true,
597
+ "steps": 64,
598
+ "time_to_success_s": 2.75,
599
+ "video": "videos/button_success_seed1099512676355.mp4"
600
+ },
601
+ {
602
+ "seed": 1099512676356,
603
+ "instruction": "press the blue button",
604
+ "success": false,
605
+ "steps": 260,
606
+ "time_to_success_s": null,
607
+ "failure": "wrong_button_pressed"
608
+ },
609
+ {
610
+ "seed": 1099512676357,
611
+ "instruction": "press the purple button",
612
+ "success": false,
613
+ "steps": 260,
614
+ "time_to_success_s": null,
615
+ "failure": "no_press"
616
+ },
617
+ {
618
+ "seed": 1099512676358,
619
+ "instruction": "press the red button",
620
+ "success": true,
621
+ "steps": 77,
622
+ "time_to_success_s": 3.4
623
+ },
624
+ {
625
+ "seed": 1099512676359,
626
+ "instruction": "press the blue button",
627
+ "success": false,
628
+ "steps": 260,
629
+ "time_to_success_s": null,
630
+ "failure": "no_press"
631
+ },
632
+ {
633
+ "seed": 1099512676360,
634
+ "instruction": "press the red button",
635
+ "success": false,
636
+ "steps": 260,
637
+ "time_to_success_s": null,
638
+ "failure": "wrong_button_pressed"
639
+ },
640
+ {
641
+ "seed": 1099512676361,
642
+ "instruction": "press the yellow button",
643
+ "success": false,
644
+ "steps": 260,
645
+ "time_to_success_s": null,
646
+ "failure": "partial_press"
647
+ },
648
+ {
649
+ "seed": 1099512676362,
650
+ "instruction": "press the red button",
651
+ "success": false,
652
+ "steps": 260,
653
+ "time_to_success_s": null,
654
+ "failure": "no_press"
655
+ },
656
+ {
657
+ "seed": 1099512676363,
658
+ "instruction": "press the purple button",
659
+ "success": true,
660
+ "steps": 64,
661
+ "time_to_success_s": 2.75
662
+ },
663
+ {
664
+ "seed": 1099512676364,
665
+ "instruction": "press the red button",
666
+ "success": false,
667
+ "steps": 260,
668
+ "time_to_success_s": null,
669
+ "failure": "no_press"
670
+ },
671
+ {
672
+ "seed": 1099512676365,
673
+ "instruction": "press the purple button",
674
+ "success": true,
675
+ "steps": 68,
676
+ "time_to_success_s": 2.95
677
+ },
678
+ {
679
+ "seed": 1099512676366,
680
+ "instruction": "press the green button",
681
+ "success": true,
682
+ "steps": 43,
683
+ "time_to_success_s": 1.7
684
+ },
685
+ {
686
+ "seed": 1099512676367,
687
+ "instruction": "press the red button",
688
+ "success": false,
689
+ "steps": 260,
690
+ "time_to_success_s": null,
691
+ "failure": "no_press"
692
+ },
693
+ {
694
+ "seed": 1099512676368,
695
+ "instruction": "press the blue button",
696
+ "success": true,
697
+ "steps": 42,
698
+ "time_to_success_s": 1.65
699
+ },
700
+ {
701
+ "seed": 1099512676369,
702
+ "instruction": "press the purple button",
703
+ "success": false,
704
+ "steps": 260,
705
+ "time_to_success_s": null,
706
+ "failure": "no_press"
707
+ },
708
+ {
709
+ "seed": 1099512676370,
710
+ "instruction": "press the yellow button",
711
+ "success": false,
712
+ "steps": 260,
713
+ "time_to_success_s": null,
714
+ "failure": "wrong_button_pressed"
715
+ },
716
+ {
717
+ "seed": 1099512676371,
718
+ "instruction": "press the red button",
719
+ "success": false,
720
+ "steps": 260,
721
+ "time_to_success_s": null,
722
+ "failure": "no_press"
723
+ },
724
+ {
725
+ "seed": 1099512676372,
726
+ "instruction": "press the red button",
727
+ "success": true,
728
+ "steps": 45,
729
+ "time_to_success_s": 1.8
730
+ },
731
+ {
732
+ "seed": 1099512676373,
733
+ "instruction": "press the purple button",
734
+ "success": false,
735
+ "steps": 260,
736
+ "time_to_success_s": null,
737
+ "failure": "no_press"
738
+ },
739
+ {
740
+ "seed": 1099512676374,
741
+ "instruction": "press the yellow button",
742
+ "success": false,
743
+ "steps": 260,
744
+ "time_to_success_s": null,
745
+ "failure": "no_press"
746
+ },
747
+ {
748
+ "seed": 1099512676375,
749
+ "instruction": "press the green button",
750
+ "success": false,
751
+ "steps": 260,
752
+ "time_to_success_s": null,
753
+ "failure": "no_press"
754
+ },
755
+ {
756
+ "seed": 1099512676376,
757
+ "instruction": "press the blue button",
758
+ "success": true,
759
+ "steps": 45,
760
+ "time_to_success_s": 1.8
761
+ },
762
+ {
763
+ "seed": 1099512676377,
764
+ "instruction": "press the yellow button",
765
+ "success": false,
766
+ "steps": 260,
767
+ "time_to_success_s": null,
768
+ "failure": "no_press"
769
+ },
770
+ {
771
+ "seed": 1099512676378,
772
+ "instruction": "press the yellow button",
773
+ "success": true,
774
+ "steps": 68,
775
+ "time_to_success_s": 2.95
776
+ },
777
+ {
778
+ "seed": 1099512676379,
779
+ "instruction": "press the green button",
780
+ "success": true,
781
+ "steps": 42,
782
+ "time_to_success_s": 1.65
783
+ },
784
+ {
785
+ "seed": 1099512676380,
786
+ "instruction": "press the yellow button",
787
+ "success": false,
788
+ "steps": 260,
789
+ "time_to_success_s": null,
790
+ "failure": "no_press"
791
+ },
792
+ {
793
+ "seed": 1099512676381,
794
+ "instruction": "press the blue button",
795
+ "success": true,
796
+ "steps": 101,
797
+ "time_to_success_s": 4.6
798
+ },
799
+ {
800
+ "seed": 1099512676382,
801
+ "instruction": "press the red button",
802
+ "success": false,
803
+ "steps": 260,
804
+ "time_to_success_s": null,
805
+ "failure": "no_press"
806
+ },
807
+ {
808
+ "seed": 1099512676383,
809
+ "instruction": "press the red button",
810
+ "success": false,
811
+ "steps": 260,
812
+ "time_to_success_s": null,
813
+ "failure": "no_press"
814
+ },
815
+ {
816
+ "seed": 1099512676384,
817
+ "instruction": "press the yellow button",
818
+ "success": true,
819
+ "steps": 43,
820
+ "time_to_success_s": 1.7
821
+ },
822
+ {
823
+ "seed": 1099512676385,
824
+ "instruction": "press the purple button",
825
+ "success": false,
826
+ "steps": 260,
827
+ "time_to_success_s": null,
828
+ "failure": "no_press"
829
+ },
830
+ {
831
+ "seed": 1099512676386,
832
+ "instruction": "press the orange button",
833
+ "success": true,
834
+ "steps": 67,
835
+ "time_to_success_s": 2.9
836
+ },
837
+ {
838
+ "seed": 1099512676387,
839
+ "instruction": "press the red button",
840
+ "success": true,
841
+ "steps": 51,
842
+ "time_to_success_s": 2.1
843
+ },
844
+ {
845
+ "seed": 1099512676388,
846
+ "instruction": "press the red button",
847
+ "success": true,
848
+ "steps": 55,
849
+ "time_to_success_s": 2.3
850
+ },
851
+ {
852
+ "seed": 1099512676389,
853
+ "instruction": "press the purple button",
854
+ "success": true,
855
+ "steps": 40,
856
+ "time_to_success_s": 1.55
857
+ },
858
+ {
859
+ "seed": 1099512676390,
860
+ "instruction": "press the purple button",
861
+ "success": false,
862
+ "steps": 260,
863
+ "time_to_success_s": null,
864
+ "failure": "no_press"
865
+ },
866
+ {
867
+ "seed": 1099512676391,
868
+ "instruction": "press the red button",
869
+ "success": false,
870
+ "steps": 260,
871
+ "time_to_success_s": null,
872
+ "failure": "no_press"
873
+ },
874
+ {
875
+ "seed": 1099512676392,
876
+ "instruction": "press the yellow button",
877
+ "success": false,
878
+ "steps": 260,
879
+ "time_to_success_s": null,
880
+ "failure": "no_press"
881
+ },
882
+ {
883
+ "seed": 1099512676393,
884
+ "instruction": "press the orange button",
885
+ "success": false,
886
+ "steps": 260,
887
+ "time_to_success_s": null,
888
+ "failure": "no_press"
889
+ },
890
+ {
891
+ "seed": 1099512676394,
892
+ "instruction": "press the blue button",
893
+ "success": false,
894
+ "steps": 260,
895
+ "time_to_success_s": null,
896
+ "failure": "no_press"
897
+ },
898
+ {
899
+ "seed": 1099512676395,
900
+ "instruction": "press the yellow button",
901
+ "success": false,
902
+ "steps": 260,
903
+ "time_to_success_s": null,
904
+ "failure": "no_press"
905
+ },
906
+ {
907
+ "seed": 1099512676396,
908
+ "instruction": "press the red button",
909
+ "success": false,
910
+ "steps": 260,
911
+ "time_to_success_s": null,
912
+ "failure": "no_press"
913
+ },
914
+ {
915
+ "seed": 1099512676397,
916
+ "instruction": "press the orange button",
917
+ "success": true,
918
+ "steps": 44,
919
+ "time_to_success_s": 1.75
920
+ },
921
+ {
922
+ "seed": 1099512676398,
923
+ "instruction": "press the purple button",
924
+ "success": false,
925
+ "steps": 260,
926
+ "time_to_success_s": null,
927
+ "failure": "no_press"
928
+ },
929
+ {
930
+ "seed": 1099512676399,
931
+ "instruction": "press the green button",
932
+ "success": true,
933
+ "steps": 37,
934
+ "time_to_success_s": 1.4
935
+ },
936
+ {
937
+ "seed": 1099512676400,
938
+ "instruction": "press the blue button",
939
+ "success": false,
940
+ "steps": 260,
941
+ "time_to_success_s": null,
942
+ "failure": "no_press"
943
+ },
944
+ {
945
+ "seed": 1099512676401,
946
+ "instruction": "press the purple button",
947
+ "success": false,
948
+ "steps": 260,
949
+ "time_to_success_s": null,
950
+ "failure": "no_press"
951
+ }
952
+ ]
953
+ },
954
+ "box": {
955
+ "n": 50,
956
+ "success": 1,
957
+ "rate": 0.02,
958
+ "wilson95": [
959
+ 0.004,
960
+ 0.105
961
+ ],
962
+ "mean_time_to_success_s": 8.55,
963
+ "failures": {
964
+ "not_lifted": 30,
965
+ "dropped_or_not_placed": 18,
966
+ "tipped": 1
967
+ },
968
+ "per_instruction": {
969
+ "lift the box and put it on the shelf": {
970
+ "success": 1,
971
+ "n": 50
972
+ }
973
+ },
974
+ "wall_s": 338.0,
975
+ "episodes": [
976
+ {
977
+ "seed": 1099513724928,
978
+ "instruction": "lift the box and put it on the shelf",
979
+ "success": false,
980
+ "steps": 440,
981
+ "time_to_success_s": null,
982
+ "failure": "not_lifted",
983
+ "video": "videos/box_failure_seed1099513724928.mp4"
984
+ },
985
+ {
986
+ "seed": 1099513724929,
987
+ "instruction": "lift the box and put it on the shelf",
988
+ "success": false,
989
+ "steps": 440,
990
+ "time_to_success_s": null,
991
+ "failure": "not_lifted"
992
+ },
993
+ {
994
+ "seed": 1099513724930,
995
+ "instruction": "lift the box and put it on the shelf",
996
+ "success": false,
997
+ "steps": 440,
998
+ "time_to_success_s": null,
999
+ "failure": "dropped_or_not_placed"
1000
+ },
1001
+ {
1002
+ "seed": 1099513724931,
1003
+ "instruction": "lift the box and put it on the shelf",
1004
+ "success": false,
1005
+ "steps": 440,
1006
+ "time_to_success_s": null,
1007
+ "failure": "not_lifted"
1008
+ },
1009
+ {
1010
+ "seed": 1099513724932,
1011
+ "instruction": "lift the box and put it on the shelf",
1012
+ "success": false,
1013
+ "steps": 440,
1014
+ "time_to_success_s": null,
1015
+ "failure": "not_lifted"
1016
+ },
1017
+ {
1018
+ "seed": 1099513724933,
1019
+ "instruction": "lift the box and put it on the shelf",
1020
+ "success": false,
1021
+ "steps": 440,
1022
+ "time_to_success_s": null,
1023
+ "failure": "dropped_or_not_placed"
1024
+ },
1025
+ {
1026
+ "seed": 1099513724934,
1027
+ "instruction": "lift the box and put it on the shelf",
1028
+ "success": true,
1029
+ "steps": 180,
1030
+ "time_to_success_s": 8.55,
1031
+ "video": "videos/box_success_seed1099513724934.mp4"
1032
+ },
1033
+ {
1034
+ "seed": 1099513724935,
1035
+ "instruction": "lift the box and put it on the shelf",
1036
+ "success": false,
1037
+ "steps": 440,
1038
+ "time_to_success_s": null,
1039
+ "failure": "dropped_or_not_placed"
1040
+ },
1041
+ {
1042
+ "seed": 1099513724936,
1043
+ "instruction": "lift the box and put it on the shelf",
1044
+ "success": false,
1045
+ "steps": 440,
1046
+ "time_to_success_s": null,
1047
+ "failure": "dropped_or_not_placed"
1048
+ },
1049
+ {
1050
+ "seed": 1099513724937,
1051
+ "instruction": "lift the box and put it on the shelf",
1052
+ "success": false,
1053
+ "steps": 440,
1054
+ "time_to_success_s": null,
1055
+ "failure": "not_lifted"
1056
+ },
1057
+ {
1058
+ "seed": 1099513724938,
1059
+ "instruction": "lift the box and put it on the shelf",
1060
+ "success": false,
1061
+ "steps": 440,
1062
+ "time_to_success_s": null,
1063
+ "failure": "dropped_or_not_placed"
1064
+ },
1065
+ {
1066
+ "seed": 1099513724939,
1067
+ "instruction": "lift the box and put it on the shelf",
1068
+ "success": false,
1069
+ "steps": 440,
1070
+ "time_to_success_s": null,
1071
+ "failure": "not_lifted"
1072
+ },
1073
+ {
1074
+ "seed": 1099513724940,
1075
+ "instruction": "lift the box and put it on the shelf",
1076
+ "success": false,
1077
+ "steps": 440,
1078
+ "time_to_success_s": null,
1079
+ "failure": "not_lifted"
1080
+ },
1081
+ {
1082
+ "seed": 1099513724941,
1083
+ "instruction": "lift the box and put it on the shelf",
1084
+ "success": false,
1085
+ "steps": 440,
1086
+ "time_to_success_s": null,
1087
+ "failure": "dropped_or_not_placed"
1088
+ },
1089
+ {
1090
+ "seed": 1099513724942,
1091
+ "instruction": "lift the box and put it on the shelf",
1092
+ "success": false,
1093
+ "steps": 440,
1094
+ "time_to_success_s": null,
1095
+ "failure": "dropped_or_not_placed"
1096
+ },
1097
+ {
1098
+ "seed": 1099513724943,
1099
+ "instruction": "lift the box and put it on the shelf",
1100
+ "success": false,
1101
+ "steps": 440,
1102
+ "time_to_success_s": null,
1103
+ "failure": "not_lifted"
1104
+ },
1105
+ {
1106
+ "seed": 1099513724944,
1107
+ "instruction": "lift the box and put it on the shelf",
1108
+ "success": false,
1109
+ "steps": 440,
1110
+ "time_to_success_s": null,
1111
+ "failure": "dropped_or_not_placed"
1112
+ },
1113
+ {
1114
+ "seed": 1099513724945,
1115
+ "instruction": "lift the box and put it on the shelf",
1116
+ "success": false,
1117
+ "steps": 440,
1118
+ "time_to_success_s": null,
1119
+ "failure": "not_lifted"
1120
+ },
1121
+ {
1122
+ "seed": 1099513724946,
1123
+ "instruction": "lift the box and put it on the shelf",
1124
+ "success": false,
1125
+ "steps": 440,
1126
+ "time_to_success_s": null,
1127
+ "failure": "dropped_or_not_placed"
1128
+ },
1129
+ {
1130
+ "seed": 1099513724947,
1131
+ "instruction": "lift the box and put it on the shelf",
1132
+ "success": false,
1133
+ "steps": 440,
1134
+ "time_to_success_s": null,
1135
+ "failure": "dropped_or_not_placed"
1136
+ },
1137
+ {
1138
+ "seed": 1099513724948,
1139
+ "instruction": "lift the box and put it on the shelf",
1140
+ "success": false,
1141
+ "steps": 440,
1142
+ "time_to_success_s": null,
1143
+ "failure": "not_lifted"
1144
+ },
1145
+ {
1146
+ "seed": 1099513724949,
1147
+ "instruction": "lift the box and put it on the shelf",
1148
+ "success": false,
1149
+ "steps": 440,
1150
+ "time_to_success_s": null,
1151
+ "failure": "tipped"
1152
+ },
1153
+ {
1154
+ "seed": 1099513724950,
1155
+ "instruction": "lift the box and put it on the shelf",
1156
+ "success": false,
1157
+ "steps": 440,
1158
+ "time_to_success_s": null,
1159
+ "failure": "dropped_or_not_placed"
1160
+ },
1161
+ {
1162
+ "seed": 1099513724951,
1163
+ "instruction": "lift the box and put it on the shelf",
1164
+ "success": false,
1165
+ "steps": 440,
1166
+ "time_to_success_s": null,
1167
+ "failure": "not_lifted"
1168
+ },
1169
+ {
1170
+ "seed": 1099513724952,
1171
+ "instruction": "lift the box and put it on the shelf",
1172
+ "success": false,
1173
+ "steps": 440,
1174
+ "time_to_success_s": null,
1175
+ "failure": "not_lifted"
1176
+ },
1177
+ {
1178
+ "seed": 1099513724953,
1179
+ "instruction": "lift the box and put it on the shelf",
1180
+ "success": false,
1181
+ "steps": 440,
1182
+ "time_to_success_s": null,
1183
+ "failure": "not_lifted"
1184
+ },
1185
+ {
1186
+ "seed": 1099513724954,
1187
+ "instruction": "lift the box and put it on the shelf",
1188
+ "success": false,
1189
+ "steps": 440,
1190
+ "time_to_success_s": null,
1191
+ "failure": "dropped_or_not_placed"
1192
+ },
1193
+ {
1194
+ "seed": 1099513724955,
1195
+ "instruction": "lift the box and put it on the shelf",
1196
+ "success": false,
1197
+ "steps": 440,
1198
+ "time_to_success_s": null,
1199
+ "failure": "not_lifted"
1200
+ },
1201
+ {
1202
+ "seed": 1099513724956,
1203
+ "instruction": "lift the box and put it on the shelf",
1204
+ "success": false,
1205
+ "steps": 440,
1206
+ "time_to_success_s": null,
1207
+ "failure": "not_lifted"
1208
+ },
1209
+ {
1210
+ "seed": 1099513724957,
1211
+ "instruction": "lift the box and put it on the shelf",
1212
+ "success": false,
1213
+ "steps": 440,
1214
+ "time_to_success_s": null,
1215
+ "failure": "not_lifted"
1216
+ },
1217
+ {
1218
+ "seed": 1099513724958,
1219
+ "instruction": "lift the box and put it on the shelf",
1220
+ "success": false,
1221
+ "steps": 440,
1222
+ "time_to_success_s": null,
1223
+ "failure": "dropped_or_not_placed"
1224
+ },
1225
+ {
1226
+ "seed": 1099513724959,
1227
+ "instruction": "lift the box and put it on the shelf",
1228
+ "success": false,
1229
+ "steps": 440,
1230
+ "time_to_success_s": null,
1231
+ "failure": "not_lifted"
1232
+ },
1233
+ {
1234
+ "seed": 1099513724960,
1235
+ "instruction": "lift the box and put it on the shelf",
1236
+ "success": false,
1237
+ "steps": 440,
1238
+ "time_to_success_s": null,
1239
+ "failure": "not_lifted"
1240
+ },
1241
+ {
1242
+ "seed": 1099513724961,
1243
+ "instruction": "lift the box and put it on the shelf",
1244
+ "success": false,
1245
+ "steps": 440,
1246
+ "time_to_success_s": null,
1247
+ "failure": "not_lifted"
1248
+ },
1249
+ {
1250
+ "seed": 1099513724962,
1251
+ "instruction": "lift the box and put it on the shelf",
1252
+ "success": false,
1253
+ "steps": 440,
1254
+ "time_to_success_s": null,
1255
+ "failure": "not_lifted"
1256
+ },
1257
+ {
1258
+ "seed": 1099513724963,
1259
+ "instruction": "lift the box and put it on the shelf",
1260
+ "success": false,
1261
+ "steps": 440,
1262
+ "time_to_success_s": null,
1263
+ "failure": "not_lifted"
1264
+ },
1265
+ {
1266
+ "seed": 1099513724964,
1267
+ "instruction": "lift the box and put it on the shelf",
1268
+ "success": false,
1269
+ "steps": 440,
1270
+ "time_to_success_s": null,
1271
+ "failure": "not_lifted"
1272
+ },
1273
+ {
1274
+ "seed": 1099513724965,
1275
+ "instruction": "lift the box and put it on the shelf",
1276
+ "success": false,
1277
+ "steps": 440,
1278
+ "time_to_success_s": null,
1279
+ "failure": "not_lifted"
1280
+ },
1281
+ {
1282
+ "seed": 1099513724966,
1283
+ "instruction": "lift the box and put it on the shelf",
1284
+ "success": false,
1285
+ "steps": 440,
1286
+ "time_to_success_s": null,
1287
+ "failure": "not_lifted"
1288
+ },
1289
+ {
1290
+ "seed": 1099513724967,
1291
+ "instruction": "lift the box and put it on the shelf",
1292
+ "success": false,
1293
+ "steps": 440,
1294
+ "time_to_success_s": null,
1295
+ "failure": "dropped_or_not_placed"
1296
+ },
1297
+ {
1298
+ "seed": 1099513724968,
1299
+ "instruction": "lift the box and put it on the shelf",
1300
+ "success": false,
1301
+ "steps": 440,
1302
+ "time_to_success_s": null,
1303
+ "failure": "not_lifted"
1304
+ },
1305
+ {
1306
+ "seed": 1099513724969,
1307
+ "instruction": "lift the box and put it on the shelf",
1308
+ "success": false,
1309
+ "steps": 440,
1310
+ "time_to_success_s": null,
1311
+ "failure": "not_lifted"
1312
+ },
1313
+ {
1314
+ "seed": 1099513724970,
1315
+ "instruction": "lift the box and put it on the shelf",
1316
+ "success": false,
1317
+ "steps": 440,
1318
+ "time_to_success_s": null,
1319
+ "failure": "dropped_or_not_placed"
1320
+ },
1321
+ {
1322
+ "seed": 1099513724971,
1323
+ "instruction": "lift the box and put it on the shelf",
1324
+ "success": false,
1325
+ "steps": 440,
1326
+ "time_to_success_s": null,
1327
+ "failure": "not_lifted"
1328
+ },
1329
+ {
1330
+ "seed": 1099513724972,
1331
+ "instruction": "lift the box and put it on the shelf",
1332
+ "success": false,
1333
+ "steps": 440,
1334
+ "time_to_success_s": null,
1335
+ "failure": "dropped_or_not_placed"
1336
+ },
1337
+ {
1338
+ "seed": 1099513724973,
1339
+ "instruction": "lift the box and put it on the shelf",
1340
+ "success": false,
1341
+ "steps": 440,
1342
+ "time_to_success_s": null,
1343
+ "failure": "not_lifted"
1344
+ },
1345
+ {
1346
+ "seed": 1099513724974,
1347
+ "instruction": "lift the box and put it on the shelf",
1348
+ "success": false,
1349
+ "steps": 440,
1350
+ "time_to_success_s": null,
1351
+ "failure": "not_lifted"
1352
+ },
1353
+ {
1354
+ "seed": 1099513724975,
1355
+ "instruction": "lift the box and put it on the shelf",
1356
+ "success": false,
1357
+ "steps": 440,
1358
+ "time_to_success_s": null,
1359
+ "failure": "not_lifted"
1360
+ },
1361
+ {
1362
+ "seed": 1099513724976,
1363
+ "instruction": "lift the box and put it on the shelf",
1364
+ "success": false,
1365
+ "steps": 440,
1366
+ "time_to_success_s": null,
1367
+ "failure": "dropped_or_not_placed"
1368
+ },
1369
+ {
1370
+ "seed": 1099513724977,
1371
+ "instruction": "lift the box and put it on the shelf",
1372
+ "success": false,
1373
+ "steps": 440,
1374
+ "time_to_success_s": null,
1375
+ "failure": "dropped_or_not_placed"
1376
+ }
1377
+ ]
1378
+ }
1379
+ },
1380
+ "inference_s": {
1381
+ "mean": 0.0944,
1382
+ "p50": 0.0941,
1383
+ "p90": 0.0954,
1384
+ "calls": 5762
1385
+ },
1386
+ "wall_s": 1795.8
1387
+ }
results/closedloop_checkpoint-6000/results.md ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Closed-loop evaluation: groot (/root/work/runs/asimov_n17/checkpoint-6000)
2
+
3
+ 50 held-out episodes per task (seeds 2**40 + task_id * 2**20 + k; checked against 926 dataset seeds: no overlap (max training seed 1200300)), 8 of 16 actions executed per inference at 20 Hz, success held 10 steps, GL egl.
4
+
5
+ | Task | Success | 95% CI | Mean time to success | Failures |
6
+ | --- | --- | --- | --- | --- |
7
+ | push | 23/50 (46%) | 0.33-0.60 | 5.83 s | target_not_moved 17, target_short_of_pad 9, target_on_wrong_pad 1 |
8
+ | button | 18/50 (36%) | 0.24-0.50 | 2.32 s | no_press 28, wrong_button_pressed 3, partial_press 1 |
9
+ | box | 1/50 (2%) | 0.00-0.10 | 8.55 s | not_lifted 30, dropped_or_not_placed 18, tipped 1 |
10
+
11
+ Policy calls: {'mean': 0.0944, 'p50': 0.0941, 'p90': 0.0954, 'calls': 5762}
12
+
13
+ ## push: per instruction
14
+
15
+ | Instruction | Success |
16
+ | --- | --- |
17
+ | push the blue block onto the blue pad | 1/2 |
18
+ | push the blue block onto the green pad | 1/2 |
19
+ | push the blue block onto the orange pad | 2/2 |
20
+ | push the blue block onto the purple pad | 0/1 |
21
+ | push the blue block onto the red pad | 1/1 |
22
+ | push the blue block onto the yellow pad | 0/1 |
23
+ | push the green block onto the blue pad | 1/1 |
24
+ | push the green block onto the green pad | 1/2 |
25
+ | push the green block onto the purple pad | 0/1 |
26
+ | push the green block onto the red pad | 0/1 |
27
+ | push the orange block onto the blue pad | 1/2 |
28
+ | push the orange block onto the purple pad | 0/1 |
29
+ | push the orange block onto the red pad | 1/2 |
30
+ | push the orange block onto the yellow pad | 2/2 |
31
+ | push the purple block onto the blue pad | 1/1 |
32
+ | push the purple block onto the green pad | 1/1 |
33
+ | push the purple block onto the orange pad | 4/5 |
34
+ | push the purple block onto the purple pad | 0/1 |
35
+ | push the purple block onto the red pad | 0/2 |
36
+ | push the purple block onto the yellow pad | 0/1 |
37
+ | push the red block onto the orange pad | 1/1 |
38
+ | push the red block onto the purple pad | 0/2 |
39
+ | push the red block onto the red pad | 0/2 |
40
+ | push the red block onto the yellow pad | 2/4 |
41
+ | push the yellow block onto the blue pad | 2/3 |
42
+ | push the yellow block onto the green pad | 1/3 |
43
+ | push the yellow block onto the purple pad | 0/1 |
44
+ | push the yellow block onto the red pad | 0/1 |
45
+ | push the yellow block onto the yellow pad | 0/1 |
46
+
47
+ Videos: videos/push_failure_seed1099511627776.mp4, videos/push_success_seed1099511627777.mp4
48
+
49
+ ## button: per instruction
50
+
51
+ | Instruction | Success |
52
+ | --- | --- |
53
+ | press the blue button | 3/8 |
54
+ | press the green button | 3/4 |
55
+ | press the orange button | 3/5 |
56
+ | press the purple button | 3/11 |
57
+ | press the red button | 4/13 |
58
+ | press the yellow button | 2/9 |
59
+
60
+ Videos: videos/button_failure_seed1099512676352.mp4, videos/button_success_seed1099512676355.mp4
61
+
62
+ ## box: per instruction
63
+
64
+ | Instruction | Success |
65
+ | --- | --- |
66
+ | lift the box and put it on the shelf | 1/50 |
67
+
68
+ Videos: videos/box_failure_seed1099513724928.mp4, videos/box_success_seed1099513724934.mp4
results/closedloop_checkpoint-6000/videos/box_failure_seed1099513724928.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fddfb46d31ef1b988f272f8f066082b84e0abea8f61b6d637527224f132c7f3f
3
+ size 291670
results/closedloop_checkpoint-6000/videos/box_success_seed1099513724934.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:61230af3c220b3a2e08bfea77229e17fbe5062d96c762064f2e793843ef5c034
3
+ size 204024
results/closedloop_checkpoint-6000/videos/button_failure_seed1099512676352.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:80d919447c46ffcc09078bdc3cedce7b6b980c1764709a28bf4ccc68632b63e9
3
+ size 256737
results/closedloop_checkpoint-6000/videos/button_success_seed1099512676355.mp4 ADDED
Binary file (87 kB). View file
 
results/closedloop_checkpoint-6000/videos/push_failure_seed1099511627776.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:37b5c269e916e9dc93c5ec1b24d1ea5ea83e5eb7342f12134c8fc6a581412022
3
+ size 395919
results/closedloop_checkpoint-6000/videos/push_success_seed1099511627777.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:42faf38340fc0ed51e174a41d3f542844d77c9e6a7dada781c157eabe56f89e6
3
+ size 113336
results/openloop_checkpoint-10000/openloop.log ADDED
@@ -0,0 +1,76 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ INFO:root:Extracted global_step 10000 from checkpoint path
2
+ /root/work/Isaac-GR00T/.venv/lib/python3.12/site-packages/albumentations/__init__.py:13: UserWarning: A new version of Albumentations is available: 2.0.8 (you have 1.4.18). Upgrade using: pip install -U albumentations. To disable automatic update checks, set the environment variable NO_ALBUMENTATIONS_UPDATE to 1.
3
+ check_for_updates()
4
+ Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLForConditionalGeneration is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)`
5
+ Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)`
6
+ Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLVisionModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)`
7
+ Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3VLTextModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)`
8
+ Total number of DiT parameters: 1091722240
9
+ INFO:gr00t.model.gr00t_n1d7.gr00t_n1d7:Using AlternateVLDiT for diffusion model
10
+ Total number of SelfAttentionTransformer parameters: 201433088
11
+
12
+ INFO:root:Using 4 denoising steps
13
+ INFO:root:Current modality config:
14
+ {'video': ModalityConfig(delta_indices=[0], modality_keys=['head'], sin_cos_embedding_keys=None, mean_std_embedding_keys=None, action_configs=None), 'state': ModalityConfig(delta_indices=[0], modality_keys=['left_arm', 'right_arm', 'waist', 'neck'], sin_cos_embedding_keys=None, mean_std_embedding_keys=None, action_configs=None), 'action': ModalityConfig(delta_indices=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], modality_keys=['left_arm', 'right_arm', 'waist', 'neck'], sin_cos_embedding_keys=None, mean_std_embedding_keys=None, action_configs=[ActionConfig(rep=<ActionRepresentation.RELATIVE: 'relative'>, type=<ActionType.NON_EEF: 'non_eef'>, format=<ActionFormat.DEFAULT: 'default'>, state_key=None), ActionConfig(rep=<ActionRepresentation.RELATIVE: 'relative'>, type=<ActionType.NON_EEF: 'non_eef'>, format=<ActionFormat.DEFAULT: 'default'>, state_key=None), ActionConfig(rep=<ActionRepresentation.RELATIVE: 'relative'>, type=<ActionType.NON_EEF: 'non_eef'>, format=<ActionFormat.DEFAULT: 'default'>, state_key=None), ActionConfig(rep=<ActionRepresentation.RELATIVE: 'relative'>, type=<ActionType.NON_EEF: 'non_eef'>, format=<ActionFormat.DEFAULT: 'default'>, state_key=None)]), 'language': ModalityConfig(delta_indices=[0], modality_keys=['annotation.human.task_description'], sin_cos_embedding_keys=None, mean_std_embedding_keys=None, action_configs=None)}
15
+ INFO:root:Dataset length: 900
16
+ INFO:root:Running evaluation on trajectories: [0, 300, 600]
17
+ INFO:root:Running trajectory: 0
18
+ INFO:root:Using 143 steps (requested: 300, trajectory length: 143)
19
+ INFO:root:inferencing at step: 0
20
+ INFO:root:inferencing at step: 16
21
+ INFO:root:inferencing at step: 32
22
+ INFO:root:inferencing at step: 48
23
+ INFO:root:inferencing at step: 64
24
+ INFO:root:inferencing at step: 80
25
+ INFO:root:inferencing at step: 96
26
+ INFO:root:inferencing at step: 112
27
+ INFO:root:inferencing at step: 128
28
+ INFO:root:Unnormalized Action MSE across single traj: 0.00448231864720583
29
+ INFO:root:Unnormalized Action MAE across single traj: 0.03181375935673714
30
+ INFO:root:state_joints vs time (143, 13)
31
+ INFO:root:gt_action_joints vs time (143, 13)
32
+ INFO:root:pred_action_joints vs time (143, 13)
33
+ INFO:root:MSE for trajectory 0: 0.00448231864720583, MAE: 0.03181375935673714
34
+ INFO:root:Running trajectory: 300
35
+ INFO:root:Using 108 steps (requested: 300, trajectory length: 108)
36
+ INFO:root:inferencing at step: 0
37
+ INFO:root:inferencing at step: 16
38
+ INFO:root:inferencing at step: 32
39
+ INFO:root:inferencing at step: 48
40
+ INFO:root:inferencing at step: 64
41
+ INFO:root:inferencing at step: 80
42
+ INFO:root:inferencing at step: 96
43
+ INFO:root:Unnormalized Action MSE across single traj: 0.002719493815675378
44
+ INFO:root:Unnormalized Action MAE across single traj: 0.0312812440097332
45
+ INFO:root:state_joints vs time (108, 13)
46
+ INFO:root:gt_action_joints vs time (108, 13)
47
+ INFO:root:pred_action_joints vs time (108, 13)
48
+ INFO:root:MSE for trajectory 300: 0.002719493815675378, MAE: 0.0312812440097332
49
+ INFO:root:Running trajectory: 600
50
+ INFO:root:Using 270 steps (requested: 300, trajectory length: 270)
51
+ INFO:root:inferencing at step: 0
52
+ INFO:root:inferencing at step: 16
53
+ INFO:root:inferencing at step: 32
54
+ INFO:root:inferencing at step: 48
55
+ INFO:root:inferencing at step: 64
56
+ INFO:root:inferencing at step: 80
57
+ INFO:root:inferencing at step: 96
58
+ INFO:root:inferencing at step: 112
59
+ INFO:root:inferencing at step: 128
60
+ INFO:root:inferencing at step: 144
61
+ INFO:root:inferencing at step: 160
62
+ INFO:root:inferencing at step: 176
63
+ INFO:root:inferencing at step: 192
64
+ INFO:root:inferencing at step: 208
65
+ INFO:root:inferencing at step: 224
66
+ INFO:root:inferencing at step: 240
67
+ INFO:root:inferencing at step: 256
68
+ INFO:root:Unnormalized Action MSE across single traj: 0.001021101139485836
69
+ INFO:root:Unnormalized Action MAE across single traj: 0.0182601697742939
70
+ INFO:root:state_joints vs time (270, 13)
71
+ INFO:root:gt_action_joints vs time (270, 13)
72
+ INFO:root:pred_action_joints vs time (270, 13)
73
+ INFO:root:MSE for trajectory 600: 0.001021101139485836, MAE: 0.0182601697742939
74
+ INFO:root:Average MSE across all trajs: 0.0027409710455685854
75
+ INFO:root:Average MAE across all trajs: 0.027118390426039696
76
+ INFO:root:Done
results/openloop_checkpoint-10000/summary.txt ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ INFO:root:Unnormalized Action MSE across single traj: 0.00448231864720583
2
+ INFO:root:Unnormalized Action MAE across single traj: 0.03181375935673714
3
+ INFO:root:MSE for trajectory 0: 0.00448231864720583, MAE: 0.03181375935673714
4
+ INFO:root:Unnormalized Action MSE across single traj: 0.002719493815675378
5
+ INFO:root:Unnormalized Action MAE across single traj: 0.0312812440097332
6
+ INFO:root:MSE for trajectory 300: 0.002719493815675378, MAE: 0.0312812440097332
7
+ INFO:root:Unnormalized Action MSE across single traj: 0.001021101139485836
8
+ INFO:root:Unnormalized Action MAE across single traj: 0.0182601697742939
9
+ INFO:root:MSE for trajectory 600: 0.001021101139485836, MAE: 0.0182601697742939
10
+ INFO:root:Average MSE across all trajs: 0.0027409710455685854
11
+ INFO:root:Average MAE across all trajs: 0.027118390426039696
results/openloop_checkpoint-10000/traj_0.jpeg ADDED

Git LFS Details

  • SHA256: 964e493d47d0ac8e9512c8ed04eaaa35b76fc1a82949a19c46b3171f4966eaaa
  • Pointer size: 131 Bytes
  • Size of remote file: 401 kB
results/openloop_checkpoint-10000/traj_300.jpeg ADDED

Git LFS Details

  • SHA256: 99a1b9fb66c3c46cc5e1e5de644666a1882030e5cd25080c23a7fbe633f990eb
  • Pointer size: 131 Bytes
  • Size of remote file: 379 kB
results/openloop_checkpoint-10000/traj_600.jpeg ADDED

Git LFS Details

  • SHA256: 6cca24a529dd79216d38b985f47cdbea1a3d44d80b70b68ff187a1ffaf34241c
  • Pointer size: 131 Bytes
  • Size of remote file: 371 kB
results/oracle_smoke/results.json ADDED
@@ -0,0 +1,97 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "policy": "oracle",
3
+ "model_path": null,
4
+ "exec_steps": 8,
5
+ "episodes_per_task": 1,
6
+ "hold_steps": 10,
7
+ "gl_backend": "egl",
8
+ "seeds": "checked against 926 dataset seeds: no overlap (max training seed 1200300)",
9
+ "seed_scheme": "2**40 + task_id * 2**20 + k",
10
+ "tasks": {
11
+ "push": {
12
+ "n": 1,
13
+ "success": 1,
14
+ "rate": 1.0,
15
+ "wilson95": [
16
+ 0.207,
17
+ 1.0
18
+ ],
19
+ "mean_time_to_success_s": 3.4,
20
+ "failures": {},
21
+ "per_instruction": {
22
+ "push the yellow block onto the green pad": {
23
+ "success": 1,
24
+ "n": 1
25
+ }
26
+ },
27
+ "wall_s": 21.2,
28
+ "episodes": [
29
+ {
30
+ "seed": 1099511627776,
31
+ "instruction": "push the yellow block onto the green pad",
32
+ "success": true,
33
+ "steps": 77,
34
+ "time_to_success_s": 3.4,
35
+ "video": "videos/push_success_seed1099511627776.mp4"
36
+ }
37
+ ]
38
+ },
39
+ "button": {
40
+ "n": 1,
41
+ "success": 1,
42
+ "rate": 1.0,
43
+ "wilson95": [
44
+ 0.207,
45
+ 1.0
46
+ ],
47
+ "mean_time_to_success_s": 2.55,
48
+ "failures": {},
49
+ "per_instruction": {
50
+ "press the blue button": {
51
+ "success": 1,
52
+ "n": 1
53
+ }
54
+ },
55
+ "wall_s": 1.4,
56
+ "episodes": [
57
+ {
58
+ "seed": 1099512676352,
59
+ "instruction": "press the blue button",
60
+ "success": true,
61
+ "steps": 60,
62
+ "time_to_success_s": 2.55,
63
+ "video": "videos/button_success_seed1099512676352.mp4"
64
+ }
65
+ ]
66
+ },
67
+ "box": {
68
+ "n": 1,
69
+ "success": 1,
70
+ "rate": 1.0,
71
+ "wilson95": [
72
+ 0.207,
73
+ 1.0
74
+ ],
75
+ "mean_time_to_success_s": 9.5,
76
+ "failures": {},
77
+ "per_instruction": {
78
+ "lift the box and put it on the shelf": {
79
+ "success": 1,
80
+ "n": 1
81
+ }
82
+ },
83
+ "wall_s": 1.6,
84
+ "episodes": [
85
+ {
86
+ "seed": 1099513724928,
87
+ "instruction": "lift the box and put it on the shelf",
88
+ "success": true,
89
+ "steps": 199,
90
+ "time_to_success_s": 9.5,
91
+ "video": "videos/box_success_seed1099513724928.mp4"
92
+ }
93
+ ]
94
+ }
95
+ },
96
+ "wall_s": 27.2
97
+ }
results/oracle_smoke/results.md ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Closed-loop evaluation: oracle
2
+
3
+ 1 held-out episodes per task (seeds 2**40 + task_id * 2**20 + k; checked against 926 dataset seeds: no overlap (max training seed 1200300)), 8 of 16 actions executed per inference at 20 Hz, success held 10 steps, GL egl.
4
+
5
+ | Task | Success | 95% CI | Mean time to success | Failures |
6
+ | --- | --- | --- | --- | --- |
7
+ | push | 1/1 (100%) | 0.21-1.00 | 3.4 s | - |
8
+ | button | 1/1 (100%) | 0.21-1.00 | 2.55 s | - |
9
+ | box | 1/1 (100%) | 0.21-1.00 | 9.5 s | - |
10
+
11
+ ## push: per instruction
12
+
13
+ | Instruction | Success |
14
+ | --- | --- |
15
+ | push the yellow block onto the green pad | 1/1 |
16
+
17
+ Videos: videos/push_success_seed1099511627776.mp4
18
+
19
+ ## button: per instruction
20
+
21
+ | Instruction | Success |
22
+ | --- | --- |
23
+ | press the blue button | 1/1 |
24
+
25
+ Videos: videos/button_success_seed1099512676352.mp4
26
+
27
+ ## box: per instruction
28
+
29
+ | Instruction | Success |
30
+ | --- | --- |
31
+ | lift the box and put it on the shelf | 1/1 |
32
+
33
+ Videos: videos/box_success_seed1099513724928.mp4
results/oracle_smoke/videos/box_success_seed1099513724928.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:682700f63033dd512fb7a05ff6376448f3b66b13bee71a4e9391d03d2729e32d
3
+ size 148209
results/oracle_smoke/videos/button_success_seed1099512676352.mp4 ADDED
Binary file (68.7 kB). View file
 
results/oracle_smoke/videos/push_success_seed1099511627776.mp4 ADDED
Binary file (84.2 kB). View file