plan-bind-d3 / results.md
thanhkt's picture
Upload results.md with huggingface_hub
6a16a92 verified
|
Raw History Blame Contribute Delete
6.72 kB

Plan-Bind D3 results: Push-T

Source: /home/25thanh.tk/plan-bind/experiments/261028-plan-bind-d3-pusht/outputs/full/eval. Training seeds 1000, 1001; values are the mean over seeds, per-seed values in brackets. Score: mean best coverage in percent.

Main comparison (latency 0, last checkpoint)

Policy Score Coverage >= 0.9 (%) Env success (%) Paper score Episodes
diffusion 83.5 (86, 81) 73.0 (76, 70) 38.2 (40, 37) 72 200
bsp10 66.4 (68, 64) 56.0 (58, 54) 34.5 (36, 33) 75 200
onestep 72.6 (71, 74) 60.8 (58, 63) 34.8 (30, 40) - 200
planbind 53.2 (55, 51) 39.0 (41, 37) 20.0 (21, 19) - 200

Checkpoint curve (latency 0, score)

Policy Seed 25000 50000 75000 100000
diffusion 1000 79.4 85.1 86.4 86.3
diffusion 1001 78.7 77.7 81.7 80.8
bsp10 1000 64.3 69.1 71.3 70.3
bsp10 1001 61.0 67.1 64.9 64.3
onestep 1000 69.9 69.9 74.4 72.4
onestep 1001 76.5 70.6 74.2 75.0
planbind 1000 54.0 61.8 62.0 57.4
planbind 1001 53.4 51.4 52.7 47.8

Call rate and commit (latency 0, score)

A call every 2 frames is the default (0.2 s); every 8 is diffusion's rate (8 actions at 10 fps).

Policy Commit Every 2 Every 8
onestep latency 72.6 (71, 74) 71.7 (73, 70)
onestep none 38.4 (38, 39) 78.7 (76, 81)
planbind latency 53.2 (55, 51) 48.6 (50, 47)
planbind none 61.4 (63, 60) 53.3 (56, 51)

Latency (commit latency, score)

bsp10 runs through the spline runtime here; its classic action queue is in the main comparison.

Policy 0 ms 100 ms 200 ms
bsp10 63.4 (63, 64) 62.2 (64, 60) 62.5 (64, 61)
onestep 72.6 (71, 74) 71.3 (73, 70) 72.0 (71, 73)
planbind 53.2 (55, 51) 49.8 (51, 49) 49.7 (51, 48)

All cells

Policy Latency Commit Every Seed Checkpoint Episodes Score >= 0.9 Env success
bsp10 0 - 2 1000 25000 100 64.3 54.0 27.0
bsp10 0 - 2 1001 25000 100 61.0 49.0 22.0
bsp10 0 - 2 1000 50000 100 69.1 60.0 31.0
bsp10 0 - 2 1001 50000 100 67.1 59.0 37.0
bsp10 0 - 2 1000 75000 100 71.3 62.0 34.0
bsp10 0 - 2 1001 75000 100 64.9 55.0 30.0
bsp10 0 - 2 1000 100000 100 70.3 61.0 40.0
bsp10 0 - 2 1001 100000 100 64.3 57.0 37.0
bsp10 0 - 2 1000 last 200 68.4 58.5 36.0
bsp10 0 - 2 1001 last 200 64.3 53.5 33.0
bsp10 0 latency 2 1000 last 200 63.3 52.5 29.5
bsp10 0 latency 2 1001 last 200 63.6 53.0 30.0
bsp10 1 latency 2 1000 last 200 64.4 53.5 26.5
bsp10 1 latency 2 1001 last 200 60.0 48.5 19.5
bsp10 2 latency 2 1000 last 200 64.4 53.5 26.5
bsp10 2 latency 2 1001 last 200 60.6 48.5 22.5
diffusion 0 - 2 1000 25000 100 79.4 68.0 31.0
diffusion 0 - 2 1001 25000 100 78.7 65.0 26.0
diffusion 0 - 2 1000 50000 100 85.1 75.0 37.0
diffusion 0 - 2 1001 50000 100 77.7 68.0 38.0
diffusion 0 - 2 1000 75000 100 86.4 76.0 40.0
diffusion 0 - 2 1001 75000 100 81.7 70.0 31.0
diffusion 0 - 2 1000 100000 100 86.3 76.0 43.0
diffusion 0 - 2 1001 100000 100 80.8 72.0 37.0
diffusion 0 - 2 1000 last 200 86.0 76.5 39.5
diffusion 0 - 2 1001 last 200 81.0 69.5 37.0
onestep 0 latency 2 1000 25000 100 69.9 60.0 21.0
onestep 0 latency 2 1001 25000 100 76.5 69.0 27.0
onestep 0 latency 2 1000 50000 100 69.9 57.0 26.0
onestep 0 latency 2 1001 50000 100 70.6 58.0 26.0
onestep 0 latency 2 1000 75000 100 74.4 60.0 35.0
onestep 0 latency 2 1001 75000 100 74.2 66.0 35.0
onestep 0 latency 2 1000 100000 100 72.4 60.0 32.0
onestep 0 latency 2 1001 100000 100 75.0 66.0 41.0
onestep 0 latency 2 1000 last 200 71.4 58.5 30.0
onestep 0 latency 2 1001 last 200 73.8 63.0 39.5
onestep 0 latency 8 1000 last 200 73.4 60.0 34.5
onestep 0 latency 8 1001 last 200 70.0 62.5 33.5
onestep 0 none 2 1000 last 200 38.3 26.0 11.5
onestep 0 none 2 1001 last 200 38.5 25.0 8.0
onestep 0 none 8 1000 last 200 76.0 66.0 36.0
onestep 0 none 8 1001 last 200 81.3 72.5 44.5
onestep 1 latency 2 1000 last 200 72.7 61.0 33.0
onestep 1 latency 2 1001 last 200 69.9 60.0 33.0
onestep 2 latency 2 1000 last 200 71.4 59.0 30.5
onestep 2 latency 2 1001 last 200 72.6 65.5 34.0
planbind 0 latency 2 1000 25000 100 54.0 41.0 10.0
planbind 0 latency 2 1001 25000 100 53.4 39.0 12.0
planbind 0 latency 2 1000 50000 100 61.8 51.0 20.0
planbind 0 latency 2 1001 50000 100 51.4 37.0 11.0
planbind 0 latency 2 1000 75000 100 62.0 47.0 24.0
planbind 0 latency 2 1001 75000 100 52.7 39.0 15.0
planbind 0 latency 2 1000 100000 100 57.4 43.0 17.0
planbind 0 latency 2 1001 100000 100 47.8 30.0 19.0
planbind 0 latency 2 1000 last 200 55.4 41.0 21.0
planbind 0 latency 2 1001 last 200 51.0 37.0 19.0
planbind 0 latency 8 1000 last 200 50.2 35.0 14.0
planbind 0 latency 8 1001 last 200 47.0 32.5 17.0
planbind 0 none 2 1000 last 200 62.7 50.5 25.0
planbind 0 none 2 1001 last 200 60.1 44.5 20.0
planbind 0 none 8 1000 last 200 55.8 41.5 17.5
planbind 0 none 8 1001 last 200 50.8 37.5 16.0
planbind 1 latency 2 1000 last 200 51.0 35.5 15.5
planbind 1 latency 2 1001 last 200 48.5 35.0 18.5
planbind 2 latency 2 1000 last 200 51.3 34.0 15.5
planbind 2 latency 2 1001 last 200 48.2 34.5 11.5