Potestates commited on
Commit
a58e013
·
verified ·
1 Parent(s): f498e22

piper README: null baselines -- the policy beats the trajectory baselines 3-4x and loses to persistence, so do not rank it by action L1

Browse files
Files changed (1) hide show
  1. piper_pick_can_10k/README.md +27 -0
piper_pick_can_10k/README.md CHANGED
@@ -48,6 +48,33 @@ protocol but it is **not** the error you will see. The table above is. Under tha
48
  curve is flat (0.0972 → 0.0930), which is why this checkpoint is the last one and not an earlier one; the open-loop
49
  error of the earlier snapshots has not been measured.
50
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
51
  ## What has NOT been established
52
 
53
  **This policy has never been run on a physical robot.** Everything above is offline. Action error on held-out
 
48
  curve is flat (0.0972 → 0.0930), which is why this checkpoint is the last one and not an earlier one; the open-loop
49
  error of the earlier snapshots has not been measured.
50
 
51
+ ## Is that error good or bad? The null baselines, same 8 episodes
52
+
53
+ Mean absolute error per joint, degrees:
54
+
55
+ | | mean per joint |
56
+ |---|---|
57
+ | constant = the training mean action | 14.70 |
58
+ | the average demonstration, time-warped to each episode | 11.02 |
59
+ | **this policy** | **3.24** |
60
+ | repeat the last command (persistence) | 2.06 |
61
+
62
+ Read it this way. The policy **beats both trajectory-level baselines by 3-4x**, so it is following each episode's
63
+ own trajectory rather than replaying an average — it has learned the task. It **loses to doing nothing**, because
64
+ the action is an ABSOLUTE joint target at 10 Hz and the arm only moves ~2 deg per step (joint 5: 0.08 deg), which
65
+ makes persistence a very strong L1 baseline that would of course fail every episode on the robot. **Do not rank this
66
+ policy by action L1** — the metric rewards standing still. What the number does say is that its precision (~3 deg)
67
+ is of the same order as the per-step motion, and that it adds ~1 deg of error on joints that barely move.
68
+
69
+ Averaging samples will not help: the flow head is stochastic, but 8 samples per observation only improve the error
70
+ from 0.1483 to 0.1452 (normalised), and more Euler steps are slightly worse — about 0.145 of the 0.148 is
71
+ systematic bias, not sampling noise. If the policy is imprecise on the robot, the levers are training-side (more
72
+ than 142 demonstrations of one task, delta instead of absolute actions, a higher control rate), not inference-side.
73
+
74
+ Temporal ensembling (on by default, as in the LIBERO evaluator) is slightly *worse* here offline, 3.69 vs 3.24 deg:
75
+ it was tuned where the action is a delta and its lag is free. Pass `ensemble=False` to try the other way — offline
76
+ cannot decide between them.
77
+
78
  ## What has NOT been established
79
 
80
  **This policy has never been run on a physical robot.** Everything above is offline. Action error on held-out