piper README: null baselines -- the policy beats the trajectory baselines 3-4x and loses to persistence, so do not rank it by action L1
Browse files- piper_pick_can_10k/README.md +27 -0
piper_pick_can_10k/README.md
CHANGED
|
@@ -48,6 +48,33 @@ protocol but it is **not** the error you will see. The table above is. Under tha
|
|
| 48 |
curve is flat (0.0972 → 0.0930), which is why this checkpoint is the last one and not an earlier one; the open-loop
|
| 49 |
error of the earlier snapshots has not been measured.
|
| 50 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 51 |
## What has NOT been established
|
| 52 |
|
| 53 |
**This policy has never been run on a physical robot.** Everything above is offline. Action error on held-out
|
|
|
|
| 48 |
curve is flat (0.0972 → 0.0930), which is why this checkpoint is the last one and not an earlier one; the open-loop
|
| 49 |
error of the earlier snapshots has not been measured.
|
| 50 |
|
| 51 |
+
## Is that error good or bad? The null baselines, same 8 episodes
|
| 52 |
+
|
| 53 |
+
Mean absolute error per joint, degrees:
|
| 54 |
+
|
| 55 |
+
| | mean per joint |
|
| 56 |
+
|---|---|
|
| 57 |
+
| constant = the training mean action | 14.70 |
|
| 58 |
+
| the average demonstration, time-warped to each episode | 11.02 |
|
| 59 |
+
| **this policy** | **3.24** |
|
| 60 |
+
| repeat the last command (persistence) | 2.06 |
|
| 61 |
+
|
| 62 |
+
Read it this way. The policy **beats both trajectory-level baselines by 3-4x**, so it is following each episode's
|
| 63 |
+
own trajectory rather than replaying an average — it has learned the task. It **loses to doing nothing**, because
|
| 64 |
+
the action is an ABSOLUTE joint target at 10 Hz and the arm only moves ~2 deg per step (joint 5: 0.08 deg), which
|
| 65 |
+
makes persistence a very strong L1 baseline that would of course fail every episode on the robot. **Do not rank this
|
| 66 |
+
policy by action L1** — the metric rewards standing still. What the number does say is that its precision (~3 deg)
|
| 67 |
+
is of the same order as the per-step motion, and that it adds ~1 deg of error on joints that barely move.
|
| 68 |
+
|
| 69 |
+
Averaging samples will not help: the flow head is stochastic, but 8 samples per observation only improve the error
|
| 70 |
+
from 0.1483 to 0.1452 (normalised), and more Euler steps are slightly worse — about 0.145 of the 0.148 is
|
| 71 |
+
systematic bias, not sampling noise. If the policy is imprecise on the robot, the levers are training-side (more
|
| 72 |
+
than 142 demonstrations of one task, delta instead of absolute actions, a higher control rate), not inference-side.
|
| 73 |
+
|
| 74 |
+
Temporal ensembling (on by default, as in the LIBERO evaluator) is slightly *worse* here offline, 3.69 vs 3.24 deg:
|
| 75 |
+
it was tuned where the action is a delta and its lag is free. Pass `ensemble=False` to try the other way — offline
|
| 76 |
+
cannot decide between them.
|
| 77 |
+
|
| 78 |
## What has NOT been established
|
| 79 |
|
| 80 |
**This policy has never been run on a physical robot.** Everything above is offline. Action error on held-out
|