ChadSaglam commited on
Commit
a0409e9
·
verified ·
1 Parent(s): c1192e7

Tighten the model card wording

Browse files
Files changed (1) hide show
  1. README.md +4 -4
README.md CHANGED
@@ -44,16 +44,16 @@ Same simulator, same episode seeds (`--seed=1000`), 200 episodes, 95% Wilson int
44
  | Policy | Training steps | Success | 95% CI |
45
  |---|---|---|---|
46
  | official [`lerobot/act_aloha_sim_transfer_cube_human`](https://huggingface.co/lerobot/act_aloha_sim_transfer_cube_human) | 80k | 81.0% (162/200) | 75.0-85.8 |
47
- | **this model** (seed 1000) | 80k | **75.5%** (151/200) | 69.1-80.9 |
48
  | same run at 50k steps | 50k | 64.0% (128/200) | 57.1-70.3 |
49
  | second seed (1001) at 50k steps | 50k | 62.0% (124/200) | 55.1-68.4 |
50
 
51
  The intervals of this model and the official checkpoint overlap; the two are statistically
52
  indistinguishable at this sample size. The 50k checkpoints are clearly below the baseline.
53
 
54
- Two things the per-checkpoint evaluations showed (details in the GitHub README):
55
- the training loss is not a proxy for task success (two seeds with identical loss curves differed by
56
- 34 points in success at 10k steps), and 50-episode evaluations cannot rank neighbouring checkpoints.
57
 
58
  ## How to use
59
 
 
44
  | Policy | Training steps | Success | 95% CI |
45
  |---|---|---|---|
46
  | official [`lerobot/act_aloha_sim_transfer_cube_human`](https://huggingface.co/lerobot/act_aloha_sim_transfer_cube_human) | 80k | 81.0% (162/200) | 75.0-85.8 |
47
+ | this model (seed 1000) | 80k | 75.5% (151/200) | 69.1-80.9 |
48
  | same run at 50k steps | 50k | 64.0% (128/200) | 57.1-70.3 |
49
  | second seed (1001) at 50k steps | 50k | 62.0% (124/200) | 55.1-68.4 |
50
 
51
  The intervals of this model and the official checkpoint overlap; the two are statistically
52
  indistinguishable at this sample size. The 50k checkpoints are clearly below the baseline.
53
 
54
+ Two things the per-checkpoint evaluations showed (details in the GitHub README): the training loss did
55
+ not predict task success (two seeds with identical loss curves differed by 34 points in success at 10k
56
+ steps), and 50-episode evaluations cannot rank neighbouring checkpoints.
57
 
58
  ## How to use
59