| --- |
| license: apache-2.0 |
| library_name: lerobot |
| pipeline_tag: robotics |
| tags: |
| - lerobot |
| - robotics |
| - libero |
| - act |
| - imitation-learning |
| datasets: |
| - lerobot/libero |
| --- |
| |
| # ACT on LIBERO-Object, butter, 20 demonstrations, seed 1000 |
|
|
| One of the 60 policies trained for [vla-vs-scratch](https://github.com/Pipe1213/vla-vs-scratch), a comparison of a fine-tuned pretrained vision-language-action model (SmolVLA) with a policy trained from scratch (ACT) on the same few demonstrations, in simulation. This model is ACT trained from scratch, on one task only: *"pick up the butter and place it in the basket"*. It is published to make the study reproducible, not as a general-purpose policy. |
|
|
| ## Results of this model |
|
|
| Each checkpoint was tested on the 50 fixed starting scenes of the task (LIBERO initial states 0 to 49), the same scenes for every model in the study. |
|
|
| | Checkpoint (training step) | Successes | Success rate | |
| | --- | ---: | ---: | |
| | 6,000 | 15 of 50 | 30% | |
| | 8,000 | 24 of 50 | 48% | |
| | 10,000 | 16 of 50 | 32% | |
| | **Mean of the 3 checkpoints** (the study's score) | **55 of 150** | **36.7%** (95% Wilson interval 29 to 45%) | |
|
|
| The repository root holds the last checkpoint (10,000 steps, 32%). The Wilson interval covers evaluation sampling only; differences between training seeds are larger, especially for ACT. |
|
|
| For context, pooled over the study's 3 tasks and all training seeds, ACT with 20 demonstrations per task scored 33.1%, and SmolVLA with the same demonstrations scored 74.2%. Full results: [README](https://github.com/Pipe1213/vla-vs-scratch#results) and [analysis](https://github.com/Pipe1213/vla-vs-scratch/blob/main/results/analysis.md). |
|
|
| ## Training |
|
|
| ACT (Zhao et al., 2023) with LeRobot's default settings: 52M parameters, chunks of 100 actions, and a ResNet-18 image encoder initialised from ImageNet weights. Everything else is trained from scratch on the demonstrations below. The only change from the defaults is at evaluation: the policy re-plans every 10 actions instead of running a whole chunk. |
|
|
| - **Task:** LIBERO-Object, simulator task id 6, *"pick up the butter and place it in the basket"*. |
| - **Demonstrations:** 20 episodes of [`lerobot/libero`](https://huggingface.co/datasets/lerobot/libero) at revision `a1aaacb7f6cd6ee5fb43120f673cebb0cfea7dd4`: 855, 873, 881, 907, 914, 917, 927, 938, 961, 1012, 1026, 1059, 1076, 1118, 1124, 1183, 1185, 1186, 1212, 1220. The smaller sets of the study are nested inside the larger ones, and both methods use the same episodes. |
| - **Settings:** batch size 8, 10,000 steps, a checkpoint every 2,000 steps, seed 1000. The step budget is the same for every number of demonstrations. |
| - **Software:** LeRobot 0.6.2 at commit `b9cb121cb7d4e3e26ec5c906d08dda68d151ba52`. |
| - **Compute:** 0.42 h on a laptop RTX 4050 (6 GB). |
|
|
| Exact command, generated from the repository's `configs/runs.yaml` by `scripts/train_grid.py`: |
|
|
| ```bash |
| lerobot-train \ |
| --dataset.repo_id=lerobot/libero \ |
| --dataset.revision=a1aaacb7f6cd6ee5fb43120f673cebb0cfea7dd4 \ |
| '--dataset.episodes=[855, 873, 881, 907, 914, 917, 927, 938, 961, 1012, 1026, 1059, 1076, 1118, 1124, 1183, 1185, 1186, 1212, 1220]' \ |
| --seed=1000 \ |
| --log_freq=200 \ |
| --output_dir=outputs/train/act_obj_t2_n20 \ |
| --job_name=act_obj_t2_n20 \ |
| --wandb.enable=true \ |
| --policy.type=act \ |
| --policy.device=cuda \ |
| --policy.push_to_hub=false \ |
| --batch_size=8 \ |
| --steps=10000 \ |
| --save_freq=2000 \ |
| --num_workers=4 |
| ``` |
|
|
| *Trained before `--wandb.disable_artifact=true` was added to the ACT settings (it only stops checkpoint uploads to Weights & Biases).* |
|
|
| ## Evaluation |
|
|
| 50 episodes per checkpoint, starting from LIBERO's fixed initial states, 256 by 256 pixel images, re-planning every 10 actions, evaluation seed 1000, MuJoCo rendering on the CPU (WSL2). To evaluate the last checkpoint: |
|
|
| ```bash |
| lerobot-eval --policy.path=Pipe1213/act_obj_t2_n20 \ |
| --env.type=libero --env.task=libero_object --env.task_ids="[6]" \ |
| --env.init_states=true --env.observation_height=256 --env.observation_width=256 \ |
| --eval.n_episodes=50 --eval.batch_size=1 --seed=1000 --policy.n_action_steps=10 |
| ``` |
|
|
| The other evaluated checkpoints are in `checkpoints/<step>/pretrained_model` (006000, 008000, 010000): download that folder and point `--policy.path` at it. |
|
|
| ## Limitations |
|
|
| - Simulation only, one task from one benchmark. Not tested on a real robot. |
| - The study compares two methods as they are used in practice. SmolVLA and ACT differ in pretraining but also in size, architecture and language input, so the gap between them cannot be put on pretraining alone. |
| - The 3 tasks were chosen among LIBERO-Object tasks that a published SmolVLA LIBERO checkpoint already solved, so results describe these tasks. |
| - States and actions are normalised with statistics from the whole `lerobot/libero` dataset (LeRobot's behaviour), not only the selected demonstrations. |
| - ACT used LeRobot's defaults and was not tuned for this task. |
| - ACT's checkpoints from the same run often perform very differently (in 30 of 36 ACT runs in the study). Evaluate a checkpoint before using it. |
|
|
| Code and full results: [https://github.com/Pipe1213/vla-vs-scratch](https://github.com/Pipe1213/vla-vs-scratch). |
|
|
