README.md

---
license: apache-2.0
pipeline_tag: robotics
tags:
  - robotics
  - imitation-learning
  - act
  - diffusion-policy
  - lerobot
  - sim-to-sim
  - so101
---

# Study 1 checkpoints: randomized SO-101 pick-and-place

Checkpoints behind the numbers of study 1 (how much a learned policy loses when only the renderer changes;
ACT and Diffusion Policy; MuJoCo, Isaac Sim and Newton as the renderer). One subfolder per checkpoint.

- Task: SO-101 puts a red cube on a green pad, cube and pad placed at random over the reachable workspace.
- Scores: success on held-out seeds, MuJoCo and Newton on 1000-1299 (300 episodes), Isaac Sim on 1000-1099
  (100 episodes). Newton here is a render-only renderer (MuJoCo physics, Newton draws the camera frames).
- Demonstrations come from a privileged scripted expert on MuJoCo (seeds 0-199, or 0-399 for the randomized
  look). The "Isaac" demonstrations are the same MuJoCo actions replayed on Isaac Sim with Isaac's frames,
  not an expert on Isaac.
- Models of the same recipe differ only in the network seed; the spread between them is part of the result. A
  folder name ends in `_seedN` when the network seed was set to N and in `_unseeded` when it was not; N is just a
  label for the seed, not a run number.

| Folder | Recipe | MuJoCo | Newton | Isaac Sim |
| --- | --- | --- | --- | --- |
| `act_nominal_unseeded` | ACT, 200 MuJoCo demos (episode seeds 0-199), chunk 100, 60k steps, network seed not set | 294/300 | 268/300 | 58/100 |
| `act_nominal_seed1` | ACT, 200 MuJoCo demos (episode seeds 0-199), chunk 100, 60k steps, network seed 1 | 290/300 | 228/300 | 62/100 |
| `act_nominal_seed2` | ACT, 200 MuJoCo demos (episode seeds 0-199), chunk 100, 60k steps, network seed 2 | 289/300 | 242/300 | 77/100 |
| `act_nominal_seed3` | ACT, 200 MuJoCo demos (episode seeds 0-199), chunk 100, 60k steps, network seed 3 | 290/300 | 258/300 | 65/100 |
| `act_randomized_look_seed0` | ACT, 400 randomized-look MuJoCo demos (episode seeds 0-399), chunk 100, 120k steps, network seed 0 | 294/300 | 295/300 | 90/100 |
| `act_randomized_look_seed1` | ACT, 400 randomized-look MuJoCo demos (episode seeds 0-399), chunk 100, 120k steps, network seed 1 | 292/300 | 289/300 | 85/100 |
| `act_randomized_look_seed2` | ACT, 400 randomized-look MuJoCo demos (episode seeds 0-399), chunk 100, 120k steps, network seed 2 | 292/300 | 289/300 | 89/100 |
| `act_both_renderers_unseeded` | ACT, 200 MuJoCo demos (episode seeds 0-199) + the same 200 replayed on Isaac Sim, 120k steps, network seed not set | 294/300 | 281/300 | 94/100 |
| `act_both_renderers_seed1` | ACT, 200 MuJoCo demos (episode seeds 0-199) + the same 200 replayed on Isaac Sim, 120k steps, network seed 1 | 294/300 | 286/300 | 91/100 |
| `diffusion_policy_v2_nominal_seed0` | Diffusion Policy v2, 200 MuJoCo demos (episode seeds 0-199), 100k steps, network seed 0, plays 16 of 64 actions | 290/300 | 14/300 | 21/100 |
| `diffusion_policy_v2_randomized_look_seed0` | Diffusion Policy v2, 400 randomized-look demos (episode seeds 0-399), 120k steps, network seed 0, plays 16 of 64 actions | 298/300 | 18/300 | 49/100 |
| `act_isaac_only_100_unseeded` | ACT, 100 MuJoCo demos (episode seeds 0-99) replayed on Isaac Sim (Isaac images only), chunk 100, 60k steps, network seed not set | 229/300 | 109/300 | 85/100 |
| `act_isaac_only_200_unseeded` | ACT, 200 demos (episode seeds 0-199) replayed on Isaac Sim, 60k steps, network seed not set | 229/300 | 105/300 | 91/100 |
| `act_isaac_only_200_seed1` | ACT, 200 demos (episode seeds 0-199) replayed on Isaac Sim, 60k steps, network seed 1 | 193/300 | 105/300 | 90/100 |
| `act_isaac_only_400_unseeded` | ACT, 400 demos (episode seeds 0-399) replayed on Isaac Sim, 120k steps, network seed not set | 253/300 | 154/300 | 97/100 |
| `act_chunk30_100demos_unseeded` | ACT, 100 MuJoCo demos (episode seeds 0-99), chunk 30, 30k steps, network seed not set (rejected on validation seeds) | 220/300 | 16/300 | 26/100 |
| `act_chunk100_100demos_unseeded` | ACT, 100 MuJoCo demos (episode seeds 0-99), chunk 100, 60k steps, network seed not set | 275/300 | 238/300 | 55/100 |

Each folder holds `model.safetensors`, `config.json`, `dataset_stats.json`, `training_meta.json`,
`checkpoint_meta.json` (and `training_log.json`). `checkpoints.json` lists the sha256 of every weights file.

## Use

With the physai framework (https://github.com/PhysAI-Robot/physai-robot-starter), `--checkpoint` takes a reference
to a folder of this repository and downloads it into `checkpoints/` (ignored by git):

```bash
python scripts/eval_policy.py --manifest research/configs/so101_randomized_pick_place.yaml --policy lerobot --checkpoint hf:rizgiak/physai-study-01-checkpoints/act_randomized_look_seed0@<commit>
python scripts/fetch_checkpoints.py hf:rizgiak/physai-study-01-checkpoints/act_randomized_look_seed0   # download only, to use offline
```

Name a commit (`@<commit>`) for a result you report; without one `main` is used. For the Diffusion Policy
checkpoints add `--policy-arg n_action_steps=16`. To use a folder without the framework's downloader, fetch it with
`huggingface_hub.snapshot_download("rizgiak/physai-study-01-checkpoints", allow_patterns=["<folder>/*"])` and pass its path to `--checkpoint`.

Three kinds of seeds appear and they are unrelated:

- **Episode seeds of the demonstrations** (`demonstration_episode_seeds`): which scene layouts the demonstrations
  were recorded on (0-199 for 200 demos, 0-399 for 400). All runs of a recipe train on the same demonstrations.
- **Network seed** (`network_seed`): seeds only the weight initialization and the order of the training batches.
  Runs of one recipe that differ only in it measure how much a single training run varies. `null` means the seed
  was not set (the first run of a recipe, trained before it was recorded).
- **Evaluation seeds**: 900-999 to choose settings, 1000-1299 for the scores above (1000-1099 on Isaac Sim). They
  never overlap the demonstration seeds.

## Dataset names in `training_meta.json`

`training_meta.json` records the local folder names of the demonstrations the model was trained on:

| Name | What it is |
| --- | --- |
| `randomized_v1` | 200 expert demonstrations on MuJoCo, seeds 0-199, the study's randomized pick-and-place task |
| `m5_visual_dr` | 400 expert demonstrations on MuJoCo, seeds 0-399, with the look (colours, floor tiles, lighting) randomized in every episode |
| `randomized_isaac_v1`, `randomized_isaac_v1b` | the actions of `randomized_v1` replayed on Isaac Sim with Isaac's frames (seeds 0-99 and 100-199) |
| `randomized_isaac_v2a`, `randomized_isaac_v2b` | the same for 200 further MuJoCo demonstrations (seeds 200-399), used by `act_isaac_only_400` |

## What is not here

- The demonstrations. Those from MuJoCo are regenerated from their seeds with the study's collection script. The
  Isaac Sim replays should not be expected to regenerate identically (Isaac runs were not deterministic in a test), which matters for the
  `act_both_renderers_*` and `act_isaac_only_*` models.
- Checkpoints of runs that are not in a table of the study (a first Diffusion Policy version that was replaced, and
  early ablations).

Reproducing the scores:

- ACT on MuJoCo: the same episodes again, step for step, given the same versions of MuJoCo, lerobot and the study's
  code (checked on one downloaded checkpoint: 10 of 10 seeds identical). Newton (MuJoCo physics, Newton frames) was not
  rechecked.
- Diffusion Policy: the policy samples random noise at every prediction and the evaluation does not fix the
  torch seed, so two runs of the same checkpoint on the same seeds differ in the details of an episode. Four runs of
  `diffusion_policy_v2_nominal_seed0` on the MuJoCo seeds 1000-1299 scored 290, 291, 292 and 293 of 300 (standard
  deviation 1.3 episodes); the Newton and Isaac Sim scores were not rerun.
- Isaac Sim, for every model: PhysX is not deterministic, so success rates reproduce within sampling noise.