Blame for README.md

---
license: apache-2.0
pipeline_tag: robotics
tags:
  - robotics
  - imitation-learning
  - act
  - diffusion-policy
  - lerobot
  - sim-to-sim
  - so101
  - viperx
---

# Study 1 checkpoints: randomized pick-and-place on the SO-101 and the ViperX 300 S

Checkpoints behind the numbers of study 1 (how much a learned policy loses when only the renderer changes; ACT and
Diffusion Policy; MuJoCo, Isaac Sim and Newton as the renderer). One subfolder per checkpoint, 34 in all: the first part
of the study (17 folders, the SO-101) and the second part, study-01-v2 (17 folders: the SO-101, the ViperX 300 S, and
the controls and mechanism tests of the second part).

- Task: the robot puts a red cube on a green pad, cube and pad placed at random over the workspace it can reach top-down
  (SO-101: `research/configs/so101_randomized_pick_place.yaml`; ViperX 300 S:
  `research/configs/viperx300s_single_cube_randomized_place.yaml`, symmetric in y).
- Demonstrations come from a privileged scripted expert on MuJoCo (seeds 0-199, or 0-399). The "Isaac" demonstrations are
  the same MuJoCo actions replayed on Isaac Sim with Isaac's frames, not an expert on Isaac.
- Models of the same recipe differ only in the network seed; the spread between them is part of the result. A folder name
  ends in `_seedN` when the network seed was set to N and in `_unseeded` when it was not; N is a label, not a run number.

## How to read the scores

Scores are successes over held-out episodes (seeds 1000-1299; seeds 900-999 were used to choose settings and are never
reported, and no demonstration seed overlaps them).

- **MuJoCo**: the training renderer, 300 episodes.
- **Newton**: a render-only renderer (MuJoCo keeps the physics, Newton draws the camera frames), 300 episodes. **The Newton
  renderer was fixed during the second part:** it drew two contact pads (collision-only geoms) and a wrongly coloured
  material that MuJoCo does not draw, and a 760-pixel difference moved the scores of some models by up to 26 of 300
  episodes. Every Newton score of the second part, and the column "Newton (fixed)" below, uses the fixed renderer. The
  column "Newton (v1)" is what the first part reported with the renderer as it was; it is kept as a sensitivity analysis,
  and 400-demonstration and randomized-look models move by 5 episodes or less.
- **Isaac Sim**: 100 episodes (seeds 1000-1099) for most models, 300 (1000-1299) where a test needed more (marked). The
  first part was scored on Isaac Sim 6.1.0.0 (driver 610.47); the second part on Isaac Sim 6.2.0.0 (driver 617.42, Windows
  build 26300). The seven first-part models that were scored again on the new stack are in the column "Isaac (new stack)":
  the mean shift of the four nominal runs is -0.75 points, inside the bridge rule of 3 points. PhysX is not
  deterministic, so an Isaac Sim score reproduces only within sampling noise.

## First part (17 folders, SO-101)

| Folder | Recipe | MuJoCo | Newton (fixed) | Newton (v1) | Isaac Sim | Isaac (new stack) |
| --- | --- | ---: | ---: | ---: | ---: | ---: |
| `act_nominal_unseeded` | ACT, 200 MuJoCo demos (episode seeds 0-199), chunk 100, 60k steps, network seed not set | 294/300 | 251/300 | 268/300 | 58/100 | 56/100 |
| `act_nominal_seed1` | ACT, 200 MuJoCo demos (episode seeds 0-199), chunk 100, 60k steps, network seed 1 | 290/300 | 220/300 | 228/300 | 62/100 | 61/100 |
| `act_nominal_seed2` | ACT, 200 MuJoCo demos (episode seeds 0-199), chunk 100, 60k steps, network seed 2 | 289/300 | 231/300 | 242/300 | 77/100 | 78/100 |
| `act_nominal_seed3` | ACT, 200 MuJoCo demos (episode seeds 0-199), chunk 100, 60k steps, network seed 3 | 290/300 | 257/300 | 258/300 | 65/100 | 72/100 |
| `act_randomized_look_seed0` | ACT, 400 randomized-look MuJoCo demos (episode seeds 0-399), chunk 100, 120k steps, network seed 0 | 294/300 | 290/300 | 295/300 | 90/100 | 88/100 |
| `act_randomized_look_seed1` | ACT, 400 randomized-look MuJoCo demos (episode seeds 0-399), chunk 100, 120k steps, network seed 1 | 292/300 | 288/300 | 289/300 | 85/100 | 88/100 |
| `act_randomized_look_seed2` | ACT, 400 randomized-look MuJoCo demos (episode seeds 0-399), chunk 100, 120k steps, network seed 2 | 292/300 | 289/300 | 289/300 | 89/100 | 91/100 |
| `act_both_renderers_unseeded` | ACT, 200 MuJoCo demos (episode seeds 0-199) + the same 200 replayed on Isaac Sim, 120k steps, network seed not set | 294/300 | 280/300 | 281/300 | 94/100 | - |
| `act_both_renderers_seed1` | ACT, 200 MuJoCo demos (episode seeds 0-199) + the same 200 replayed on Isaac Sim, 120k steps, network seed 1 | 294/300 | 279/300 | 286/300 | 91/100 | - |
| `diffusion_policy_v2_nominal_seed0` | Diffusion Policy v2, 200 MuJoCo demos (episode seeds 0-199), 100k steps, network seed 0, plays 16 of 64 actions | 290/300 | 11/300 | 14/300 | 21/100 | - |
| `diffusion_policy_v2_randomized_look_seed0` | Diffusion Policy v2, 400 randomized-look demos (episode seeds 0-399), 120k steps, network seed 0, plays 16 of 64 actions | 298/300 | 32/300 | 18/300 | 49/100 | - |
| `act_isaac_only_100_unseeded` | ACT, 100 MuJoCo demos (episode seeds 0-99) replayed on Isaac Sim (Isaac images only), chunk 100, 60k steps, network seed not set | 229/300 | 88/300 | 109/300 | 85/100 | - |
| `act_isaac_only_200_unseeded` | ACT, 200 demos (episode seeds 0-199) replayed on Isaac Sim, 60k steps, network seed not set | 229/300 | 91/300 | 105/300 | 91/100 | - |
| `act_isaac_only_200_seed1` | ACT, 200 demos (episode seeds 0-199) replayed on Isaac Sim, 60k steps, network seed 1 | 193/300 | 112/300 | 105/300 | 90/100 | - |
| `act_isaac_only_400_unseeded` | ACT, 400 demos (episode seeds 0-399) replayed on Isaac Sim, 120k steps, network seed not set | 253/300 | 140/300 | 154/300 | 97/100 | - |
| `act_chunk30_100demos_unseeded` | ACT, 100 MuJoCo demos (episode seeds 0-99), chunk 30, 30k steps, network seed not set (rejected on validation seeds) | 220/300 | 5/300 | 16/300 | 26/100 | - |
| `act_chunk100_100demos_unseeded` | ACT, 100 MuJoCo demos (episode seeds 0-99), chunk 100, 60k steps, network seed not set | 275/300 | 235/300 | 238/300 | 55/100 | - |

## Second part, study-01-v2 (17 folders)

Counted from the per-seed files in `results/<folder>/` (see "Result files" below), so the paired tests
of the study (exact McNemar, rank tests) can be recomputed. `checkpoints_v2.json` lists the recipe, the seeds, the scores
and the sha256 of every weights file of this part. The Isaac Sim column says how many seeds were scored.

| Folder | Recipe | MuJoCo | Newton | Isaac Sim | Other |
| --- | --- | ---: | ---: | ---: | --- |
| `act_nominal400_seed1` | ACT, 400 MuJoCo demos (episode seeds 0-399), nominal look, chunk 100, 120k steps, network seed 1 | 296/300 | 267/300 | 181/300 (300 seeds; first 100: 61) | - |
| `act_nominal400_seed2` | ACT, 400 MuJoCo demos (episode seeds 0-399), nominal look, chunk 100, 120k steps, network seed 2 | 292/300 | 264/300 | 239/300 (300 seeds; first 100: 74) | - |
| `act_nominal400_seed3` | ACT, 400 MuJoCo demos (episode seeds 0-399), nominal look, chunk 100, 120k steps, network seed 3 | 296/300 | 288/300 | 270/300 (300 seeds; first 100: 90) | - |
| `act_nominal_seed4` | ACT, 200 MuJoCo demos (episode seeds 0-199), nominal look, chunk 100, 60k steps, network seed 4 | 291/300 | 277/300 | 214/300 (300 seeds; first 100: 71) | - |
| `act_nominal_seed5` | ACT, 200 MuJoCo demos (episode seeds 0-199), nominal look, chunk 100, 60k steps, network seed 5 | 288/300 | 255/300 | 221/300 (300 seeds; first 100: 75) | - |
| `act_randomized_look_seed3` | ACT, 400 MuJoCo demos (episode seeds 0-399), look randomized in every episode, chunk 100, 120k steps, network seed 3 | 290/300 | 286/300 | 89/100 | - |
| `act_randomized_look_seed4` | ACT, 400 MuJoCo demos (episode seeds 0-399), look randomized in every episode, chunk 100, 120k steps, network seed 4 | 292/300 | 287/300 | 93/100 | - |
| `viperx_act_nominal_seed1` | ViperX 300 S, ACT, 400 MuJoCo demos (episode seeds 0-399), nominal look, chunk 100, 120k steps, network seed 1 | 284/300 | 238/300 | 255/300 (300 seeds; first 100: 88) | newton_heldout_look 59/300 |
| `viperx_act_nominal_seed2` | ViperX 300 S, ACT, 400 MuJoCo demos (episode seeds 0-399), nominal look, chunk 100, 120k steps, network seed 2 | 284/300 | 215/300 | 248/300 (300 seeds; first 100: 85) | newton_heldout_look 16/300 |
| `viperx_act_randomized_look_seed1` | ViperX 300 S, ACT, 400 MuJoCo demos (episode seeds 0-399), look randomized in every episode, chunk 100, 120k steps, network seed 1 | 282/300 | 264/300 | 275/300 (300 seeds; first 100: 91) | newton_heldout_look 215/300 |
| `viperx_act_randomized_look_seed2` | ViperX 300 S, ACT, 400 MuJoCo demos (episode seeds 0-399), look randomized in every episode, chunk 100, 120k steps, network seed 2 | 275/300 | 247/300 | 266/300 (300 seeds; first 100: 93) | newton_heldout_look 201/300 |
| `act_randomized_look_bw_seed1` | ACT, 400 MuJoCo demos (episode seeds 0-399), look randomized in every episode and the floor tiles black and white in 30% of them, chunk 100, 120k steps, network seed 1 | 296/300 | 280/300 | 80/100 | newton_heldout_look 282/300 |
| `act_randomized_look_bw_seed2` | ACT, 400 MuJoCo demos (episode seeds 0-399), look randomized in every episode and the floor tiles black and white in 30% of them, chunk 100, 120k steps, network seed 2 | 297/300 | 286/300 | 89/100 | newton_heldout_look 250/300 |
| `act_randomized_look_bw_seed3` | ACT, 400 MuJoCo demos (episode seeds 0-399), look randomized in every episode and the floor tiles black and white in 30% of them, chunk 100, 120k steps, network seed 3 | 291/300 | 279/300 | 94/100 | newton_heldout_look 266/300 |
| `diffusion_policy_v2_nominal_seed2` | Diffusion Policy, 200 MuJoCo demos (episode seeds 0-199), nominal look, chunk 64, 100k steps, network seed 2 (add --policy-arg n_action_steps=16) | 294/300 | 9/300 | 43/100 | - |
| `diffusion_policy_v2_nominal_seed3` | Diffusion Policy, 200 MuJoCo demos (episode seeds 0-199), nominal look, chunk 64, 100k steps, network seed 3 (add --policy-arg n_action_steps=16) | 298/300 | 17/300 | 57/100 | - |
| `act_pad_visible_seed1` | ACT, 200 MuJoCo demos (episode seeds 0-199), nominal look, the two contact pads drawn opaque, chunk 100, 60k steps, network seed 1 | 284/300 | 197/300 | - | mujoco_pad_visible 283/300; newton_pad_visible 221/300 |

Notes on the second part: `act_nominal400_*` is the nominal look with the same 400 demonstrations and 120k steps as the
randomized look (the control of the study); `act_randomized_look_bw_*` adds black and white floor tiles to 30% of the
episodes and is also scored on a look it never saw (wood-coloured tiles, dark blue sky, mirrored key light; "Other");
`act_pad_visible_*` was trained on images that show the SO-101's two contact pads, and its "Other" column gives its scores
with the pad drawn (MuJoCo, Newton before the fix) and without it; the `viperx_*` models are for the ViperX 300 S manifest.

Each folder holds `model.safetensors`, `config.json`, `dataset_stats.json`, `training_meta.json` and `checkpoint_meta.json`
(and `training_log.json`). `checkpoints.json` (first part) and `checkpoints_v2.json` (second part) list the sha256 of every
weights file.

## Result files

`results/<checkpoint folder>/<setting>.json` holds the outcome of every evaluation episode of that checkpoint, for **all 34 folders**: a JSON object `seed -> true|false`
(no paths, no logs), from which the success counts, the paired tests (exact McNemar) and the intervals of the study are recomputed. `results/INDEX.tsv` has one row per file: the checkpoint, the file, the successes,
the number of episodes, the seeds and what the setting is. The files were written by `stage_results.py` of the research repository from the per-episode result files, and for the first part's 17 folders every
MuJoCo and Isaac Sim file was checked against the score in `checkpoints.json`.

| File name (in every folder unless noted) | What it is |
| --- | --- |
| `mujoco.json` | MuJoCo, seeds 1000-1299 (the engine the policy was trained on) |
| `isaac.json` | Isaac Sim, seeds 1000-1099 (no file for `act_pad_visible_seed1`, which was not run there) |
| `newton.json` | MuJoCo physics drawn by Newton, with the fixed renderer, seeds 1000-1299 |
| `newton_old_renderer.json` | the same with the renderer before the fix, only for the first part's folders where that score has a file (7 of 17); the other first-part Newton scores in `checkpoints.json` have no per-seed file that matches their count |
| `newton_heldout_look.json`, `newton_look_*.json` | Newton (fixed renderer) on a look outside the ranges the randomized-look models were trained on: the wood-coloured held-out look; the look outside every range (fine tiles, black and white tiles, dark sky and low light together) and each of its four changes alone |
| `mujoco_physics_isaac_pictures.json` | cell **MI**: MuJoCo simulates, Isaac Sim draws the pictures the policy sees (`act_nominal_seed1`, `act_nominal_seed2`, `act_randomized_look_seed0`, seeds 1000-1099; `viperx_act_nominal_seed1`, `viperx_act_nominal_seed2`, seeds 1000-1299) |
| `isaac_physics_mujoco_pictures.json` | cell **IM**: Isaac Sim simulates, MuJoCo draws the pictures (the same five folders) |
| `mujoco_seeds_1000_1099_<edit>.json` | MuJoCo with one change to what the policy sees: blurred cameras, lighting 0.5 and 1.4 times, a 10 mm camera shift, and for the SO-101 an orange-tinted arm, a dimmed background and a plain floor; `..._unedited.json` is the same seeds without an edit (29 ACT folders; the sensitivity pilot that did not predict the Isaac Sim loss) |
| `mujoco_seeds_900_999_<edit>.json` | the same kind of edits, and lighting levels, on the first nominal ACT (`act_nominal_unseeded`) on the seeds the settings were chosen on |

The four cells are named by physics, then pictures: MM = MuJoCo's own run (`mujoco.json`), II = Isaac Sim's own run (`isaac.json`), MI and IM as above.

Notes on the files:

- The first-part MuJoCo files and the unedited rerun of the pilot (`mujoco_seeds_1000_1099_unedited.json`) differ on 1 to 4 of the 100 seeds for four folders (`act_chunk30_100demos_unseeded`, `act_isaac_only_100_unseeded`,
  `act_isaac_only_200_unseeded`, `viperx_act_nominal_seed1`); the study's tables use `mujoco.json`.
- The first gate of cell IM found the target pad drawn in the wrong place (8 to 118 px off); it was fixed in the code (commit `510f773` of `PhysAI-Robot/physai-robot-starter`) before any policy was scored in that cell, and the files here are after the fix.
  The gates (the first frames of seeds 1000-1004 against the engine's own run) held except on the cube centroid in MI for the SO-101 (1.7 and 1.8 px on two seeds; bar 1 px) and on the target centroid of one ViperX seed in MI (2.95 px);
  the scoring ran regardless, and the study's findings file states each miss.
- Other first-part experiments (sweeps of the camera and the lighting, replays, clutter, the scripted expert and the visual servo) have per-seed files in the study's outputs that are not published here; their counts are in the paper's appendix tables.

## Use

With the physai framework (https://github.com/PhysAI-Robot/physai-robot-starter), `--checkpoint` takes a reference to a
folder of this repository and downloads it into `checkpoints/` (ignored by git):

```bash
python scripts/eval_policy.py --manifest research/configs/so101_randomized_pick_place.yaml --policy lerobot --checkpoint hf:rizgiak/physai-study-01-checkpoints/act_randomized_look_seed0@<commit>
python scripts/eval_policy.py --manifest research/configs/viperx300s_single_cube_randomized_place.yaml --policy lerobot --checkpoint hf:rizgiak/physai-study-01-checkpoints/viperx_act_nominal_seed1@<commit>
python scripts/fetch_checkpoints.py hf:rizgiak/physai-study-01-checkpoints/act_randomized_look_seed0   # download only, to use offline
```

Add `--sim isaac` for Isaac Sim, or `--sim mujoco --render newton` for Newton. Name a commit (`@<commit>`) for a result you
report; without one `main` is used (the second part is commit `ece89d5` and later). For the Diffusion Policy checkpoints add
`--policy-arg n_action_steps=16`. To use a folder without the framework's downloader, fetch it with
`huggingface_hub.snapshot_download("rizgiak/physai-study-01-checkpoints", allow_patterns=["<folder>/*"])` and pass its path
to `--checkpoint`. `act_pad_visible_seed1` is scored with the pads drawn through `research/scripts/with_pad_visible.py`.

Three kinds of seeds appear and they are unrelated:

- **Episode seeds of the demonstrations** (`demonstration_episode_seeds`): which scene layouts the demonstrations were
  recorded on (0-199 for 200 demos, 0-399 for 400). All runs of a recipe train on the same demonstrations.
- **Network seed** (`network_seed`): seeds only the weight initialization and the order of the training batches. Runs of one
  recipe that differ only in it measure how much a single training run varies. `null` means the seed was not set (the first
  run of a recipe, trained before it was recorded).
- **Evaluation seeds**: 900-999 to choose settings, 1000-1299 for the scores above. They never overlap the demonstration seeds.

## Dataset names in `training_meta.json`

`training_meta.json` records the local folder names of the demonstrations the model was trained on:

| Name | What it is |
| --- | --- |
| `randomized_v1`, `mujoco_nominal_200_seeds0-199` | 200 expert demonstrations on MuJoCo, seeds 0-199, the SO-101's randomized pick-and-place task |
| `mujoco_nominal_200_seeds200-399` | 200 further expert demonstrations (seeds 200-399); with the previous one, the 400 of `act_nominal400_*` |
| `m5_visual_dr`, `mujoco_randomized-look_400_seeds0-399` | 400 expert demonstrations on MuJoCo, seeds 0-399, with the look (colours, floor tiles, lighting) randomized in every episode |
| `mujoco_randomized-look-bw_400_seeds0-399` | the same with black and white floor tiles in 30% of the episodes |
| `mujoco_nominal_padvisible_200_seeds0-199` | the 200 nominal demonstrations with the contact pads drawn opaque |
| `viperx_nominal_400_seeds0-399`, `viperx_randomized-look_400_seeds0-399` | 400 expert demonstrations on the ViperX 300 S, nominal and randomized look |
| `randomized_isaac_v1`, `randomized_isaac_v1b` | the actions of `randomized_v1` replayed on Isaac Sim with Isaac's frames (seeds 0-99 and 100-199) |
| `randomized_isaac_v2a`, `randomized_isaac_v2b` | the same for 200 further MuJoCo demonstrations (seeds 200-399), used by `act_isaac_only_400` |

## What is not here

- The demonstrations. Those from MuJoCo are regenerated from their seeds with the study's collection script. The Isaac Sim
  replays should not be expected to regenerate identically (Isaac runs were not deterministic in a test), which matters for
  the `act_both_renderers_*` and `act_isaac_only_*` models.
- Checkpoints of runs that are not in a table of the study (a first Diffusion Policy version that was replaced, and early
  ablations).

Reproducing the scores:

- ACT on MuJoCo: the same episodes again, step for step, given the same versions of MuJoCo, lerobot and the study's code
  (checked on one downloaded checkpoint: 10 of 10 seeds identical). Newton with the fixed renderer is deterministic too: one
  model scored twice on one day gives the same outcome on all 300 seeds.
- Diffusion Policy: the policy samples random noise at every prediction and the evaluation does not fix the torch seed, so
  two runs of the same checkpoint on the same seeds differ in the details of an episode. Four runs of
  `diffusion_policy_v2_nominal_seed0` on the MuJoCo seeds 1000-1299 scored 290, 291, 292 and 293 of 300 (standard deviation
  1.3 episodes), and two scorings on Newton (fixed) gave 11 and 10 of 300.
- Isaac Sim, for every model: PhysX is not deterministic, so success rates reproduce within sampling noise. On Isaac Sim the
  cube settles 3 mm (SO-101) or 8.6 mm (ViperX) from its drawn position in the first control step whenever it is placed at a
  new place, a small difference of the Isaac Sim task that the camera images show.