README.md

metadata

license: apache-2.0
pipeline_tag: robotics
tags:
  - robotics
  - imitation-learning
  - act
  - diffusion-policy
  - lerobot
  - sim-to-sim
  - so101

Study 1 checkpoints: randomized SO-101 pick-and-place

Checkpoints behind the numbers of study 1 (how much a learned policy loses when only the renderer changes; ACT and Diffusion Policy; MuJoCo, Isaac Sim and Newton as the renderer). One subfolder per checkpoint.

• Task: SO-101 puts a red cube on a green pad, cube and pad placed at random over the reachable workspace.
• Scores: success on held-out seeds, MuJoCo and Newton on 1000-1299 (300 episodes), Isaac Sim on 1000-1099 (100 episodes). Newton here is a render-only renderer (MuJoCo physics, Newton draws the camera frames).
• Demonstrations come from a privileged scripted expert on MuJoCo (seeds 0-199, or 0-399 for the randomized look). The "Isaac" demonstrations are the same MuJoCo actions replayed on Isaac Sim with Isaac's frames, not an expert on Isaac.
• Models of the same recipe differ only in the network seed; the spread between them is part of the result. A folder name ends in _seedN when the network seed was set to N and in _unseeded when it was not; N is just a label for the seed, not a run number.

Folder	Recipe	MuJoCo	Newton	Isaac Sim
act_nominal_unseeded	ACT, 200 MuJoCo demos (episode seeds 0-199), chunk 100, 60k steps, network seed not set	294/300	268/300	58/100
act_nominal_seed1	ACT, 200 MuJoCo demos (episode seeds 0-199), chunk 100, 60k steps, network seed 1	290/300	228/300	62/100
act_nominal_seed2	ACT, 200 MuJoCo demos (episode seeds 0-199), chunk 100, 60k steps, network seed 2	289/300	242/300	77/100
act_nominal_seed3	ACT, 200 MuJoCo demos (episode seeds 0-199), chunk 100, 60k steps, network seed 3	290/300	258/300	65/100
act_randomized_look_seed0	ACT, 400 randomized-look MuJoCo demos (episode seeds 0-399), chunk 100, 120k steps, network seed 0	294/300	295/300	90/100
act_randomized_look_seed1	ACT, 400 randomized-look MuJoCo demos (episode seeds 0-399), chunk 100, 120k steps, network seed 1	292/300	289/300	85/100
act_randomized_look_seed2	ACT, 400 randomized-look MuJoCo demos (episode seeds 0-399), chunk 100, 120k steps, network seed 2	292/300	289/300	89/100
act_both_renderers_unseeded	ACT, 200 MuJoCo demos (episode seeds 0-199) + the same 200 replayed on Isaac Sim, 120k steps, network seed not set	294/300	281/300	94/100
act_both_renderers_seed1	ACT, 200 MuJoCo demos (episode seeds 0-199) + the same 200 replayed on Isaac Sim, 120k steps, network seed 1	294/300	286/300	91/100
diffusion_policy_v2_nominal_seed0	Diffusion Policy v2, 200 MuJoCo demos (episode seeds 0-199), 100k steps, network seed 0, plays 16 of 64 actions	290/300	14/300	21/100
diffusion_policy_v2_randomized_look_seed0	Diffusion Policy v2, 400 randomized-look demos (episode seeds 0-399), 120k steps, network seed 0, plays 16 of 64 actions	298/300	18/300	49/100
act_isaac_only_100_unseeded	ACT, 100 MuJoCo demos (episode seeds 0-99) replayed on Isaac Sim (Isaac images only), chunk 100, 60k steps, network seed not set	229/300	109/300	85/100
act_isaac_only_200_unseeded	ACT, 200 demos (episode seeds 0-199) replayed on Isaac Sim, 60k steps, network seed not set	229/300	105/300	91/100
act_isaac_only_200_seed1	ACT, 200 demos (episode seeds 0-199) replayed on Isaac Sim, 60k steps, network seed 1	193/300	105/300	90/100
act_isaac_only_400_unseeded	ACT, 400 demos (episode seeds 0-399) replayed on Isaac Sim, 120k steps, network seed not set	253/300	154/300	97/100
act_chunk30_100demos_unseeded	ACT, 100 MuJoCo demos (episode seeds 0-99), chunk 30, 30k steps, network seed not set (rejected on validation seeds)	220/300	16/300	26/100
act_chunk100_100demos_unseeded	ACT, 100 MuJoCo demos (episode seeds 0-99), chunk 100, 60k steps, network seed not set	275/300	238/300	55/100

Each folder holds model.safetensors, config.json, dataset_stats.json, training_meta.json, checkpoint_meta.json (and training_log.json). checkpoints.json lists the sha256 of every weights file.

Use

With the physai framework (https://github.com/PhysAI-Robot/physai-robot-starter), --checkpoint takes a reference to a folder of this repository and downloads it into checkpoints/ (ignored by git):

python scripts/eval_policy.py --manifest research/configs/so101_randomized_pick_place.yaml --policy lerobot --checkpoint hf:rizgiak/physai-study-01-checkpoints/act_randomized_look_seed0@<commit>
python scripts/fetch_checkpoints.py hf:rizgiak/physai-study-01-checkpoints/act_randomized_look_seed0   # download only, to use offline

Name a commit (@<commit>) for a result you report; without one main is used. For the Diffusion Policy checkpoints add --policy-arg n_action_steps=16. To use a folder without the framework's downloader, fetch it with huggingface_hub.snapshot_download("rizgiak/physai-study-01-checkpoints", allow_patterns=["<folder>/*"]) and pass its path to --checkpoint.

Three kinds of seeds appear and they are unrelated:

• Episode seeds of the demonstrations (demonstration_episode_seeds): which scene layouts the demonstrations were recorded on (0-199 for 200 demos, 0-399 for 400). All runs of a recipe train on the same demonstrations.
• Network seed (network_seed): seeds only the weight initialization and the order of the training batches. Runs of one recipe that differ only in it measure how much a single training run varies. null means the seed was not set (the first run of a recipe, trained before it was recorded).
• Evaluation seeds: 900-999 to choose settings, 1000-1299 for the scores above (1000-1099 on Isaac Sim). They never overlap the demonstration seeds.

Dataset names in training_meta.json

training_meta.json records the local folder names of the demonstrations the model was trained on:

Name	What it is
randomized_v1	200 expert demonstrations on MuJoCo, seeds 0-199, the study's randomized pick-and-place task
m5_visual_dr	400 expert demonstrations on MuJoCo, seeds 0-399, with the look (colours, floor tiles, lighting) randomized in every episode
randomized_isaac_v1, randomized_isaac_v1b	the actions of randomized_v1 replayed on Isaac Sim with Isaac's frames (seeds 0-99 and 100-199)
randomized_isaac_v2a, randomized_isaac_v2b	the same for 200 further MuJoCo demonstrations (seeds 200-399), used by act_isaac_only_400

What is not here

• The demonstrations. Those from MuJoCo are regenerated from their seeds with the study's collection script. The Isaac Sim replays should not be expected to regenerate identically (Isaac runs were not deterministic in a test), which matters for the act_both_renderers_* and act_isaac_only_* models.
• Checkpoints of runs that are not in a table of the study (a first Diffusion Policy version that was replaced, and early ablations).

Reproducing the scores:

• ACT on MuJoCo: the same episodes again, step for step, given the same versions of MuJoCo, lerobot and the study's code (checked on one downloaded checkpoint: 10 of 10 seeds identical). Newton (MuJoCo physics, Newton frames) was not rechecked.
• Diffusion Policy: the policy samples random noise at every prediction and the evaluation does not fix the torch seed, so two runs of the same checkpoint on the same seeds differ in the details of an episode. Four runs of diffusion_policy_v2_nominal_seed0 on the MuJoCo seeds 1000-1299 scored 290, 291, 292 and 293 of 300 (standard deviation 1.3 episodes); the Newton and Isaac Sim scores were not rerun.
• Isaac Sim, for every model: PhysX is not deterministic, so success rates reproduce within sampling noise.