Add ACT + M3 checkpoint (step 20,000) and model card

Browse files

Files changed (4) hide show

README.md +105 -0
config.json +69 -0
model.safetensors +3 -0
train_config.json +175 -0

README.md ADDED Viewed

	@@ -0,0 +1,105 @@

+---
+license: apache-2.0
+library_name: lerobot
+pipeline_tag: robotics
+tags:
+  - LeRobot
+  - act
+  - yamlab
+  - bimanual
+  - yam
+  - isaac-sim
+  - simulation
+  - modality-masking
+datasets:
+  - kabilanKB/yam_put_pot
+---
+# ACT + M3 for bimanual YAM: PutPotOnCooktop
+An [ACT](https://arxiv.org/abs/2304.13705) policy for the simulated bimanual YAM robot, trained
+on [`kabilanKB/yam_put_pot`](https://huggingface.co/datasets/kabilanKB/yam_put_pot) with M3
+modality masking, adapted from "Robust Bimanual Vision-Language-Action Models via Embarrassingly
+Simple Modality Masking" ([arXiv 2608.22419](https://arxiv.org/abs/2608.22419)).
+The task is PutPotOnCooktop in Isaac Sim 6 / Isaac Lab 3: both arms grasp a pot by its handles,
+lift it and place it on a cooktop.
+Code: [kabilankb/Isaac-Lab-3-and-add-SO-101-leader-teleoperation](https://github.com/kabilankb/Isaac-Lab-3-and-add-SO-101-leader-teleoperation)
+(see `ACT_M3.md`).
+> **This is an early research checkpoint.** It completes the task in roughly one episode in
+> four, and M3 masking showed no measurable benefit over plain ACT in the evaluation below.
+## Model
+| | |
+|---|---|
+| Inputs | Three 320×240 RGB cameras (`top_rgb`, `left_rgb`, `right_rgb`) and the 14-value joint state |
+| Output | Chunk of 100 actions × 14 joint-position targets (6 joints + gripper per arm) |
+| Backbone | ImageNet-pretrained ResNet-18, shared across cameras |
+| Transformer | 4 encoder layers, 4 decoder layers, width 512, 8 heads, VAE latent 32 |
+| Size | 68M parameters |
+M3 is applied during training only; this checkpoint is an ordinary ACT checkpoint.
+- **Wrist-camera masking:** for 30% of training samples, both wrist cameras are hidden from
+  attention together. The top camera is never hidden.
+- **Query masking:** each of the 100 action queries is hidden from the others with probability
+  0.1, and the visible ones are rescaled by 1/0.9.
+- The paper's language masking does not apply, since ACT has no language input.
+## Training
+| | |
+|---|---|
+| Data | `kabilanKB/yam_put_pot`: 20 MimicGen episodes, 7,779 frames at 30 FPS, domain randomization on |
+| Steps / batch size | 20,000 / 32 (about 82 passes over the data) |
+| Optimizer | AdamW, learning rate 1e-5 |
+| Seed | 1000 (single run) |
+| Final training loss | 0.041 |
+## Evaluation
+PutPotOnCooktop-v0 with `pot_000` / `cooktop_000`, plain scene (no domain randomization),
+25 actions executed per prediction, 900-step limit, 50 episodes, 10 parallel environments.
+| Model | Full task succeeded | Pot lifted |
+|---|---|---|
+| ACT + M3 (this checkpoint) | 11 of 50 (22%) | 13 of 50 (26%) |
+| Plain ACT, same settings | 12 of 50 (24%) | 15 of 50 (30%) |
+- The two models are indistinguishable on this test.
+- The rates are slightly optimistic: the run stopped at the first 50 finished episodes, and
+  successes finish sooner than timeouts.
+- Not tested: randomized or cluttered scenes, which is where the paper claims M3 helps.
+- The dataset is small (two demonstrations per pot and cooktop pair); more data is the most
+  likely way to raise the success rate.
+## Usage
+Requires the LeRobot v2.0 fork used by YAMLab (`RogerDAI1217/lerobot`, branch `lerobotv2.0`).
+Its `config.json` has no `type` key, so pass the config explicitly:
+```python
+import draccus
+from huggingface_hub import snapshot_download
+from lerobot.common.policies.act.configuration_act import ACTConfig
+from lerobot.common.policies.act.modeling_act import ACTPolicy
+ckpt = snapshot_download("kabilanKB/yam_put_pot_act_m3")
+with draccus.config_type("json"):
+    config = draccus.parse(ACTConfig, f"{ckpt}/config.json", args=[])
+config.n_action_steps = 25
+policy = ACTPolicy.from_pretrained(ckpt, config=config)
+```
+To run it in the simulator, download the checkpoint and use `scripts/eval/eval_act.py` from the
+code repository:
+```bash
+D=yamlab_datasets/tasks_data/PutPotOnCooktop/objects
+python scripts/eval/eval_act.py --checkpoint <downloaded folder> --n_action_steps 25 \
+    --task PutPotOnCooktop-v0 --asset pot=$D/Pot/pot_000 --asset cooktop=$D/Cooktop/cooktop_000 \
+    --num_episodes 5 --enable_gripper_clamp --enable_cameras --viz kit
+```

config.json ADDED Viewed

	@@ -0,0 +1,69 @@

+{
+    "n_obs_steps": 1,
+    "normalization_mapping": {
+        "VISUAL": "MEAN_STD",
+        "STATE": "MEAN_STD",
+        "ACTION": "MEAN_STD"
+    },
+    "input_features": {
+        "observation.state": {
+            "type": "STATE",
+            "shape": [
+                14
+            ]
+        },
+        "observation.images.top_rgb": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                240,
+                320
+            ]
+        },
+        "observation.images.left_rgb": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                240,
+                320
+            ]
+        },
+        "observation.images.right_rgb": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                240,
+                320
+            ]
+        }
+    },
+    "output_features": {
+        "action": {
+            "type": "ACTION",
+            "shape": [
+                14
+            ]
+        }
+    },
+    "chunk_size": 100,
+    "n_action_steps": 100,
+    "vision_backbone": "resnet18",
+    "pretrained_backbone_weights": "ResNet18_Weights.IMAGENET1K_V1",
+    "replace_final_stride_with_dilation": false,
+    "pre_norm": false,
+    "dim_model": 512,
+    "n_heads": 8,
+    "dim_feedforward": 3200,
+    "feedforward_activation": "relu",
+    "n_encoder_layers": 4,
+    "n_decoder_layers": 4,
+    "use_vae": true,
+    "latent_dim": 32,
+    "n_vae_encoder_layers": 4,
+    "temporal_ensemble_coeff": null,
+    "dropout": 0.1,
+    "kl_weight": 10.0,
+    "optimizer_lr": 1e-05,
+    "optimizer_weight_decay": 0.0001,
+    "optimizer_lr_backbone": 1e-05
+}

model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5300c356ad22abe27869d5fa8cff56c28ef51f39d572851d717cb1861c22d340
+size 271391216

train_config.json ADDED Viewed

	@@ -0,0 +1,175 @@

+{
+    "dataset": {
+        "repo_id": "yam_put_pot",
+        "episodes": null,
+        "image_transforms": {
+            "enable": false,
+            "max_num_transforms": 3,
+            "random_order": false,
+            "tfs": {
+                "brightness": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "brightness": [
+                            0.8,
+                            1.2
+                        ]
+                    }
+                },
+                "contrast": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "contrast": [
+                            0.8,
+                            1.2
+                        ]
+                    }
+                },
+                "saturation": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "saturation": [
+                            0.5,
+                            1.5
+                        ]
+                    }
+                },
+                "hue": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "hue": [
+                            -0.05,
+                            0.05
+                        ]
+                    }
+                },
+                "sharpness": {
+                    "weight": 1.0,
+                    "type": "SharpnessJitter",
+                    "kwargs": {
+                        "sharpness": [
+                            0.5,
+                            1.5
+                        ]
+                    }
+                }
+            }
+        },
+        "local_files_only": true,
+        "use_imagenet_stats": false,
+        "video_backend": "pyav"
+    },
+    "env": null,
+    "policy": {
+        "type": "act",
+        "n_obs_steps": 1,
+        "normalization_mapping": {
+            "VISUAL": "MEAN_STD",
+            "STATE": "MEAN_STD",
+            "ACTION": "MEAN_STD"
+        },
+        "input_features": {
+            "observation.state": {
+                "type": "STATE",
+                "shape": [
+                    14
+                ]
+            },
+            "observation.images.top_rgb": {
+                "type": "VISUAL",
+                "shape": [
+                    3,
+                    240,
+                    320
+                ]
+            },
+            "observation.images.left_rgb": {
+                "type": "VISUAL",
+                "shape": [
+                    3,
+                    240,
+                    320
+                ]
+            },
+            "observation.images.right_rgb": {
+                "type": "VISUAL",
+                "shape": [
+                    3,
+                    240,
+                    320
+                ]
+            }
+        },
+        "output_features": {
+            "action": {
+                "type": "ACTION",
+                "shape": [
+                    14
+                ]
+            }
+        },
+        "chunk_size": 100,
+        "n_action_steps": 100,
+        "vision_backbone": "resnet18",
+        "pretrained_backbone_weights": "ResNet18_Weights.IMAGENET1K_V1",
+        "replace_final_stride_with_dilation": false,
+        "pre_norm": false,
+        "dim_model": 512,
+        "n_heads": 8,
+        "dim_feedforward": 3200,
+        "feedforward_activation": "relu",
+        "n_encoder_layers": 4,
+        "n_decoder_layers": 4,
+        "use_vae": true,
+        "latent_dim": 32,
+        "n_vae_encoder_layers": 4,
+        "temporal_ensemble_coeff": null,
+        "dropout": 0.1,
+        "kl_weight": 10.0,
+        "optimizer_lr": 1e-05,
+        "optimizer_weight_decay": 0.0001,
+        "optimizer_lr_backbone": 1e-05
+    },
+    "output_dir": "outputs/act_m3_fast",
+    "job_name": "act",
+    "resume": false,
+    "device": "cuda",
+    "use_amp": false,
+    "seed": 1000,
+    "num_workers": 4,
+    "batch_size": 32,
+    "steps": 20000,
+    "eval_freq": 20000,
+    "log_freq": 100,
+    "save_checkpoint": true,
+    "save_freq": 5000,
+    "use_policy_training_preset": true,
+    "optimizer": {
+        "type": "adamw",
+        "lr": 1e-05,
+        "weight_decay": 0.0001,
+        "grad_clip_norm": 10.0,
+        "betas": [
+            0.9,
+            0.999
+        ],
+        "eps": 1e-08
+    },
+    "scheduler": null,
+    "eval": {
+        "n_episodes": 50,
+        "batch_size": 50,
+        "use_async_envs": false
+    },
+    "wandb": {
+        "enable": false,
+        "disable_artifact": false,
+        "project": "lerobot",
+        "entity": null,
+        "notes": null
+    }
+}