twanghcmut/molmoact2-so101-left-arm-multitask

MolmoAct2 SO101 Left Arm, 11-task

MolmoAct2-SO100_101 fully finetuned on the 11 datasets of the SO101_LeftArmOnly_Task collection, merged into one multitask set, with the LeRobot molmoact2 policy. One checkpoint covers all 11 tasks; the language instruction selects the behaviour.

Same data and sample budget (768k) as twanghcmut/pi05-so101-left-arm-multitask and twanghcmut/gr00t-n1d7-so101-left-arm-multitask.

License: this is a derivative of allenai/MolmoAct2-SO100_101; its terms apply.

Tasks

Use the exact wording, including trailing punctuation; the model has not been trained on paraphrases.

#	Instruction	Episodes
0	Pick up the red cube and place it into the steel pot.	50
1	Pick up the blue cube and place it into the steel pot	50
2	Take the banana and place it into the steel pot then close the lid.	50
3	Pick up the blue cube and place it next to the blue cube.	50
4	Take the blue cube out of the steel pot and place it on the blue cube.	51
5	Take the red cube out of the steel pot and place it on the red cube	50
6	Stack the yellow cube on top of the orange cube.	50
7	Pick up one orange block and stack it neatly on top of the other orange block	50
8	Pick up the cube that makes the number of red and blue cubes on the wooden board equal, and place it in the empty corner of the wooden board.	50
9	Pick up the red cube from the wooden board, put it into the orange hollow cube, then pick up the orange hollow cube and place it on the wooden board	50
10	Take out the blue cubes from shortest to tallest and place it on the table	50

551 episodes, 253,745 frames in total.

Robot

SO101 follower arm (left), 5 joints plus gripper, 30 fps.

State / action dim	6: shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll + gripper
Cameras	observation.images.top (overhead, 360×640), observation.images.wrist (480×640)
Action	absolute joint targets, chunk of 30 (n_action_steps=30)
Prompt metadata	setup_type="single so100/so101 robotic arm in molmoact2", control_mode="absolute joint pose" (same as the base's SO100/101 tag)
Normalization	this dataset's own q01/q99 (QUANTILES) for state and action, stored in the pre/post-processor files; gripper normalized too

joint_signs / joint_offsets are not used. This dataset's calibration differs from the base's SO100/101 pretraining mix (e.g. shoulder_lift mean −25.6 here), but quantile normalization with this dataset's statistics absorbs the offsets, and the LeRobot port applies those transforms to the state only, not to training action targets.

Usage

from lerobot.policies.molmoact2.modeling_molmoact2 import MolmoAct2Policy
from lerobot.policies.factory import make_pre_post_processors
from lerobot.configs.policies import PreTrainedConfig

repo = "twanghcmut/molmoact2-so101-left-arm-multitask"
policy = MolmoAct2Policy.from_pretrained(repo).to("cuda").eval()
pre, post = make_pre_post_processors(PreTrainedConfig.from_pretrained(repo), pretrained_path=repo)

batch = {
    "observation.images.top":   top,     # (1, 3, 360, 640) float in [0, 1]
    "observation.images.wrist": wrist,   # (1, 3, 480, 640) float in [0, 1]
    "observation.state":        state,   # (1, 6) float32, raw joint units
    "task": "Stack the yellow cube on top of the orange cube.",
}
actions = post(policy.predict_action_chunk(pre(batch)))  # (1, 30, 6) absolute joint targets

policy.select_action returns one step at a time from the same chunk queue.

Training

Trainer	upstream huggingface/lerobot main at commit b6ec0060 (2026-09-10), lerobot-train --policy.type=molmoact2. This is the fine-tuning path the official allenai/molmoact2 README prescribes ("MolmoAct2 is integrated into LeRobot as a policy implementation, so users can train ... with standard LeRobot workflows"), and it includes upstream's fix(molmoact2): align continuous training with official recipe (2026-08-21), which the allenai README's 2026-08-22 note refers to. The allenai/lerobot@molmoact2-policy fork branch is older (last commit 2026-06-02) and allenai/molmoact2/experiments/lerobot is the authors' replication copy; neither was used.
Base	allenai/MolmoAct2-SO100_101, full finetune (train_mode_vlm=fft), embeddings frozen, 5.05B trainable params
Steps	24,000
Batch size	32 (768k samples = 3.0 epochs)
LR	cosine with 200 warmup, peak 1e-5 (VLM) / 5e-5 (action expert), decays to 1e-6
Optimizer	AdamW (0.9, 0.95), weight decay 0, per-group grad clip 1.0
Flow matching	8 flow timesteps per sample; knowledge insulation off
Augmentation	RandomRotation ±5°, ColorJitter (no square RandomResizedCrop: frames are 16:9 / 4:3)
Precision	bfloat16, gradient checkpointing, torch.compile
Hardware	1× H100 80GB, 55.6 GB peak
Runtime	24.3 h at 3.6 s/step
Final loss	0.004

Loss at steps 2k / 4k / 8k / 12k / 16k / 20k / 24k: 0.019 → 0.015 → 0.009 → 0.007 → 0.005 → 0.004 → 0.004.

Open-loop evaluation

One trajectory per task, walked in 16-step strides, the first 16 predicted steps of each chunk compared against the ground-truth actions, up to 400 frames per trajectory. Same trajectories and horizon as the π₀.₅ and GR00T checkpoints above. Errors are in the dataset's own action units.

traj	task	MAE (h=16)	π₀.₅	GR00T N1.7
0	0 red cube → pot	0.835	2.479	2.098
50	1 blue cube → pot	1.015	2.079	2.327
100	2 banana → pot, close lid	1.474	3.258	3.258
150	3 blue cube next to blue cube	1.469	3.257	3.372
200	4 blue cube out of pot → blue cube	1.149	2.459	2.374
251	5 red cube out of pot → red cube	1.214	2.659	2.941
301	6 stack yellow on orange	0.933	2.149	2.604
351	7 stack orange blocks	1.406	3.700	3.567
401	8 balance red/blue on board	1.151	2.847	3.717
451	9 red cube via hollow cube	1.608	2.846	2.760
501	10 blue cubes shortest → tallest	1.527	3.252	3.385
	average	1.253 (MSE 5.91)	2.817	2.946

Divided by the absolute action std (mean 28.67 across dims) that is 4.37% of action scale at horizon 16, against 9.82% (π₀.₅) and 10.27% (GR00T N1.7).

For scale, two naive baselines on the same anchors: repeating the current state scores 19.2%, and repeating the ground-truth action at the anchor (which peeks at the label) scores 11.4%.

Each model was scored by its own framework's open-loop script; the scripts follow the same procedure (stride, horizon, unnormalized units), but are not one shared harness.

Limitations

No held-out split. All 551 episodes were used for training, so the numbers above are measured on training data. They show the policy fits its data and is not degenerate; they do not measure generalization and they are not a success rate. A lower open-loop error on training data can also mean stronger memorization. Treat real-robot performance as unmeasured.

One trajectory per task is a small sample; per-task numbers above are single episodes.

Deployment note

The data is 30 fps. A client interpolating between model steps must derive its sub-step count from that: smooth_step = control_hz / 30.