SO-101 MolmoAct2: front RGB + EEF delta, 20k

This is the completed 20,000-update MolmoAct2 continuous-action run trained on 1,565 causal segments (349,611 frames) exported from 477 front-view Task 0-9 trajectories. It replaces the previous 5k checkpoint in this repository.

Input contract

Each policy observation contains:

  • observation.images.rgb: one front RGB image, resized to 256x256 and represented as float BCHW in [0,1].
  • observation.state: six measured SO-101 joints ordered as shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper.
  • task: one of exactly three causal prompts:
    • put balloon onto tray
    • put balloon onto tray. I have released the object
    • put balloon onto tray. I have held the object

The released suffix is used only for Task 3 (h2r-air) on or after the recorded release edge. The held suffix is used only for Task 5 (r2h) on or after the recorded grasp-confirmed edge. All other frames and tasks use the base prompt. These events are supplied by the task state and must not be inferred from pixels.

Use policy_preprocessor.json and its safetensors state. RGB normalization is IDENTITY, the six joints use the embedded QUANTILES normalization, and actions use IDENTITY normalization.

Output contract

predict_action_chunk returns 50 continuous 7D actions ordered [dx, dy, dz, dRx, dRy, dRz, gripper]. Decode every executed action against the latest measured EEF pose using action_contract.json:

a = np.clip(action, -1.0, 1.0)
target_position_m = latest_position_m + a[:3] * 0.004
delta_rotvec_rad = a[3:6] * 0.5
target_rotation = Rotation.from_rotvec(delta_rotvec_rad).as_matrix() @ latest_rotation
fraction_closed = (a[6] + 1.0) / 2.0
gripper_units = 100.0 - fraction_closed * 100.0

Gripper -1 is open and +1 is closed. Apply calibrated SO-101 IK and joint limits after decoding.

Training recipe

  • 20,000 optimizer updates; effective batch 8 on four GPUs
  • 80% globally frame-uniform anchors and 20% balanced Task-3/Task-5 authority-boundary anchors
  • semantic replacement only for labeled wall pixels
  • measured sim-real RGB bridge on every example
  • camera color, illumination, blur, noise, and JPEG pools ramped linearly with optimizer progress
  • no crops, flips, warps, dropout masks, or label changes

Final observed action-flow loss was 0.008. This is not a physical task-success measurement. Exact configuration and preprocessing are in train_config.json, policy_preprocessor.json, eval_contract.json, and the augmentation and sampling contract files.

Downloads last month
90
Safetensors
Model size
6B params
Tensor type
F32
·
BF16
·
Video Preview
loading