Instructions to use nhatcm/hitbench-molmoact2-20k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LeRobot
How to use nhatcm/hitbench-molmoact2-20k with LeRobot:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
SO-101 MolmoAct2: front RGB + EEF delta, 20k
This is the completed 20,000-update MolmoAct2 continuous-action run trained on 1,565 causal segments (349,611 frames) exported from 477 front-view Task 0-9 trajectories. It replaces the previous 5k checkpoint in this repository.
Input contract
Each policy observation contains:
observation.images.rgb: one front RGB image, resized to 256x256 and represented as float BCHW in[0,1].observation.state: six measured SO-101 joints ordered asshoulder_pan,shoulder_lift,elbow_flex,wrist_flex,wrist_roll,gripper.task: one of exactly three causal prompts:put balloon onto trayput balloon onto tray. I have released the objectput balloon onto tray. I have held the object
The released suffix is used only for Task 3 (h2r-air) on or after the
recorded release edge. The held suffix is used only for Task 5 (r2h) on or
after the recorded grasp-confirmed edge. All other frames and tasks use the
base prompt. These events are supplied by the task state and must not be
inferred from pixels.
Use policy_preprocessor.json and its safetensors state. RGB normalization is
IDENTITY, the six joints use the embedded QUANTILES normalization, and actions
use IDENTITY normalization.
Output contract
predict_action_chunk returns 50 continuous 7D actions ordered
[dx, dy, dz, dRx, dRy, dRz, gripper]. Decode every executed action against
the latest measured EEF pose using action_contract.json:
a = np.clip(action, -1.0, 1.0)
target_position_m = latest_position_m + a[:3] * 0.004
delta_rotvec_rad = a[3:6] * 0.5
target_rotation = Rotation.from_rotvec(delta_rotvec_rad).as_matrix() @ latest_rotation
fraction_closed = (a[6] + 1.0) / 2.0
gripper_units = 100.0 - fraction_closed * 100.0
Gripper -1 is open and +1 is closed. Apply calibrated SO-101 IK and joint
limits after decoding.
Training recipe
- 20,000 optimizer updates; effective batch 8 on four GPUs
- 80% globally frame-uniform anchors and 20% balanced Task-3/Task-5 authority-boundary anchors
- semantic replacement only for labeled wall pixels
- measured sim-real RGB bridge on every example
- camera color, illumination, blur, noise, and JPEG pools ramped linearly with optimizer progress
- no crops, flips, warps, dropout masks, or label changes
Final observed action-flow loss was 0.008. This is not a physical task-success
measurement. Exact configuration and preprocessing are in train_config.json,
policy_preprocessor.json, eval_contract.json, and the augmentation and
sampling contract files.
- Downloads last month
- 90