You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

Open Laptop Policies and Visual Encoders

Inference weights for opening a laptop in RoboTwin. The bundle includes an ACT base policy, three residual policies, and three visual encoders. These checkpoints are for simulator inference; they do not include training state and have not been validated on a real robot.

Quick start

Request access on this page, then follow the reproduction guide to set up RoboTwin and apply the rl-garden source patch. Keep the downloaded files together. Choose the weights for your use case:

  • ACT only: act-stride-two.pt. Use the ACT-only evaluator.
  • Privileged-state residual: act-stride-two.pt and either privileged-state-150.pt or privileged-state-200.pt. Use the launcher mode matching the residual filename without .pt.
  • Visual residual: act-stride-two.pt, visual-state.pt, and encoder-visual-policy.pt. Use launcher mode visual-state.
  • Geometry estimation only: pick one encoder from the table below and follow the encoder loading guide. The standalone 320×240 encoders are not drop-in replacements for the paired visual-policy encoder.

Use the short simulator check to check all seven weights together, or follow the policy evaluation instructions to run full episodes with a JSON scene list. Evaluation does not require a warmup checkpoint or a replay buffer.

How the models work

ACT proposes actions from three RGB views and a 14-dimensional robot control state. Following ResFiT's action composition, the residual policy adds a scaled correction to the normalized ACT action. The combined action is clipped to the valid range before it is converted back to the units used by the optimized RoboTwin delta end-effector executor.

The privileged-state policies use simulator object state and robot feedback (146 dimensions). The visual policy instead uses geometry estimated from three calibrated RGBD views and measured robot feedback (163 dimensions); its residual actor and critic do not receive privileged object state. See the training recipes and observation interfaces for the required inputs and action settings.

Training data

ACT was trained on 50 demo_clean demonstrations, keeping every second observation and recomputing actions between retained frames.

The visual encoders were pretrained through supervised learning on 1,064 simulator trajectories with 169,912 synchronized frames. They use four frames from three calibrated views, with either RGBD or RGB inputs, to estimate part poses, hinge geometry, lid opening angle, and interaction points. Ground-truth geometry from the simulator supplies the training labels; it is not an input to the encoder.

The residual policies were trained online in RoboTwin with Soft Actor-Critic (SAC) while ACT remained frozen. The visual route also kept its pretrained encoder frozen. Training data are not included in this release.

Evaluation

Task success in RoboTwin

Task success rate over 500 episodes per policy: five fixed sets of 100 scene seeds, with up to 700 decisions per episode. These results use demo_clean, not demo_randomized. The 150- and 200-decision caps in the table apply to training, not evaluation.

Policy weights Configuration Task success rate
act-stride-two.pt ACT only 54.4%
privileged-state-200.pt Privileged-state residual, 200-decision training cap 99.0%
privileged-state-150.pt Privileged-state residual, 150-decision training cap 99.6%
visual-state.pt Visual residual with the paired encoder 95.4%

Offline geometry estimation

Encoder errors are measured on a validation set of 100 fixed scenes, disjoint from training, with 1,600 four-frame windows in total. These scenes were used for model selection. The errors measure geometry prediction, not task success.

Encoder weights Camera input Mean operation-point error Mean absolute lid-angle error
encoder-visual-policy.pt 128×128 RGBD, paired with visual-state.pt 5.41 mm 1.16°
encoder-rgbd-320x240.pt 320×240 RGBD, standalone 2.41 mm 0.67°
encoder-rgb-320x240.pt 320×240 RGB, standalone 5.12 mm 1.75°

References

Downloads last month
2
Video Preview
loading

Papers for Nole326/RoboTwin-Open_Laptop