Open Laptop Policies and Visual Encoders
Inference weights for opening a laptop in RoboTwin. The bundle includes an ACT base policy, three residual policies, and three visual encoders. These checkpoints are for simulator inference; they do not include training state and have not been validated on a real robot.
Quick start
Request access on this page, then follow the reproduction guide to set up RoboTwin and apply the rl-garden source patch. Keep the downloaded files together. Choose the weights for your use case:
- ACT only:
act-stride-two.pt. Use the ACT-only evaluator. - Privileged-state residual:
act-stride-two.ptand eitherprivileged-state-150.ptorprivileged-state-200.pt. Use the launcher mode matching the residual filename without.pt. - Visual residual:
act-stride-two.pt,visual-state.pt, andencoder-visual-policy.pt. Use launcher modevisual-state. - Geometry estimation only: pick one encoder from the table below and follow the encoder loading guide. The standalone 320×240 encoders are not drop-in replacements for the paired visual-policy encoder.
Use the short simulator check to check all seven weights together, or follow the policy evaluation instructions to run full episodes with a JSON scene list. Evaluation does not require a warmup checkpoint or a replay buffer.
How the models work
ACT proposes actions from three RGB views and a 14-dimensional robot control state. Following ResFiT's action composition, the residual policy adds a scaled correction to the normalized ACT action. The combined action is clipped to the valid range before it is converted back to the units used by the optimized RoboTwin delta end-effector executor.
The privileged-state policies use simulator object state and robot feedback (146 dimensions). The visual policy instead uses geometry estimated from three calibrated RGBD views and measured robot feedback (163 dimensions); its residual actor and critic do not receive privileged object state. See the training recipes and observation interfaces for the required inputs and action settings.
Training data
ACT was trained on 50 demo_clean demonstrations, keeping every second observation and recomputing actions between retained frames.
The visual encoders were pretrained through supervised learning on 1,064 simulator trajectories with 169,912 synchronized frames. They use four frames from three calibrated views, with either RGBD or RGB inputs, to estimate part poses, hinge geometry, lid opening angle, and interaction points. Ground-truth geometry from the simulator supplies the training labels; it is not an input to the encoder.
The residual policies were trained online in RoboTwin with Soft Actor-Critic (SAC) while ACT remained frozen. The visual route also kept its pretrained encoder frozen. Training data are not included in this release.
Evaluation
Task success in RoboTwin
Task success rate over 500 episodes per policy: five fixed sets of 100 scene seeds, with up to 700 decisions per episode. These results use demo_clean, not demo_randomized. The 150- and 200-decision caps in the table apply to training, not evaluation.
| Policy weights | Configuration | Task success rate |
|---|---|---|
act-stride-two.pt |
ACT only | 54.4% |
privileged-state-200.pt |
Privileged-state residual, 200-decision training cap | 99.0% |
privileged-state-150.pt |
Privileged-state residual, 150-decision training cap | 99.6% |
visual-state.pt |
Visual residual with the paired encoder | 95.4% |
Offline geometry estimation
Encoder errors are measured on a validation set of 100 fixed scenes, disjoint from training, with 1,600 four-frame windows in total. These scenes were used for model selection. The errors measure geometry prediction, not task success.
| Encoder weights | Camera input | Mean operation-point error | Mean absolute lid-angle error |
|---|---|---|---|
encoder-visual-policy.pt |
128×128 RGBD, paired with visual-state.pt |
5.41 mm | 1.16° |
encoder-rgbd-320x240.pt |
320×240 RGBD, standalone | 2.41 mm | 0.67° |
encoder-rgb-320x240.pt |
320×240 RGB, standalone | 5.12 mm | 1.75° |
References
- Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware — ACT base policy.
- Soft Actor-Critic — off-policy algorithm used to train the residual policies.
- Residual Off-Policy RL for Finetuning Behavior Cloning Policies — reference for residual action composition.
- RoboTwin 2.0 — simulator, task assets, and benchmark.
- Downloads last month
- 2