SO101 ACT pick-and-place: 25,000-step simulation checkpoint

Stock Hugging Face LeRobot 0.4.4 ACT architecture trained from fresh ACT initialization (with the configured ImageNet ResNet18 backbone) for 25,000 optimizer steps on one NVIDIA L40S.

  • Data: 600 clean and 200 perturbed/recovery scripted-IK simulation episodes; 133,400 frames at 30 Hz.
  • Full official appearance, placement and physics randomization; recovery data uses broader ranges.
  • No real-world demonstrations in this training run.
  • Task instruction: Put the cube on the tray. Cube side: 25 mm.
  • Inputs: scene and wrist RGB images, 640 x 480, and six joint positions.
  • Actions: absolute joint targets; five arm joints in degrees, gripper in 0..100 interface units.
  • Batch 8; learning rate 1e-5; chunk size 30; training n_action_steps 30; KL weight 10; seed 2026.
  • ResNet18; hidden 512; feedforward 3200; four encoder layers, one decoder layer, eight heads.
  • Checkpoints retained every 5,000 steps. This repository contains the 25,000-step weights.

Inference

Keep the complete processor/configuration files alongside the model weights. The supplied custom policy.py wraps the stock ACT model. Its default inference uses overlapping chunks with temporal ensemble coefficient 0.01 and one output action per query, bounds arm targets to +/-175 degrees, bounds the gripper to 5..45 units, rejects non-finite outputs, and leaves optional gripper snapping off. These deployment settings require measured validation and do not guarantee hardware safety. Optional inference.json can select temporal_ensemble_coeff, n_action_steps, gripper_min, gripper_max, and snap_gripper. A null coefficient disables ensembling.

For later weight fine-tuning use lerobot-train --policy.path=CHECKPOINT_DIRECTORY with a new output directory and a compatible dataset; this does not by itself resume the previous optimizer state. Keep the chunk-size architecture compatible.

Evaluation and provenance

Completed local baseline evaluation: 26/30 successes (86.7%), with pooled mean successful completion time 6.1474 seconds. This missed the preparation target of at least 28/30 successes and mean time below 5.5 seconds.

Seed Successes Mean successful time
2026 7/10 6.79 s
2027 9/10 5.63 s
2028 10/10 6.17 s

These are local practice results, not official qualification. A repeatability run with the same weights, inference settings and seeds is underway; the first baseline remains preserved under evaluation/baseline_30_rollouts/. Training loss is not rollout success. This is a private preparation checkpoint, not an official competition submission. Eligibility of pre-event work must be confirmed with the organisers.

ACT: Tony Z. Zhao, Vikash Kumar, Sergey Levine and Chelsea Finn, Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (2023). Implementation: Hugging Face LeRobot. Simulation task: NWaves SO-101 hackathon.

Downloads last month
4
Safetensors
Model size
51.6M params
Tensor type
F32
ยท
Video Preview
loading