omx_tomato_act

ACT (Action Chunking with Transformers) policy trained with LeRobot 0.6.1. A ROBOTIS OMX follower arm picks up a tomato and puts it into a bin, using two USB cameras (top and wrist).

OMX ํŒ”๋กœ์›Œ๋กœ ํ† ๋งˆํ† ๋ฅผ ์ง‘์–ด ํ†ต์— ๋„ฃ๋Š” ACT ์ •์ฑ…์ด๋‹ค. ์ˆ˜์—… ์‹ค์Šต์šฉ ์†Œ๊ทœ๋ชจ ๋ชจ๋ธ์ด๋‹ค.

Summary

Item Value
Policy ACT, resnet18 backbone, chunk_size 100, VAE on
Training data minodori/omx_tomato (50 episodes, 34,432 frames, 30 fps)
Training 30,000 steps, batch size 8, seed 1000
Inputs observation.state (6 joints), observation.images.top, observation.images.wrist (640x480, 30 fps)
Output action (6 joints: pan, lift, elbow, wrist_flex, wrist_roll, gripper)
Hardware OMX leader (teleoperation) and OMX follower, 2 USB cameras

The tomato was placed at 5 positions during recording (10 episodes each).

Usage

from lerobot.policies.act.modeling_act import ACTPolicy

policy = ACTPolicy.from_pretrained("minodori/omx_tomato_act")

For real-robot rollout, enable temporal ensembling. The saved config has it off (chunk_size 100 = 3.3 s at 30 fps), and without it the arm jumps at every chunk boundary. Pass these options to the rollout command:

--policy.temporal_ensemble_coeff=0.01 --policy.n_action_steps=1

Temporal ensembling runs inference every step, so it needs a GPU (about 15 ms per inference on an RTX 5060 Laptop; about 200 ms on CPU, too slow for a 30 fps loop).

Limitations (what we saw on the real robot)

This model is sensitive to how the deployment setup differs from the recording.

  • Top camera position. A few pixels of shift changed where the gripper went. We aligned the live view over a recorded first frame before each test.
  • Start pose. Starting from a hand-placed home pose that differed from the demonstrations caused a swing of about 5 degrees at the start. Match the recorded start pose (mean of the 50 demos: pan 2.4, lift -63.7, elbow 55.7, wrist_flex 45.0, wrist_roll 1.6, gripper 50.6).
  • Background in the top image. A hand in view, the cable route, and the background on the right side shifted the target to the right by 10 to 30 mm.
  • Position results (final settings: step 30000, temporal ensembling on, start pose matched, camera aligned, background and chair position matched to the recording, hands out of view). Small trial counts; treat as anecdotal.
Position Result
1 (center) gripper lands slightly right of the tomato (under 20 mm, not measured precisely); the wide-open gripper still grasps it in most trials; 4 of 5 grasps succeeded
3 about 10 mm right; succeeded
4 about 10 mm right; 3 of 3 succeeded
5, 6 almost no offset; succeeded
untrained points between 1-3 and 1-6 succeeded

Not tested: other objects, other tables, other lighting, a different camera mount. Wrist roll barely varies in the training data, so rotated objects will likely fail.

Demo videos

Real-robot rollouts with the final settings (1080p, about 20 seconds each, filmed by hand, so the view moves):

The files are in the assets/ folder of this repository. They are HEVC (H.265) mp4 files. Some browsers cannot play HEVC; if the link does not play, download the file and open it with a video player.

Reproduce

Train with the dataset above using lerobot-train (policy type act, 30,000 steps, batch size 8). The scripts used for recording, training, and rollout are in the GitHub repository minodori/omx-tomato-act.

Downloads last month
46
Safetensors
Model size
51.7M params
Tensor type
F32
ยท
Video Preview
loading

Dataset used to train minodori/omx_tomato_act