Instructions to use minodori/omx_tomato_act with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LeRobot
How to use minodori/omx_tomato_act with LeRobot:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
omx_tomato_act
ACT (Action Chunking with Transformers) policy trained with LeRobot 0.6.1. A ROBOTIS OMX follower arm picks up a tomato and puts it into a bin, using two USB cameras (top and wrist).
OMX ํ๋ก์๋ก ํ ๋งํ ๋ฅผ ์ง์ด ํต์ ๋ฃ๋ ACT ์ ์ฑ ์ด๋ค. ์์ ์ค์ต์ฉ ์๊ท๋ชจ ๋ชจ๋ธ์ด๋ค.
Summary
| Item | Value |
|---|---|
| Policy | ACT, resnet18 backbone, chunk_size 100, VAE on |
| Training data | minodori/omx_tomato (50 episodes, 34,432 frames, 30 fps) |
| Training | 30,000 steps, batch size 8, seed 1000 |
| Inputs | observation.state (6 joints), observation.images.top, observation.images.wrist (640x480, 30 fps) |
| Output | action (6 joints: pan, lift, elbow, wrist_flex, wrist_roll, gripper) |
| Hardware | OMX leader (teleoperation) and OMX follower, 2 USB cameras |
The tomato was placed at 5 positions during recording (10 episodes each).
Usage
from lerobot.policies.act.modeling_act import ACTPolicy
policy = ACTPolicy.from_pretrained("minodori/omx_tomato_act")
For real-robot rollout, enable temporal ensembling. The saved config has it off (chunk_size 100 = 3.3 s at 30 fps), and without it the arm jumps at every chunk boundary. Pass these options to the rollout command:
--policy.temporal_ensemble_coeff=0.01 --policy.n_action_steps=1
Temporal ensembling runs inference every step, so it needs a GPU (about 15 ms per inference on an RTX 5060 Laptop; about 200 ms on CPU, too slow for a 30 fps loop).
Limitations (what we saw on the real robot)
This model is sensitive to how the deployment setup differs from the recording.
- Top camera position. A few pixels of shift changed where the gripper went. We aligned the live view over a recorded first frame before each test.
- Start pose. Starting from a hand-placed home pose that differed from the demonstrations caused a swing of about 5 degrees at the start. Match the recorded start pose (mean of the 50 demos: pan 2.4, lift -63.7, elbow 55.7, wrist_flex 45.0, wrist_roll 1.6, gripper 50.6).
- Background in the top image. A hand in view, the cable route, and the background on the right side shifted the target to the right by 10 to 30 mm.
- Position results (final settings: step 30000, temporal ensembling on, start pose matched, camera aligned, background and chair position matched to the recording, hands out of view). Small trial counts; treat as anecdotal.
| Position | Result |
|---|---|
| 1 (center) | gripper lands slightly right of the tomato (under 20 mm, not measured precisely); the wide-open gripper still grasps it in most trials; 4 of 5 grasps succeeded |
| 3 | about 10 mm right; succeeded |
| 4 | about 10 mm right; 3 of 3 succeeded |
| 5, 6 | almost no offset; succeeded |
| untrained points between 1-3 and 1-6 | succeeded |
Not tested: other objects, other tables, other lighting, a different camera mount. Wrist roll barely varies in the training data, so rotated objects will likely fail.
Demo videos
Real-robot rollouts with the final settings (1080p, about 20 seconds each, filmed by hand, so the view moves):
The files are in the assets/ folder of this repository. They are HEVC (H.265)
mp4 files. Some browsers cannot play HEVC; if the link does not play, download
the file and open it with a video player.
Reproduce
Train with the dataset above using lerobot-train (policy type act,
30,000 steps, batch size 8). The scripts used for recording, training, and
rollout are in the GitHub repository
minodori/omx-tomato-act.
- Downloads last month
- 46