license: apache-2.0
language:
- en
- zh
tags:
- robotics
- umi
- bimanual
- openpi
- flux
- inference
- lerobot-v3
- cloth-folding
- robot-rollout
iUMI Inference · Bimanual UMI Inference and Deployment
English · 简体中文
Model inference and robot deployment code for Crobotia iUMI, with OpenPI (π0.5) and FLUX backends, configuration templates, launch scripts and a nominal dual-arm URDF.
Demos and sample rollouts
This repository includes OpenPI π0.5 and FLUX cloth-folding inference checkpoints, deployment code, two demo videos and one robot rollout per backend. The task is fold_the_clothes on a dual-arm Flexiv Rizon 4 setup with left/right ego cameras.
| Backend | Muted presentation edit | Complete rollout | Frames at 30 Hz | Recording duration |
|---|---|---|---|---|
| OpenPI π0.5 | 4 min 15.4 sec | openpi_pi05 | 6,540 | 218 sec |
| FLUX | 4 min 29.7 sec | flux | 30,619 | 1,020.633 sec |
OpenPI π0.5
FLUX
Presentation edits have their own timelines; they are not frame-aligned to the datasets. The two rollouts contain 37,159 frames / 20 min 38.6 sec in total. Each retains left/right 960×960 / 30 Hz AV1 video, 16-D states/actions, joint data, bilateral IMU, calibration and robot H5 records. Audio and rear-camera views are excluded.
Download only the demos and rollouts, without the approximately 21.37 GB of model files:
pip install -U huggingface_hub
hf download Crobotia/iUMI_Inference --local-dir iUMI_Inference --include "demos/*" "rollout_data/*"
The media and rollout files total approximately 2.35 GB. Read rollout_data/openpi_pi05/ and rollout_data/flux/ as separate LeRobot v3 roots; both start at episode 0. See the rollout guide and demo guide. These assets live in this model repository; downloads use the default repo_type="model".
Intended use and model interface
The backends support bimanual UMI cloth-folding research, offline integration and deployment validation on a matching setup. Inference consumes both wrist images, previous/current poses and task text. The matching adapter restores predicted action chunks to robot target poses.
| Item | Convention |
|---|---|
| Recorded state / action | 16-D, left then right; each hand [x, y, z, qw, qx, qy, qz, gripper]; metres / wxyz / gripper degrees |
| Rollout action meaning | action_mode=next_state: action[t] = observation.state[t+1], excluding the last frame; not raw policy predictions |
| Internal UMI representation | 20-D total: translation 3 + first two rotation columns 6 + gripper 1 per hand |
| FLUX action chunk | 32 steps at 30 Hz, using both wrist images |
| π0.5 action chunk | 50 model steps; a 32-D model head padded from 20-D UMI targets, restored to dual-arm target poses by the adapter |
Recorded pose provenance is robot_fk_cad. Camera optical poses, robot flange poses and model-relative representations require explicit conversion. See the frame and input/output contract.
Training provenance, evaluation and limitations
The released FLUX checkpoint is the step-14418 BF16 inference export; π0.5 is the epoch-2 pi05_umi_v3 checkpoint. A complete training corpus inventory, split definition and exact run configuration are not supplied with these added assets. The two rollouts are execution examples and do not identify the training set.
The new assets pass file-hash checks, video stream/packet counts, Parquet index and next-state checks, IMU checks and H5 structure inspection; see rollout validation. There is one example per model, with no benchmark protocol, task-success measurement or complete motion QC. Exact checkpoint hashes and sampling settings used for the recordings, and frame mappings between presentation edits and datasets, were not supplied. Current launcher defaults therefore do not constitute an exact reproduction recipe for the videos.
Existing checkpoint/source validation remains documented in validation scope. This upload does not rerun GPU inference or the robot. Physical deployment requires matching encoder assets, vendor SDKs, device calibration, workspace constraints and on-site validation.
Get the code
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/Crobotia/iUMI_Inference
cd iUMI_Inference
| Directory | Purpose |
|---|---|
| flexiv-umi-infer/ | Policy service, robot/camera/gripper interfaces and recording |
| flux-umi/ | FLUX models and UMI representation |
| openpi-umi/ | OpenPI models, data transforms and client |
| scripts/ | Policy-server and robot-agent launchers |
| configs/ | Inference configuration templates |
| checkpoints/ | FLUX BF16 and π0.5 cloth-folding inference weights |
| demos/ | Two muted cloth-folding demo videos |
| rollout_data/ | One complete execution recording per backend |
For OpenPI, use Linux (glibc ≥2.31) and Python 3.12 as pinned in openpi-umi/.python-version and run uv sync --frozen --python 3.12 from that directory.
Launch FLUX with scripts/run_flux_server.sh, OpenPI with scripts/run_pi05_server.sh, and the robot agent with scripts/run_robot.sh. Use separate environments for the model backends and follow each project's dependency instructions. Configure model resource paths and cell calibration before launch. The robot script defaults to dry run.
Before running, prepare required encoders, vendor SDKs and cell calibration. See docs/INFERENCE_CONTRACT.md for input/output conventions and the subproject READMEs for installation and device configuration.
Service deployment
Each service process selects one backend. Both launchers run umi_infer.policy_server: FLUX loads through flux-umi, while π0.5 loads through openpi-umi. The UMI service wrappers live under flexiv-umi-infer/. On the Flexiv host, robot_agent sends observations and receives actions over WebSocket without loading model weights. Input encoding, normalization and action-frame restoration run in the server wrapper; servo execution runs locally on the robot host.
For separate hosts, set SERVER_HOST=0.0.0.0 and SERVER_PORT=8765 on the GPU host, then set SERVER_URI=ws://<GPU-host>:8765 on the Flexiv host and run bash scripts/run_robot.sh. Use different ports for simultaneous FLUX and π0.5 servers. Make UMI_INFER_DATA calibration files available on the relevant hosts. The robot launcher defaults to dry run.
Use the matching UMI server/client pair. The native OpenPI WebSocket service uses a different handshake and message envelope and cannot directly serve this robot_agent.
Download weights and launch
Both inference checkpoints are included under checkpoints/ (about 21.37 GB total). From the repository root:
pip install -U huggingface_hub
hf download Crobotia/iUMI_Inference --local-dir . --include "checkpoints/*"
| Backend | Default checkpoint directory | Model-file size |
|---|---|---|
| FLUX | checkpoints/flux_fold_clothes_bf16/ |
13,894,466,592 B |
| π0.5 | checkpoints/pi05_fold_clothes/epoch_2/ |
7,473,091,464 B |
The launchers use these directories automatically; override FLUX_EXPORT_DIR or PI05_CHECKPOINT_DIR if needed. Install the backend dependencies, then copy and edit the configuration:
cp configs/inference.env.example .env
# Set FLUX_RESOURCES, TOKENIZER_DIR, UMI_INFER_DATA and the Python executable paths.
set -a
source .env
set +a
bash scripts/run_flux_server.sh
# Or from the π0.5 environment: bash scripts/run_pi05_server.sh
Provide FLUX_RESOURCES/video_vae.safetensors and FLUX_RESOURCES/text_encoder/ separately. π0.5 requires TOKENIZER_DIR/paligemma-3b-pt-224/tokenizer.model. These external resources are not included in this weight release. Checkpoint-specific normalization statistics are included and loaded automatically.
The FLUX export has no saved sampling preset. The launcher uses Euler, 4 steps, shift 42 and video/action guidance 1.0; these are adjustable launcher defaults, with cloth-folding quality still unverified. π0.5 uses 10 denoising steps by default. Keep VLA_QUAT_PERM=0 for this checkpoint's xyz_quat_wxyz pose configuration.
Download integrity, file structure and sanitization have been checked. CPU checks do not establish GPU inference or physical-robot performance. See FLUX, π0.5 and validation.
License
Source code is licensed under Apache-2.0, with upstream notices retained in each component. Gemma, model resources and vendor SDKs retain their respective terms. See LICENSES.md.
No separate media/data license declaration accompanies the added demos and rollouts. The root Apache-2.0 tag describes source code and does not automatically license media, data or model resources on the same terms. Confirm applicable use and redistribution terms with Crobotia; licenses of separate human-demonstration datasets are not applied here.