Make model card English by default with a Simplified Chinese README link

1718854 verified 3 days ago

9.5 kB

---
license: apache-2.0
language:
  - en
  - zh
tags:
  - robotics
  - umi
  - bimanual
  - openpi
  - flux
  - inference
  - lerobot-v3
  - cloth-folding
  - robot-rollout
---
# iUMI Inference · Bimanual UMI Inference and Deployment

**English** · [简体中文](README_ZH.md)

Model inference and robot deployment code for Crobotia iUMI, with OpenPI (π0.5) and FLUX backends, configuration templates, launch scripts and a nominal dual-arm URDF.

## Demos and sample rollouts

This repository includes OpenPI π0.5 and FLUX cloth-folding inference checkpoints, deployment code, two demo videos and one robot rollout per backend. The task is `fold_the_clothes` on a dual-arm Flexiv Rizon 4 setup with left/right ego cameras.

| Backend | Muted presentation edit | Complete rollout | Frames at 30 Hz | Recording duration |
| --- | --- | --- | ---: | ---: |
| OpenPI π0.5 | [4 min 15.4 sec](demos/openpi_fold_demo.mp4) | [openpi_pi05](rollout_data/openpi_pi05) | 6,540 | 218 sec |
| FLUX | [4 min 29.7 sec](demos/flux_fold_demo.mp4) | [flux](rollout_data/flux) | 30,619 | 1,020.633 sec |

**OpenPI π0.5**

<video src="https://huggingface.co/Crobotia/iUMI_Inference/resolve/main/demos/openpi_fold_demo.mp4" controls preload="metadata" width="100%"></video>

**FLUX**

<video src="https://huggingface.co/Crobotia/iUMI_Inference/resolve/main/demos/flux_fold_demo.mp4" controls preload="metadata" width="100%"></video>

Presentation edits have their own timelines; they are not frame-aligned to the datasets. The two rollouts contain 37,159 frames / 20 min 38.6 sec in total. Each retains left/right 960×960 / 30 Hz AV1 video, 16-D states/actions, joint data, bilateral IMU, calibration and robot H5 records. Audio and rear-camera views are excluded.

Download only the demos and rollouts, without the approximately 21.37 GB of model files:

```bash
pip install -U huggingface_hub
hf download Crobotia/iUMI_Inference --local-dir iUMI_Inference --include "demos/*" "rollout_data/*"
```

The media and rollout files total approximately 2.35 GB. Read `rollout_data/openpi_pi05/` and `rollout_data/flux/` as separate LeRobot v3 roots; both start at episode 0. See the [rollout guide](rollout_data/README.md) and [demo guide](demos/README.md). These assets live in this model repository; downloads use the default `repo_type="model"`.

## Intended use and model interface

The backends support bimanual UMI cloth-folding research, offline integration and deployment validation on a matching setup. Inference consumes both wrist images, previous/current poses and task text. The matching adapter restores predicted action chunks to robot target poses.

| Item | Convention |
| --- | --- |
| Recorded state / action | 16-D, left then right; each hand `[x, y, z, qw, qx, qy, qz, gripper]`; metres / `wxyz` / gripper degrees |
| Rollout action meaning | `action_mode=next_state`: `action[t] = observation.state[t+1]`, excluding the last frame; not raw policy predictions |
| Internal UMI representation | 20-D total: translation 3 + first two rotation columns 6 + gripper 1 per hand |
| FLUX action chunk | 32 steps at 30 Hz, using both wrist images |
| π0.5 action chunk | 50 model steps; a 32-D model head padded from 20-D UMI targets, restored to dual-arm target poses by the adapter |

Recorded pose provenance is `robot_fk_cad`. Camera optical poses, robot flange poses and model-relative representations require explicit conversion. See the [frame and input/output contract](docs/INFERENCE_CONTRACT.md).

## Training provenance, evaluation and limitations

The released FLUX checkpoint is the step-14418 BF16 inference export; π0.5 is the epoch-2 `pi05_umi_v3` checkpoint. A complete training corpus inventory, split definition and exact run configuration are not supplied with these added assets. The two rollouts are execution examples and do not identify the training set.

The new assets pass file-hash checks, video stream/packet counts, Parquet index and next-state checks, IMU checks and H5 structure inspection; see [rollout validation](docs/ROLLOUT_VALIDATION.md). There is one example per model, with no benchmark protocol, task-success measurement or complete motion QC. Exact checkpoint hashes and sampling settings used for the recordings, and frame mappings between presentation edits and datasets, were not supplied. Current launcher defaults therefore do not constitute an exact reproduction recipe for the videos.

Existing checkpoint/source validation remains documented in [validation scope](docs/VALIDATION.md). This upload does not rerun GPU inference or the robot. Physical deployment requires matching encoder assets, vendor SDKs, device calibration, workspace constraints and on-site validation.

## Get the code

```bash
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/Crobotia/iUMI_Inference
cd iUMI_Inference
```

| Directory | Purpose |
| --- | --- |
| flexiv-umi-infer/ | Policy service, robot/camera/gripper interfaces and recording |
| flux-umi/ | FLUX models and UMI representation |
| openpi-umi/ | OpenPI models, data transforms and client |
| scripts/ | Policy-server and robot-agent launchers |
| configs/ | Inference configuration templates |
| checkpoints/ | FLUX BF16 and π0.5 cloth-folding inference weights |
| demos/ | Two muted cloth-folding demo videos |
| rollout_data/ | One complete execution recording per backend |
For OpenPI, use Linux (glibc ≥2.31) and Python 3.12 as pinned in `openpi-umi/.python-version` and run `uv sync --frozen --python 3.12` from that directory.
Launch FLUX with `scripts/run_flux_server.sh`, OpenPI with `scripts/run_pi05_server.sh`, and the robot agent with `scripts/run_robot.sh`. Use separate environments for the model backends and follow each project's dependency instructions. Configure model resource paths and cell calibration before launch. The robot script defaults to dry run.

Before running, prepare required encoders, vendor SDKs and cell calibration. See `docs/INFERENCE_CONTRACT.md` for input/output conventions and the subproject READMEs for installation and device configuration.

## Service deployment

Each service process selects one backend. Both launchers run `umi_infer.policy_server`: FLUX loads through `flux-umi`, while π0.5 loads through `openpi-umi`. The UMI service wrappers live under `flexiv-umi-infer/`. On the Flexiv host, `robot_agent` sends observations and receives actions over WebSocket without loading model weights. Input encoding, normalization and action-frame restoration run in the server wrapper; servo execution runs locally on the robot host.

For separate hosts, set `SERVER_HOST=0.0.0.0` and `SERVER_PORT=8765` on the GPU host, then set `SERVER_URI=ws://<GPU-host>:8765` on the Flexiv host and run `bash scripts/run_robot.sh`. Use different ports for simultaneous FLUX and π0.5 servers. Make `UMI_INFER_DATA` calibration files available on the relevant hosts. The robot launcher defaults to dry run.

Use the matching UMI server/client pair. The native OpenPI WebSocket service uses a different handshake and message envelope and cannot directly serve this `robot_agent`.

## Download weights and launch

Both inference checkpoints are included under `checkpoints/` (about 21.37 GB total). From the repository root:

```bash
pip install -U huggingface_hub
hf download Crobotia/iUMI_Inference --local-dir . --include "checkpoints/*"
```

| Backend | Default checkpoint directory | Model-file size |
| --- | --- | --- |
| FLUX | `checkpoints/flux_fold_clothes_bf16/` | 13,894,466,592 B |
| π0.5 | `checkpoints/pi05_fold_clothes/epoch_2/` | 7,473,091,464 B |

The launchers use these directories automatically; override `FLUX_EXPORT_DIR` or `PI05_CHECKPOINT_DIR` if needed. Install the backend dependencies, then copy and edit the configuration:

```bash
cp configs/inference.env.example .env
# Set FLUX_RESOURCES, TOKENIZER_DIR, UMI_INFER_DATA and the Python executable paths.
set -a
source .env
set +a
bash scripts/run_flux_server.sh
# Or from the π0.5 environment: bash scripts/run_pi05_server.sh
```

Provide `FLUX_RESOURCES/video_vae.safetensors` and `FLUX_RESOURCES/text_encoder/` separately. π0.5 requires `TOKENIZER_DIR/paligemma-3b-pt-224/tokenizer.model`. These external resources are not included in this weight release. Checkpoint-specific normalization statistics are included and loaded automatically.

The FLUX export has no saved sampling preset. The launcher uses Euler, 4 steps, shift 42 and video/action guidance 1.0; these are adjustable launcher defaults, with cloth-folding quality still unverified. π0.5 uses 10 denoising steps by default. Keep `VLA_QUAT_PERM=0` for this checkpoint's `xyz_quat_wxyz` pose configuration.

Download integrity, file structure and sanitization have been checked. CPU checks do not establish GPU inference or physical-robot performance. See [FLUX](docs/CHECKPOINTS_FLUX.md), [π0.5](docs/CHECKPOINTS_PI05.md) and [validation](docs/VALIDATION.md).

## License

Source code is licensed under Apache-2.0, with upstream notices retained in each component. Gemma, model resources and vendor SDKs retain their respective terms. See [LICENSES.md](LICENSES.md).

No separate media/data license declaration accompanies the added demos and rollouts. The root Apache-2.0 tag describes source code and does not automatically license media, data or model resources on the same terms. Confirm applicable use and redistribution terms with Crobotia; licenses of separate human-demonstration datasets are not applied here.

	---
	license: apache-2.0
	language:
	- en
	- zh
	tags:
	- robotics
	- umi
	- bimanual
	- openpi
	- flux
	- inference
	- lerobot-v3
	- cloth-folding
	- robot-rollout
	---

	# iUMI Inference · Bimanual UMI Inference and Deployment

	English · [简体中文](README_ZH.md)

	Model inference and robot deployment code for Crobotia iUMI, with OpenPI (π0.5) and FLUX backends, configuration templates, launch scripts and a nominal dual-arm URDF.

	## Demos and sample rollouts

	This repository includes OpenPI π0.5 and FLUX cloth-folding inference checkpoints, deployment code, two demo videos and one robot rollout per backend. The task is `fold_the_clothes` on a dual-arm Flexiv Rizon 4 setup with left/right ego cameras.

	\| Backend \| Muted presentation edit \| Complete rollout \| Frames at 30 Hz \| Recording duration \|
	\| --- \| --- \| --- \| ---: \| ---: \|
	\| OpenPI π0.5 \| [4 min 15.4 sec](demos/openpi_fold_demo.mp4) \| [openpi_pi05](rollout_data/openpi_pi05) \| 6,540 \| 218 sec \|
	\| FLUX \| [4 min 29.7 sec](demos/flux_fold_demo.mp4) \| [flux](rollout_data/flux) \| 30,619 \| 1,020.633 sec \|

	OpenPI π0.5

	<video src="https://huggingface.co/Crobotia/iUMI_Inference/resolve/main/demos/openpi_fold_demo.mp4" controls preload="metadata" width="100%"></video>

	FLUX

	<video src="https://huggingface.co/Crobotia/iUMI_Inference/resolve/main/demos/flux_fold_demo.mp4" controls preload="metadata" width="100%"></video>

	Presentation edits have their own timelines; they are not frame-aligned to the datasets. The two rollouts contain 37,159 frames / 20 min 38.6 sec in total. Each retains left/right 960×960 / 30 Hz AV1 video, 16-D states/actions, joint data, bilateral IMU, calibration and robot H5 records. Audio and rear-camera views are excluded.

	Download only the demos and rollouts, without the approximately 21.37 GB of model files:

	```bash
	pip install -U huggingface_hub
	hf download Crobotia/iUMI_Inference --local-dir iUMI_Inference --include "demos/" "rollout_data/"
	```

	The media and rollout files total approximately 2.35 GB. Read `rollout_data/openpi_pi05/` and `rollout_data/flux/` as separate LeRobot v3 roots; both start at episode 0. See the [rollout guide](rollout_data/README.md) and [demo guide](demos/README.md). These assets live in this model repository; downloads use the default `repo_type="model"`.

	## Intended use and model interface

	The backends support bimanual UMI cloth-folding research, offline integration and deployment validation on a matching setup. Inference consumes both wrist images, previous/current poses and task text. The matching adapter restores predicted action chunks to robot target poses.

	\| Item \| Convention \|
	\| --- \| --- \|
	\| Recorded state / action \| 16-D, left then right; each hand `[x, y, z, qw, qx, qy, qz, gripper]`; metres / `wxyz` / gripper degrees \|
	\| Rollout action meaning \| `action_mode=next_state`: `action[t] = observation.state[t+1]`, excluding the last frame; not raw policy predictions \|
	\| Internal UMI representation \| 20-D total: translation 3 + first two rotation columns 6 + gripper 1 per hand \|
	\| FLUX action chunk \| 32 steps at 30 Hz, using both wrist images \|
	\| π0.5 action chunk \| 50 model steps; a 32-D model head padded from 20-D UMI targets, restored to dual-arm target poses by the adapter \|

	Recorded pose provenance is `robot_fk_cad`. Camera optical poses, robot flange poses and model-relative representations require explicit conversion. See the [frame and input/output contract](docs/INFERENCE_CONTRACT.md).

	## Training provenance, evaluation and limitations

	The released FLUX checkpoint is the step-14418 BF16 inference export; π0.5 is the epoch-2 `pi05_umi_v3` checkpoint. A complete training corpus inventory, split definition and exact run configuration are not supplied with these added assets. The two rollouts are execution examples and do not identify the training set.

	The new assets pass file-hash checks, video stream/packet counts, Parquet index and next-state checks, IMU checks and H5 structure inspection; see [rollout validation](docs/ROLLOUT_VALIDATION.md). There is one example per model, with no benchmark protocol, task-success measurement or complete motion QC. Exact checkpoint hashes and sampling settings used for the recordings, and frame mappings between presentation edits and datasets, were not supplied. Current launcher defaults therefore do not constitute an exact reproduction recipe for the videos.

	Existing checkpoint/source validation remains documented in [validation scope](docs/VALIDATION.md). This upload does not rerun GPU inference or the robot. Physical deployment requires matching encoder assets, vendor SDKs, device calibration, workspace constraints and on-site validation.

	## Get the code

	```bash
	GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/Crobotia/iUMI_Inference
	cd iUMI_Inference
	```

	\| Directory \| Purpose \|
	\| --- \| --- \|
	\| flexiv-umi-infer/ \| Policy service, robot/camera/gripper interfaces and recording \|
	\| flux-umi/ \| FLUX models and UMI representation \|
	\| openpi-umi/ \| OpenPI models, data transforms and client \|
	\| scripts/ \| Policy-server and robot-agent launchers \|
	\| configs/ \| Inference configuration templates \|
	\| checkpoints/ \| FLUX BF16 and π0.5 cloth-folding inference weights \|
	\| demos/ \| Two muted cloth-folding demo videos \|
	\| rollout_data/ \| One complete execution recording per backend \|

	For OpenPI, use Linux (glibc ≥2.31) and Python 3.12 as pinned in `openpi-umi/.python-version` and run `uv sync --frozen --python 3.12` from that directory.

	Launch FLUX with `scripts/run_flux_server.sh`, OpenPI with `scripts/run_pi05_server.sh`, and the robot agent with `scripts/run_robot.sh`. Use separate environments for the model backends and follow each project's dependency instructions. Configure model resource paths and cell calibration before launch. The robot script defaults to dry run.

	Before running, prepare required encoders, vendor SDKs and cell calibration. See `docs/INFERENCE_CONTRACT.md` for input/output conventions and the subproject READMEs for installation and device configuration.

	## Service deployment

	Each service process selects one backend. Both launchers run `umi_infer.policy_server`: FLUX loads through `flux-umi`, while π0.5 loads through `openpi-umi`. The UMI service wrappers live under `flexiv-umi-infer/`. On the Flexiv host, `robot_agent` sends observations and receives actions over WebSocket without loading model weights. Input encoding, normalization and action-frame restoration run in the server wrapper; servo execution runs locally on the robot host.

	For separate hosts, set `SERVER_HOST=0.0.0.0` and `SERVER_PORT=8765` on the GPU host, then set `SERVER_URI=ws://<GPU-host>:8765` on the Flexiv host and run `bash scripts/run_robot.sh`. Use different ports for simultaneous FLUX and π0.5 servers. Make `UMI_INFER_DATA` calibration files available on the relevant hosts. The robot launcher defaults to dry run.

	Use the matching UMI server/client pair. The native OpenPI WebSocket service uses a different handshake and message envelope and cannot directly serve this `robot_agent`.

	## Download weights and launch

	Both inference checkpoints are included under `checkpoints/` (about 21.37 GB total). From the repository root:

	```bash
	pip install -U huggingface_hub
	hf download Crobotia/iUMI_Inference --local-dir . --include "checkpoints/*"
	```

	\| Backend \| Default checkpoint directory \| Model-file size \|
	\| --- \| --- \| --- \|
	\| FLUX \| `checkpoints/flux_fold_clothes_bf16/` \| 13,894,466,592 B \|
	\| π0.5 \| `checkpoints/pi05_fold_clothes/epoch_2/` \| 7,473,091,464 B \|

	The launchers use these directories automatically; override `FLUX_EXPORT_DIR` or `PI05_CHECKPOINT_DIR` if needed. Install the backend dependencies, then copy and edit the configuration:

	```bash
	cp configs/inference.env.example .env
	# Set FLUX_RESOURCES, TOKENIZER_DIR, UMI_INFER_DATA and the Python executable paths.
	set -a
	source .env
	set +a
	bash scripts/run_flux_server.sh
	# Or from the π0.5 environment: bash scripts/run_pi05_server.sh
	```

	Provide `FLUX_RESOURCES/video_vae.safetensors` and `FLUX_RESOURCES/text_encoder/` separately. π0.5 requires `TOKENIZER_DIR/paligemma-3b-pt-224/tokenizer.model`. These external resources are not included in this weight release. Checkpoint-specific normalization statistics are included and loaded automatically.

	The FLUX export has no saved sampling preset. The launcher uses Euler, 4 steps, shift 42 and video/action guidance 1.0; these are adjustable launcher defaults, with cloth-folding quality still unverified. π0.5 uses 10 denoising steps by default. Keep `VLA_QUAT_PERM=0` for this checkpoint's `xyz_quat_wxyz` pose configuration.

	Download integrity, file structure and sanitization have been checked. CPU checks do not establish GPU inference or physical-robot performance. See [FLUX](docs/CHECKPOINTS_FLUX.md), [π0.5](docs/CHECKPOINTS_PI05.md) and [validation](docs/VALIDATION.md).

	## License

	Source code is licensed under Apache-2.0, with upstream notices retained in each component. Gemma, model resources and vendor SDKs retain their respective terms. See [LICENSES.md](LICENSES.md).

	No separate media/data license declaration accompanies the added demos and rollouts. The root Apache-2.0 tag describes source code and does not automatically license media, data or model resources on the same terms. Confirm applicable use and redistribution terms with Crobotia; licenses of separate human-demonstration datasets are not applied here.