Make model card English by default with a Simplified Chinese README link
Browse filesMove the Chinese translation to README_ZH.md with reciprocal language links. Keep both demo players on the English model card, preserve model metadata, and synchronize checksums.
- README.md +10 -139
- README_ZH.md +132 -0
- checksums.sha256 +2 -1
README.md
CHANGED
|
@@ -15,24 +15,20 @@ tags:
|
|
| 15 |
- robot-rollout
|
| 16 |
---
|
| 17 |
|
| 18 |
-
#
|
| 19 |
-
|
| 20 |
-
[中文](#zh) · [English](#en)
|
| 21 |
-
|
| 22 |
-
<a id="zh"></a>
|
| 23 |
|
| 24 |
-
|
| 25 |
|
| 26 |
-
Crobotia iUMI
|
| 27 |
|
| 28 |
-
##
|
| 29 |
|
| 30 |
-
|
| 31 |
|
| 32 |
-
|
|
| 33 |
| --- | --- | --- | ---: | ---: |
|
| 34 |
-
| OpenPI π0.5 | [4
|
| 35 |
-
| FLUX | [4
|
| 36 |
|
| 37 |
**OpenPI π0.5**
|
| 38 |
|
|
@@ -42,128 +38,7 @@ Crobotia iUMI的模型推理与机器人部署代码,支持OpenPI(π0.5)
|
|
| 42 |
|
| 43 |
<video src="https://huggingface.co/Crobotia/iUMI_Inference/resolve/main/demos/flux_fold_demo.mp4" controls preload="metadata" width="100%"></video>
|
| 44 |
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
只下载视频和 rollout,无需下载约 21.37 GB 的模型文件:
|
| 48 |
-
|
| 49 |
-
```bash
|
| 50 |
-
pip install -U huggingface_hub
|
| 51 |
-
hf download Crobotia/iUMI_Inference --local-dir iUMI_Inference --include "demos/*" "rollout_data/*"
|
| 52 |
-
```
|
| 53 |
-
|
| 54 |
-
媒体与 rollout 合计约 2.35 GB。分别以 `rollout_data/openpi_pi05/` 和 `rollout_data/flux/` 为独立 LeRobot v3 数据集根目录;两者都从 episode 0 开始。见 [rollout 读取说明](rollout_data/README.md) 和 [视频说明](demos/README.md)。这些文件存放在本模型仓库中,下载时使用默认 `repo_type="model"`。
|
| 55 |
-
|
| 56 |
-
## 模型用途与数据约定
|
| 57 |
-
|
| 58 |
-
两种后端用于双臂 UMI 衣物折叠的研究、离线接口集成和匹配设备上的部署验证。每次推理使用左右腕部图像、前一/当前位姿和任务文本,输出动作块,再由配套适配器完成坐标还原。
|
| 59 |
-
|
| 60 |
-
| 项目 | 约定 |
|
| 61 |
-
| --- | --- |
|
| 62 |
-
| 记录的状态 / 动作 | 16 维,左手在前;每手 `[x, y, z, qw, qx, qy, qz, gripper]`;米 / `wxyz` / 夹爪角度 |
|
| 63 |
-
| rollout 动作语义 | `action_mode=next_state`,`action[t] = observation.state[t+1]`(末帧除外),不是原始策略输出 |
|
| 64 |
-
| 模型内部 UMI 表示 | 两手合计 20 维;每手相对平移 3 + 旋转前两列 6 + 夹爪 1 |
|
| 65 |
-
| FLUX 动作块 | 32 步 / 30 Hz,使用双腕图像 |
|
| 66 |
-
| π0.5 动作块 | 50 个模型步;32 维模型头由 20 维 UMI 目标补齐,适配器还原为双臂目标位姿 |
|
| 67 |
-
|
| 68 |
-
记录位姿来源为 `robot_fk_cad`。相机光学坐标、机器人法兰坐标和模型相对表示需要显式转换,见 [输入输出与坐标约定](docs/INFERENCE_CONTRACT.md)。
|
| 69 |
-
|
| 70 |
-
## 训练来源、验证与局限
|
| 71 |
-
|
| 72 |
-
发布的 FLUX 为 step 14418 的 BF16 推理导出;π0.5 为 `pi05_umi_v3` 的 epoch 2 checkpoint。完整训练语料清单、划分和精确运行配置未随本次新增材料提供;这两条 rollout 是机器人执行样例,不能据此反推训练集。
|
| 73 |
-
|
| 74 |
-
本次新增材料通过文件哈希、视频流/包计数、Parquet 索引与下一帧动作关系、IMU 和 H5 结构检查,详见 [rollout 验证](docs/ROLLOUT_VALIDATION.md)。每个模型只有 1 条样例,未建立评测协议或任务成功率,也未运行完整运动质检。演示记录的确切 checkpoint 哈希、采样配置和剪辑到数据的逐帧映射未随材料提供,因此不能把当前启动脚本默认值视为视频的精确复现配方。
|
| 75 |
-
|
| 76 |
-
现有权重与源码的验证记录仍见 [验证范围](docs/VALIDATION.md);此次上传没有重新运行 GPU 推理或机器人。实机部署需要匹配的编码器、厂商 SDK、设备标定、工作空间限制与现场验证。
|
| 77 |
-
|
| 78 |
-
## 获取代码
|
| 79 |
-
|
| 80 |
-
```bash
|
| 81 |
-
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/Crobotia/iUMI_Inference
|
| 82 |
-
cd iUMI_Inference
|
| 83 |
-
```
|
| 84 |
-
|
| 85 |
-
| 目录 | 内容 |
|
| 86 |
-
| --- | --- |
|
| 87 |
-
| flexiv-umi-infer/ | 策略服务、机器人/相机/夹爪接口及数据记录 |
|
| 88 |
-
| flux-umi/ | FLUX模型与UMI数据表示 |
|
| 89 |
-
| openpi-umi/ | OpenPI模型、数据变换与客户端 |
|
| 90 |
-
| scripts/ | 模型服务和机器人端启动脚本 |
|
| 91 |
-
| configs/ | 推理配置模板 |
|
| 92 |
-
| checkpoints/ | FLUX BF16 和 π0.5 叠衣推理权重 |
|
| 93 |
-
| demos/ | 两段静音叠衣演示视频 |
|
| 94 |
-
| rollout_data/ | OpenPI π0.5 / FLUX 各 1 条完整执行记录 |
|
| 95 |
-
|
| 96 |
-
OpenPI使用Linux(glibc ≥2.31);按`openpi-umi/.python-version`配置Python3.12,在该目录运行`uv sync --frozen --python 3.12`。
|
| 97 |
-
|
| 98 |
-
FLUX使用`scripts/run_flux_server.sh`,OpenPI使用`scripts/run_pi05_server.sh`,机器人端使用`scripts/run_robot.sh`。两类模型使用独立环境,按各子项目说明安装依赖。启动前配置模型资源路径及设备标定;机器人脚本默认为dry run。
|
| 99 |
-
|
| 100 |
-
运行前准备所需编码器、厂商SDK及设备标定。输入输出约定见`docs/INFERENCE_CONTRACT.md`,安装与设备配置见各子项目README。
|
| 101 |
-
|
| 102 |
-
## 服务部署关系
|
| 103 |
-
|
| 104 |
-
```mermaid
|
| 105 |
-
flowchart LR
|
| 106 |
-
Robot["Flexiv 主机:robot_agent / 相机 / 本地伺服"] <-->|"WebSocket Observation / Action"| Server["GPU 主机:UMI policy_server"]
|
| 107 |
-
Server --> Flux["FLUX 后端:flux-umi + FLUX checkpoint"]
|
| 108 |
-
Server --> Pi["π0.5 后端:openpi-umi + OpenPI checkpoint"]
|
| 109 |
-
```
|
| 110 |
-
|
| 111 |
-
每个服务进程选择一个后端。`run_flux_server.sh` 和 `run_pi05_server.sh` 都启动 `umi_infer.policy_server`,分别调用对应的模型加载器;UMI 服务包装层位于 `flexiv-umi-infer/`。机器人端只运行 `robot_agent`,通过接口请求推理,不加载权重。模型输入编码、归一化和动作坐标还原在服务包装层执行;伺服执行在 Flexiv 主机上进行。
|
| 112 |
-
|
| 113 |
-
跨机器部署时,GPU 主机设置 `SERVER_HOST=0.0.0.0`、`SERVER_PORT=8765`;Flexiv 主机设置 `SERVER_URI=ws://<GPU主机地址>:8765`,再运行 `bash scripts/run_robot.sh`。同时部署两种模型时使用不同端口。设备标定路径 `UMI_INFER_DATA` 应在对应主机可访问;机器人脚本默认为 dry run。
|
| 114 |
-
|
| 115 |
-
请使用这套匹配的服务/客户端。OpenPI 原生 WebSocket 服务有不同的握手和数据封装,不能直接连接此 `robot_agent`。
|
| 116 |
-
|
| 117 |
-
## 下载权重与启动
|
| 118 |
-
|
| 119 |
-
两份权重已放在本仓库的 `checkpoints/` 下,共约 21.37 GB。在仓库根目录下载(需要 Hugging Face CLI):
|
| 120 |
-
|
| 121 |
-
```bash
|
| 122 |
-
pip install -U huggingface_hub
|
| 123 |
-
hf download Crobotia/iUMI_Inference --local-dir . --include "checkpoints/*"
|
| 124 |
-
```
|
| 125 |
-
|
| 126 |
-
| 后端 | 启动脚本默认权重目录 | 模型文件大小 |
|
| 127 |
-
| --- | --- | --- |
|
| 128 |
-
| FLUX | `checkpoints/flux_fold_clothes_bf16/` | 13,894,466,592 B |
|
| 129 |
-
| π0.5 | `checkpoints/pi05_fold_clothes/epoch_2/` | 7,473,091,464 B |
|
| 130 |
-
|
| 131 |
-
启动脚本自动使用上述目录;也可用 `FLUX_EXPORT_DIR` 或 `PI05_CHECKPOINT_DIR` 指定其他位置。按后端安装依赖后,复制配置并填写资源、Python 环境和设备标定路径:
|
| 132 |
-
|
| 133 |
-
```bash
|
| 134 |
-
cp configs/inference.env.example .env
|
| 135 |
-
# 编辑 .env:FLUX_RESOURCES、TOKENIZER_DIR、UMI_INFER_DATA 及 Python 路径
|
| 136 |
-
set -a
|
| 137 |
-
source .env
|
| 138 |
-
set +a
|
| 139 |
-
bash scripts/run_flux_server.sh
|
| 140 |
-
# 或在 π0.5 环境中运行:bash scripts/run_pi05_server.sh
|
| 141 |
-
```
|
| 142 |
-
|
| 143 |
-
FLUX 仍需单独准备 `FLUX_RESOURCES/video_vae.safetensors` 和 `FLUX_RESOURCES/text_encoder/`;π0.5 仍需 `TOKENIZER_DIR/paligemma-3b-pt-224/tokenizer.model`。这些外部资源未包含在此次权重发布中。归一化参数随权重提供,启动时自动加载对应文件。
|
| 144 |
-
|
| 145 |
-
FLUX 导出未保存采样参数。当前脚本使用 Euler、4 步、shift 42、video/action guidance 1.0;这是可覆盖的脚本默认值,尚未验证叠衣效果。π0.5 默认 10 步去噪,`VLA_QUAT_PERM=0` 对应该权重的 `xyz_quat_wxyz` 配置。
|
| 146 |
-
|
| 147 |
-
权重已完成下载校验、文件结构及脱敏检查;CPU 检查不代表 GPU 推理或实机叠衣验证。详情见 [FLUX 权重说明](docs/CHECKPOINTS_FLUX.md)、[π0.5 权重说明](docs/CHECKPOINTS_PI05.md) 和 [验证范围](docs/VALIDATION.md)。
|
| 148 |
-
|
| 149 |
-
---
|
| 150 |
-
|
| 151 |
-
<a id="en"></a>
|
| 152 |
-
|
| 153 |
-
# iUMI Inference · Bimanual UMI Inference and Deployment
|
| 154 |
-
|
| 155 |
-
Model inference and robot deployment code for Crobotia iUMI, with OpenPI (π0.5) and FLUX backends, configuration templates, launch scripts and a nominal dual-arm URDF.
|
| 156 |
-
|
| 157 |
-
## Demos and sample rollouts
|
| 158 |
-
|
| 159 |
-
This repository includes OpenPI π0.5 and FLUX cloth-folding inference checkpoints, deployment code, two demo videos and one robot rollout per backend. The task is `fold_the_clothes` on a dual-arm Flexiv Rizon 4 setup with left/right ego cameras.
|
| 160 |
-
|
| 161 |
-
| Backend | Muted presentation edit | Complete rollout | Frames at 30 Hz | Recording duration |
|
| 162 |
-
| --- | --- | --- | ---: | ---: |
|
| 163 |
-
| OpenPI π0.5 | [4 min 15.4 sec](demos/openpi_fold_demo.mp4) | [openpi_pi05](rollout_data/openpi_pi05) | 6,540 | 218 sec |
|
| 164 |
-
| FLUX | [4 min 29.7 sec](demos/flux_fold_demo.mp4) | [flux](rollout_data/flux) | 30,619 | 1,020.633 sec |
|
| 165 |
-
|
| 166 |
-
The embedded previews are above. Presentation edits have their own timelines; they are not frame-aligned to the datasets. The two rollouts contain 37,159 frames / 20 min 38.6 sec in total. Each retains left/right 960×960 / 30 Hz AV1 video, 16-D states/actions, joint data, bilateral IMU, calibration and robot H5 records. Audio and rear-camera views are excluded.
|
| 167 |
|
| 168 |
Download only the demos and rollouts, without the approximately 21.37 GB of model files:
|
| 169 |
|
|
@@ -260,12 +135,8 @@ The FLUX export has no saved sampling preset. The launcher uses Euler, 4 steps,
|
|
| 260 |
|
| 261 |
Download integrity, file structure and sanitization have been checked. CPU checks do not establish GPU inference or physical-robot performance. See [FLUX](docs/CHECKPOINTS_FLUX.md), [π0.5](docs/CHECKPOINTS_PI05.md) and [validation](docs/VALIDATION.md).
|
| 262 |
|
| 263 |
-
##
|
| 264 |
-
|
| 265 |
-
源码采用Apache-2.0许可,各子项目保留原有声明;Gemma、模型资源及厂商SDK遵循各自条款。详见[LICENSES.md](LICENSES.md)。
|
| 266 |
|
| 267 |
Source code is licensed under Apache-2.0, with upstream notices retained in each component. Gemma, model resources and vendor SDKs retain their respective terms. See [LICENSES.md](LICENSES.md).
|
| 268 |
|
| 269 |
-
新增 demo 与 rollout 未附独立的媒体/数据许可声明;根目录 Apache-2.0 标签用于源码,不自动授予这些媒体、数据或模型资源的相同许可。使用或再分发前请向 Crobotia 确认适用条款;不将其他人工演示数据集的 CC BY-SA 声明套用于本次材料。
|
| 270 |
-
|
| 271 |
No separate media/data license declaration accompanies the added demos and rollouts. The root Apache-2.0 tag describes source code and does not automatically license media, data or model resources on the same terms. Confirm applicable use and redistribution terms with Crobotia; licenses of separate human-demonstration datasets are not applied here.
|
|
|
|
| 15 |
- robot-rollout
|
| 16 |
---
|
| 17 |
|
| 18 |
+
# iUMI Inference · Bimanual UMI Inference and Deployment
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
|
| 20 |
+
**English** · [简体中文](README_ZH.md)
|
| 21 |
|
| 22 |
+
Model inference and robot deployment code for Crobotia iUMI, with OpenPI (π0.5) and FLUX backends, configuration templates, launch scripts and a nominal dual-arm URDF.
|
| 23 |
|
| 24 |
+
## Demos and sample rollouts
|
| 25 |
|
| 26 |
+
This repository includes OpenPI π0.5 and FLUX cloth-folding inference checkpoints, deployment code, two demo videos and one robot rollout per backend. The task is `fold_the_clothes` on a dual-arm Flexiv Rizon 4 setup with left/right ego cameras.
|
| 27 |
|
| 28 |
+
| Backend | Muted presentation edit | Complete rollout | Frames at 30 Hz | Recording duration |
|
| 29 |
| --- | --- | --- | ---: | ---: |
|
| 30 |
+
| OpenPI π0.5 | [4 min 15.4 sec](demos/openpi_fold_demo.mp4) | [openpi_pi05](rollout_data/openpi_pi05) | 6,540 | 218 sec |
|
| 31 |
+
| FLUX | [4 min 29.7 sec](demos/flux_fold_demo.mp4) | [flux](rollout_data/flux) | 30,619 | 1,020.633 sec |
|
| 32 |
|
| 33 |
**OpenPI π0.5**
|
| 34 |
|
|
|
|
| 38 |
|
| 39 |
<video src="https://huggingface.co/Crobotia/iUMI_Inference/resolve/main/demos/flux_fold_demo.mp4" controls preload="metadata" width="100%"></video>
|
| 40 |
|
| 41 |
+
Presentation edits have their own timelines; they are not frame-aligned to the datasets. The two rollouts contain 37,159 frames / 20 min 38.6 sec in total. Each retains left/right 960×960 / 30 Hz AV1 video, 16-D states/actions, joint data, bilateral IMU, calibration and robot H5 records. Audio and rear-camera views are excluded.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 42 |
|
| 43 |
Download only the demos and rollouts, without the approximately 21.37 GB of model files:
|
| 44 |
|
|
|
|
| 135 |
|
| 136 |
Download integrity, file structure and sanitization have been checked. CPU checks do not establish GPU inference or physical-robot performance. See [FLUX](docs/CHECKPOINTS_FLUX.md), [π0.5](docs/CHECKPOINTS_PI05.md) and [validation](docs/VALIDATION.md).
|
| 137 |
|
| 138 |
+
## License
|
|
|
|
|
|
|
| 139 |
|
| 140 |
Source code is licensed under Apache-2.0, with upstream notices retained in each component. Gemma, model resources and vendor SDKs retain their respective terms. See [LICENSES.md](LICENSES.md).
|
| 141 |
|
|
|
|
|
|
|
| 142 |
No separate media/data license declaration accompanies the added demos and rollouts. The root Apache-2.0 tag describes source code and does not automatically license media, data or model resources on the same terms. Confirm applicable use and redistribution terms with Crobotia; licenses of separate human-demonstration datasets are not applied here.
|
README_ZH.md
ADDED
|
@@ -0,0 +1,132 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# iUMI Inference · 双臂UMI推理与部署
|
| 2 |
+
|
| 3 |
+
[English](README.md) · **简体中文**
|
| 4 |
+
|
| 5 |
+
Crobotia iUMI的模型推理与机器人部署代码,支持OpenPI(π0.5)和FLUX后端,包含配置模板、启动脚本及双臂名义URDF。
|
| 6 |
+
|
| 7 |
+
## 模型演示与 sample rollouts
|
| 8 |
+
|
| 9 |
+
本仓库提供 OpenPI π0.5 和 FLUX 的叠衣推理权重、部署源码、两段演示视频,以及每个后端各 1 条机器人执行样例。任务为 `fold_the_clothes`,硬件为 Flexiv Rizon 4 双臂与左右 ego 相机。
|
| 10 |
+
|
| 11 |
+
| 后端 | 演示视频(静音剪辑) | 完整 rollout | 帧数 / 30 Hz | 完整记录时长 |
|
| 12 |
+
| --- | --- | --- | ---: | ---: |
|
| 13 |
+
| OpenPI π0.5 | [4分15.4秒](demos/openpi_fold_demo.mp4) | [openpi_pi05](rollout_data/openpi_pi05) | 6,540 | 218 秒 |
|
| 14 |
+
| FLUX | [4分29.7秒](demos/flux_fold_demo.mp4) | [flux](rollout_data/flux) | 30,619 | 1,020.633 秒 |
|
| 15 |
+
|
| 16 |
+
**OpenPI π0.5**
|
| 17 |
+
|
| 18 |
+
<video src="https://huggingface.co/Crobotia/iUMI_Inference/resolve/main/demos/openpi_fold_demo.mp4" controls preload="metadata" width="100%"></video>
|
| 19 |
+
|
| 20 |
+
**FLUX**
|
| 21 |
+
|
| 22 |
+
<video src="https://huggingface.co/Crobotia/iUMI_Inference/resolve/main/demos/flux_fold_demo.mp4" controls preload="metadata" width="100%"></video>
|
| 23 |
+
|
| 24 |
+
视频用于快速浏览,剪辑时间轴不与 rollout 逐帧对应。两条 rollout 合计 37,159 帧、约 20分38.6秒,保留左右两路 960×960 / 30 Hz AV1 视频、16 维状态/动作、关节数据、双侧 IMU、标定和机器人 H5;不包含音频或 rear 相机。
|
| 25 |
+
|
| 26 |
+
只下载视频和 rollout,无需下载约 21.37 GB 的模型文件:
|
| 27 |
+
|
| 28 |
+
```bash
|
| 29 |
+
pip install -U huggingface_hub
|
| 30 |
+
hf download Crobotia/iUMI_Inference --local-dir iUMI_Inference --include "demos/*" "rollout_data/*"
|
| 31 |
+
```
|
| 32 |
+
|
| 33 |
+
媒体与 rollout 合计约 2.35 GB。分别以 `rollout_data/openpi_pi05/` 和 `rollout_data/flux/` 为独立 LeRobot v3 数据集根目录;两者都从 episode 0 开始。见 [rollout 读取说明](rollout_data/README.md) 和 [视频说明](demos/README.md)。这些文件存放在本模型仓库中,下载时使用默认 `repo_type="model"`。
|
| 34 |
+
|
| 35 |
+
## 模型用途与数据约定
|
| 36 |
+
|
| 37 |
+
两种后端用于双臂 UMI 衣物折叠的研究、离线接口集成和匹配设备上的部署验证。每次推理使用左右腕部图像、前一/当前位姿和任务文本,输出动作块,再由配套适配器完成坐标还原。
|
| 38 |
+
|
| 39 |
+
| 项目 | 约定 |
|
| 40 |
+
| --- | --- |
|
| 41 |
+
| 记录的状态 / 动作 | 16 维,左手在前;每手 `[x, y, z, qw, qx, qy, qz, gripper]`;米 / `wxyz` / 夹爪角度 |
|
| 42 |
+
| rollout 动作语义 | `action_mode=next_state`,`action[t] = observation.state[t+1]`(末帧除外),不是原始策略输出 |
|
| 43 |
+
| 模型内部 UMI 表示 | 两手合计 20 维;每手相对平移 3 + 旋转前两列 6 + 夹爪 1 |
|
| 44 |
+
| FLUX 动作块 | 32 步 / 30 Hz,使用双腕图像 |
|
| 45 |
+
| π0.5 动作块 | 50 个模型步;32 维模型头由 20 维 UMI 目标补齐,适配器还原为双臂目标位姿 |
|
| 46 |
+
|
| 47 |
+
记录位姿来源为 `robot_fk_cad`。相机光学坐标、机器人法兰坐标和模型相对表示需要显式转换,见 [输入输出与坐标约定](docs/INFERENCE_CONTRACT.md)。
|
| 48 |
+
|
| 49 |
+
## 训练来源、验证与局限
|
| 50 |
+
|
| 51 |
+
发布的 FLUX 为 step 14418 的 BF16 推理导出;π0.5 为 `pi05_umi_v3` 的 epoch 2 checkpoint。完整训练语料清单、划分和精确运行配置未随本次新增材料提供;这两条 rollout 是机器人执行样例,不能据此反推训练集。
|
| 52 |
+
|
| 53 |
+
本次新增材料通过文件哈希、视频流/包计数、Parquet 索引与下一帧动作关系、IMU 和 H5 结构检查,详见 [rollout 验证](docs/ROLLOUT_VALIDATION.md)。每个模型只有 1 条样例,未建立评测协议或任务成功率,也未运行完整运动质检。演示记录的确切 checkpoint 哈希、采样配置和剪辑到数据的逐帧映射未随材料提供,因此不能把当前启动脚本默认值视为视频的精确复现配方。
|
| 54 |
+
|
| 55 |
+
现有权重与源码的验证记录仍见 [验证范围](docs/VALIDATION.md);此次上传没有重新运行 GPU 推理或机器人。实机部署需要匹配的编码器、厂商 SDK、设备标定、工作空间限制与现场验证。
|
| 56 |
+
|
| 57 |
+
## 获取代码
|
| 58 |
+
|
| 59 |
+
```bash
|
| 60 |
+
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/Crobotia/iUMI_Inference
|
| 61 |
+
cd iUMI_Inference
|
| 62 |
+
```
|
| 63 |
+
|
| 64 |
+
| 目录 | 内容 |
|
| 65 |
+
| --- | --- |
|
| 66 |
+
| flexiv-umi-infer/ | 策略服务、机器人/相机/夹爪接口及数据记录 |
|
| 67 |
+
| flux-umi/ | FLUX模型与UMI数据表示 |
|
| 68 |
+
| openpi-umi/ | OpenPI模型、数据变换与客户端 |
|
| 69 |
+
| scripts/ | 模型服务和机器人端启动脚本 |
|
| 70 |
+
| configs/ | 推理配置模板 |
|
| 71 |
+
| checkpoints/ | FLUX BF16 和 π0.5 叠衣推理权重 |
|
| 72 |
+
| demos/ | 两段静音叠衣演示视频 |
|
| 73 |
+
| rollout_data/ | OpenPI π0.5 / FLUX 各 1 条完整执行记录 |
|
| 74 |
+
|
| 75 |
+
OpenPI使用Linux(glibc ≥2.31);按`openpi-umi/.python-version`配置Python3.12,在该目录运行`uv sync --frozen --python 3.12`。
|
| 76 |
+
|
| 77 |
+
FLUX使用`scripts/run_flux_server.sh`,OpenPI使用`scripts/run_pi05_server.sh`,机器人端使用`scripts/run_robot.sh`。两类模型使用独立环境,按各子项目说明安装依赖。启动前配置模型资源路径及设备标定;机器人脚本默认为dry run。
|
| 78 |
+
|
| 79 |
+
运行前准备所需编码器、厂商SDK及设备标定。输入输出约定见`docs/INFERENCE_CONTRACT.md`,安装与设备配置见各子项目README。
|
| 80 |
+
|
| 81 |
+
## 服务部署关系
|
| 82 |
+
|
| 83 |
+
```mermaid
|
| 84 |
+
flowchart LR
|
| 85 |
+
Robot["Flexiv 主机:robot_agent / 相机 / 本地伺服"] <-->|"WebSocket Observation / Action"| Server["GPU 主机:UMI policy_server"]
|
| 86 |
+
Server --> Flux["FLUX 后端:flux-umi + FLUX checkpoint"]
|
| 87 |
+
Server --> Pi["π0.5 后端:openpi-umi + OpenPI checkpoint"]
|
| 88 |
+
```
|
| 89 |
+
|
| 90 |
+
每个服务进程选择一个后端。`run_flux_server.sh` 和 `run_pi05_server.sh` 都启动 `umi_infer.policy_server`,分别调用对应的模型加载器;UMI 服务包装层位于 `flexiv-umi-infer/`。机器人端只运行 `robot_agent`,通过接口请求推理,不加载权重。模型输入编码、归一化和动作坐标还原在服务包装层执行;伺服执行在 Flexiv 主机上进行。
|
| 91 |
+
|
| 92 |
+
跨机器部署时,GPU 主机设置 `SERVER_HOST=0.0.0.0`、`SERVER_PORT=8765`;Flexiv 主机设置 `SERVER_URI=ws://<GPU主机地址>:8765`,再运行 `bash scripts/run_robot.sh`。同时部署两种模型时使用不同端口。设备标定路径 `UMI_INFER_DATA` 应在对应主机可访问;机器人脚本默认为 dry run。
|
| 93 |
+
|
| 94 |
+
请使用这套匹配的服务/客户端。OpenPI 原生 WebSocket 服务有不同的握手和数据封装,不能直接连接此 `robot_agent`。
|
| 95 |
+
|
| 96 |
+
## 下载权重与启动
|
| 97 |
+
|
| 98 |
+
两份权重已放在本仓库的 `checkpoints/` 下,共约 21.37 GB。在仓库根目录下载(需要 Hugging Face CLI):
|
| 99 |
+
|
| 100 |
+
```bash
|
| 101 |
+
pip install -U huggingface_hub
|
| 102 |
+
hf download Crobotia/iUMI_Inference --local-dir . --include "checkpoints/*"
|
| 103 |
+
```
|
| 104 |
+
|
| 105 |
+
| 后端 | 启动脚本默认权重目录 | 模型文件大小 |
|
| 106 |
+
| --- | --- | --- |
|
| 107 |
+
| FLUX | `checkpoints/flux_fold_clothes_bf16/` | 13,894,466,592 B |
|
| 108 |
+
| π0.5 | `checkpoints/pi05_fold_clothes/epoch_2/` | 7,473,091,464 B |
|
| 109 |
+
|
| 110 |
+
启动脚本自动使用上述目录;也可用 `FLUX_EXPORT_DIR` 或 `PI05_CHECKPOINT_DIR` 指定其他位置。按后端安装依赖后,复制配置并填写资源、Python 环境和设备标定路径:
|
| 111 |
+
|
| 112 |
+
```bash
|
| 113 |
+
cp configs/inference.env.example .env
|
| 114 |
+
# 编辑 .env:FLUX_RESOURCES、TOKENIZER_DIR、UMI_INFER_DATA 及 Python 路径
|
| 115 |
+
set -a
|
| 116 |
+
source .env
|
| 117 |
+
set +a
|
| 118 |
+
bash scripts/run_flux_server.sh
|
| 119 |
+
# 或在 π0.5 环境中运行:bash scripts/run_pi05_server.sh
|
| 120 |
+
```
|
| 121 |
+
|
| 122 |
+
FLUX 仍需单独准备 `FLUX_RESOURCES/video_vae.safetensors` 和 `FLUX_RESOURCES/text_encoder/`;π0.5 仍需 `TOKENIZER_DIR/paligemma-3b-pt-224/tokenizer.model`。这些外部资源未包含在此次权重发布中。归一化参数随权重提供,启动时自动加载对应文件。
|
| 123 |
+
|
| 124 |
+
FLUX 导出未保存采样参数。当前脚本使用 Euler、4 步、shift 42、video/action guidance 1.0;这是可覆盖的脚本默认值,尚未验证叠衣效果。π0.5 默认 10 步去噪,`VLA_QUAT_PERM=0` 对应该权重的 `xyz_quat_wxyz` 配置。
|
| 125 |
+
|
| 126 |
+
权重已完成下载校验、文件结构及脱敏检查;CPU 检查不代表 GPU 推理或实机叠衣验证。详情见 [FLUX 权重说明](docs/CHECKPOINTS_FLUX.md)、[π0.5 权重说明](docs/CHECKPOINTS_PI05.md) 和 [验证范围](docs/VALIDATION.md)。
|
| 127 |
+
|
| 128 |
+
## 许可证
|
| 129 |
+
|
| 130 |
+
源码采用Apache-2.0许可,各子项目保留原有声明;Gemma、模型资源及厂商SDK遵循各自条款。详见[LICENSES.md](LICENSES.md)。
|
| 131 |
+
|
| 132 |
+
新增 demo 与 rollout 未附独立的媒体/数据许可声明;根目录 Apache-2.0 标签用于源码,不自动授予这些媒体、数据或模型资源的相同许可。使用或再分发前请向 Crobotia 确认适用条款;不将其他人工演示数据集的 CC BY-SA 声明套用于本次材料。
|
checksums.sha256
CHANGED
|
@@ -2,7 +2,8 @@
|
|
| 2 |
581892afe64b0306e0a6588e0b02fd796bb53fb3c28a083a958fbb9df07a01ea .gitignore
|
| 3 |
c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4 LICENSE
|
| 4 |
08ff9cea9ff5857fe955722d67ae76185706c424007e3906c290a98c99e83489 LICENSES.md
|
| 5 |
-
|
|
|
|
| 6 |
f92f3c5e00e48f18829c88f41693c33651ace74b40ff363d9efdf88ffe972890 checkpoints/flux_fold_clothes_bf16/config.json
|
| 7 |
f2fd3587be719f1c8499cd9a1647387db194d5a0f5cdd62ab5abc55de15a0dfa checkpoints/flux_fold_clothes_bf16/manifest.json
|
| 8 |
4378d2ffd9f90f6b2efc0767512c29c934cd9f78c1e66997b551b52ade4b1c45 checkpoints/flux_fold_clothes_bf16/model.safetensors
|
|
|
|
| 2 |
581892afe64b0306e0a6588e0b02fd796bb53fb3c28a083a958fbb9df07a01ea .gitignore
|
| 3 |
c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4 LICENSE
|
| 4 |
08ff9cea9ff5857fe955722d67ae76185706c424007e3906c290a98c99e83489 LICENSES.md
|
| 5 |
+
3ffe9dbc25673559c488be3e21112a56742d2f766e2c56b16fb3753269c0fb91 README.md
|
| 6 |
+
620ea4826e4a8623c82aab9d5a4c85e4b12c27e5c008bea99318f68b95bb429f README_ZH.md
|
| 7 |
f92f3c5e00e48f18829c88f41693c33651ace74b40ff363d9efdf88ffe972890 checkpoints/flux_fold_clothes_bf16/config.json
|
| 8 |
f2fd3587be719f1c8499cd9a1647387db194d5a0f5cdd62ab5abc55de15a0dfa checkpoints/flux_fold_clothes_bf16/manifest.json
|
| 9 |
4378d2ffd9f90f6b2efc0767512c29c934cd9f78c1e66997b551b52ade4b1c45 checkpoints/flux_fold_clothes_bf16/model.safetensors
|