| --- |
| license: mit |
| pipeline_tag: robotics |
| tags: |
| - robotics |
| - vision-language-action |
| - world-model |
| - manipulation |
| - bimanual |
| - yam |
| - flow-matching |
| - preview |
| datasets: |
| - XDOF/ABC-130k |
| - allenai/MolmoAct2-BimanualYAM-Dataset |
| - agibot-world/AgiBotWorld-Beta |
| - flex-pi/put_plate_on_the_rack |
| - flex-pi/soft_bag_zipping |
| - flex-pi/kitchen_organization |
| - flex-pi/self_repair_gripper_bc |
| - flex-pi/self_repair_gripper_dagger |
| --- |
| |
| # Flex-π · Pretrain (preview) — RGB + DINO + depth |
|
|
| <p align="center"> |
| <a href="https://arxiv.org/abs/2608.10860"><img src="https://img.shields.io/badge/arXiv-2608.10860-b31b1b.svg" alt="arXiv"></a> |
| <a href="https://flex-pi.github.io/"><img src="https://img.shields.io/badge/Project_Page-flex--pi-2ea44f.svg" alt="Project Page"></a> |
| <a href="https://github.com/geyan21/flex-pi"><img src="https://img.shields.io/badge/GitHub-flex--pi-181717.svg" alt="Code"></a> |
| <img src="https://img.shields.io/badge/status-preview-orange.svg" alt="Preview"> |
| </p> |
|
|
| > **Preview.** An early release of the pretrained Flex-π model; a stronger pretrained |
| > model will follow. |
|
|
| [Flex-π](https://arxiv.org/abs/2608.10860) is a 6B-parameter world-action model |
| that supervises three visual futures — RGB, 3D pointmaps and DINOv3 semantics — |
| alongside actions. A single checkpoint runs on any subset of its streams, so the |
| compute/accuracy operating point is a runtime flag. |
|
|
| This checkpoint is a large-scale pretrain for bimanual 6-DoF YAM-family robots |
| with one head camera and two wrist cameras. It trains on RGB + DINO + depth |
| modalities; depth is optional at inference. Training depth is DA3 for ABC-130k, |
| MolmoAct2 and AgiBot World, and sensor depth for the YAM data. It is a starting |
| point for fine-tuning; zero-shot use is possible on a matching rig. The sibling |
| without depth is |
| [`flex-pi/flexpi-pretrain-rgb-preview`](https://huggingface.co/flex-pi/flexpi-pretrain-rgb-preview). |
|
|
| ## Model details |
|
|
| | | | |
| |---|---| |
| | Architecture | Mixture-of-Transformers: Wan2.2-TI2V-5B video expert (5.00 B) + ActionDiT action expert (1.02 B), 30 layers, HBridge 7 / 16 / 7 | |
| | Cameras | `cam_high` 256×320, `cam_left_wrist` and `cam_right_wrist` 224×224, composited at 384×320 (`robotwin_uniform14`), 33 frames | |
| | Modalities | RGB + DINO + depth (depth optional at inference) | |
| | Visual streams | RGB and 3D pointmap through the frozen Wan2.2 VAE; DINOv3 ViT-B/16 (frozen), temporal stride 1 | |
| | Flex regime | `p_present_* = p_j* = 0.5` for video, DINO and pointmap; cross-modal forcing on | |
| | Action head | 34-D, shared by 6- and 7-joint arms. Channels 26 and 33 are each arm's 7th joint, which a YAM arm does not have: they are left out of the loss on YAM data and held at 0 when serving a YAM robot, which sends and receives 32-D | |
| | Wire format | 32-D YAM end-effector state in, 32 × 32 absolute action chunk out | |
| | Normalization | frame-pooled q01/q99 over 34 channels (`dataset_stats.json`), normalized targets unclamped (`action_norm_clamp: inf`) | |
| | Training data | ~4,600 hours of robot data: ABC-130k, MolmoAct2, AgiBot World and in-house YAM teleoperation data | |
| | Training depth | DA3 for ABC-130k, MolmoAct2 and AgiBot World; sensor depth for YAM | |
| | Checkpoint | `step_190000`, bf16, 12,082,212,409 bytes, sha256 `b168eb68f255b98bf4805ad762dd870e06dd5f0f6f044f3e4280622338799eb2` | |
|
|
| ## Usage |
|
|
| The serving and fine-tuning code is in the [Flex-π repository](https://github.com/geyan21/flex-pi); |
| the zero-shot guide is |
| [`docs/PRETRAINED.md`](https://github.com/geyan21/flex-pi/blob/main/docs/PRETRAINED.md). |
|
|
| ```bash |
| huggingface-cli download flex-pi/flexpi-pretrain-preview --local-dir ./checkpoints/flexpi-pretrain-preview |
| ``` |
|
|
| `config.yaml` and `dataset_stats.json` stay beside the weights: the server reads |
| the architecture back from the saved config, and `--dataset-stats-path` names |
| the statistics the checkpoint was trained with. |
|
|
| ### Zero-shot server |
|
|
| ```bash |
| D=./checkpoints/flexpi-pretrain-preview |
| python -u scripts/serve_yam_flexpi.py \ |
| --ckpt-path $D/checkpoints/weights/step_190000.pt \ |
| --dataset-stats-path $D/dataset_stats.json \ |
| --default-prompt "Pick up the plates from the table one by one and place each into a slot on the dish rack." \ |
| --host 0.0.0.0 --port 8000 --no-torch-compile --offload-text-encoder \ |
| --action-horizon 32 --mixed-precision bf16 --num-inference-steps 10 \ |
| <regime flags> |
| ``` |
|
|
| `--num-inference-steps` is 10 in the pre-release robot deploy of this |
| architecture and 4 in the repository's `configs/real_yam.yaml`; neither has been |
| measured on this checkpoint. |
|
|
| | Regime | Regime flags | Client sends depth + K | |
| |---|---|---| |
| | no depth, action only | `--infer-present-video true --infer-present-dino true --infer-present-pointmap false --infer-joint-video false --infer-joint-dino false --infer-joint-pointmap false` | no | |
| | with depth, action only | same, with `--infer-present-pointmap true` | yes | |
| | no depth, joint video + DINO | `--infer-present-pointmap false --infer-joint-video true --infer-joint-dino true --infer-joint-pointmap false` | no | |
| | with depth, full joint | none (the trained default) | yes | |
|
|
| With no regime flags the model runs full joint generation and requires depth; |
| the connect metadata reports this as `depth_required`. Depth is uint16 |
| millimetres for all three cameras or none, on the `per_cam_hw` grid from the |
| metadata, with intrinsics `[fx, fy, cx, cy]` on the same grid. The server wraps |
| the bare instruction in the training prompt template. A correct start logs |
| `action_dim_mask: channels [26, 33] zero-filled` and a |
| `FlexPi inference regime:` line. The client contract (state layout, frames, |
| gripper sense) is in |
| [`docs/YAM.md`](https://github.com/geyan21/flex-pi/blob/main/docs/YAM.md). |
|
|
| ### Fine-tuning |
|
|
| [`docs/FINETUNE.md`](https://github.com/geyan21/flex-pi/blob/main/docs/FINETUNE.md) |
| covers fine-tuning from this checkpoint with the task preset |
| `configs/task/yam_finetune_pretrain_preview_3cam_34d_rel_1e-4.yaml`, which reads this |
| checkpoint's statistics from `dataset_stats_rel32.json` (the same statistics |
| without channels 26 and 33, the 7th-joint slots a YAM arm does not use). |
|
|
| ## Scope |
|
|
| - Zero-shot use needs a bimanual 6-DoF YAM-family rig with one head and two wrist |
| cameras speaking the 32-D wire format; LIBERO, RoboTwin and AgiBot need |
| fine-tuning. |
|
|
| ## License |
|
|
| The weights are released under the MIT license, as is the code. The training |
| data carries its own terms: ABC-130k and MolmoAct2 BimanualYAM are Apache-2.0, |
| AgiBot World is gated and AgiBot's terms are CC BY-NC-SA 4.0, and the released |
| flex-pi YAM datasets are CC BY 4.0. At inference the model downloads DINOv3 |
| `timm/vit_base_patch16_dinov3.lvd1689m`, which is under the DINOv3 License, and |
| Wan2.2 components, which are Apache-2.0. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{yan2026flexpi, |
| title = {Flex-$\pi$: A Multi-Stream World-Action Model with Compute Flexibility}, |
| author = {Yan, Ge and Liu, Jinghao and Fan, Yuzhi and Cai, Lei and Liao, Minwen |
| and Zhang, Jesse and Fox, Dieter}, |
| journal = {arXiv preprint arXiv:2608.10860}, |
| year = {2026}, |
| url = {https://arxiv.org/abs/2608.10860} |
| } |
| ``` |
|
|
