Model card: simpler inputs wording
Browse files
README.md
CHANGED
|
@@ -39,15 +39,9 @@ alongside actions. A single checkpoint runs on any subset of its streams, so the
|
|
| 39 |
compute/accuracy operating point is a runtime flag.
|
| 40 |
|
| 41 |
This checkpoint is a large-scale pretrain for bimanual 6-DoF YAM-family robots
|
| 42 |
-
with one head camera and two wrist cameras.
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
DINOv3 features from it.
|
| 46 |
-
- **Depth is optional.** With depth and intrinsics the 3D pointmap stream is on as
|
| 47 |
-
well; without them the model runs on RGB and DINO alone.
|
| 48 |
-
- **Training depth** was DA3 (Depth Anything 3) estimates for ABC-130k, MolmoAct2
|
| 49 |
-
and AgiBot World, and the YAM recordings' own sensor depth.
|
| 50 |
-
|
| 51 |
It is a starting point for fine-tuning; zero-shot use is possible on a matching
|
| 52 |
rig. The sibling without depth is
|
| 53 |
[`flex-pi/flexpi-pretrain-rgb-preview`](https://huggingface.co/flex-pi/flexpi-pretrain-rgb-preview).
|
|
@@ -58,14 +52,14 @@ rig. The sibling without depth is
|
|
| 58 |
|---|---|
|
| 59 |
| Architecture | Mixture-of-Transformers: Wan2.2-TI2V-5B video expert (5.00 B) + ActionDiT action expert (1.02 B), 30 layers, HBridge 7 / 16 / 7 |
|
| 60 |
| Cameras | `cam_high` 256×320, `cam_left_wrist` and `cam_right_wrist` 224×224, composited at 384×320 (`robotwin_uniform14`), 33 frames |
|
| 61 |
-
| Inputs | three RGB cameras,
|
| 62 |
| Visual streams | RGB and 3D pointmap through the frozen Wan2.2 VAE; DINOv3 ViT-B/16 (frozen), temporal stride 1 |
|
| 63 |
| Flex regime | `p_present_* = p_j* = 0.5` for video, DINO and pointmap; cross-modal forcing on |
|
| 64 |
| Action head | 34-D, shared by 6- and 7-joint arms. Channels 26 and 33 are each arm's 7th joint, which a YAM arm does not have: they are left out of the loss on YAM data and held at 0 when serving a YAM robot, which sends and receives 32-D |
|
| 65 |
| Wire format | 32-D YAM end-effector state in, 32 × 32 absolute action chunk out |
|
| 66 |
| Normalization | frame-pooled q01/q99 over 34 channels (`dataset_stats.json`), normalized targets unclamped (`action_norm_clamp: inf`) |
|
| 67 |
| Training data | ~4,600 hours of robot data: ABC-130k, MolmoAct2, AgiBot World and in-house YAM teleoperation data |
|
| 68 |
-
| Training depth | DA3
|
| 69 |
| Checkpoint | `step_190000`, bf16, 12,082,212,409 bytes, sha256 `b168eb68f255b98bf4805ad762dd870e06dd5f0f6f044f3e4280622338799eb2` |
|
| 70 |
|
| 71 |
## Usage
|
|
|
|
| 39 |
compute/accuracy operating point is a runtime flag.
|
| 40 |
|
| 41 |
This checkpoint is a large-scale pretrain for bimanual 6-DoF YAM-family robots
|
| 42 |
+
with one head camera and two wrist cameras. It takes RGB, with optional depth,
|
| 43 |
+
and always uses DINO, which the model computes from the RGB. Its training depth
|
| 44 |
+
is DA3 for ABC-130k, MolmoAct2 and AgiBot World, and sensor depth for the YAM data.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 45 |
It is a starting point for fine-tuning; zero-shot use is possible on a matching
|
| 46 |
rig. The sibling without depth is
|
| 47 |
[`flex-pi/flexpi-pretrain-rgb-preview`](https://huggingface.co/flex-pi/flexpi-pretrain-rgb-preview).
|
|
|
|
| 52 |
|---|---|
|
| 53 |
| Architecture | Mixture-of-Transformers: Wan2.2-TI2V-5B video expert (5.00 B) + ActionDiT action expert (1.02 B), 30 layers, HBridge 7 / 16 / 7 |
|
| 54 |
| Cameras | `cam_high` 256×320, `cam_left_wrist` and `cam_right_wrist` 224×224, composited at 384×320 (`robotwin_uniform14`), 33 frames |
|
| 55 |
+
| Inputs | three RGB cameras, optional depth; DINO is computed from the RGB |
|
| 56 |
| Visual streams | RGB and 3D pointmap through the frozen Wan2.2 VAE; DINOv3 ViT-B/16 (frozen), temporal stride 1 |
|
| 57 |
| Flex regime | `p_present_* = p_j* = 0.5` for video, DINO and pointmap; cross-modal forcing on |
|
| 58 |
| Action head | 34-D, shared by 6- and 7-joint arms. Channels 26 and 33 are each arm's 7th joint, which a YAM arm does not have: they are left out of the loss on YAM data and held at 0 when serving a YAM robot, which sends and receives 32-D |
|
| 59 |
| Wire format | 32-D YAM end-effector state in, 32 × 32 absolute action chunk out |
|
| 60 |
| Normalization | frame-pooled q01/q99 over 34 channels (`dataset_stats.json`), normalized targets unclamped (`action_norm_clamp: inf`) |
|
| 61 |
| Training data | ~4,600 hours of robot data: ABC-130k, MolmoAct2, AgiBot World and in-house YAM teleoperation data |
|
| 62 |
+
| Training depth | DA3 for ABC-130k, MolmoAct2 and AgiBot World; sensor depth for YAM |
|
| 63 |
| Checkpoint | `step_190000`, bf16, 12,082,212,409 bytes, sha256 `b168eb68f255b98bf4805ad762dd870e06dd5f0f6f044f3e4280622338799eb2` |
|
| 64 |
|
| 65 |
## Usage
|