| --- |
| tags: |
| - robotics |
| - video-generation |
| - vision-language-action |
| - push-t |
| - franka |
| library_name: pytorch |
| --- |
| |
| # Push-T FlashWAM (from scratch, 30 epochs) |
|
|
| Public archival bundle for a FROM-SCRATCH run on the real-hardware Push-T task: |
| push an orange T-shaped block across a table until it aligns with a pink T |
| outline, on a Franka Emika Panda. The gripper is clamped shut on a marker pen |
| that acts as a single-point pusher, so this is the classic Push-T task run on |
| real hardware rather than in simulation. |
|
|
| Trained on [`SleepMastger/pusht-manipulation`] |
| (https://huggingface.co/datasets/SleepMastger/pusht-manipulation) — 100 human |
| teleoperation demonstrations, 32,131 frames at 10 Hz, converted to LeRobot |
| v2.1. Companion model: [`SleepMastger/pusht-fastwam`](https://huggingface.co/SleepMastger/pusht-fastwam) |
| — identical data and recipe, different architecture, so the two are directly |
| comparable. |
|
|
| Architecture: FlashWAM M1 decoupled MoT: fused-KV KV source, fixed action-aligned video RoPE, **1-layer** action expert (randomly initialised — fused-KV modes cannot use layer-selective init), 30-layer Wan2.2 video expert. |
|
|
| ## Training |
|
|
| | | | |
| |---|---| |
| | init | from scratch (`resume: null`) | |
| | GPUs | 4 x GPU | |
| | batch | 8 per GPU x 4 x accum 1 = global 32 | |
| | schedule | cosine, lr 1e-4, weight decay 1e-2, bf16 | |
| | epochs | 30 | |
| | steps | 1,005/epoch -> **30,150** total | |
| | wall clock | 8h47m | |
| | validation | none (`val_set_proportion: 0.0`) — all 100 episodes are train | |
|
|
| Completed all 30 epochs. Checkpoints were saved every 5 epochs: |
|
|
| - `checkpoints/weights/step_005025.pt` (epoch 5) — SHA-256 `7732ce6db1e2199f67b6b587f027c359c8c0bebefb9bf6f34394e5e76e5a5720` |
| - `checkpoints/weights/step_010050.pt` (epoch 10) — SHA-256 `4ddef751a683df6b073b3c8e2818f17a03d4e4c502b80d3e2ba167a587ba23b7` |
| - `checkpoints/weights/step_015075.pt` (epoch 15) — SHA-256 `2a9f762196f66ece1cb2fe6d399912428c49a32d9cd99e234f7a079dfff630c9` |
| - `checkpoints/weights/step_020100.pt` (epoch 20) — SHA-256 `f13b996c9ce49f1abf6d48db7a6f678c658866a7caf84592ca66bbfb1878f2b5` |
| - `checkpoints/weights/step_025125.pt` (epoch 25) — SHA-256 `a76af418ff3bdd83547694f8551c30be20d1c43a6f5ac5cdd75d88a5f5598815` |
| - `checkpoints/weights/step_030150.pt` (epoch 30) — SHA-256 `61829d8bc1cda5ad3e59ed4281d8f76495d322fc928c4786d61491a2410ad440` |
|
|
| The model has not been evaluated on a robot. |
|
|
| ## Conditioning |
|
|
| Exact task text (**must byte-match at deployment**): |
|
|
| > push the T block to the target outline |
|
|
| No trailing period. `conditioning/text_embedding.pt` is the exact cached T5 |
| embedding consumed during training (Wan text stack, context length 128); its |
| filename hash in the training cache is |
| `71cb088954da46da4e3cb6c6f73ac812690e9bccf886c7858377c61918b4e3e8`. The |
| resolved config sets `load_text_encoder: false`, so this cached tensor is part |
| of the required inference bundle. |
|
|
| ## Input processing and normalization |
|
|
| - Two 256x256 RGB cameras with raw LeRobot keys `image` / `wrist_image`, |
| resized to 224x224 and concatenated horizontally to 224x448. |
| - Dataset is LeRobot **v2.1** (per-episode parquet + AV1 mp4), staged |
| node-locally by `training_code/stage_pusht_local.sh`. |
| - Horizon: 33 observations; 32 action transitions at **10 Hz**. |
| - State is 8-D: `eef_pos(3) + quat2axisangle(eef_quat)(3) + |
| [gripper_width/2, -gripper_width/2]`. |
| - Action is 7-D: delta XYZ (metres per 100 ms step), delta rotation XYZ, grasp. |
| Raw grasp uses the robosuite `{-1 open, +1 close}` convention and is remapped |
| `g -> (1-g)/2` to the LIBERO/RLDS `{1 open, 0 close}` convention at conversion. |
| - `dataset_stats.json` holds the exact min/max statistics used by this run. |
|
|
| ### ⚠️ Six of the fifteen channels are constant — read before deploying |
|
|
| The teleoperation rig commanded translation only, and the pen stayed clamped |
| for the whole task. Measured across all 100 episodes / 32,131 frames: |
|
|
| | field | dims | raw range | | |
| |---|---|---|---| |
| | action | `dx, dy, dz` | 3.5e-02 … 3.6e-02 | carries the signal | |
| | action | `drx, dry, drz` | **0.0** | constant | |
| | action | `grasp` | **0.0** (raw +1 -> 0.0) | constant, gripper never opens | |
| | state | `x, y, z, rx, ry, rz` | 8.9e-02 … 4.4e-01 | carries the signal | |
| | state | `gripL, gripR` | **3.2e-05** | constant | |
|
|
| This does **not** produce NaNs. `SingleFieldLinearNormalizer` |
| (`training_code/normalizer.py`) has `ignore_dim = input_range < range_tol` with |
| `range_tol = 1e-4`, which catches all six — the tightest, the gripper state |
| dims at 3.2e-05, clears it by about 3x. An ignored dim gets `scale = 1.0` and |
| `offset = -min`, i.e. it normalizes to `x - min`: exactly 0 for the four action |
| dims whose raw range is exactly 0.0, and `[0, 3.2e-05]` for the two gripper |
| state dims. Finite and negligible beside the ±1 live channels either way. |
|
|
| Consequences for deployment: |
|
|
| 1. **Only `dx, dy, dz` are meaningful outputs.** Dims 3-6 were constant in |
| training, so the model has learned to emit their constant value; denormalize |
| as usual and they come back as the constants. Do not interpret them as |
| commanded rotation or grasp. |
| 2. **The gripper must stay closed.** The policy never saw it open and cannot |
| command it to. Clamp the pusher before handing control to the policy. |
| 3. Do not lower `range_tol` below 3.2e-05 — the gripper state dims would then |
| be treated as live and divide by ~0. |
|
|
| The full 7-D action / 8-D proprio space was kept deliberately, so the |
| architecture is byte-identical to the dish-utensil and fruit-picking runs and |
| the results stay comparable. |
|
|
| ## Attention masks |
|
|
| - `video_attention_mask_mode: first_frame_causal` |
| - `action_group_causal_mask_mode: group_diagonal` |
| - Text cross-attention is enabled for the action expert. |
|
|
| The exact implementations, configs, dataset converter, staging and launch |
| scripts are included under `training_code/`; the resolved config is |
| `config.yaml`. `training_code/preflight.py` is the CPU-only check that verifies |
| the tensor contract and the degenerate-channel behaviour described above. No |
| license is asserted here for the bundled upstream code; its original terms |
| continue to apply. |
|
|
| ## Base components |
|
|
| This weights-only checkpoint is not standalone. It references |
| `Wan-AI/Wan2.2-TI2V-5B` and requires the matching Wan VAE plus the included |
| FastWAM code/configuration. PyTorch `.pt` files may contain pickled objects; |
| load only in a trusted environment. |
|
|
