| --- |
| license: cc-by-nc-4.0 |
| library_name: pytorch |
| tags: |
| - robotics |
| - robot-learning |
| - latent-action-policy |
| --- |
| |
| # UniLAP |
|
|
| **A Unified Latent Action Policy for Cross-Embodiment Manipulation** |
|
|
| UniLAP predicts robot action chunks from visual observations, language |
| instructions, and robot state. This repository provides robot-specific |
| checkpoints containing the latent action policy and action decoder, together |
| with the configuration and normalization statistics needed for inference. |
|
|
| [Project and paper information](https://rayyoh.github.io/UniLAP/) · |
| [Companion code](https://github.com/RayYoh/UniLAP) |
|
|
| ## Checkpoints |
|
|
| | Directory | Robot | Benchmark | Tasks | |
| | --- | --- | --- | --- | |
| | `bridge/` | WidowX | SimplerEnv | Cube stacking, carrot placement, spoon placement, eggplant placement | |
| | `fractal/` | Google Robot | SimplerEnv | Move near, pick up a coke can, open and close drawers | |
| | `gr1/` | GR1 | RoboCasa | Pick-and-place tasks with a bottle, can, cup, milk, potato, and wine | |
|
|
| The Google Robot checkpoint is shared by the Visual Matching (VM) and |
| Variant Aggregation (VA) evaluations. |
|
|
| ## Files |
|
|
| ```text |
| UniLAP/ |
| ├── README.md |
| ├── bridge/ |
| │ ├── config.yaml |
| │ └── model.ckpt |
| ├── fractal/ |
| │ ├── config.yaml |
| │ └── model.ckpt |
| ├── gr1/ |
| │ ├── config.yaml |
| │ └── model.ckpt |
| └── metadata/ |
| ├── bridge_stats.json |
| ├── fractal_stats.json |
| └── gr1_stats.json |
| ``` |
|
|
| - `model.ckpt`: latent policy and action decoder weights. |
| - `config.yaml`: model dimensions, pretrained encoder identifiers, action |
| representation, and normalization settings used by the inference loader. |
| - `metadata/*_stats.json`: action and state normalization statistics. |
|
|
| Keep each checkpoint beside its `config.yaml` and preserve the directory layout. |
|
|
| ## Download |
|
|
| Use the [Hugging Face CLI](https://huggingface.co/docs/huggingface_hub/guides/cli). |
| Run these commands from the companion code directory. |
|
|
| Download all checkpoints: |
|
|
| ```bash |
| hf download RayYoh/UniLAP --local-dir checkpoints/UniLAP |
| ``` |
|
|
| Or download one robot and its supporting files: |
|
|
| ```bash |
| # WidowX |
| hf download RayYoh/UniLAP \ |
| bridge/model.ckpt bridge/config.yaml metadata/bridge_stats.json README.md \ |
| --local-dir checkpoints/UniLAP |
| |
| # Google Robot |
| hf download RayYoh/UniLAP \ |
| fractal/model.ckpt fractal/config.yaml metadata/fractal_stats.json README.md \ |
| --local-dir checkpoints/UniLAP |
| |
| # GR1 |
| hf download RayYoh/UniLAP \ |
| gr1/model.ckpt gr1/config.yaml metadata/gr1_stats.json README.md \ |
| --local-dir checkpoints/UniLAP |
| ``` |
|
|
| ## Inference and evaluation |
|
|
| The companion code release is being prepared at |
| [RayYoh/UniLAP](https://github.com/RayYoh/UniLAP). The commands below require a |
| code checkout containing `eval/run_lap_server.py` and `scripts/eval_paper.sh`. |
| Follow its installation instructions for the `unilap` policy environment and |
| `eval/envs/README.md` for the `simpler` and `gr1` simulator environments. |
|
|
| Run from the code checkout root: |
|
|
| ```bash |
| conda activate unilap |
| CHECKPOINT_ROOT="$(pwd)/checkpoints/UniLAP" |
| ``` |
|
|
| To start a policy server independently, select `bridge`, `fractal`, or `gr1`: |
|
|
| ```bash |
| ROBOT=bridge |
| python eval/run_lap_server.py \ |
| --ckpt_path "$CHECKPOINT_ROOT/$ROBOT/model.ckpt" \ |
| --stat_path "$CHECKPOINT_ROOT/metadata/${ROBOT}_stats.json" \ |
| --robot "$ROBOT" \ |
| --num_inference_steps 4 \ |
| --device cuda \ |
| --port 5555 |
| ``` |
|
|
| The server loads DINOv2 ViT-B/14, SigLIP ViT-B/16 |
| (`vit_base_patch16_siglip_224.v2_webli`), and Flan-T5-large through the companion |
| code. Their pretrained files must be available locally or downloadable on first use. |
|
|
| For closed-loop evaluation, use the following entrypoints. They start the policy |
| server and simulator client automatically. For SimplerEnv, set `SIMPLER_PATH` to |
| your checkout containing the RGB overlay assets. |
|
|
| ```bash |
| export SIMPLER_PATH=/path/to/SimplerEnv |
| |
| # WidowX |
| STAT_PATH="$CHECKPOINT_ROOT/metadata/bridge_stats.json" \ |
| bash scripts/eval_paper.sh bridge "$CHECKPOINT_ROOT/bridge/model.ckpt" |
| |
| # Google Robot: Visual Matching |
| STAT_PATH="$CHECKPOINT_ROOT/metadata/fractal_stats.json" \ |
| bash scripts/eval_paper.sh fractal_vm "$CHECKPOINT_ROOT/fractal/model.ckpt" |
| |
| # Google Robot: Variant Aggregation |
| STAT_PATH="$CHECKPOINT_ROOT/metadata/fractal_stats.json" \ |
| bash scripts/eval_paper.sh fractal_va "$CHECKPOINT_ROOT/fractal/model.ckpt" |
| |
| # GR1 |
| STAT_PATH="$CHECKPOINT_ROOT/metadata/gr1_stats.json" \ |
| bash scripts/eval_paper.sh gr1 "$CHECKPOINT_ROOT/gr1/model.ckpt" |
| ``` |
|
|
| ## Citation |
|
|
| Please cite UniLAP when using these checkpoints in your research. See the |
| [project page](https://rayyoh.github.io/UniLAP/) for publication updates. |
|
|
| ```bibtex |
| @misc{unilap2026, |
| title = {UniLAP: A Unified Latent Action Policy for Cross-Embodiment Manipulation}, |
| author = {Lei Yao and Qiao Feng and Mutian Tong and Qihao Yun and Lap-Pui Chau and Lingjie Liu}, |
| year = {2026}, |
| url = {https://rayyoh.github.io/UniLAP/} |
| } |
| ``` |
|
|
| ## License |
|
|
| The UniLAP-authored weights, configurations, and documentation in this repository |
| are licensed under |
| [Creative Commons Attribution-NonCommercial 4.0 International](https://creativecommons.org/licenses/by-nc/4.0/legalcode.en). |
| This permits noncommercial use, sharing, and adaptation with attribution under |
| the license terms. |
|
|
| The companion code uses the MIT License. Referenced pretrained models, datasets, |
| and simulators remain subject to their respective original licenses and terms. |
|
|
