--- license: apache-2.0 base_model: Qwen/Qwen3-VL-4B-Instruct library_name: peft pipeline_tag: robotics language: - en tags: - robotics - vision-language-action - latent-action - libero - lora datasets: - Mo-ZheHan/FreeAct-Data - Mo-ZheHan/droid ---
Demonstration-Free Robot Adaptation
via Action-Grounded Generated Videos
NeurIPS 2026
Paper PDF · Project Page · Code · Data
FreeAct adapts a pretrained vision-language-action (VLA) policy to a new environment without target-domain expert demonstrations. A physically grounded latent action model (LAM) pseudo-labels generated videos of the target tasks with latent action tokens, and the policy is trained on these tokens together with source-domain action supervision. This repository hosts the checkpoints from the paper. ## Checkpoints | Checkpoint | Description | Tasks | Success (%) | |---|---|---|---:| | [`lam/`](https://huggingface.co/Mo-ZheHan/FreeAct/tree/main/lam) | Stage-1 latent action model | – | – | | [`stage2/`](https://huggingface.co/Mo-ZheHan/FreeAct/tree/main/stage2) | Stage-2 pretrained VLA | – | – | | **[`stage3/`](https://huggingface.co/Mo-ZheHan/FreeAct/tree/main/stage3)** | Adapted policy | A–E / B (camera shift) / C (ketchup bottle) | 56.8 / 74 / 60 | | **[`self_evolve/`](https://huggingface.co/Mo-ZheHan/FreeAct/tree/main/self_evolve)** | Self-evolved policy | A–E | 73.6 | | [`self_evolve_camera/`](https://huggingface.co/Mo-ZheHan/FreeAct/tree/main/self_evolve_camera) | Self-evolved policy, camera-shift environment | B (camera shift) | 92 | | [`self_evolve_bottle/`](https://huggingface.co/Mo-ZheHan/FreeAct/tree/main/self_evolve_bottle) | Self-evolved policy, ketchup-bottle environment | C (ketchup bottle) | 94 | | [`baselines/univla/`](https://huggingface.co/Mo-ZheHan/FreeAct/tree/main/baselines/univla) | Stages 2–3 with the UniVLA tokenizer | A–E | 4.0 | | [`baselines/lapa/`](https://huggingface.co/Mo-ZheHan/FreeAct/tree/main/baselines/lapa) | Stages 2–3 with the LAPA tokenizer | A–E | 12.0 | Success rates are over 50 simulated rollouts per task; without adaptation, the source policy reaches 1.6% on Tasks A–E. ## Model details - **Latent action model:** a VQ-VAE on frozen [DINOv3](https://github.com/facebookresearch/dinov3) ViT-B/16 features that encodes two frames 1 s apart into 4 tokens from a 64-entry codebook, grounded in camera-frame end-effector motion during training. - **Policy:** [Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) with LoRA adapters (rank 64) and 64 added latent action tokens. From an agent-view image and an instruction it predicts 4 latent action tokens, and a continuous action head (`action_decoder.pt`, with `action_stats.json`) decodes their hidden states into a chunk of 7D end-effector actions. - **Training data:** DROID, LIBERO demonstrations, Wan 2.6 generated videos and MLLM-selected rollouts; see [FreeAct-Data](https://huggingface.co/datasets/Mo-ZheHan/FreeAct-Data) and [DROID](https://huggingface.co/datasets/Mo-ZheHan/droid). ## Usage With the [PyTorch](https://github.com/Mo-ZheHan/FreeAct) or [Jittor](https://github.com/Mo-ZheHan/FreeAct-Jittor) implementation installed: ```bash hf download Mo-ZheHan/FreeAct --include 'stage3/*' --local-dir checkpoints/FreeAct bash scripts/eval.sh checkpoints/FreeAct/stage3 ``` ## Intended use and limitations The checkpoints are intended for research on robot policy adaptation. They were trained and evaluated in simulation only (LIBERO, a Franka arm on a tabletop) and are specific to the paper's target tasks; deployment on real robots is out of scope without further validation. ## Citation ```bibtex @inproceedings{mo2026freeact, title = {FreeAct: Demonstration-Free Robot Adaptation via Action-Grounded Generated Videos}, author = {Mo, Zhe-Han and Li, Jia-Ning and Song, Jiaxi and Guo, Meng-Hao and Hu, Shi-Min}, booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems}, year = {2026}, url = {https://openreview.net/forum?id=wNWxVVntG2} } ``` ## Acknowledgements The policy is built on [Qwen3-VL](https://github.com/QwenLM/Qwen3-VL) and the latent action model on [DINOv3](https://github.com/facebookresearch/dinov3). The models were trained on [DROID](https://droid-dataset.github.io/), [LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO) (as regenerated by [OpenVLA](https://github.com/openvla/openvla)) and videos generated with [Wan](https://github.com/Wan-Video) 2.6; the baselines use the [UniVLA](https://github.com/OpenDriveLab/UniVLA) and [LAPA](https://github.com/LatentActionPretraining/LAPA) tokenizers. ## License The checkpoints are released under Apache-2.0, except that `lam/lam.ckpt` contains Meta's DINOv3 ViT-B/16 weights and is subject to the [DINOv3 License](https://huggingface.co/Mo-ZheHan/FreeAct/blob/main/lam/DINOv3_LICENSE.md). The policies are LoRA adapters of Qwen3-VL-4B-Instruct (Apache-2.0).