Instructions to use Mo-ZheHan/FreeAct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use Mo-ZheHan/FreeAct with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
FreeAct · Models
Demonstration-Free Robot Adaptation
via Action-Grounded Generated Videos
NeurIPS 2026
Paper PDF · Project Page · Code · Data
FreeAct adapts a pretrained vision-language-action (VLA) policy to a new environment without target-domain expert demonstrations. A physically grounded latent action model (LAM) pseudo-labels generated videos of the target tasks with latent action tokens, and the policy is trained on these tokens together with source-domain action supervision. This repository hosts the checkpoints from the paper.
Checkpoints
| Checkpoint | Description | Tasks | Success (%) |
|---|---|---|---|
lam/ |
Stage-1 latent action model | – | – |
stage2/ |
Stage-2 pretrained VLA | – | – |
stage3/ |
Adapted policy | A–E / B (camera shift) / C (ketchup bottle) | 56.8 / 74 / 60 |
self_evolve/ |
Self-evolved policy | A–E | 73.6 |
self_evolve_camera/ |
Self-evolved policy, camera-shift environment | B (camera shift) | 92 |
self_evolve_bottle/ |
Self-evolved policy, ketchup-bottle environment | C (ketchup bottle) | 94 |
baselines/univla/ |
Stages 2–3 with the UniVLA tokenizer | A–E | 4.0 |
baselines/lapa/ |
Stages 2–3 with the LAPA tokenizer | A–E | 12.0 |
Success rates are over 50 simulated rollouts per task; without adaptation, the source policy reaches 1.6% on Tasks A–E.
Model details
- Latent action model: a VQ-VAE on frozen DINOv3 ViT-B/16 features that encodes two frames 1 s apart into 4 tokens from a 64-entry codebook, grounded in camera-frame end-effector motion during training.
- Policy: Qwen3-VL-4B-Instruct with LoRA adapters (rank 64) and 64 added latent action tokens. From an agent-view image and an instruction it predicts 4 latent action tokens, and a continuous action head (
action_decoder.pt, withaction_stats.json) decodes their hidden states into a chunk of 7D end-effector actions. - Training data: DROID, LIBERO demonstrations, Wan 2.6 generated videos and MLLM-selected rollouts; see FreeAct-Data and DROID.
Usage
With the PyTorch or Jittor implementation installed:
hf download Mo-ZheHan/FreeAct --include 'stage3/*' --local-dir checkpoints/FreeAct
bash scripts/eval.sh checkpoints/FreeAct/stage3
Intended use and limitations
The checkpoints are intended for research on robot policy adaptation. They were trained and evaluated in simulation only (LIBERO, a Franka arm on a tabletop) and are specific to the paper's target tasks; deployment on real robots is out of scope without further validation.
Citation
@inproceedings{mo2026freeact,
title = {FreeAct: Demonstration-Free Robot Adaptation via Action-Grounded Generated Videos},
author = {Mo, Zhe-Han and Li, Jia-Ning and Song, Jiaxi and Guo, Meng-Hao and Hu, Shi-Min},
booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems},
year = {2026},
url = {https://openreview.net/forum?id=wNWxVVntG2}
}
Acknowledgements
The policy is built on Qwen3-VL and the latent action model on DINOv3. The models were trained on DROID, LIBERO (as regenerated by OpenVLA) and videos generated with Wan 2.6; the baselines use the UniVLA and LAPA tokenizers.
License
The checkpoints are released under Apache-2.0, except that lam/lam.ckpt contains Meta's DINOv3 ViT-B/16 weights and is subject to the DINOv3 License. The policies are LoRA adapters of Qwen3-VL-4B-Instruct (Apache-2.0).
- Downloads last month
- -
Model tree for Mo-ZheHan/FreeAct
Base model
Qwen/Qwen3-VL-4B-Instruct