FreeAct · Models

Demonstration-Free Robot Adaptation
via Action-Grounded Generated Videos

NeurIPS 2026

Paper PDF  ·  Project Page  ·  Code  ·  Data

FreeAct adapts a pretrained vision-language-action (VLA) policy to a new environment without target-domain expert demonstrations. A physically grounded latent action model (LAM) pseudo-labels generated videos of the target tasks with latent action tokens, and the policy is trained on these tokens together with source-domain action supervision. This repository hosts the checkpoints from the paper.

Checkpoints

Checkpoint Description Tasks Success (%)
lam/ Stage-1 latent action model – –
stage2/ Stage-2 pretrained VLA – –
stage3/ Adapted policy A–E / B (camera shift) / C (ketchup bottle) 56.8 / 74 / 60
self_evolve/ Self-evolved policy A–E 73.6
self_evolve_camera/ Self-evolved policy, camera-shift environment B (camera shift) 92
self_evolve_bottle/ Self-evolved policy, ketchup-bottle environment C (ketchup bottle) 94
baselines/univla/ Stages 2–3 with the UniVLA tokenizer A–E 4.0
baselines/lapa/ Stages 2–3 with the LAPA tokenizer A–E 12.0

Success rates are over 50 simulated rollouts per task; without adaptation, the source policy reaches 1.6% on Tasks A–E.

Model details

  • Latent action model: a VQ-VAE on frozen DINOv3 ViT-B/16 features that encodes two frames 1 s apart into 4 tokens from a 64-entry codebook, grounded in camera-frame end-effector motion during training.
  • Policy: Qwen3-VL-4B-Instruct with LoRA adapters (rank 64) and 64 added latent action tokens. From an agent-view image and an instruction it predicts 4 latent action tokens, and a continuous action head (action_decoder.pt, with action_stats.json) decodes their hidden states into a chunk of 7D end-effector actions.
  • Training data: DROID, LIBERO demonstrations, Wan 2.6 generated videos and MLLM-selected rollouts; see FreeAct-Data and DROID.

Usage

With the PyTorch or Jittor implementation installed:

hf download Mo-ZheHan/FreeAct --include 'stage3/*' --local-dir checkpoints/FreeAct
bash scripts/eval.sh checkpoints/FreeAct/stage3

Intended use and limitations

The checkpoints are intended for research on robot policy adaptation. They were trained and evaluated in simulation only (LIBERO, a Franka arm on a tabletop) and are specific to the paper's target tasks; deployment on real robots is out of scope without further validation.

Citation

@inproceedings{mo2026freeact,
  title     = {FreeAct: Demonstration-Free Robot Adaptation via Action-Grounded Generated Videos},
  author    = {Mo, Zhe-Han and Li, Jia-Ning and Song, Jiaxi and Guo, Meng-Hao and Hu, Shi-Min},
  booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems},
  year      = {2026},
  url       = {https://openreview.net/forum?id=wNWxVVntG2}
}

Acknowledgements

The policy is built on Qwen3-VL and the latent action model on DINOv3. The models were trained on DROID, LIBERO (as regenerated by OpenVLA) and videos generated with Wan 2.6; the baselines use the UniVLA and LAPA tokenizers.

License

The checkpoints are released under Apache-2.0, except that lam/lam.ckpt contains Meta's DINOv3 ViT-B/16 weights and is subject to the DINOv3 License. The policies are LoRA adapters of Qwen3-VL-4B-Instruct (Apache-2.0).

Downloads last month
-
Video Preview
loading

Model tree for Mo-ZheHan/FreeAct

Adapter
(255)
this model

Datasets used to train Mo-ZheHan/FreeAct