OKayQi/AED

Learning Skills from Historical Action Trajectories: Action Experience Dictionary for World Action Models

This repository hosts the checkpoints for Learning Skills from Historical Action Trajectories: Action Experience Dictionary for World Action Models. AED builds on the Wan2.2 video backbone and learns a dictionary of action experience from historical action trajectories, which the World Action Model uses when predicting future actions.

Code, training scripts, and evaluation scripts are in the GitHub repository.

Checkpoints

Directory	Benchmark	Training steps	Action / state dim
libero_last/	LIBERO (Spatial, Object, Goal, 10)	42K	7 / 8
robotwin_latest/	RoboTwin 2.0	50K	14 / 14

Each directory contains:

• step_last.pt: model weights (about 12 GB).
• dataset_stats.json: state and action normalization statistics. Use the file from the same directory as the checkpoint at evaluation time.

Usage

Set up the environment, prepare the data, and install LIBERO or RoboTwin by following the GitHub README. Then download a checkpoint:

pip install -U huggingface_hub

hf download OKayQi/AED \
  libero_last/step_last.pt libero_last/dataset_stats.json \
  --local-dir checkpoints

hf download OKayQi/AED \
  robotwin_latest/step_last.pt robotwin_latest/dataset_stats.json \
  --local-dir checkpoints

Evaluate on LIBERO:

python experiments/libero/run_libero_manager.py \
  task=libero_aed_wam_vae_memory_2cam224_1e-4 \
  ckpt=./checkpoints/libero_last/step_last.pt \
  EVALUATION.dataset_stats_path=./checkpoints/libero_last/dataset_stats.json \
  MULTIRUN.num_gpus=8

Evaluate on RoboTwin:

python experiments/robotwin/run_robotwin_manager.py \
  task=robotwin_aed_wam_vae_memory_3cam384_1e-4 \
  ckpt=./checkpoints/robotwin_latest/step_last.pt \
  EVALUATION.dataset_stats_path=./checkpoints/robotwin_latest/dataset_stats.json \
  MULTIRUN.num_gpus=8

Training data

• LIBERO: yuanty/LIBERO-fastwam
• RoboTwin 2.0: yuanty/robotwin2.0-fastwam

Dependencies

• Base video model: Wan-AI/Wan2.2-TI2V-5B
• Frozen vision target: Robbyant/lingbot-vision-vit-base

These models are downloaded separately and keep their own licenses.

License

The code and the checkpoints in this repository are released under the MIT License.

Citation

@misc{lyu2026learningskillshistoricalaction,
      title={Learning Skills from Historical Action Trajectories: Action Experience Dictionary for World Action Models},
      author={Qi Lyu and Jiahua Dong and Hao Shen and Xudong Wang and Hongyuan Yu and Baichen Liu and Henghui Ding and Zhi Han and Nicu Sebe and Ivan Laptev and Fahad Shahbaz Khan and Salman Khan},
      year={2026},
      eprint={2609.40219},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2609.40219},
}