--- base_model: - Wan-AI/Wan2.2-TI2V-5B license: other pipeline_tag: robotics tags: - pytorch - end-to-end-autonomous-driving - world-action-model - navsim ---

SimWAM: A Simple World Action Model for End-to-End Autonomous Driving

GitHub repository Hugging Face model Paper
Zongchuang Zhao1, Xin Zhou1, Tianyang Xu1, Zhengyang Sun1, Kaixuan Zhou2, Honglin Li2, Dingkang Liang1†, Xiang Bai1
1 Huazhong University of Science and Technology
2 Dongfeng Research & Development Institute
† Project lead
## 🔍 Overview **SimWAM** is a World-Action Model for end-to-end autonomous driving. It transfers video-dynamics priors to trajectory prediction by jointly training a pretrained video expert and a lightweight action diffusion transformer with flow matching. An isolated attention mask prevents the action branch from accessing future-frame tokens. As a result, the video-generation branch is used only as a training signal and can be removed at inference time, leaving a self-contained action expert that directly predicts future driving trajectories. This repository contains the official supervised and reinforcement-learning checkpoints for SimWAM on NAVSIM.
Method Summary SimWAM jointly trains a pretrained video expert and an action expert through a unified attention interface. Both experts attend to the current visual observation, while future-video tokens and action tokens remain isolated from each other. This design transfers motion and traffic-dynamics knowledge from video prediction to trajectory planning without requiring future-scene generation during deployment. The action expert can subsequently be optimized using FlowGRPO and the compositional NAVSIM PDM reward.
### ✨ Key Ideas - Joint flow-matching training of a pretrained video expert and a lightweight action diffusion transformer. - Isolated attention between future-video tokens and action tokens. - Direct trajectory prediction without future-video generation at inference time. - Action-only reinforcement-learning fine-tuning with FlowGRPO. - Independent scaling of the video and action experts without shared model parameters. ## 🎯 Task, Inputs, and Outputs SimWAM performs end-to-end ego-trajectory planning from a single front-facing camera observation and vehicle-conditioning information. Because autonomous-driving trajectory planning does not correspond to a standard Hugging Face inference pipeline, this model repository uses `pipeline_tag: robotics`. | Benchmark | Inputs | Output | Prediction horizon | Sensor configuration | | --- | --- | --- | ---: | --- | | NAVSIM | Front RGB image, scene-context embedding, 8-D ego state | Normalized future ego trajectory with shape `[8, 3]` | 8 steps | One front-facing camera at `384 × 672` | ### Inputs The official NAVSIM configuration uses: - One front-facing RGB camera observation resized to `384 × 672` - Precomputed text or scene-context embeddings - An 8-dimensional ego-state representation - Model and evaluation configuration supplied by the official SimWAM codebase ### Outputs The action expert predicts: - An eight-step future ego trajectory - Three trajectory values for each predicted step - A tensor with shape `[8, 3]` for each evaluated sample The released checkpoints predict normalized trajectory values. The official dataset statistics and NAVSIM evaluation pipeline are required to convert predictions into the benchmark action representation. ## 📦 Released Checkpoints | Training stage | Checkpoint | Description | | --- | --- | --- | | Supervised learning | `weights/SimWAM.pt` | SimWAM checkpoint trained with joint video-action flow matching | | Reinforcement learning | `weights/SimWAM-RL.pt` | SimWAM checkpoint fine-tuned with FlowGRPO and the NAVSIM PDM reward | The released files are custom PyTorch checkpoints. They require the official SimWAM implementation and cannot be loaded directly with `transformers.AutoModel.from_pretrained()`. ## 🛠️ Usage You can download the complete model repository using either the Hugging Face CLI or the Python SDK. ### Option 1: Hugging Face CLI ```bash pip install -U huggingface_hub hf download H-EmbodVis/SimWAM \ --repo-type model \ --local-dir SimWAM-model ``` ### Option 2: Python SDK ```python from huggingface_hub import snapshot_download snapshot_download( repo_id="H-EmbodVis/SimWAM", repo_type="model", local_dir="SimWAM-model", ) ``` To download only the supervised checkpoint: ```python from huggingface_hub import hf_hub_download checkpoint_path = hf_hub_download( repo_id="H-EmbodVis/SimWAM", repo_type="model", filename="weights/SimWAM.pt", ) print(checkpoint_path) ``` ### Installation and Evaluation Clone and install the official implementation: ```bash git clone https://github.com/H-EmbodVis/SimWAM.git cd SimWAM conda create -n simwam python=3.10 -y conda activate simwam python -m pip install -r requirements.txt python -m pip install -e navsim --no-deps python -m pip install -e . --no-deps ``` Place the downloaded checkpoints in the `weights/` directory. A one-sample evaluation of the supervised checkpoint can be launched with: ```bash CKPT=./weights/SimWAM.pt \ TASK=navsim_uncond_front_384x672_1e-4 \ NPROC_PER_NODE=1 \ bash experiments/navsim/run_eval_navsim.sh \ EVALUATION.max_samples=1 \ EVALUATION.num_inference_steps=2 \ EVALUATION.save_videos=false ``` NAVSIM, nuPlan/OpenScene data, maps, sensor blobs, precomputed embeddings, and normalization statistics must be prepared according to the official GitHub documentation. ## 📈 Performance Using one front-facing camera at `384 × 672`, SimWAM reports a **91.5 PDMS** score on the NAVSIM `navtest` benchmark after reinforcement-learning fine-tuning. | Configuration | PDMS | | --- | ---: | | Action-only baseline | 86.6 | | Joint video-action training | 90.3 | | Joint training with FlowGRPO | **91.5** | For complete evaluation settings and comparisons, refer to the official implementation: https://github.com/H-EmbodVis/SimWAM ## 📂 Repository Contents | Path | Description | | --- | --- | | `weights/SimWAM.pt` | Supervised SimWAM checkpoint | | `weights/SimWAM-RL.pt` | FlowGRPO reinforcement-learning checkpoint | | `config.json` | Minimal Hugging Face model metadata | | `README.md` | Hugging Face model card | | `.gitattributes` | Hugging Face/Xet large-file configuration | Expected repository structure: ```text . ├── .gitattributes ├── config.json ├── README.md └── weights ├── SimWAM.pt └── SimWAM-RL.pt ``` ## ⚠️ Limitations and Safety - The checkpoints require the official SimWAM implementation and its pinned dependencies. - The model is designed for the NAVSIM data format and evaluation protocol. - Performance outside the training and evaluation distributions is not guaranteed. - The model may produce unsafe or invalid trajectories under unfamiliar road, weather, traffic, or sensor conditions. - Predictions must not be deployed directly on a physical vehicle without extensive validation, safety constraints, fallback systems, and human oversight. - NAVSIM, nuPlan, OpenScene, Wan2.2, and other upstream resources retain their respective licenses and distribution terms. ## 🔗 Links - **Paper:** https://huggingface.co/papers/2608.07468 - **Hugging Face:** https://huggingface.co/H-EmbodVis/SimWAM - **GitHub:** https://github.com/H-EmbodVis/SimWAM - **NAVSIM:** https://github.com/autonomousvision/navsim - **Wan2.2:** https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B ## ⚖️ License The SimWAM source repository is released under the MIT License. Third-party code, datasets, pretrained models, and other upstream assets remain subject to their respective licenses and usage terms. ## 📖 Citation If you find SimWAM useful in your research, please consider citing: ```bibtex @article{zhao2026simwam, title={SimWAM: A Simple World Action Model for End-to-End Autonomous Driving}, author={Zongchuang Zhao and Xin Zhou and Tianyang Xu and Zhengyang Sun and Kaixuan Zhou and Honglin Li and Dingkang Liang and Xiang Bai}, journal={arXiv preprint arXiv:2608.07468}, year = {2026} } ```