SC2_WM / README.md
zhuyuze
Add model card with paper link and checkpoint details
95a4208
|
Raw History Blame Contribute Delete
3.41 kB
---
license: mit
language:
- en
pipeline_tag: robotics
tags:
- vision-and-language-navigation
- vln-ce
- world-model
- embodied-ai
- habitat
- navigation
- robotics
---
# SC²-WM
Official checkpoints for **SC²-WM: A Self-Correcting World Model with Closed-Loop Feedback for Vision-and-Language Navigation in Continuous Environments** (ICML 2026).
📄 Paper: https://arxiv.org/abs/2608.07548
💻 Code: https://github.com/sunrise-yuze/SC2_WM
Authors: Xuan Yao, Yuze Zhu, Junyu Gao, Zongmeng Wang, Changsheng Xu
## Overview
Vision-and-Language Navigation in Continuous Environments (VLN-CE) requires agents to make fine-grained navigation decisions under partial observability. However, most existing methods rely on open-loop execution, lacking mechanisms to detect and correct internal state drift during inference. We propose **SC²-WM, a self-correcting world model framework that introduces internal feedback for closed-loop decision making** in VLN-CE. Our method derives feedback from world-model foresight to perform state-level plan refinement before action execution. To handle challenging scenarios, we further introduce conditional world-aware adaptation, which enables model-level correction by selectively updating the world model at test time when feedback indicates model capacity insufficiency. The framework is built on Habitat-Sim and is evaluated on the R2R-CE and RxR-CE benchmarks.
## Checkpoints
This repository hosts the released weights for the **R2R-CE** setting.
| File | Description |
|---|---|
| `ckpt.46600.pth` | R2R-CE main checkpoint (fine-tuned SC²-WM model) |
| `ViT-B-16.pt` | CLIP ViT-B/16 vision encoder |
| `pretrained/model_step_100000.pt` | VLN-BERT (R2R variant) |
| `pretrained/model_step_82500.pt` | VLN-BERT (default) |
| `pretrained/cwp_predictor.pth` | Candidate waypoint predictor |
| `pretrained/NeRF_p16_8x8.pth` | NeRF rendering module |
| `pretrained/segm.pt` | Image segmentation module |
| `pretrained/resnet18-f37072fd.pth` | ResNet-18 backbone |
## Results
### R2R-CE (val_unseen)
| Method | SR | SPL | NE ↓ |
|---|---|---|---|
| Baseline (VLN-3DFF) | 44.9 | 30.4 | 5.95 |
| **SC²-WM (Ours)** | **50.9** | **37.2** | **5.37** |
### RxR-CE (val_unseen)
| Method | SR | SPL | NE ↓ |
|---|---|---|---|
| Baseline (VLN-3DFF) | 25.5 | 18.1 | 8.79 |
| **SC²-WM (Ours)** | **35.8** | **27.2** | **8.36** |
## Usage
Please refer to our [GitHub repository](https://github.com/sunrise-yuze/SC2_WM) for installation, data preparation, and evaluation instructions. Download the checkpoints into the directory layout described in the repository README.
You can download the weights with the `huggingface_hub` library:
```python
from huggingface_hub import snapshot_download
snapshot_download(repo_id="zhuyuze/SC2_WM", local_dir="SC2_WM")
```
## Citation
If you find this project useful in your research, please consider citing:
```bibtex
@inproceedings{yao2026sc2wm,
title = {{$SC^2$-WM}: A Self-Correcting World Model with Closed-Loop Feedback for Vision-and-Language Navigation in Continuous Environments},
author = {Yao, Xuan and Zhu, Yuze and Gao, Junyu and Wang, Zongmeng and Xu, Changsheng},
booktitle = {Proceedings of the 43rd International Conference on Machine Learning (ICML)},
year = {2026}
}
```
## License
Released under the [MIT License](https://github.com/sunrise-yuze/SC2_WM/blob/main/LICENSE).