|
Download README.md from zhuyuze/SC2_WM: direct link, hf CLI and curl.
- Browser
- Download file 3.41 kB
-
https://huggingface.co/zhuyuze/SC2_WM/resolve/main/README.md
- Command line
-
hf download hf://zhuyuze/SC2_WM/README.md
-
curl -L -o README.md https://huggingface.co/zhuyuze/SC2_WM/resolve/main/README.md
3.41 kB
| license: mit | |
| language: | |
| - en | |
| pipeline_tag: robotics | |
| tags: | |
| - vision-and-language-navigation | |
| - vln-ce | |
| - world-model | |
| - embodied-ai | |
| - habitat | |
| - navigation | |
| - robotics | |
| # SC²-WM | |
| Official checkpoints for **SC²-WM: A Self-Correcting World Model with Closed-Loop Feedback for Vision-and-Language Navigation in Continuous Environments** (ICML 2026). | |
| 📄 Paper: https://arxiv.org/abs/2608.07548 | |
| 💻 Code: https://github.com/sunrise-yuze/SC2_WM | |
| Authors: Xuan Yao, Yuze Zhu, Junyu Gao, Zongmeng Wang, Changsheng Xu | |
| ## Overview | |
| Vision-and-Language Navigation in Continuous Environments (VLN-CE) requires agents to make fine-grained navigation decisions under partial observability. However, most existing methods rely on open-loop execution, lacking mechanisms to detect and correct internal state drift during inference. We propose **SC²-WM, a self-correcting world model framework that introduces internal feedback for closed-loop decision making** in VLN-CE. Our method derives feedback from world-model foresight to perform state-level plan refinement before action execution. To handle challenging scenarios, we further introduce conditional world-aware adaptation, which enables model-level correction by selectively updating the world model at test time when feedback indicates model capacity insufficiency. The framework is built on Habitat-Sim and is evaluated on the R2R-CE and RxR-CE benchmarks. | |
| ## Checkpoints | |
| This repository hosts the released weights for the **R2R-CE** setting. | |
| | File | Description | | |
| |---|---| | |
| | `ckpt.46600.pth` | R2R-CE main checkpoint (fine-tuned SC²-WM model) | | |
| | `ViT-B-16.pt` | CLIP ViT-B/16 vision encoder | | |
| | `pretrained/model_step_100000.pt` | VLN-BERT (R2R variant) | | |
| | `pretrained/model_step_82500.pt` | VLN-BERT (default) | | |
| | `pretrained/cwp_predictor.pth` | Candidate waypoint predictor | | |
| | `pretrained/NeRF_p16_8x8.pth` | NeRF rendering module | | |
| | `pretrained/segm.pt` | Image segmentation module | | |
| | `pretrained/resnet18-f37072fd.pth` | ResNet-18 backbone | | |
| ## Results | |
| ### R2R-CE (val_unseen) | |
| | Method | SR | SPL | NE ↓ | | |
| |---|---|---|---| | |
| | Baseline (VLN-3DFF) | 44.9 | 30.4 | 5.95 | | |
| | **SC²-WM (Ours)** | **50.9** | **37.2** | **5.37** | | |
| ### RxR-CE (val_unseen) | |
| | Method | SR | SPL | NE ↓ | | |
| |---|---|---|---| | |
| | Baseline (VLN-3DFF) | 25.5 | 18.1 | 8.79 | | |
| | **SC²-WM (Ours)** | **35.8** | **27.2** | **8.36** | | |
| ## Usage | |
| Please refer to our [GitHub repository](https://github.com/sunrise-yuze/SC2_WM) for installation, data preparation, and evaluation instructions. Download the checkpoints into the directory layout described in the repository README. | |
| You can download the weights with the `huggingface_hub` library: | |
| ```python | |
| from huggingface_hub import snapshot_download | |
| snapshot_download(repo_id="zhuyuze/SC2_WM", local_dir="SC2_WM") | |
| ``` | |
| ## Citation | |
| If you find this project useful in your research, please consider citing: | |
| ```bibtex | |
| @inproceedings{yao2026sc2wm, | |
| title = {{$SC^2$-WM}: A Self-Correcting World Model with Closed-Loop Feedback for Vision-and-Language Navigation in Continuous Environments}, | |
| author = {Yao, Xuan and Zhu, Yuze and Gao, Junyu and Wang, Zongmeng and Xu, Changsheng}, | |
| booktitle = {Proceedings of the 43rd International Conference on Machine Learning (ICML)}, | |
| year = {2026} | |
| } | |
| ``` | |
| ## License | |
| Released under the [MIT License](https://github.com/sunrise-yuze/SC2_WM/blob/main/LICENSE). | |