| --- |
| license: other |
| language: |
| - en |
| tags: |
| - vision-and-language-navigation |
| - embodied-ai |
| - pytorch |
| --- |
| |
| # User-Feedback-Driven Adaptation for Vision-and-Language Navigation |
|
|
| Data and checkpoint for the paper *User-Feedback-Driven Adaptation for Vision-and-Language Navigation*, IEEE Transactions on Multimedia, 2026. |
|
|
| [[Paper (IEEE TMM)](https://doi.org/10.1109/TMM.2026.3724731)] [[arXiv](https://arxiv.org/abs/2512.10322)] [[Code](https://github.com/seaotter999/UFD)] |
|
|
| ## Download |
|
|
| ```bash |
| huggingface-cli download Peachilk/UFD --local-dir data |
| ``` |
|
|
| This gives the `data/` directory used by the scripts in the [code repository](https://github.com/seaotter999/UFD). Some files come from other projects and are not included here: the connectivity graphs and CLIP features (ScaleVLN), `prevalent_aug_train_enc.json` (GSA-R2R) and the initial GR-DUET checkpoint `best_val_unseen` (GSA-VLN). The code README explains where to get them. |
|
|
| ## Contents |
|
|
| | Path | Description | |
| |---|---| |
| | `ckpts/best_overall_iter_23500` | GR-DUET fine-tuned on paths reconstructed from user feedback in the Basic setting (`rec_basic`); the checkpoint with the best average validation SPL (iteration 23,500). It is also the model evaluated with the memory bank. | |
| | `split_basic/split_test_500_<scan>.json` | 500 instructions per scan on which user feedback is collected; input to path reconstruction (24 scans, 12,000 entries). | |
| | `split_mixed/split_test_500_<scan>.json` | Same for the mixed user styles (21 scans, 10,500 entries). | |
| | `rec_basic/merged_train_dataset.json` | Training set of reconstructed paths with 5 to 7 viewpoints (24 scans, 11,342 entries). | |
| | `rec_mixed/merged_train_dataset.json` | Same for the mixed user styles (21 scans, 9,766 entries). | |
| | `rec_basic/merged_500_test_dataset.json`, `rec_mixed/merge_500_test_dataset_mixed.json` | The 500 instructions of all scans in one file, used to build the memory bank. | |
| | `GSA_Dataset/Validation/Residential/Reconstruction/validation_set_100.json` | Validation set: up to 100 reconstructed paths per scan (24 scans, 2,390 entries). | |
| | `GSA_Dataset/Test/Residential/Reconstruction/merged_test_dataset.json` | Test set: 100 held-out instructions per scan (24 scans, 2,400 entries). | |
| | `GSA_Dataset/Test/Residential/Recmixed/merged_test_dataset.json` | Test set for the mixed user styles (21 scans, 2,100 entries). | |
| | `GSA_Dataset/Train/R2R_train_enc.json` | R2R training instructions with BERT token ids. | |
| | `scanvp_cands_relangles_with_habitat.json` | Candidate viewpoints and relative angles of every viewpoint. | |
| | `pano_inputs_habitats_more_scan.h5` | Precomputed panorama inputs of every viewpoint, used to build full graphs during training. | |
|
|
| `basic` uses the Basic instructions of GSA-R2R; `mixed` combines instructions in five user styles (child, keith, moira, rachel, sheldon). |
|
|
| ## License |
|
|
| The data and checkpoint are for non-commercial research use only. Please also follow the terms of use of the original datasets: R2R, Matterport3D, HM3D and GSA-R2R. The code is released under the [MIT License](https://github.com/seaotter999/UFD/blob/main/LICENSE). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{yu2026userfeedback, |
| title = {User-Feedback-Driven Adaptation for Vision-and-Language Navigation}, |
| author = {Yu, Yongqiang and Li, Xuhui and Mahmood, Hazza and Zhou, Jinxing and Hong, Haodong and Jiang, Longtao and Xu, Zhiqiang and Wu, Qi and Chang, Xiaojun}, |
| journal = {IEEE Transactions on Multimedia}, |
| year = {2026}, |
| doi = {10.1109/TMM.2026.3724731} |
| } |
| ``` |
|
|
