| --- |
| license: other |
| license_name: vips |
| license_link: LICENSE |
| language: |
| - en |
| base_model: |
| - Qwen/Qwen3-VL-8B-Instruct |
| datasets: |
| - nyu-visionx/VSI-Bench |
| tags: |
| - vips |
| - spatial-understanding |
| - video-question-answering |
| - visual-priors |
| - lora |
| - arxiv:2607.15054 |
| --- |
| |
| # Beyond Single Expert: Harmonizing Diverse Visual Priors in MLLMs for Spatial Understanding |
|
|
| **NeurIPS 2026** |
|
|
| Xiao Lin, Xiaohu Huang, Kai Han |
|
|
| Visual AI Lab, The University of Hong Kong |
|
|
| [Paper](https://arxiv.org/abs/2607.15054) · [Project page](https://visual-ai.github.io/vips/) · [Code](https://github.com/Visual-AI/ViPS) |
|
|
| ## Checkpoints |
|
|
| | Location | Description | |
| | --- | --- | |
| | `vips/` | ViPS checkpoint. | |
| | `vggt_multihead.pt` | Prior proxy checkpoint for prior feature extraction. | |
|
|
| ## Download |
|
|
| ViPS requires [Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct). Run these commands from the [code repository](https://github.com/Visual-AI/ViPS) root: |
|
|
| ```bash |
| hf download leeiioolin/ViPS --local-dir checkpoints |
| hf download Qwen/Qwen3-VL-8B-Instruct --local-dir models/Qwen3-VL-8B-Instruct |
| ``` |
|
|
| See the [code README](https://github.com/Visual-AI/ViPS#readme) for setup and usage. |
|
|
| ## Evaluation of the ViPS checkpoint |
|
|
| VSI-Bench results: |
|
|
| | Model | Overall | Obj. Count | Abs. Dist. | Obj. Size | Room Size | Rel. Dist. | Rel. Dir. | Route Plan | Appr. Order | |
| | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | |
| | ViPS | **63.7** | 69.6 | 54.7 | 71.1 | 72.5 | 60.1 | 81.9 | 45.9 | 53.6 | |
|
|
| ## License |
|
|
| | Component | License | |
| |:--|:--| |
| | Original ViPS code and original adapter / fusion-module contributions | [MIT](LICENSE) | |
| | Qwen-derived components | [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) | |
| | Prior Proxy (`vggt_multihead.pt`, derived from VGGT-1B) | [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) — non-commercial | |
|
|
| See [LICENSE](LICENSE) for attribution, scope, and full terms. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{Lin2026ViPS, |
| author = {Xiao Lin and Xiaohu Huang and Kai Han}, |
| title = {Beyond Single Expert: Harmonizing Diverse Visual Priors in MLLMs for Spatial Understanding}, |
| journal = {arXiv preprint arXiv:2607.15054}, |
| year = {2026} |
| } |
| ``` |
|
|
