README.md

---
license: apache-2.0
library_name: pytorch
tags:
- panoramic
- bird-eye-view
- semantic-mapping
- indoor-scenes
- vision
- wacv-2024
- segformer
- segnext
- 360mapper
pipeline_tag: image-segmentation
---

# 360Mapper: Panoramic Semantic Mapping for Indoor Bird's-Eye View

[IEEE/CVF WACV 2024] Official Model Checkpoints Repository for **360BEV** and **360Mapper**.

- **GitHub Repository:** https://github.com/InSAI-Lab/360BEV
- **Project Page:** https://jamycheung.github.io/360BEV.html
- **Dataset Repository:** [InSAI-Lab/360BEV](https://huggingface.co/datasets/InSAI-Lab/360BEV)
- **Paper:** [arXiv:2303.11910](https://arxiv.org/abs/2303.11910)
- **Collection:** [360BEV & 360Mapper (WACV 2024)](https://huggingface.co/collections/InSAI-Lab/360bev-and-360mapper-wacv-2024-6ad1bdcdeb1718eb4753c447)

---

## Model Description

**360Mapper** is designed for end-to-end panoramic semantic mapping to indoor Bird's-Eye View (BEV). It bridges 360° panoramic observations with top-down allocentric metric maps using cross-attention mechanisms, deformable attention, and high-capacity backbones (SegFormer MiT and SegNeXt MSCAN).

## Benchmark Performance

### 1. 360FV Stanford-2D3D (Front-View Semantic Segmentation)
| Model | Backbone | Input | mIoU (%) | Checkpoint |
| :--- | :--- | :---: | :---: | :--- |
| Trans4PASS | MiT-B2 | RGB | 52.10 | -- |
| CBFC | ResNet-101 | RGB | 52.20 | -- |
| **Ours (360FV)** | **MiT-B2** | **RGB** | **54.30** | [`360FV_Stanford/`](https://huggingface.co/InSAI-Lab/360BEV/tree/main/360FV_Stanford) |

### 2. 360FV Matterport3D (Front-View Semantic Segmentation)
| Model | Backbone | Input | mIoU (%) | Checkpoint |
| :--- | :--- | :---: | :---: | :--- |
| HoHoNet | ResNet-101 | RGB-D | 44.85 | -- |
| SegFormer | MiT-B2 | RGB | 45.53 | -- |
| **Ours (360FV)** | **MiT-B2** | **RGB** | **46.35** | [`360FV_Matterport/fv_mp3d.pkl`](https://huggingface.co/InSAI-Lab/360BEV/resolve/main/360FV_Matterport/fv_mp3d.pkl) |

### 3. 360BEV-Stanford (Panoramic BEV Semantic Mapping)
| Method | Backbone | Acc (%) | mRecall (%) | mPrecision (%) | mIoU (%) | Checkpoint |
| :--- | :---: | :---: | :---: | :---: | :---: | :--- |
| Trans4Map | MiT-B0 | 86.41 | 40.45 | 57.47 | 32.26 | -- |
| Trans4Map | MiT-B2 | 86.53 | 45.28 | 62.61 | 36.08 | -- |
| **360Mapper (Ours)** | **MiT-B0** | 92.07 | 50.14 | 65.37 | 42.42 | [`360BEV_Stanford/360BEV_s2d3d_segformer_b0.pkl`](https://huggingface.co/InSAI-Lab/360BEV/resolve/main/360BEV_Stanford/360BEV_s2d3d_segformer_b0.pkl) |
| **360Mapper (Ours)** | **MiT-B2** | **92.80** | 53.56 | 67.72 | 45.78 | [`360BEV_Stanford/360BEV_s2d3d_segformer_b2.pkl`](https://huggingface.co/InSAI-Lab/360BEV/resolve/main/360BEV_Stanford/360BEV_s2d3d_segformer_b2.pkl) |
| **360Mapper (Ours)** | **MSCA-B** | 92.67 | **55.02** | **68.02** | **46.44** | [`360BEV_Stanford/360BEV_s2d3d_segnext_B.pkl`](https://huggingface.co/InSAI-Lab/360BEV/resolve/main/360BEV_Stanford/360BEV_s2d3d_segnext_B.pkl) |

### 4. 360BEV-Matterport (Panoramic BEV Semantic Mapping)
| Method | Backbone | Acc (%) | mRecall (%) | mPrecision (%) | mIoU (%) | Checkpoint |
| :--- | :---: | :---: | :---: | :---: | :---: | :--- |
| Trans4Map | MiT-B0 | 70.19 | 44.31 | 50.39 | 31.92 | -- |
| Trans4Map | MiT-B2 | 73.28 | 51.60 | 53.02 | 36.72 | -- |
| **360Mapper (Ours)** | **MiT-B0** | 75.44 | 48.80 | 56.01 | 36.98 | [`360BEV_Matterport/360BEV_mp3d_segformer_b0.pkl`](https://huggingface.co/InSAI-Lab/360BEV/resolve/main/360BEV_Matterport/360BEV_mp3d_segformer_b0.pkl) |
| **360Mapper (Ours)** | **MiT-B2** | 78.80 | 59.54 | 59.97 | 44.32 | [`360BEV_Matterport/360BEV_mp3d_segformer_b2.pkl`](https://huggingface.co/InSAI-Lab/360BEV/resolve/main/360BEV_Matterport/360BEV_mp3d_segformer_b2.pkl) |
| **360Mapper (Ours)** | **MSCA-B** | **78.93** | **60.51** | **62.83** | **46.31** | [`360BEV_Matterport/360BEV_mp3d_segnext_B.pkl`](https://huggingface.co/InSAI-Lab/360BEV/resolve/main/360BEV_Matterport/360BEV_mp3d_segnext_B.pkl) |

## Checkpoints Organization

```text
├── 360FV_Stanford/
│   ├── fv_s2d3d_fold1.pkl
│   ├── fv_s2d3d_fold2.pkl
│   └── fv_s2d3d_fold3.pkl
├── 360FV_Matterport/
│   └── fv_mp3d.pkl
├── 360BEV_Stanford/
│   ├── 360BEV_s2d3d_segformer_b0.pkl
│   ├── 360BEV_s2d3d_segformer_b2.pkl
│   └── 360BEV_s2d3d_segnext_B.pkl
└── 360BEV_Matterport/
    ├── 360BEV_mp3d_segformer_b0.pkl
    ├── 360BEV_mp3d_segformer_b2.pkl
    └── 360BEV_mp3d_segnext_B.pkl
```

## Quick Start & Evaluation

### 1. Download Weights

Using Hugging Face CLI:
```bash
# Download all checkpoints
hf download InSAI-Lab/360BEV --local-dir ./checkpoints

# Or download specific model
hf download InSAI-Lab/360BEV 360BEV_Stanford/360BEV_s2d3d_segformer_b2.pkl --local-dir ./checkpoints
```

Using Python:
```python
from huggingface_hub import hf_hub_download

model_path = hf_hub_download(
    repo_id="InSAI-Lab/360BEV",
    filename="360BEV_Stanford/360BEV_s2d3d_segformer_b2.pkl",
    local_dir="./checkpoints"
)
```

### 2. Evaluation

In the [360BEV codebase](https://github.com/InSAI-Lab/360BEV):

```bash
# 360BEV Matterport
python test_360BEV_Matterport.py --config configs/model_360BEV_mp3d.yml

# 360BEV Stanford-2D3D
python test_360BEV_S2d3d.py --config configs/model_360BEV_s2d3d.yml

# Stanford2D3D (360FV)
python test_pano_360Attention_S2d3d.py --config configs/model_fv_s2d3d.yml

# 360FV-Matterport
python test_pano_360Attention_Matterport.py --config configs/model_fv_mp3d.yml
```

## Citation

```bibtex
@inproceedings{teng2024_360bev,
  title={360BEV: Panoramic Semantic Mapping for Indoor Bird's-Eye View}, 
  author={Teng, Zhifeng and Zhang, Jiaming and Yang, Kailun and Peng, Kunyu and Shi, Hao and Reiß, Simon and Cao, Ke and Stiefelhagen, Rainer},
  booktitle={2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year={2024}
}
```

## License

This repository is licensed under the [Apache-2.0 License](https://www.apache.org/licenses/LICENSE-2.0).