| --- |
| pipeline_tag: image-segmentation |
| library_name: pytorch |
| license: apache-2.0 |
| tags: |
| - vision-transformer |
| - bird-eye-view |
| - semantic-mapping |
| - egocentric-vision |
| - matterport3d |
| - replica |
| - pytorch |
| - wacv-2023 |
| --- |
| |
| # Trans4Map Model Zoo |
|
|
| Official model repository for **Trans4Map: Revisiting Holistic Bird's-Eye-View Mapping from Egocentric Images to Allocentric Semantics with Vision Transformers** (WACV 2023). |
|
|
| - **GitHub Repository:** [InSAI-Lab/Trans4Map](https://github.com/InSAI-Lab/Trans4Map) |
| - **Paper:** [arXiv:2207.06205](https://arxiv.org/abs/2207.06205) |
| - **Dataset / Ground Truth:** [InSAI-Lab/Trans4Map (Dataset Hub)](https://huggingface.co/datasets/InSAI-Lab/Trans4Map) |
| - **Demo Video:** [Trans4Map Video Demo](https://yangkailun.com/videos/trans4map.mp4) |
|
|
| --- |
|
|
| ## Architecture |
|
|
| Trans4Map proposes an end-to-end one-stage Transformer-based framework for Bird's-Eye-View mapping: |
| 1. **Transformer Encoder:** Efficient Transformer backbone (e.g. SegFormer MiT-B2, MiT-B4) extracts rich contextual features from egocentric image batches. |
| 2. **Bidirectional Allocentric Memory (BAM):** Projects egocentric features into an allocentric memory space. |
| 3. **Map Decoder:** Parses accumulated memory and predicts the complete top-down semantic segmentation map. |
|
|
| --- |
|
|
| ## Benchmark Results (Matterport3D) |
|
|
| | Method | Backbone | mIoU (%) | Checkpoint | |
| | :--- | :--- | :---: | :---: | |
| | ConvNeXt | ConvNeXt-T | 35.91 | - | |
| | ConvNeXt | ConvNeXt-S | 36.49 | - | |
| | FAN | FAN-T | 31.07 | - | |
| | FAN | FAN-S | 34.62 | - | |
| | Swin | Swin-T | 34.19 | - | |
| | Swin | Swin-S | 36.80 | - | |
| | **Trans4Map** | **MiT-B2** | **40.02** | [`Tran4Map_B2_20points.pkl`](https://huggingface.co/InSAI-Lab/Trans4Map/resolve/main/Tran4Map_B2_20points.pkl) | |
| | **Trans4Map** | **MiT-B4** | **40.88** | [`Tran4Map_B4_20points.pkl`](https://huggingface.co/InSAI-Lab/Trans4Map/resolve/main/Tran4Map_B4_20points.pkl) | |
|
|
| --- |
|
|
| ## Quick Start & Evaluation |
|
|
| ### Download Checkpoints: |
|
|
| ```bash |
| # Download MiT-B2 model |
| hf download InSAI-Lab/Trans4Map Tran4Map_B2_20points.pkl --local-dir ./checkpoints |
| |
| # Download MiT-B4 model |
| hf download InSAI-Lab/Trans4Map Tran4Map_B4_20points.pkl --local-dir ./checkpoints |
| ``` |
|
|
| ### Python Evaluation Example: |
|
|
| ```bash |
| python build_test_data_feature.py |
| python test.py |
| python eval/eval.py |
| python eval/eval_bfscore.py |
| ``` |
|
|
| --- |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{chen2023trans4map, |
| title={Trans4Map: Revisiting Holistic Bird's-Eye-View Mapping from Egocentric Images to Allocentric Semantics with Vision Transformers}, |
| author={Chen, Chang and Zhang, Jiaming and Yang, Kailun and Peng, Kunyu and Stiefelhagen, Rainer}, |
| booktitle={2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)}, |
| year={2023} |
| } |
| ``` |
|
|
| ## License |
|
|
| This repository is licensed under the [Apache-2.0 License](https://www.apache.org/licenses/LICENSE-2.0). |
|
|
