README.md

metadata

pipeline_tag: image-segmentation
library_name: pytorch
license: apache-2.0
tags:
  - vision-transformer
  - bird-eye-view
  - semantic-mapping
  - egocentric-vision
  - matterport3d
  - replica
  - pytorch
  - wacv-2023

Trans4Map Model Zoo

Official model repository for Trans4Map: Revisiting Holistic Bird's-Eye-View Mapping from Egocentric Images to Allocentric Semantics with Vision Transformers (WACV 2023).

• GitHub Repository: InSAI-Lab/Trans4Map
• Paper: arXiv:2207.06205
• Dataset / Ground Truth: InSAI-Lab/Trans4Map (Dataset Hub)
• Demo Video: Trans4Map Video Demo

---

Architecture

Trans4Map proposes an end-to-end one-stage Transformer-based framework for Bird's-Eye-View mapping:

1. Transformer Encoder: Efficient Transformer backbone (e.g. SegFormer MiT-B2, MiT-B4) extracts rich contextual features from egocentric image batches.
2. Bidirectional Allocentric Memory (BAM): Projects egocentric features into an allocentric memory space.
3. Map Decoder: Parses accumulated memory and predicts the complete top-down semantic segmentation map.

---

Benchmark Results (Matterport3D)

Method	Backbone	mIoU (%)	Checkpoint
ConvNeXt	ConvNeXt-T	35.91	-
ConvNeXt	ConvNeXt-S	36.49	-
FAN	FAN-T	31.07	-
FAN	FAN-S	34.62	-
Swin	Swin-T	34.19	-
Swin	Swin-S	36.80	-
Trans4Map	MiT-B2	40.02	Tran4Map_B2_20points.pkl
Trans4Map	MiT-B4	40.88	Tran4Map_B4_20points.pkl

---

Quick Start & Evaluation

Download Checkpoints:

# Download MiT-B2 model
hf download InSAI-Lab/Trans4Map Tran4Map_B2_20points.pkl --local-dir ./checkpoints

# Download MiT-B4 model
hf download InSAI-Lab/Trans4Map Tran4Map_B4_20points.pkl --local-dir ./checkpoints

Python Evaluation Example:

python build_test_data_feature.py
python test.py
python eval/eval.py
python eval/eval_bfscore.py

---

Citation

@inproceedings{chen2023trans4map,
  title={Trans4Map: Revisiting Holistic Bird's-Eye-View Mapping from Egocentric Images to Allocentric Semantics with Vision Transformers},
  author={Chen, Chang and Zhang, Jiaming and Yang, Kailun and Peng, Kunyu and Stiefelhagen, Rainer},
  booktitle={2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year={2023}
}

License

This repository is licensed under the Apache-2.0 License.