metadata
license: apache-2.0
library_name: pytorch
tags:
- panoramic
- vision-transformer
- semantic-segmentation
- domain-adaptation
- trans4pass
- trans4pass-plus
- cvpr-2022
pipeline_tag: image-segmentation
Trans4PASS & Trans4PASS+: Transformers for Panoramic Semantic Segmentation
Official Model Checkpoints Repository for Trans4PASS and Trans4PASS+.
- GitHub Repository: https://github.com/InSAI-Lab/Trans4PASS
- Paper (Trans4PASS, CVPR 2022): arXiv:2203.01452
- Paper (Trans4PASS+, arXiv 2022): arXiv:2207.11860
- Dataset Repository: InSAI-Lab/SynPASS
Checkpoints Organization
This repository contains all pretrained backbones, supervised weights, and domain adaptation checkpoints matching the Trans4PASS project directory structure:
├── pretrained/
│ ├── mit_b1.pth
│ └── mit_b2.pth
├── adaptations/
│ └── snapshots/
│ ├── CS2DensePASS_Trans4PASS_v1_MPA/BestCS2DensePASS_G.pth
│ ├── CS2DensePASS_Trans4PASS_v2_MPA/BestCS2DensePASS_G.pth
│ ├── CS132CS132DP13_Trans4PASS_plus_v2_MPA/BestCS132DP13_G.pth
│ ├── CS2DP_Trans4PASS_plus_v1_MPA/BestCS2DensePASS_G.pth
│ └── CS2DP_Trans4PASS_plus_v2_MPA/BestCS2DensePASS_G.pth
└── workdirs/
├── cityscapes/ (trans4pass & plus, tiny & small)
├── cityscapes13/ (trans4pass & plus, tiny & small)
├── stanford2d3d/ (trans4pass & plus, tiny & small)
├── stanford2d3d8/ (trans4pass & plus, tiny & small)
├── stanford2d3d_pan/ (folds F1, F2, F3)
├── structured3d8/ (trans4pass & plus, tiny & small)
├── synpass/ (trans4pass & plus, tiny & small)
└── synpass13/ (trans4pass & plus, tiny & small)
Benchmark Results
Cityscapes -> DensePASS (Domain Adaptation)
| Model | CS mIoU (%) | DP mIoU (%) |
|---|---|---|
| Trans4PASS (Tiny) | 72.49 | 45.89 |
| Trans4PASS (Small) | 72.84 | 51.38 |
| Trans4PASS+ (Tiny) | 72.67 | 50.23 |
| Trans4PASS+ (Small) | 75.24 | 51.41 |
SynPASS Benchmark
| Model | Val mIoU (%) | Test mIoU (%) |
|---|---|---|
| Trans4PASS (Tiny) | 43.68 | 38.53 |
| Trans4PASS (Small) | 44.80 | 38.57 |
| Trans4PASS+ (Tiny) | 45.21 | 38.85 |
| Trans4PASS+ (Small) | 46.47 | 39.16 |
Download & Usage
Using hf:
# Download all models to the local directory
hf download InSAI-Lab/Trans4PASS --local-dir .
Evaluation example:
# In Trans4PASS repository
python tools/eval_sp.py --config-file configs/synpass/trans4pass_plus_tiny_512x512.yaml
Citation
@inproceedings{zhang2022bending,
title={Bending Reality: Distortion-aware Transformers for Adapting to Panoramic Semantic Segmentation},
author={Zhang, Jiaming and Yang, Kailun and Ma, Chaoxiang and Rei{\ss}, Simon and Peng, Kunyu and Stiefelhagen, Rainer},
booktitle={CVPR},
pages={16917--16927},
year={2022}
}
@article{zhang2022behind,
title={Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation},
author={Zhang, Jiaming and Yang, Kailun and Shi, Hao and Rei{\ss}, Simon and Peng, Kunyu and Ma, Chaoxiang and Fu, Haodong and Wang, Kaiwei and Stiefelhagen, Rainer},
journal={arXiv preprint arXiv:2207.11860},
year={2022}
}