README.md

---
license: apache-2.0
library_name: pytorch
tags:
- panoramic
- vision-transformer
- semantic-segmentation
- domain-adaptation
- trans4pass
- trans4pass-plus
- cvpr-2022
pipeline_tag: image-segmentation
---

# Trans4PASS & Trans4PASS+: Transformers for Panoramic Semantic Segmentation

Official Model Checkpoints Repository for **Trans4PASS** and **Trans4PASS+**.

- **GitHub Repository:** https://github.com/InSAI-Lab/Trans4PASS
- **Paper (Trans4PASS, CVPR 2022):** [arXiv:2203.01452](https://arxiv.org/abs/2203.01452)
- **Paper (Trans4PASS+, arXiv 2022):** [arXiv:2207.11860](https://arxiv.org/abs/2207.11860)
- **Dataset Repository:** [InSAI-Lab/SynPASS](https://huggingface.co/datasets/InSAI-Lab/SynPASS)

---

## Checkpoints Organization

This repository contains all pretrained backbones, supervised weights, and domain adaptation checkpoints matching the Trans4PASS project directory structure:

```text
├── pretrained/
│   ├── mit_b1.pth
│   └── mit_b2.pth
├── adaptations/
│   └── snapshots/
│       ├── CS2DensePASS_Trans4PASS_v1_MPA/BestCS2DensePASS_G.pth
│       ├── CS2DensePASS_Trans4PASS_v2_MPA/BestCS2DensePASS_G.pth
│       ├── CS132CS132DP13_Trans4PASS_plus_v2_MPA/BestCS132DP13_G.pth
│       ├── CS2DP_Trans4PASS_plus_v1_MPA/BestCS2DensePASS_G.pth
│       └── CS2DP_Trans4PASS_plus_v2_MPA/BestCS2DensePASS_G.pth
└── workdirs/
    ├── cityscapes/ (trans4pass & plus, tiny & small)
    ├── cityscapes13/ (trans4pass & plus, tiny & small)
    ├── stanford2d3d/ (trans4pass & plus, tiny & small)
    ├── stanford2d3d8/ (trans4pass & plus, tiny & small)
    ├── stanford2d3d_pan/ (folds F1, F2, F3)
    ├── structured3d8/ (trans4pass & plus, tiny & small)
    ├── synpass/ (trans4pass & plus, tiny & small)
    └── synpass13/ (trans4pass & plus, tiny & small)
```

## Benchmark Results

### Cityscapes -> DensePASS (Domain Adaptation)
| Model | CS mIoU (%) | DP mIoU (%) |
| :--- | :---: | :---: |
| Trans4PASS (Tiny) | 72.49 | 45.89 |
| Trans4PASS (Small) | 72.84 | 51.38 |
| Trans4PASS+ (Tiny) | 72.67 | 50.23 |
| Trans4PASS+ (Small) | **75.24** | **51.41** |

### SynPASS Benchmark
| Model | Val mIoU (%) | Test mIoU (%) |
| :--- | :---: | :---: |
| Trans4PASS (Tiny) | 43.68 | 38.53 |
| Trans4PASS (Small) | 44.80 | 38.57 |
| Trans4PASS+ (Tiny) | 45.21 | 38.85 |
| Trans4PASS+ (Small) | **46.47** | **39.16** |

## Download & Usage

Using `hf`:

```bash
# Download all models to the local directory
hf download InSAI-Lab/Trans4PASS --local-dir .
```

Evaluation example:

```bash
# In Trans4PASS repository
python tools/eval_sp.py --config-file configs/synpass/trans4pass_plus_tiny_512x512.yaml
```

## Citation

```bibtex
@inproceedings{zhang2022bending,
  title={Bending Reality: Distortion-aware Transformers for Adapting to Panoramic Semantic Segmentation},
  author={Zhang, Jiaming and Yang, Kailun and Ma, Chaoxiang and Rei{\ss}, Simon and Peng, Kunyu and Stiefelhagen, Rainer},
  booktitle={CVPR},
  pages={16917--16927},
  year={2022}
}

@article{zhang2022behind,
  title={Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation},
  author={Zhang, Jiaming and Yang, Kailun and Shi, Hao and Rei{\ss}, Simon and Peng, Kunyu and Ma, Chaoxiang and Fu, Haodong and Wang, Kaiwei and Stiefelhagen, Rainer},
  journal={arXiv preprint arXiv:2207.11860},
  year={2022}
}
```