README.md

---
pipeline_tag: feature-matching
library_name: pytorch
license: apache-2.0
tags:
- transformer
- feature-matching
- accv-2022
- vision
- scannet
- megadepth
datasets:
- scannet
- megadepth
---

# MatchFormer: Interleaving Attention in Transformers for Feature Matching (ACCV 2022)

Official model checkpoints for **MatchFormer: Interleaving Attention in Transformers for Feature Matching** (ACCV 2022 Oral).

<div align="center">

[![Paper](https://img.shields.io/badge/arXiv-2203.09645-B31B1B.svg)](https://arxiv.org/abs/2203.09645)
[![GitHub](https://img.shields.io/badge/GitHub-MatchFormer-181717.svg?logo=github)](https://github.com/InSAI-Lab/MatchFormer)
[![Collection](https://img.shields.io/badge/HF-Collection-FFD21E.svg)](https://huggingface.co/collections/InSAI-Lab)

</div>

---

## 📌 Introduction

In this work, we propose a novel hierarchical extract-and-match transformer, termed **MatchFormer**. Inside each stage of the hierarchical encoder, we interleave self-attention for feature extraction and cross-attention for feature matching, enabling an intuitive and powerful **extract-and-match** scheme. MatchFormer establishes robust and accurate dense correspondences across diverse indoor and outdoor challenging scenes.

![MatchFormer Architecture](https://raw.githubusercontent.com/InSAI-Lab/MatchFormer/main/matchformer.png)

---

## 📂 Model Zoo & File Structure

```text
InSAI-Lab/MatchFormer
├── indoor-large-SEA.ckpt    # Indoor large model with SEA (Spatial-Entropy Attention) (89.5 MB)
├── indoor-lite-LA.ckpt      # Indoor lightweight model with LA (Linear Attention) (77.4 MB)
├── outdoor-large-LA.ckpt    # Outdoor large model with LA (77.4 MB)
└── outdoor-lite-SEA.ckpt    # Outdoor lightweight model with SEA (103.5 MB)
```

---

## 📊 Benchmark Results

### 1. Indoor Relative Pose Estimation on ScanNet (AUC)

| Model | Backbone | Resolution | AUC @ 5° | AUC @ 10° | AUC @ 20° | Weights |
| :--- | :--- | :---: | :---: | :---: | :---: | :--- |
| **MatchFormer-Large** | largesea | (8, 2) | **43.3** | **63.4** | **77.1** | [`indoor-large-SEA.ckpt`](https://huggingface.co/InSAI-Lab/MatchFormer/resolve/main/indoor-large-SEA.ckpt) |
| **MatchFormer-Lite** | litela | (8, 4) | **38.8** | **59.2** | **74.0** | [`indoor-lite-LA.ckpt`](https://huggingface.co/InSAI-Lab/MatchFormer/resolve/main/indoor-lite-LA.ckpt) |

### 2. Outdoor Relative Pose Estimation on MegaDepth (AUC)

| Model | Backbone | Resolution | AUC @ 5° | AUC @ 10° | AUC @ 20° | Weights |
| :--- | :--- | :---: | :---: | :---: | :---: | :--- |
| **MatchFormer-Large** | largela | (8, 2) | **53.3** | **69.7** | **81.8** | [`outdoor-large-LA.ckpt`](https://huggingface.co/InSAI-Lab/MatchFormer/resolve/main/outdoor-large-LA.ckpt) |
| **MatchFormer-Lite** | litesea | (8, 4) | **48.2** | **65.1** | **78.4** | [`outdoor-lite-SEA.ckpt`](https://huggingface.co/InSAI-Lab/MatchFormer/resolve/main/outdoor-lite-SEA.ckpt) |

---

## ⚡ Quick Download Guide

### Option 1: Hugging Face CLI

```bash
# Download all MatchFormer weights to model/weights/
hf download InSAI-Lab/MatchFormer --local-dir model/weights

# Or download a specific checkpoint
hf download InSAI-Lab/MatchFormer indoor-large-SEA.ckpt --local-dir model/weights
```

### Option 2: Python `huggingface_hub`

```python
from huggingface_hub import hf_hub_download

# Download indoor large SEA checkpoint
ckpt_path = hf_hub_download(
    repo_id="InSAI-Lab/MatchFormer",
    filename="indoor-large-SEA.ckpt",
    local_dir="model/weights"
)
```

---

## 📖 Citation

```bibtex
@inproceedings{wang2022matchformer,
  title={MatchFormer: Interleaving Attention in Transformers for Feature Matching},
  author={Wang, Qing and Zhang, Jiaming and Yang, Kailun and Peng, Kunyu and Stiefelhagen, Rainer},
  booktitle={Asian Conference on Computer Vision (ACCV)},
  year={2022}
}
```