File size: 4,553 Bytes
8fb1903 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 | ---
license: apache-2.0
library_name: pytorch
tags:
- multimodal
- semantic-segmentation
- vision
- cvpr-2023
- cmnext
- segformer
- swin-transformer
pipeline_tag: image-segmentation
---
# CMNeXt: Delivering Arbitrary-Modal Semantic Segmentation
[CVPR 2023] Official Model Checkpoints Repository for **CMNeXt**.
- **Project Page:** https://jamycheung.github.io/DELIVER.html
- **GitHub Repository:** https://github.com/InSAI-Lab/DELIVER
- **Paper:** [arXiv:2303.01480](https://arxiv.org/abs/2303.01480)
- **Dataset Repository:** [InSAI-Lab/DELIVER](https://huggingface.co/datasets/InSAI-Lab/DELIVER)
---
## Model Description
**CMNeXt** is an arbitrary cross-modal semantic segmentation model operating in the Hub2Fuse paradigm with asymmetric branches:
- Multi-Head Self-Attention (MHSA) blocks in the RGB branch
- Parallel Pooling Mixer (PPX) blocks in the accompanying modality branch
- Self-Query Hub selects informative supplementary features
- Feature Rectification Module (FRM) and Feature Fusion Module (FFM) fuse features dynamically across 1 to 81 modalities
## Checkpoints Organization
This repository contains trained weights and pretrained backbones matching the official DELIVER codebase structure:
```text
├── DELIVER/
│ ├── cmnext_b2_deliver_rgb.pth
│ ├── cmnext_b2_deliver_rgbd.pth
│ ├── cmnext_b2_deliver_rgbde.pth
│ ├── cmnext_b2_deliver_rgbdel.pth
│ ├── cmnext_b2_deliver_rgbdl.pth
│ ├── cmnext_b2_deliver_rgbe.pth
│ └── cmnext_b2_deliver_rgbl.pth
├── KITTI360/
│ ├── cmnext_b2_kitti360_rgb.pth
│ ├── cmnext_b2_kitti360_rgbd.pth
│ ├── cmnext_b2_kitti360_rgbde.pth
│ ├── cmnext_b2_kitti360_rgbdel.pth
│ ├── cmnext_b2_kitti360_rgbdl.pth
│ ├── cmnext_b2_kitti360_rgbe.pth
│ └── cmnext_b2_kitti360_rgbl.pth
├── MCubeS/
│ ├── cmnext_b2_mcubes_rgb.pth
│ ├── cmnext_b2_mcubes_rgba.pth
│ ├── cmnext_b2_mcubes_rgbad.pth
│ └── cmnext_b2_mcubes_rgbadn.pth
├── MFNet/
│ └── cmnext_b4_mfnet_rgbt.pth
├── NYU_Depth_V2/
│ └── cmnext_b4_nyu_rgbd.pth
├── UrbanLF/
│ ├── cmnext_b4_urbanlf_real_rgblf1.pth
│ ├── cmnext_b4_urbanlf_real_rgblf33.pth
│ ├── cmnext_b4_urbanlf_real_rgblf8.pth
│ ├── cmnext_b4_urbanlf_real_rgblf80.pth
│ ├── cmnext_b4_urbanlf_syn_rgblf1.pth
│ ├── cmnext_b4_urbanlf_syn_rgblf33.pth
│ ├── cmnext_b4_urbanlf_syn_rgblf8.pth
│ └── cmnext_b4_urbanlf_syn_rgblf80.pth
└── pretrained/
├── segformers/
│ ├── mit_b0.pth ~ mit_b5.pth
└── swintransformer/
├── swin_base_patch4_window12_384_22k.pth
├── swin_large_patch4_window12_384_22k.pth
└── swin_small_patch4_window7_224.pth
```
## Benchmark Results
### DELIVER Benchmark
| Model-Modal | #Params(M) | GFLOPs | mIoU (%) | Checkpoint |
| :--- | :--- | :--- | :--- | :--- |
| CMNeXt-RGB | 25.79 | 38.93 | 57.20 | `DELIVER/cmnext_b2_deliver_rgb.pth` |
| CMNeXt-RGB-E | 58.69 | 62.94 | 57.48 | `DELIVER/cmnext_b2_deliver_rgbe.pth` |
| CMNeXt-RGB-L | 58.69 | 62.94 | 58.04 | `DELIVER/cmnext_b2_deliver_rgbl.pth` |
| CMNeXt-RGB-D | 58.69 | 62.94 | 63.58 | `DELIVER/cmnext_b2_deliver_rgbd.pth` |
| CMNeXt-RGB-D-E | 58.72 | 64.19 | 64.44 | `DELIVER/cmnext_b2_deliver_rgbde.pth` |
| CMNeXt-RGB-D-L | 58.72 | 64.19 | 65.50 | `DELIVER/cmnext_b2_deliver_rgbdl.pth` |
| CMNeXt-RGB-D-E-L | 58.73 | 65.42 | **66.30** | `DELIVER/cmnext_b2_deliver_rgbdel.pth` |
### Other Benchmarks
- **KITTI-360:** CMNeXt-RGB-D-E-L achieves 67.84% mIoU
- **NYU Depth V2:** CMNeXt-RGB-D (MiT-B4) achieves 56.90% mIoU
- **MFNet (RGB-T):** CMNeXt-RGB-T (MiT-B4) achieves 59.90% mIoU
- **UrbanLF:** Up to 83.22% mIoU (Real) / 81.02% mIoU (Synthetic)
- **MCubeS:** CMNeXt-RGB-A-D-N achieves 51.54% mIoU
## Download & Usage
Using `hf`:
```bash
# Download all checkpoints into output directory
hf download InSAI-Lab/CMNeXt --local-dir output/
```
Evaluate with DELIVER repository:
```bash
cd DELIVER
CUDA_VISIBLE_DEVICES=0 python tools/val_mm.py --cfg configs/deliver_rgbdel.yaml
```
## Citation
```bibtex
@inproceedings{zhang2023delivering,
title={Delivering Arbitrary-Modal Semantic Segmentation},
author={Zhang, Jiaming and Liu, Ruiping and Shi, Hao and Yang, Kailun and Rei{\ss}, Simon and Peng, Kunyu and Fu, Haodong and Wang, Kaiwei and Stiefelhagen, Rainer},
booktitle={CVPR},
year={2023}
}
```
|