CMNeXt: Delivering Arbitrary-Modal Semantic Segmentation

[CVPR 2023] Official Model Checkpoints Repository for CMNeXt.

• Project Page: https://jamycheung.github.io/DELIVER.html
• GitHub Repository: https://github.com/InSAI-Lab/DELIVER
• Paper: arXiv:2303.01480
• Dataset Repository: InSAI-Lab/DELIVER

---

Model Description

CMNeXt is an arbitrary cross-modal semantic segmentation model operating in the Hub2Fuse paradigm with asymmetric branches:

• Multi-Head Self-Attention (MHSA) blocks in the RGB branch
• Parallel Pooling Mixer (PPX) blocks in the accompanying modality branch
• Self-Query Hub selects informative supplementary features
• Feature Rectification Module (FRM) and Feature Fusion Module (FFM) fuse features dynamically across 1 to 81 modalities

Checkpoints Organization

This repository contains trained weights and pretrained backbones matching the official DELIVER codebase structure:

├── DELIVER/
│   ├── cmnext_b2_deliver_rgb.pth
│   ├── cmnext_b2_deliver_rgbd.pth
│   ├── cmnext_b2_deliver_rgbde.pth
│   ├── cmnext_b2_deliver_rgbdel.pth
│   ├── cmnext_b2_deliver_rgbdl.pth
│   ├── cmnext_b2_deliver_rgbe.pth
│   └── cmnext_b2_deliver_rgbl.pth
├── KITTI360/
│   ├── cmnext_b2_kitti360_rgb.pth
│   ├── cmnext_b2_kitti360_rgbd.pth
│   ├── cmnext_b2_kitti360_rgbde.pth
│   ├── cmnext_b2_kitti360_rgbdel.pth
│   ├── cmnext_b2_kitti360_rgbdl.pth
│   ├── cmnext_b2_kitti360_rgbe.pth
│   └── cmnext_b2_kitti360_rgbl.pth
├── MCubeS/
│   ├── cmnext_b2_mcubes_rgb.pth
│   ├── cmnext_b2_mcubes_rgba.pth
│   ├── cmnext_b2_mcubes_rgbad.pth
│   └── cmnext_b2_mcubes_rgbadn.pth
├── MFNet/
│   └── cmnext_b4_mfnet_rgbt.pth
├── NYU_Depth_V2/
│   └── cmnext_b4_nyu_rgbd.pth
├── UrbanLF/
│   ├── cmnext_b4_urbanlf_real_rgblf1.pth
│   ├── cmnext_b4_urbanlf_real_rgblf33.pth
│   ├── cmnext_b4_urbanlf_real_rgblf8.pth
│   ├── cmnext_b4_urbanlf_real_rgblf80.pth
│   ├── cmnext_b4_urbanlf_syn_rgblf1.pth
│   ├── cmnext_b4_urbanlf_syn_rgblf33.pth
│   ├── cmnext_b4_urbanlf_syn_rgblf8.pth
│   └── cmnext_b4_urbanlf_syn_rgblf80.pth
└── pretrained/
    ├── segformers/
    │   ├── mit_b0.pth ~ mit_b5.pth
    └── swintransformer/
        ├── swin_base_patch4_window12_384_22k.pth
        ├── swin_large_patch4_window12_384_22k.pth
        └── swin_small_patch4_window7_224.pth

Benchmark Results

DELIVER Benchmark

Model-Modal	#Params(M)	GFLOPs	mIoU (%)	Checkpoint
CMNeXt-RGB	25.79	38.93	57.20	DELIVER/cmnext_b2_deliver_rgb.pth
CMNeXt-RGB-E	58.69	62.94	57.48	DELIVER/cmnext_b2_deliver_rgbe.pth
CMNeXt-RGB-L	58.69	62.94	58.04	DELIVER/cmnext_b2_deliver_rgbl.pth
CMNeXt-RGB-D	58.69	62.94	63.58	DELIVER/cmnext_b2_deliver_rgbd.pth
CMNeXt-RGB-D-E	58.72	64.19	64.44	DELIVER/cmnext_b2_deliver_rgbde.pth
CMNeXt-RGB-D-L	58.72	64.19	65.50	DELIVER/cmnext_b2_deliver_rgbdl.pth
CMNeXt-RGB-D-E-L	58.73	65.42	66.30	DELIVER/cmnext_b2_deliver_rgbdel.pth

Other Benchmarks

• KITTI-360: CMNeXt-RGB-D-E-L achieves 67.84% mIoU
• NYU Depth V2: CMNeXt-RGB-D (MiT-B4) achieves 56.90% mIoU
• MFNet (RGB-T): CMNeXt-RGB-T (MiT-B4) achieves 59.90% mIoU
• UrbanLF: Up to 83.22% mIoU (Real) / 81.02% mIoU (Synthetic)
• MCubeS: CMNeXt-RGB-A-D-N achieves 51.54% mIoU

Download & Usage

Using hf:

# Download all checkpoints into output directory
hf download InSAI-Lab/CMNeXt --local-dir output/

Evaluate with DELIVER repository:

cd DELIVER
CUDA_VISIBLE_DEVICES=0 python tools/val_mm.py --cfg configs/deliver_rgbdel.yaml

Citation

@inproceedings{zhang2023delivering,
  title={Delivering Arbitrary-Modal Semantic Segmentation},
  author={Zhang, Jiaming and Liu, Ruiping and Shi, Hao and Yang, Kailun and Rei{\ss}, Simon and Peng, Kunyu and Fu, Haodong and Wang, Kaiwei and Stiefelhagen, Rainer},
  booktitle={CVPR},
  year={2023}
}