Hugging Face's logo

InSAI-Lab
/
CMNeXt

Image Segmentation
PyTorch
multimodal
semantic-segmentation
vision
cvpr-2023
cmnext
segformer
swin-transformer
Model card Files
xet
 Community
CMNeXt / README.md
insailab's picture
insailab
docs: add CMNeXt model card
8fb1903 verified
preview code
|
Raw History Blame Contribute Delete
4.55 kB
metadata 
license: apache-2.0
library_name: pytorch
tags:
  - multimodal
  - semantic-segmentation
  - vision
  - cvpr-2023
  - cmnext
  - segformer
  - swin-transformer
pipeline_tag: image-segmentation

CMNeXt: Delivering Arbitrary-Modal Semantic Segmentation

[CVPR 2023] Official Model Checkpoints Repository for CMNeXt.

Model Description

CMNeXt is an arbitrary cross-modal semantic segmentation model operating in the Hub2Fuse paradigm with asymmetric branches:

  • Multi-Head Self-Attention (MHSA) blocks in the RGB branch
  • Parallel Pooling Mixer (PPX) blocks in the accompanying modality branch
  • Self-Query Hub selects informative supplementary features
  • Feature Rectification Module (FRM) and Feature Fusion Module (FFM) fuse features dynamically across 1 to 81 modalities

Checkpoints Organization

This repository contains trained weights and pretrained backbones matching the official DELIVER codebase structure:

├── DELIVER/
│   ├── cmnext_b2_deliver_rgb.pth
│   ├── cmnext_b2_deliver_rgbd.pth
│   ├── cmnext_b2_deliver_rgbde.pth
│   ├── cmnext_b2_deliver_rgbdel.pth
│   ├── cmnext_b2_deliver_rgbdl.pth
│   ├── cmnext_b2_deliver_rgbe.pth
│   └── cmnext_b2_deliver_rgbl.pth
├── KITTI360/
│   ├── cmnext_b2_kitti360_rgb.pth
│   ├── cmnext_b2_kitti360_rgbd.pth
│   ├── cmnext_b2_kitti360_rgbde.pth
│   ├── cmnext_b2_kitti360_rgbdel.pth
│   ├── cmnext_b2_kitti360_rgbdl.pth
│   ├── cmnext_b2_kitti360_rgbe.pth
│   └── cmnext_b2_kitti360_rgbl.pth
├── MCubeS/
│   ├── cmnext_b2_mcubes_rgb.pth
│   ├── cmnext_b2_mcubes_rgba.pth
│   ├── cmnext_b2_mcubes_rgbad.pth
│   └── cmnext_b2_mcubes_rgbadn.pth
├── MFNet/
│   └── cmnext_b4_mfnet_rgbt.pth
├── NYU_Depth_V2/
│   └── cmnext_b4_nyu_rgbd.pth
├── UrbanLF/
│   ├── cmnext_b4_urbanlf_real_rgblf1.pth
│   ├── cmnext_b4_urbanlf_real_rgblf33.pth
│   ├── cmnext_b4_urbanlf_real_rgblf8.pth
│   ├── cmnext_b4_urbanlf_real_rgblf80.pth
│   ├── cmnext_b4_urbanlf_syn_rgblf1.pth
│   ├── cmnext_b4_urbanlf_syn_rgblf33.pth
│   ├── cmnext_b4_urbanlf_syn_rgblf8.pth
│   └── cmnext_b4_urbanlf_syn_rgblf80.pth
└── pretrained/
    ├── segformers/
    │   ├── mit_b0.pth ~ mit_b5.pth
    └── swintransformer/
        ├── swin_base_patch4_window12_384_22k.pth
        ├── swin_large_patch4_window12_384_22k.pth
        └── swin_small_patch4_window7_224.pth

Benchmark Results

DELIVER Benchmark

Model-Modal #Params(M) GFLOPs mIoU (%) Checkpoint
CMNeXt-RGB 25.79 38.93 57.20 DELIVER/cmnext_b2_deliver_rgb.pth
CMNeXt-RGB-E 58.69 62.94 57.48 DELIVER/cmnext_b2_deliver_rgbe.pth
CMNeXt-RGB-L 58.69 62.94 58.04 DELIVER/cmnext_b2_deliver_rgbl.pth
CMNeXt-RGB-D 58.69 62.94 63.58 DELIVER/cmnext_b2_deliver_rgbd.pth
CMNeXt-RGB-D-E 58.72 64.19 64.44 DELIVER/cmnext_b2_deliver_rgbde.pth
CMNeXt-RGB-D-L 58.72 64.19 65.50 DELIVER/cmnext_b2_deliver_rgbdl.pth
CMNeXt-RGB-D-E-L 58.73 65.42 66.30 DELIVER/cmnext_b2_deliver_rgbdel.pth

Other Benchmarks

  • KITTI-360: CMNeXt-RGB-D-E-L achieves 67.84% mIoU
  • NYU Depth V2: CMNeXt-RGB-D (MiT-B4) achieves 56.90% mIoU
  • MFNet (RGB-T): CMNeXt-RGB-T (MiT-B4) achieves 59.90% mIoU
  • UrbanLF: Up to 83.22% mIoU (Real) / 81.02% mIoU (Synthetic)
  • MCubeS: CMNeXt-RGB-A-D-N achieves 51.54% mIoU

Download & Usage

Using hf:

# Download all checkpoints into output directory
hf download InSAI-Lab/CMNeXt --local-dir output/

Evaluate with DELIVER repository:

cd DELIVER
CUDA_VISIBLE_DEVICES=0 python tools/val_mm.py --cfg configs/deliver_rgbdel.yaml

Citation 

@inproceedings{zhang2023delivering,
  title={Delivering Arbitrary-Modal Semantic Segmentation},
  author={Zhang, Jiaming and Liu, Ruiping and Shi, Hao and Yang, Kailun and Rei{\ss}, Simon and Peng, Kunyu and Fu, Haodong and Wang, Kaiwei and Stiefelhagen, Rainer},
  booktitle={CVPR},
  year={2023}
}