| --- |
| license: apache-2.0 |
| library_name: pytorch |
| tags: |
| - multimodal |
| - semantic-segmentation |
| - vision |
| - cvpr-2023 |
| - cmnext |
| - segformer |
| - swin-transformer |
| pipeline_tag: image-segmentation |
| --- |
| |
| # CMNeXt: Delivering Arbitrary-Modal Semantic Segmentation |
|
|
| [CVPR 2023] Official Model Checkpoints Repository for **CMNeXt**. |
|
|
| - **Project Page:** https://jamycheung.github.io/DELIVER.html |
| - **GitHub Repository:** https://github.com/InSAI-Lab/DELIVER |
| - **Paper:** [arXiv:2303.01480](https://arxiv.org/abs/2303.01480) |
| - **Dataset Repository:** [InSAI-Lab/DELIVER](https://huggingface.co/datasets/InSAI-Lab/DELIVER) |
|
|
| --- |
|
|
| ## Model Description |
|
|
| **CMNeXt** is an arbitrary cross-modal semantic segmentation model operating in the Hub2Fuse paradigm with asymmetric branches: |
| - Multi-Head Self-Attention (MHSA) blocks in the RGB branch |
| - Parallel Pooling Mixer (PPX) blocks in the accompanying modality branch |
| - Self-Query Hub selects informative supplementary features |
| - Feature Rectification Module (FRM) and Feature Fusion Module (FFM) fuse features dynamically across 1 to 81 modalities |
|
|
| ## Checkpoints Organization |
|
|
| This repository contains trained weights and pretrained backbones matching the official DELIVER codebase structure: |
|
|
| ```text |
| ├── DELIVER/ |
| │ ├── cmnext_b2_deliver_rgb.pth |
| │ ├── cmnext_b2_deliver_rgbd.pth |
| │ ├── cmnext_b2_deliver_rgbde.pth |
| │ ├── cmnext_b2_deliver_rgbdel.pth |
| │ ├── cmnext_b2_deliver_rgbdl.pth |
| │ ├── cmnext_b2_deliver_rgbe.pth |
| │ └── cmnext_b2_deliver_rgbl.pth |
| ├── KITTI360/ |
| │ ├── cmnext_b2_kitti360_rgb.pth |
| │ ├── cmnext_b2_kitti360_rgbd.pth |
| │ ├── cmnext_b2_kitti360_rgbde.pth |
| │ ├── cmnext_b2_kitti360_rgbdel.pth |
| │ ├── cmnext_b2_kitti360_rgbdl.pth |
| │ ├── cmnext_b2_kitti360_rgbe.pth |
| │ └── cmnext_b2_kitti360_rgbl.pth |
| ├── MCubeS/ |
| │ ├── cmnext_b2_mcubes_rgb.pth |
| │ ├── cmnext_b2_mcubes_rgba.pth |
| │ ├── cmnext_b2_mcubes_rgbad.pth |
| │ └── cmnext_b2_mcubes_rgbadn.pth |
| ├── MFNet/ |
| │ └── cmnext_b4_mfnet_rgbt.pth |
| ├── NYU_Depth_V2/ |
| │ └── cmnext_b4_nyu_rgbd.pth |
| ├── UrbanLF/ |
| │ ├── cmnext_b4_urbanlf_real_rgblf1.pth |
| │ ├── cmnext_b4_urbanlf_real_rgblf33.pth |
| │ ├── cmnext_b4_urbanlf_real_rgblf8.pth |
| │ ├── cmnext_b4_urbanlf_real_rgblf80.pth |
| │ ├── cmnext_b4_urbanlf_syn_rgblf1.pth |
| │ ├── cmnext_b4_urbanlf_syn_rgblf33.pth |
| │ ├── cmnext_b4_urbanlf_syn_rgblf8.pth |
| │ └── cmnext_b4_urbanlf_syn_rgblf80.pth |
| └── pretrained/ |
| ├── segformers/ |
| │ ├── mit_b0.pth ~ mit_b5.pth |
| └── swintransformer/ |
| ├── swin_base_patch4_window12_384_22k.pth |
| ├── swin_large_patch4_window12_384_22k.pth |
| └── swin_small_patch4_window7_224.pth |
| ``` |
|
|
| ## Benchmark Results |
|
|
| ### DELIVER Benchmark |
| | Model-Modal | #Params(M) | GFLOPs | mIoU (%) | Checkpoint | |
| | :--- | :--- | :--- | :--- | :--- | |
| | CMNeXt-RGB | 25.79 | 38.93 | 57.20 | `DELIVER/cmnext_b2_deliver_rgb.pth` | |
| | CMNeXt-RGB-E | 58.69 | 62.94 | 57.48 | `DELIVER/cmnext_b2_deliver_rgbe.pth` | |
| | CMNeXt-RGB-L | 58.69 | 62.94 | 58.04 | `DELIVER/cmnext_b2_deliver_rgbl.pth` | |
| | CMNeXt-RGB-D | 58.69 | 62.94 | 63.58 | `DELIVER/cmnext_b2_deliver_rgbd.pth` | |
| | CMNeXt-RGB-D-E | 58.72 | 64.19 | 64.44 | `DELIVER/cmnext_b2_deliver_rgbde.pth` | |
| | CMNeXt-RGB-D-L | 58.72 | 64.19 | 65.50 | `DELIVER/cmnext_b2_deliver_rgbdl.pth` | |
| | CMNeXt-RGB-D-E-L | 58.73 | 65.42 | **66.30** | `DELIVER/cmnext_b2_deliver_rgbdel.pth` | |
|
|
| ### Other Benchmarks |
| - **KITTI-360:** CMNeXt-RGB-D-E-L achieves 67.84% mIoU |
| - **NYU Depth V2:** CMNeXt-RGB-D (MiT-B4) achieves 56.90% mIoU |
| - **MFNet (RGB-T):** CMNeXt-RGB-T (MiT-B4) achieves 59.90% mIoU |
| - **UrbanLF:** Up to 83.22% mIoU (Real) / 81.02% mIoU (Synthetic) |
| - **MCubeS:** CMNeXt-RGB-A-D-N achieves 51.54% mIoU |
|
|
| ## Download & Usage |
|
|
| Using `hf`: |
|
|
| ```bash |
| # Download all checkpoints into output directory |
| hf download InSAI-Lab/CMNeXt --local-dir output/ |
| ``` |
|
|
| Evaluate with DELIVER repository: |
|
|
| ```bash |
| cd DELIVER |
| CUDA_VISIBLE_DEVICES=0 python tools/val_mm.py --cfg configs/deliver_rgbdel.yaml |
| ``` |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{zhang2023delivering, |
| title={Delivering Arbitrary-Modal Semantic Segmentation}, |
| author={Zhang, Jiaming and Liu, Ruiping and Shi, Hao and Yang, Kailun and Rei{\ss}, Simon and Peng, Kunyu and Fu, Haodong and Wang, Kaiwei and Stiefelhagen, Rainer}, |
| booktitle={CVPR}, |
| year={2023} |
| } |
| ``` |
|
|
