docs: add CMNeXt model card

README.md

@@ -0,0 +1,127 @@
+---
+license: apache-2.0
+library_name: pytorch
+tags:
+- multimodal
+- semantic-segmentation
+- vision
+- cvpr-2023
+- cmnext
+- segformer
+- swin-transformer
+pipeline_tag: image-segmentation
+---
+
+# CMNeXt: Delivering Arbitrary-Modal Semantic Segmentation
+
+[CVPR 2023] Official Model Checkpoints Repository for **CMNeXt**.
+
+- **Project Page:** https://jamycheung.github.io/DELIVER.html
+- **GitHub Repository:** https://github.com/InSAI-Lab/DELIVER
+- **Paper:** [arXiv:2303.01480](https://arxiv.org/abs/2303.01480)
+- **Dataset Repository:** [InSAI-Lab/DELIVER](https://huggingface.co/datasets/InSAI-Lab/DELIVER)
+
+---
+
+## Model Description
+
+**CMNeXt** is an arbitrary cross-modal semantic segmentation model operating in the Hub2Fuse paradigm with asymmetric branches:
+- Multi-Head Self-Attention (MHSA) blocks in the RGB branch
+- Parallel Pooling Mixer (PPX) blocks in the accompanying modality branch
+- Self-Query Hub selects informative supplementary features
+- Feature Rectification Module (FRM) and Feature Fusion Module (FFM) fuse features dynamically across 1 to 81 modalities
+
+## Checkpoints Organization
+
+This repository contains trained weights and pretrained backbones matching the official DELIVER codebase structure:
+
+```text
+├── DELIVER/
+│   ├── cmnext_b2_deliver_rgb.pth
+│   ├── cmnext_b2_deliver_rgbd.pth
+│   ├── cmnext_b2_deliver_rgbde.pth
+│   ├── cmnext_b2_deliver_rgbdel.pth
+│   ├── cmnext_b2_deliver_rgbdl.pth
+│   ├── cmnext_b2_deliver_rgbe.pth
+│   └── cmnext_b2_deliver_rgbl.pth
+├── KITTI360/
+│   ├── cmnext_b2_kitti360_rgb.pth
+│   ├── cmnext_b2_kitti360_rgbd.pth
+│   ├── cmnext_b2_kitti360_rgbde.pth
+│   ├── cmnext_b2_kitti360_rgbdel.pth
+│   ├── cmnext_b2_kitti360_rgbdl.pth
+│   ├── cmnext_b2_kitti360_rgbe.pth
+│   └── cmnext_b2_kitti360_rgbl.pth
+├── MCubeS/
+│   ├── cmnext_b2_mcubes_rgb.pth
+│   ├── cmnext_b2_mcubes_rgba.pth
+│   ├── cmnext_b2_mcubes_rgbad.pth
+│   └── cmnext_b2_mcubes_rgbadn.pth
+├── MFNet/
+│   └── cmnext_b4_mfnet_rgbt.pth
+├── NYU_Depth_V2/
+│   └── cmnext_b4_nyu_rgbd.pth
+├── UrbanLF/
+│   ├── cmnext_b4_urbanlf_real_rgblf1.pth
+│   ├── cmnext_b4_urbanlf_real_rgblf33.pth
+│   ├── cmnext_b4_urbanlf_real_rgblf8.pth
+│   ├── cmnext_b4_urbanlf_real_rgblf80.pth
+│   ├── cmnext_b4_urbanlf_syn_rgblf1.pth
+│   ├── cmnext_b4_urbanlf_syn_rgblf33.pth
+│   ├── cmnext_b4_urbanlf_syn_rgblf8.pth
+│   └── cmnext_b4_urbanlf_syn_rgblf80.pth
+└── pretrained/
+    ├── segformers/
+    │   ├── mit_b0.pth ~ mit_b5.pth
+    └── swintransformer/
+        ├── swin_base_patch4_window12_384_22k.pth
+        ├── swin_large_patch4_window12_384_22k.pth
+        └── swin_small_patch4_window7_224.pth
+```
+
+## Benchmark Results
+
+### DELIVER Benchmark
+| Model-Modal | #Params(M) | GFLOPs | mIoU (%) | Checkpoint |
+| :--- | :--- | :--- | :--- | :--- |
+| CMNeXt-RGB | 25.79 | 38.93 | 57.20 | `DELIVER/cmnext_b2_deliver_rgb.pth` |
+| CMNeXt-RGB-E | 58.69 | 62.94 | 57.48 | `DELIVER/cmnext_b2_deliver_rgbe.pth` |
+| CMNeXt-RGB-L | 58.69 | 62.94 | 58.04 | `DELIVER/cmnext_b2_deliver_rgbl.pth` |
+| CMNeXt-RGB-D | 58.69 | 62.94 | 63.58 | `DELIVER/cmnext_b2_deliver_rgbd.pth` |
+| CMNeXt-RGB-D-E | 58.72 | 64.19 | 64.44 | `DELIVER/cmnext_b2_deliver_rgbde.pth` |
+| CMNeXt-RGB-D-L | 58.72 | 64.19 | 65.50 | `DELIVER/cmnext_b2_deliver_rgbdl.pth` |
+| CMNeXt-RGB-D-E-L | 58.73 | 65.42 | **66.30** | `DELIVER/cmnext_b2_deliver_rgbdel.pth` |
+
+### Other Benchmarks
+- **KITTI-360:** CMNeXt-RGB-D-E-L achieves 67.84% mIoU
+- **NYU Depth V2:** CMNeXt-RGB-D (MiT-B4) achieves 56.90% mIoU
+- **MFNet (RGB-T):** CMNeXt-RGB-T (MiT-B4) achieves 59.90% mIoU
+- **UrbanLF:** Up to 83.22% mIoU (Real) / 81.02% mIoU (Synthetic)
+- **MCubeS:** CMNeXt-RGB-A-D-N achieves 51.54% mIoU
+
+## Download & Usage
+
+Using `hf`:
+
+```bash
+# Download all checkpoints into output directory
+hf download InSAI-Lab/CMNeXt --local-dir output/
+```
+
+Evaluate with DELIVER repository:
+
+```bash
+cd DELIVER
+CUDA_VISIBLE_DEVICES=0 python tools/val_mm.py --cfg configs/deliver_rgbdel.yaml
+```
+
+## Citation
+
+```bibtex
+@inproceedings{zhang2023delivering,
+  title={Delivering Arbitrary-Modal Semantic Segmentation},
+  author={Zhang, Jiaming and Liu, Ruiping and Shi, Hao and Yang, Kailun and Rei{\ss}, Simon and Peng, Kunyu and Fu, Haodong and Wang, Kaiwei and Stiefelhagen, Rainer},
+  booktitle={CVPR},
+  year={2023}
+}
+```