CMX (SegFormer, MiT-B2) trained on NYUv2
Demo: predictions of this checkpoint on NYUv2 test scenes from our dataset mirror (RGB, hha input, prediction overlay; per-frame pixel accuracy vs. ground truth), rendered with the official code.
Unofficial mirror of the official CMX weights from huaaaliu/RGBX_Semantic_Segmentation. The weights are unchanged; this repository adds a weights-only
model.safetensorscopy and aconfig.json.
Model
- Architecture: CMX: CMX (SegFormer, MiT-B2)
- Task: RGB-X (here RGB-D) semantic segmentation
- Trained on: NYUv2 (mirror on the Hub)
- Parameters: 66.6M
- Paper: CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers (IEEE T-ITS 2023)
- Code: https://github.com/huaaaliu/RGBX_Semantic_Segmentation
- License: MIT (from the official repository)
SegFormer (MiT-B2) dual-branch encoder with cross-modal feature rectification and fusion modules.
Results (reported by the authors)
| Metric | Value |
|---|---|
| mIoU (single-scale, %) | 54.1 |
| mIoU (multi-scale + flip, %) | 54.4 |
See the official repository for full tables and evaluation details.
Files
| File | Content |
|---|---|
NYUDV2_CMX+Segformer-B2.pth |
original checkpoint, exactly as released |
model.safetensors |
the model weights only (bit-exact copy of the checkpoint's model tensors; no optimizer state) |
config.json |
architecture, dataset, results and provenance |
Usage
import json
from huggingface_hub import hf_hub_download
repo = "dronefreak/cmx-mit-b2-nyuv2"
config = json.load(open(hf_hub_download(repo, "config.json")))
ckpt = hf_hub_download(repo, "NYUDV2_CMX+Segformer-B2.pth") # original format: use it with the official code
print(config["architecture"], config["dataset"], ckpt)
Build the network with the official code (https://github.com/huaaaliu/RGBX_Semantic_Segmentation) using the matching config for this variant, then load
NYUDV2_CMX+Segformer-B2.pth exactly as you would the original download, or load model.safetensors with
safetensors.torch.load_file into the same model.
Citation
If you use these weights, please cite the original work:
@article{zhang2023cmx,
title = {CMX: Cross-modal fusion for RGB-X semantic segmentation with transformers},
author = {Zhang, Jiaming and Liu, Huayao and Yang, Kailun and Hu, Xinxin and Liu, Ruiping and Stiefelhagen, Rainer},
journal = {IEEE Transactions on Intelligent Transportation Systems},
year = {2023}
}
Disclaimer
This is not an official release. All credit goes to the original authors (Jiaming Zhang, Huayao Liu, Kailun Yang, Xinxin Hu, Ruiping Liu, Rainer Stiefelhagen; KIT). The weights are redistributed under the MIT license of the official repository. If you are an author and want this mirror changed or removed, please open a discussion.
- Downloads last month
- -