--- license: mit library_name: pytorch pipeline_tag: image-segmentation tags: - rgb-d - semantic-segmentation - indoor-scene-understanding - dformer - sunrgbd - community-mirror datasets: - dronefreak/SUN-RGBD --- # DFormer-T trained on SUN RGB-D
*Demo: predictions of this checkpoint on SUN RGB-D test scenes from our dataset mirror (RGB, depth input (8-bit), prediction overlay; per-frame pixel accuracy vs. ground truth), rendered with the official code.* > **Unofficial mirror** of the official DFormer weights from [VCIP-RGBD/DFormer](https://github.com/VCIP-RGBD/DFormer). The weights are **unchanged**; this repository adds a weights-only `model.safetensors` copy and a `config.json`. ## Model - **Architecture:** DFormer: DFormer-T - **Task:** RGB-D semantic segmentation - **Trained on:** [SUN RGB-D](https://huggingface.co/datasets/dronefreak/SUN-RGBD) (mirror on the Hub) - **Parameters:** 6.0M - **Paper:** [DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation](https://arxiv.org/abs/2309.09668) (ICLR 2024) - **Code:** https://github.com/VCIP-RGBD/DFormer - **License:** MIT (from the official repository) RGB-D pretrained encoder (ImageNet-1K with depth) plus a lightweight decoder head. The encoder used for initialisation is included under `pretrained/` (its `.pth.tar` stores training args, so load it with `weights_only=False`). ## Results (reported by the authors) | Metric | Value | |---|---| | mIoU (SUN RGB-D, single-scale) | 48.8 | See the [official repository](https://github.com/VCIP-RGBD/DFormer) for full tables and evaluation details. ## Files | File | Content | |---|---| | `SUNRGBD_DFormer_Tiny.pth` | original checkpoint, exactly as released | | `model.safetensors` | the model weights only (bit-exact copy of the checkpoint's `model` tensors; no optimizer state) | | `config.json` | architecture, dataset, results and provenance | | `pretrained/DFormer_Tiny.pth.tar` | DFormer-T encoder, RGB-D pretrained on ImageNet-1K | ## Usage ```python import json from huggingface_hub import hf_hub_download repo = "dronefreak/dformer-tiny-sunrgbd" config = json.load(open(hf_hub_download(repo, "config.json"))) ckpt = hf_hub_download(repo, "SUNRGBD_DFormer_Tiny.pth") # original format: use it with the official code print(config["architecture"], config["dataset"], ckpt) ``` Build the network with the official code (https://github.com/VCIP-RGBD/DFormer) using the matching config for this variant, then load `SUNRGBD_DFormer_Tiny.pth` exactly as you would the original download, or load `model.safetensors` with `safetensors.torch.load_file` into the same model. ## Citation If you use these weights, please cite the original work: ```bibtex @inproceedings{yin2024dformer, title = {DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation}, author = {Yin, Bowen and Zhang, Xuying and Li, Zhong-Yu and Liu, Li and Cheng, Ming-Ming and Hou, Qibin}, booktitle = {ICLR}, year = {2024} } ``` ## Disclaimer This is not an official release. All credit goes to the original authors (Bowen Yin, Xuying Zhang, Zhong-Yu Li, Li Liu, Ming-Ming Cheng, Qibin Hou; Nankai University (VCIP)). The weights are redistributed under the MIT license of the official repository. If you are an author and want this mirror changed or removed, please open a discussion.