dronefreak's picture
Add demo banner (official model on test scenes) (#2)
03a4962
|
Raw History Blame Contribute Delete
3.82 kB
metadata
license: mit
library_name: pytorch
pipeline_tag: image-segmentation
tags:
  - rgb-d
  - semantic-segmentation
  - indoor-scene-understanding
  - dformer
  - sunrgbd
  - community-mirror
datasets:
  - dronefreak/SUN-RGBD

DFormer-T trained on SUN RGB-D

Demo: predictions of this checkpoint on SUN RGB-D test scenes from our dataset mirror (RGB, depth input (8-bit), prediction overlay; per-frame pixel accuracy vs. ground truth), rendered with the official code.

Unofficial mirror of the official DFormer weights from VCIP-RGBD/DFormer. The weights are unchanged; this repository adds a weights-only model.safetensors copy and a config.json.

Model

RGB-D pretrained encoder (ImageNet-1K with depth) plus a lightweight decoder head. The encoder used for initialisation is included under pretrained/ (its .pth.tar stores training args, so load it with weights_only=False).

Results (reported by the authors)

Metric Value
mIoU (SUN RGB-D, single-scale) 48.8

See the official repository for full tables and evaluation details.

Files

File Content
SUNRGBD_DFormer_Tiny.pth original checkpoint, exactly as released
model.safetensors the model weights only (bit-exact copy of the checkpoint's model tensors; no optimizer state)
config.json architecture, dataset, results and provenance
pretrained/DFormer_Tiny.pth.tar DFormer-T encoder, RGB-D pretrained on ImageNet-1K

Usage

import json
from huggingface_hub import hf_hub_download

repo = "dronefreak/dformer-tiny-sunrgbd"
config = json.load(open(hf_hub_download(repo, "config.json")))
ckpt = hf_hub_download(repo, "SUNRGBD_DFormer_Tiny.pth")   # original format: use it with the official code
print(config["architecture"], config["dataset"], ckpt)

Build the network with the official code (https://github.com/VCIP-RGBD/DFormer) using the matching config for this variant, then load SUNRGBD_DFormer_Tiny.pth exactly as you would the original download, or load model.safetensors with safetensors.torch.load_file into the same model.

Citation

If you use these weights, please cite the original work:

@inproceedings{yin2024dformer,
  title     = {DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation},
  author    = {Yin, Bowen and Zhang, Xuying and Li, Zhong-Yu and Liu, Li and Cheng, Ming-Ming and Hou, Qibin},
  booktitle = {ICLR},
  year      = {2024}
}

Disclaimer

This is not an official release. All credit goes to the original authors (Bowen Yin, Xuying Zhang, Zhong-Yu Li, Li Liu, Ming-Ming Cheng, Qibin Hou; Nankai University (VCIP)). The weights are redistributed under the MIT license of the official repository. If you are an author and want this mirror changed or removed, please open a discussion.