File size: 3,820 Bytes
e6862a1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
03a4962
 
 
 
e6862a1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
---
license: mit
library_name: pytorch
pipeline_tag: image-segmentation
tags:
- rgb-d
- semantic-segmentation
- indoor-scene-understanding
- dformer
- sunrgbd
- community-mirror
datasets:
- dronefreak/SUN-RGBD
---

# DFormer-T trained on SUN RGB-D

<p align="center"><video controls autoplay loop muted playsinline width="900" poster="https://huggingface.co/dronefreak/dformer-tiny-sunrgbd/resolve/main/assets/demo_banner_poster.jpg" src="https://huggingface.co/dronefreak/dformer-tiny-sunrgbd/resolve/main/assets/demo_banner.mp4"><img src="https://huggingface.co/dronefreak/dformer-tiny-sunrgbd/resolve/main/assets/demo_banner_poster.jpg" alt="DFormer-T predictions on SUN RGB-D test scenes" width="900"></video></p>

*Demo: predictions of this checkpoint on SUN RGB-D test scenes from our dataset mirror (RGB, depth input (8-bit), prediction overlay; per-frame pixel accuracy vs. ground truth), rendered with the official code.*

> **Unofficial mirror** of the official DFormer weights from [VCIP-RGBD/DFormer](https://github.com/VCIP-RGBD/DFormer). The weights are **unchanged**; this repository adds a weights-only `model.safetensors` copy and a `config.json`.

## Model

- **Architecture:** DFormer: DFormer-T
- **Task:** RGB-D semantic segmentation
- **Trained on:** [SUN RGB-D](https://huggingface.co/datasets/dronefreak/SUN-RGBD) (mirror on the Hub)
- **Parameters:** 6.0M
- **Paper:** [DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation](https://arxiv.org/abs/2309.09668) (ICLR 2024)
- **Code:** https://github.com/VCIP-RGBD/DFormer
- **License:** MIT (from the official repository)

RGB-D pretrained encoder (ImageNet-1K with depth) plus a lightweight decoder head. The encoder used for initialisation is included under `pretrained/` (its `.pth.tar` stores training args, so load it with `weights_only=False`).

## Results (reported by the authors)

| Metric | Value |
|---|---|
| mIoU (SUN RGB-D, single-scale) | 48.8 |

See the [official repository](https://github.com/VCIP-RGBD/DFormer) for full tables and evaluation details.

## Files

| File | Content |
|---|---|
| `SUNRGBD_DFormer_Tiny.pth` | original checkpoint, exactly as released |
| `model.safetensors` | the model weights only (bit-exact copy of the checkpoint's `model` tensors; no optimizer state) |
| `config.json` | architecture, dataset, results and provenance |
| `pretrained/DFormer_Tiny.pth.tar` | DFormer-T encoder, RGB-D pretrained on ImageNet-1K |

## Usage

```python
import json
from huggingface_hub import hf_hub_download

repo = "dronefreak/dformer-tiny-sunrgbd"
config = json.load(open(hf_hub_download(repo, "config.json")))
ckpt = hf_hub_download(repo, "SUNRGBD_DFormer_Tiny.pth")   # original format: use it with the official code
print(config["architecture"], config["dataset"], ckpt)
```

Build the network with the official code (https://github.com/VCIP-RGBD/DFormer) using the matching config for this variant, then load
`SUNRGBD_DFormer_Tiny.pth` exactly as you would the original download, or load `model.safetensors` with
`safetensors.torch.load_file` into the same model.

## Citation

If you use these weights, please cite the original work:

```bibtex
@inproceedings{yin2024dformer,
  title     = {DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation},
  author    = {Yin, Bowen and Zhang, Xuying and Li, Zhong-Yu and Liu, Li and Cheng, Ming-Ming and Hou, Qibin},
  booktitle = {ICLR},
  year      = {2024}
}
```

## Disclaimer

This is not an official release. All credit goes to the original authors (Bowen Yin, Xuying Zhang, Zhong-Yu Li, Li Liu, Ming-Ming Cheng, Qibin Hou; Nankai University (VCIP)). The weights are
redistributed under the MIT license of the official repository. If you are an author and want this mirror changed or
removed, please open a discussion.