File size: 3,820 Bytes
e6862a1 03a4962 e6862a1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 | ---
license: mit
library_name: pytorch
pipeline_tag: image-segmentation
tags:
- rgb-d
- semantic-segmentation
- indoor-scene-understanding
- dformer
- sunrgbd
- community-mirror
datasets:
- dronefreak/SUN-RGBD
---
# DFormer-T trained on SUN RGB-D
<p align="center"><video controls autoplay loop muted playsinline width="900" poster="https://huggingface.co/dronefreak/dformer-tiny-sunrgbd/resolve/main/assets/demo_banner_poster.jpg" src="https://huggingface.co/dronefreak/dformer-tiny-sunrgbd/resolve/main/assets/demo_banner.mp4"><img src="https://huggingface.co/dronefreak/dformer-tiny-sunrgbd/resolve/main/assets/demo_banner_poster.jpg" alt="DFormer-T predictions on SUN RGB-D test scenes" width="900"></video></p>
*Demo: predictions of this checkpoint on SUN RGB-D test scenes from our dataset mirror (RGB, depth input (8-bit), prediction overlay; per-frame pixel accuracy vs. ground truth), rendered with the official code.*
> **Unofficial mirror** of the official DFormer weights from [VCIP-RGBD/DFormer](https://github.com/VCIP-RGBD/DFormer). The weights are **unchanged**; this repository adds a weights-only `model.safetensors` copy and a `config.json`.
## Model
- **Architecture:** DFormer: DFormer-T
- **Task:** RGB-D semantic segmentation
- **Trained on:** [SUN RGB-D](https://huggingface.co/datasets/dronefreak/SUN-RGBD) (mirror on the Hub)
- **Parameters:** 6.0M
- **Paper:** [DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation](https://arxiv.org/abs/2309.09668) (ICLR 2024)
- **Code:** https://github.com/VCIP-RGBD/DFormer
- **License:** MIT (from the official repository)
RGB-D pretrained encoder (ImageNet-1K with depth) plus a lightweight decoder head. The encoder used for initialisation is included under `pretrained/` (its `.pth.tar` stores training args, so load it with `weights_only=False`).
## Results (reported by the authors)
| Metric | Value |
|---|---|
| mIoU (SUN RGB-D, single-scale) | 48.8 |
See the [official repository](https://github.com/VCIP-RGBD/DFormer) for full tables and evaluation details.
## Files
| File | Content |
|---|---|
| `SUNRGBD_DFormer_Tiny.pth` | original checkpoint, exactly as released |
| `model.safetensors` | the model weights only (bit-exact copy of the checkpoint's `model` tensors; no optimizer state) |
| `config.json` | architecture, dataset, results and provenance |
| `pretrained/DFormer_Tiny.pth.tar` | DFormer-T encoder, RGB-D pretrained on ImageNet-1K |
## Usage
```python
import json
from huggingface_hub import hf_hub_download
repo = "dronefreak/dformer-tiny-sunrgbd"
config = json.load(open(hf_hub_download(repo, "config.json")))
ckpt = hf_hub_download(repo, "SUNRGBD_DFormer_Tiny.pth") # original format: use it with the official code
print(config["architecture"], config["dataset"], ckpt)
```
Build the network with the official code (https://github.com/VCIP-RGBD/DFormer) using the matching config for this variant, then load
`SUNRGBD_DFormer_Tiny.pth` exactly as you would the original download, or load `model.safetensors` with
`safetensors.torch.load_file` into the same model.
## Citation
If you use these weights, please cite the original work:
```bibtex
@inproceedings{yin2024dformer,
title = {DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation},
author = {Yin, Bowen and Zhang, Xuying and Li, Zhong-Yu and Liu, Li and Cheng, Ming-Ming and Hou, Qibin},
booktitle = {ICLR},
year = {2024}
}
```
## Disclaimer
This is not an official release. All credit goes to the original authors (Bowen Yin, Xuying Zhang, Zhong-Yu Li, Li Liu, Ming-Ming Cheng, Qibin Hou; Nankai University (VCIP)). The weights are
redistributed under the MIT license of the official repository. If you are an author and want this mirror changed or
removed, please open a discussion.
|