DFormer-B trained on NYUv2
Demo: predictions of this checkpoint on NYUv2 test scenes from our dataset mirror (RGB, depth input (8-bit), prediction overlay; per-frame pixel accuracy vs. ground truth), rendered with the official code.
Unofficial mirror of the official DFormer weights from VCIP-RGBD/DFormer. The weights are unchanged; this repository adds a weights-only
model.safetensorscopy and aconfig.json.
Model
- Architecture: DFormer: DFormer-B
- Task: RGB-D semantic segmentation
- Trained on: NYUv2 (mirror on the Hub)
- Parameters: 29.5M
- Paper: DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation (ICLR 2024)
- Code: https://github.com/VCIP-RGBD/DFormer
- License: MIT (from the official repository)
RGB-D pretrained encoder (ImageNet-1K with depth) plus a lightweight decoder head. The encoder used for initialisation is included under pretrained/ (its .pth.tar stores training args, so load it with weights_only=False).
Results (reported by the authors)
| Metric | Value |
|---|---|
| mIoU (NYUv2, single-scale) | 55.6 |
See the official repository for full tables and evaluation details.
Files
| File | Content |
|---|---|
NYUv2_DFormer_Base.pth |
original checkpoint, exactly as released |
model.safetensors |
the model weights only (bit-exact copy of the checkpoint's model tensors; no optimizer state) |
config.json |
architecture, dataset, results and provenance |
pretrained/DFormer_Base.pth.tar |
DFormer-B encoder, RGB-D pretrained on ImageNet-1K |
Usage
import json
from huggingface_hub import hf_hub_download
repo = "dronefreak/dformer-base-nyuv2"
config = json.load(open(hf_hub_download(repo, "config.json")))
ckpt = hf_hub_download(repo, "NYUv2_DFormer_Base.pth") # original format: use it with the official code
print(config["architecture"], config["dataset"], ckpt)
Build the network with the official code (https://github.com/VCIP-RGBD/DFormer) using the matching config for this variant, then load
NYUv2_DFormer_Base.pth exactly as you would the original download, or load model.safetensors with
safetensors.torch.load_file into the same model.
Citation
If you use these weights, please cite the original work:
@inproceedings{yin2024dformer,
title = {DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation},
author = {Yin, Bowen and Zhang, Xuying and Li, Zhong-Yu and Liu, Li and Cheng, Ming-Ming and Hou, Qibin},
booktitle = {ICLR},
year = {2024}
}
Disclaimer
This is not an official release. All credit goes to the original authors (Bowen Yin, Xuying Zhang, Zhong-Yu Li, Li Liu, Ming-Ming Cheng, Qibin Hou; Nankai University (VCIP)). The weights are redistributed under the MIT license of the official repository. If you are an author and want this mirror changed or removed, please open a discussion.
- Downloads last month
- -