README.md

---
license: cc-by-nc-4.0
library_name: transformers
base_model: depth-anything/camera-depth-model-kinect
tags:
- camera-depth-model
- depth-estimation
- rgbd
- robotics
pipeline_tag: depth-estimation
---

# Camera-Depth-Model-Kinect — transformers format (original code)

Unofficial transformers port of the **Camera Depth Model** (CDM) `depth-anything/camera-depth-model-kinect` (`cdm_kinect.ckpt`) that runs
the **original** code. Loads with `trust_remote_code`. `model.safetensors` holds the checkpoint's model tensors under
their original names (every tensor checked equal). Trained for the **Microsoft Azure Kinect** (camera-specific noise model).

> **Not an official release.** Camera Depth Models were created by Minghuan Liu, Zhengbang Zhu, Xiaoshen Han, Peng Hu,
> Haotong Lin, Xinyao Li, Jingxiao Chen, Jiafeng Xu, Yichu Yang, Yunfeng Lin, Xinghang Li, Yong Yu, Weinan Zhang, Tao
> Kong and Bingyi Kang (ByteDance Seed) — https://github.com/ByteDance-Seed/manip-as-in-sim-suite, paper *"Manipulation
> as in Simulation: Enabling Accurate Geometry Perception in Robots"* (arXiv 2509.02530).

## What it is

Denoised metric depth from an RGB image plus the raw depth of a depth camera. `CameraDepthModelForDepthEstimation`
holds the official `RGBDDepth`'s submodules (`pretrained` / `depth_pretrained`: two DINOv2 ViT-L/14, `crossAtts`,
`depth_head_rgbd`) and computes depth with that class's own `forward`, returning the standard `DepthEstimatorOutput`
(`predicted_depth`: **inverse depth**; meters = `1 / predicted_depth`, as the original `infer.py`); the official
`infer_image` and `image2tensor` are kept. Code vendored from ByteDance-Seed/manip-as-in-sim-suite @ 757b2ea2 (`cdm/rgbddepth/`)
into `cdm_dinov2.py` / `cdm_dpt.py`; edits: imports, and `Block` → `NestedTensorBlock` where the original imported
`NestedTensorBlock as Block`. `CameraDepthModelProcessor` (`processing_camera_depth_model.py`, `processor_config.json`)
wraps `CameraDepthModelImageProcessor`: `infer.py`'s `load_images` (raw depth / 1000, values above 25 m set to 0,
inverse depth) and `image2tensor`.

As in the original `infer.py` and README API, the image is given as RGB and `image2tensor` applies
`cv2.COLOR_BGR2RGB` to it, so the network sees the channels swapped. This port keeps that behaviour unchanged.

Verified against the official code (same process, V100): preprocessing (image and depth), `predicted_depth` (also as
a tuple), the hidden states, `infer_image`, the post-processing and fp16 autocast outputs are **bit-identical**.

## Usage

```python
import torch
from transformers import AutoModelForDepthEstimation, AutoProcessor

model = AutoModelForDepthEstimation.from_pretrained("Camera-Depth-Model-Kinect", trust_remote_code=True).eval().cuda()
processor = AutoProcessor.from_pretrained("Camera-Depth-Model-Kinect", trust_remote_code=True)
# image: path / PIL / RGB uint8 array; depth: the camera's raw depth png (millimeters, depth_scale=1000) or meters
inputs = processor(images="rgb.png", depth="depth.png", return_tensors="pt").to("cuda")
with torch.no_grad():
    outputs = model(**inputs)                       # outputs.predicted_depth: (1, H', W') inverse depth
inverse = processor.post_process_depth_estimation(outputs, target_sizes=[(height, width)])[0]["predicted_depth"]
depth_m = 1 / inverse
```

- `AutoModel` loads the same class. `output_hidden_states=True` adds the DINOv2 features of the RGB and depth branches; `return_dict=False` returns a tuple.
- `post_process_depth_estimation` resizes as the original `infer_image` (nearest).
- `depth_scale` / `max_depth` of the processor: the original's `--depth-scale` (1000) and the 25 m cut-off `infer.py` uses.

Runtime packages beyond torch / transformers: `opencv-python`, `torchvision` (as the original).

## Licensing

- Weights (`depth-anything/camera-depth-model-kinect`): **CC-BY-NC-4.0** (non-commercial) — [`LICENSE`](LICENSE).
- Code (vendored files and the port): **Apache-2.0** — [`LICENSE.code`](LICENSE.code).
- Attribution, sources and the changes made here: [`NOTICE`](NOTICE).

## Citation

If you use this model, please cite the original work, as its authors ask:

```bibtex
@article{liu2025manipulation,
  title={Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robots},
  author={Liu, Minghuan and Zhu, Zhengbang and Han, Xiaoshen and Hu, Peng and Lin, Haotong and 
          Li, Xinyao and Chen, Jingxiao and Xu, Jiafeng and Yang, Yichu and Lin, Yunfeng and 
          Li, Xinghang and Yu, Yong and Zhang, Weinan and Kong, Tao and Kang, Bingyi},
  journal={arXiv preprint},
  year={2025}
}
```