| --- |
| license: cc-by-nc-4.0 |
| library_name: transformers |
| base_model: depth-anything/camera-depth-model-d405 |
| tags: |
| - camera-depth-model |
| - depth-estimation |
| - rgbd |
| - robotics |
| pipeline_tag: depth-estimation |
| --- |
| |
| # Camera-Depth-Model-D405 — transformers format (original code) |
|
|
| Unofficial transformers port of the **Camera Depth Model** (CDM) `depth-anything/camera-depth-model-d405` (`cdm_d405.ckpt`) that runs |
| the **original** code. Loads with `trust_remote_code`. `model.safetensors` holds the checkpoint's model tensors under |
| their original names (every tensor checked equal). Trained for the **Intel RealSense D405** (camera-specific noise model). |
|
|
| > **Not an official release.** Camera Depth Models were created by Minghuan Liu, Zhengbang Zhu, Xiaoshen Han, Peng Hu, |
| > Haotong Lin, Xinyao Li, Jingxiao Chen, Jiafeng Xu, Yichu Yang, Yunfeng Lin, Xinghang Li, Yong Yu, Weinan Zhang, Tao |
| > Kong and Bingyi Kang (ByteDance Seed) — https://github.com/ByteDance-Seed/manip-as-in-sim-suite, paper *"Manipulation |
| > as in Simulation: Enabling Accurate Geometry Perception in Robots"* (arXiv 2509.02530). |
|
|
| ## What it is |
|
|
| Denoised metric depth from an RGB image plus the raw depth of a depth camera. `CameraDepthModelForDepthEstimation` |
| holds the official `RGBDDepth`'s submodules (`pretrained` / `depth_pretrained`: two DINOv2 ViT-L/14, `crossAtts`, |
| `depth_head_rgbd`) and computes depth with that class's own `forward`, returning the standard `DepthEstimatorOutput` |
| (`predicted_depth`: **inverse depth**; meters = `1 / predicted_depth`, as the original `infer.py`); the official |
| `infer_image` and `image2tensor` are kept. Code vendored from ByteDance-Seed/manip-as-in-sim-suite @ 757b2ea2 (`cdm/rgbddepth/`) |
| into `cdm_dinov2.py` / `cdm_dpt.py`; edits: imports, and `Block` → `NestedTensorBlock` where the original imported |
| `NestedTensorBlock as Block`. `CameraDepthModelProcessor` (`processing_camera_depth_model.py`, `processor_config.json`) |
| wraps `CameraDepthModelImageProcessor`: `infer.py`'s `load_images` (raw depth / 1000, values above 25 m set to 0, |
| inverse depth) and `image2tensor`. |
|
|
| As in the original `infer.py` and README API, the image is given as RGB and `image2tensor` applies |
| `cv2.COLOR_BGR2RGB` to it, so the network sees the channels swapped. This port keeps that behaviour unchanged. |
|
|
| Verified against the official code (same process, V100): preprocessing (image and depth), `predicted_depth` (also as |
| a tuple), the hidden states, `infer_image`, the post-processing and fp16 autocast outputs are **bit-identical**. |
|
|
| ## Usage |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForDepthEstimation, AutoProcessor |
| |
| model = AutoModelForDepthEstimation.from_pretrained("Camera-Depth-Model-D405", trust_remote_code=True).eval().cuda() |
| processor = AutoProcessor.from_pretrained("Camera-Depth-Model-D405", trust_remote_code=True) |
| # image: path / PIL / RGB uint8 array; depth: the camera's raw depth png (millimeters, depth_scale=1000) or meters |
| inputs = processor(images="rgb.png", depth="depth.png", return_tensors="pt").to("cuda") |
| with torch.no_grad(): |
| outputs = model(**inputs) # outputs.predicted_depth: (1, H', W') inverse depth |
| inverse = processor.post_process_depth_estimation(outputs, target_sizes=[(height, width)])[0]["predicted_depth"] |
| depth_m = 1 / inverse |
| ``` |
|
|
| - `AutoModel` loads the same class. `output_hidden_states=True` adds the DINOv2 features of the RGB and depth branches; `return_dict=False` returns a tuple. |
| - `post_process_depth_estimation` resizes as the original `infer_image` (nearest). |
| - `depth_scale` / `max_depth` of the processor: the original's `--depth-scale` (1000) and the 25 m cut-off `infer.py` uses. |
|
|
| Runtime packages beyond torch / transformers: `opencv-python`, `torchvision` (as the original). |
|
|
| ## Licensing |
|
|
| - Weights (`depth-anything/camera-depth-model-d405`): **CC-BY-NC-4.0** (non-commercial) — [`LICENSE`](LICENSE). |
| - Code (vendored files and the port): **Apache-2.0** — [`LICENSE.code`](LICENSE.code). |
| - Attribution, sources and the changes made here: [`NOTICE`](NOTICE). |
|
|
| ## Citation |
|
|
| If you use this model, please cite the original work, as its authors ask: |
|
|
| ```bibtex |
| @article{liu2025manipulation, |
| title={Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robots}, |
| author={Liu, Minghuan and Zhu, Zhengbang and Han, Xiaoshen and Hu, Peng and Lin, Haotong and |
| Li, Xinyao and Chen, Jingxiao and Xu, Jiafeng and Yang, Yichu and Lin, Yunfeng and |
| Li, Xinghang and Yu, Yong and Zhang, Weinan and Kong, Tao and Kang, Bingyi}, |
| journal={arXiv preprint}, |
| year={2025} |
| } |
| ``` |
|
|
