Instructions to use christianengel/DepthPro2-S-ONNX with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- TensorRT
How to use christianengel/DepthPro2-S-ONNX with TensorRT:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Depth Pro 2 S ONNX, focal-free
Unofficial ONNX export of apple/DepthPro2-S for calibrated cameras. Pass the focal length f_px; the 23.5M-parameter focal-length network is gone. Output is metric depth in meters. Every graph passes a parity gate against PyTorch. Try it in the browser. Not affiliated with or endorsed by Apple.
Research use only. These weights are a Model Derivative under the Apple Machine Learning Research Model License: non-commercial research and academic use only. Apple Machine Learning Research Model is licensed under the Apple Machine Learning Research Model License Agreement.
Quick start
pip install onnxruntime-gpu huggingface_hub pillow numpy # onnxruntime for CPU
import json
import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download
repo = "christianengel/DepthPro2-S-ONNX"
config = json.load(open(hf_hub_download(repo, "config.json")))
session = ort.InferenceSession(
hf_hub_download(repo, "onnx/model_fp16.onnx"),
providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)
# pixel_values: RGB in [0, 1], shape (1, 3, 480, 640). f_px: fx of that 640x480 image in pixels.
depth = session.run(None, {"pixel_values": pixels, "f_px": np.array([fx], np.float32)})[0]
example.py resizes and crops any photo and scales f_px with it. From ROS 2, f_px is CameraInfo.k[0] of the rectified image.
Files
Static shapes, opset 18. AbsRel and delta1 are against PyTorch FP32 on a 17-frame fixture.
| File | Input | Frames | Precision | MB | AbsRel | delta1 |
|---|---|---|---|---|---|---|
onnx/model.onnx |
480x640 | 1 frame | FP32 | 98 | 0.0004 | 1.0000 |
onnx/model_fp16.onnx |
480x640 | 1 frame | FP16 | 51 | 0.0008 | 1.0000 |
onnx/model_int8.onnx |
480x640 | 1 frame | INT8 | 38 | 0.0049 | 1.0000 |
onnx/model_clip4.onnx |
480x640 | 4 frames attending to each other | FP32 | 100 | 0.0002 | 1.0000 |
onnx/model_clip4_fp16.onnx |
480x640 | 4 frames attending to each other | FP16 | 53 | 0.0005 | 1.0000 |
onnx/model_stream4.onnx |
480x640 | 1 frame + 3-frame cache as state | FP32 | 98 | 0.0003 | 1.0000 |
onnx/model_stream4_fp16.onnx |
480x640 | 1 frame + 3-frame cache as state | FP16 | 51 | 0.0005 | 1.0000 |
onnx/384x672/model.onnx |
384x672 | 1 frame | FP32 | 98 | 0.0005 | 1.0000 |
onnx/384x672/model_fp16.onnx |
384x672 | 1 frame | FP16 | 51 | 0.0008 | 1.0000 |
onnx/384x672/model_int8.onnx |
384x672 | 1 frame | INT8 | 37 | 0.0042 | 1.0000 |
onnx/384x672/model_stream4.onnx |
384x672 | 1 frame + 3-frame cache as state | FP32 | 98 | 0.0004 | 1.0000 |
onnx/384x672/model_stream4_fp16.onnx |
384x672 | 1 frame + 3-frame cache as state | FP16 | 50 | 0.0006 | 1.0000 |
- Single frame (
model*.onnx): each frame on its own. - Stream (
model_stream4*.onnx): one frame per call, attending to the previous three; Depth Pro 2's unbuffered video mode. The cache goes in aspast_*and comes out aspresent_*; zeros for the first frame. - Clip (
model_clip4*.onnx): four frames jointly, asDepthPro2.inferon a(4, 3, H, W)clip.
I/O:
pixel_values: float32 RGB in [0, 1], NCHW, not normalized.f_px: float32(1,), horizontal focal length in pixels ofpixel_values.depth_meters: float32(T, H, W), z-depth in meters;infwhere infinitely far (sky).
Speed
NVIDIA GeForce RTX 3090, ONNX Runtime 1.23.2, 480x640, one frame at a time, host array to host array. Peak VRAM is the whole process (NVML).
| Model | Runtime | Latency (ms/frame) | FPS | Peak VRAM (MB) | Disk (MB) | AbsRel vs PyTorch FP32 |
|---|---|---|---|---|---|---|
| DepthPro2-S | PyTorch BF16 (Apple default) | 10.3 | 96.9 | 622 | - | 0.0030 |
| DepthPro2-S | PyTorch FP32 | 10.5 | 95.1 | 588 | - | 0.0000 |
| DepthPro2-S | ORT CUDA FP32 | 11.6 | 86.2 | 798 | 99 | 0.0004 |
| DepthPro2-S | ORT CUDA FP16 | 8.8 | 114.1 | 548 | 51 | 0.0008 |
| DepthPro2-S | ORT CUDA INT8 | 11.8 | 84.8 | 796 | 38 | 0.0049 |
| DepthPro2-S | TensorRT FP16 engine | 3.6 | 277.8 | 480 | 56 | 0.0014 |
| DepthPro2-S | TensorRT FP16 engine, stream window 4 | 15.1 | 66.2 | 546 | 57 | 0.0011 |
| DepthPro2-S | ORT CUDA FP16 stream, window 4 | 11.5 | 87.0 | 1378 | 51 | 0.0005 |
| DepthPro2-S | ORT CUDA FP16 clip of 4 | 8.8 | 113.2 | 2352 | 53 | 0.0005 |
| DepthPro2-S | ORT CPU FP32 | 259.4 | 3.9 | - | 99 | 0.0000 |
| DepthPro2-S | ORT CPU INT8 | 303.1 | 3.3 | - | 38 | 0.0049 |
TensorRT rows are engines from build_tensorrt.sh on that GPU. The stream engine copies its state through host memory each frame; ONNX Runtime keeps it on the GPU.
TensorRT and Jetson
On the device: ./build_tensorrt.sh onnx/model_fp16.onnx (script, trtexec --stronglyTyped). Do not use --fp16 or ONNX Runtime's TensorRT provider: both may run block 0's attention (logits up to 2e6) or B's LayerNorms in FP16 and return NaN.
ROS 2
Node: subscribes to Image and CameraInfo, takes f_px from k[0], publishes 32FC1 depth in meters, matching CameraInfo, optional PointCloud2.
Changes from the original
- Focal-length network (apple/DepthPro2-focal) removed;
f_pxis an input. - ImageNet normalization and
depth = f_px / (W * inverse_depth)are in the graph. - No padding: sides must be multiples of 32.
- FP16: FP16 weights; block 0 attention, RoPE tables, depth head and final division stay FP32.
- INT8: weight-only 8-bit MatMul weights (symmetric, blocks of 32). Dynamic activation INT8 collapses this backbone (delta1 < 0.5).
- Stream graphs follow the unbuffered step of Apple's Core ML exporter, with the cache as explicit tensors.
Source: apple/DepthPro2-S@30e6471, 24.3M parameters. Export code: https://github.com/protortyp/depthpro2-onnx (Apache-2.0).
Citation
@article{DepthPro2:2026:arxiv,
title = {Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth},
author = {Peiyun Hu and Marko Mihajlovic and Lyndon Duong and Lahav Lipson and Wei Dong and Ama\"{e}l Delaunoy and Stephan R. Richter and Vladlen Koltun},
journal = {arXiv preprint arXiv:2611.26603},
year = {2026},
}
- Downloads last month
- -
Model tree for christianengel/DepthPro2-S-ONNX
Base model
apple/DepthPro2-S