Depth Pro 2 B ONNX, focal-free

Unofficial ONNX export of apple/DepthPro2-B for calibrated cameras. Pass the focal length f_px; the 23.5M-parameter focal-length network is gone. Output is metric depth in meters. Every graph passes a parity gate against PyTorch. Try it in the browser. Not affiliated with or endorsed by Apple.

Research use only. These weights are a Model Derivative under the Apple Machine Learning Research Model License: non-commercial research and academic use only. Apple Machine Learning Research Model is licensed under the Apple Machine Learning Research Model License Agreement.

Quick start

pip install onnxruntime-gpu huggingface_hub pillow numpy   # onnxruntime for CPU
import json
import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download

repo = "christianengel/DepthPro2-B-ONNX"
config = json.load(open(hf_hub_download(repo, "config.json")))
session = ort.InferenceSession(
    hf_hub_download(repo, "onnx/model_fp16.onnx"),
    providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)

# pixel_values: RGB in [0, 1], shape (1, 3, 480, 640). f_px: fx of that 640x480 image in pixels.
depth = session.run(None, {"pixel_values": pixels, "f_px": np.array([fx], np.float32)})[0]

example.py resizes and crops any photo and scales f_px with it. From ROS 2, f_px is CameraInfo.k[0] of the rectified image.

Files

Static shapes, opset 18. AbsRel and delta1 are against PyTorch FP32 on a 17-frame fixture.

File Input Frames Precision MB AbsRel delta1
onnx/model.onnx 480x640 1 frame FP32 387 0.0002 1.0000
onnx/model_fp16.onnx 480x640 1 frame FP16 199 0.0007 1.0000
onnx/model_int8.onnx 480x640 1 frame INT8 143 0.0032 1.0000
onnx/model_clip4.onnx 480x640 4 frames attending to each other FP32 389 0.0002 1.0000
onnx/model_clip4_fp16.onnx 480x640 4 frames attending to each other FP16 201 0.0006 1.0000
onnx/model_stream4.onnx 480x640 1 frame + 3-frame cache as state FP32 387 0.0001 1.0000
onnx/model_stream4_fp16.onnx 480x640 1 frame + 3-frame cache as state FP16 199 0.0005 1.0000
onnx/384x672/model.onnx 384x672 1 frame FP32 387 0.0003 1.0000
onnx/384x672/model_fp16.onnx 384x672 1 frame FP16 199 0.0007 1.0000
onnx/384x672/model_int8.onnx 384x672 1 frame INT8 143 0.0026 1.0000
onnx/384x672/model_stream4.onnx 384x672 1 frame + 3-frame cache as state FP32 387 0.0002 1.0000
onnx/384x672/model_stream4_fp16.onnx 384x672 1 frame + 3-frame cache as state FP16 199 0.0005 1.0000
  • Single frame (model*.onnx): each frame on its own.
  • Stream (model_stream4*.onnx): one frame per call, attending to the previous three; Depth Pro 2's unbuffered video mode. The cache goes in as past_* and comes out as present_*; zeros for the first frame.
  • Clip (model_clip4*.onnx): four frames jointly, as DepthPro2.infer on a (4, 3, H, W) clip.

I/O:

  • pixel_values: float32 RGB in [0, 1], NCHW, not normalized.
  • f_px: float32 (1,), horizontal focal length in pixels of pixel_values.
  • depth_meters: float32 (T, H, W), z-depth in meters; inf where infinitely far (sky).

Speed

NVIDIA GeForce RTX 3090, ONNX Runtime 1.23.2, 480x640, one frame at a time, host array to host array. Peak VRAM is the whole process (NVML).

Model Runtime Latency (ms/frame) FPS Peak VRAM (MB) Disk (MB) AbsRel vs PyTorch FP32
DepthPro2-B PyTorch BF16 (Apple default) 11.5 86.6 1198 - 0.0022
DepthPro2-B PyTorch FP32 24.9 40.1 1112 - 0.0000
DepthPro2-B ORT CUDA FP32 22.4 44.6 1314 387 0.0002
DepthPro2-B ORT CUDA FP16 14.9 67.1 1322 199 0.0007
DepthPro2-B ORT CUDA INT8 23.1 43.2 1308 143 0.0030
DepthPro2-B TensorRT FP16 engine 7.1 140.5 700 204 0.0009
DepthPro2-B TensorRT FP16 engine, stream window 4 38.9 25.7 830 204 0.0009
DepthPro2-B ORT CUDA FP16 stream, window 4 21.7 46.0 2472 199 0.0005
DepthPro2-B ORT CUDA FP16 clip of 4 19.2 52.1 5430 201 0.0006

TensorRT rows are engines from build_tensorrt.sh on that GPU. The stream engine copies its state through host memory each frame; ONNX Runtime keeps it on the GPU.

TensorRT and Jetson

On the device: ./build_tensorrt.sh onnx/model_fp16.onnx (script, trtexec --stronglyTyped). Do not use --fp16 or ONNX Runtime's TensorRT provider: both may run block 0's attention (logits up to 2e6) or B's LayerNorms in FP16 and return NaN.

ROS 2

Node: subscribes to Image and CameraInfo, takes f_px from k[0], publishes 32FC1 depth in meters, matching CameraInfo, optional PointCloud2.

Changes from the original

  • Focal-length network (apple/DepthPro2-focal) removed; f_px is an input.
  • ImageNet normalization and depth = f_px / (W * inverse_depth) are in the graph.
  • No padding: sides must be multiples of 32.
  • FP16: FP16 weights; block 0 attention, RoPE tables, depth head and final division stay FP32.
  • INT8: weight-only 8-bit MatMul weights (symmetric, blocks of 32). Dynamic activation INT8 collapses this backbone (delta1 < 0.5).
  • Stream graphs follow the unbuffered step of Apple's Core ML exporter, with the cache as explicit tensors.

Source: apple/DepthPro2-B@281dc98, 96.5M parameters. Export code: https://github.com/protortyp/depthpro2-onnx (Apache-2.0).

Citation

@article{DepthPro2:2026:arxiv,
  title   = {Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth},
  author  = {Peiyun Hu and Marko Mihajlovic and Lyndon Duong and Lahav Lipson and Wei Dong and Ama\"{e}l Delaunoy and Stephan R. Richter and Vladlen Koltun},
  journal = {arXiv preprint arXiv:2611.26603},
  year    = {2026},
}
Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for christianengel/DepthPro2-B-ONNX

Quantized
(1)
this model

Paper for christianengel/DepthPro2-B-ONNX