Depth Pro 2 S ONNX, focal-free

Unofficial ONNX export of apple/DepthPro2-S for calibrated cameras. Pass the focal length f_px; the 23.5M-parameter focal-length network is gone. Output is metric depth in meters. Every graph passes a parity gate against PyTorch. Try it in the browser. Not affiliated with or endorsed by Apple.

Research use only. These weights are a Model Derivative under the Apple Machine Learning Research Model License: non-commercial research and academic use only. Apple Machine Learning Research Model is licensed under the Apple Machine Learning Research Model License Agreement.

Quick start

pip install onnxruntime-gpu huggingface_hub pillow numpy   # onnxruntime for CPU
import json
import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download

repo = "christianengel/DepthPro2-S-ONNX"
config = json.load(open(hf_hub_download(repo, "config.json")))
session = ort.InferenceSession(
    hf_hub_download(repo, "onnx/model_fp16.onnx"),
    providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)

# pixel_values: RGB in [0, 1], shape (1, 3, 480, 640). f_px: fx of that 640x480 image in pixels.
depth = session.run(None, {"pixel_values": pixels, "f_px": np.array([fx], np.float32)})[0]

example.py resizes and crops any photo and scales f_px with it. From ROS 2, f_px is CameraInfo.k[0] of the rectified image.

Files

Static shapes, opset 18. AbsRel and delta1 are against PyTorch FP32 on a 17-frame fixture.

File Input Frames Precision MB AbsRel delta1
onnx/model.onnx 480x640 1 frame FP32 98 0.0004 1.0000
onnx/model_fp16.onnx 480x640 1 frame FP16 51 0.0008 1.0000
onnx/model_int8.onnx 480x640 1 frame INT8 38 0.0049 1.0000
onnx/model_clip4.onnx 480x640 4 frames attending to each other FP32 100 0.0002 1.0000
onnx/model_clip4_fp16.onnx 480x640 4 frames attending to each other FP16 53 0.0005 1.0000
onnx/model_stream4.onnx 480x640 1 frame + 3-frame cache as state FP32 98 0.0003 1.0000
onnx/model_stream4_fp16.onnx 480x640 1 frame + 3-frame cache as state FP16 51 0.0005 1.0000
onnx/384x672/model.onnx 384x672 1 frame FP32 98 0.0005 1.0000
onnx/384x672/model_fp16.onnx 384x672 1 frame FP16 51 0.0008 1.0000
onnx/384x672/model_int8.onnx 384x672 1 frame INT8 37 0.0042 1.0000
onnx/384x672/model_stream4.onnx 384x672 1 frame + 3-frame cache as state FP32 98 0.0004 1.0000
onnx/384x672/model_stream4_fp16.onnx 384x672 1 frame + 3-frame cache as state FP16 50 0.0006 1.0000
  • Single frame (model*.onnx): each frame on its own.
  • Stream (model_stream4*.onnx): one frame per call, attending to the previous three; Depth Pro 2's unbuffered video mode. The cache goes in as past_* and comes out as present_*; zeros for the first frame.
  • Clip (model_clip4*.onnx): four frames jointly, as DepthPro2.infer on a (4, 3, H, W) clip.

I/O:

  • pixel_values: float32 RGB in [0, 1], NCHW, not normalized.
  • f_px: float32 (1,), horizontal focal length in pixels of pixel_values.
  • depth_meters: float32 (T, H, W), z-depth in meters; inf where infinitely far (sky).

Speed

NVIDIA GeForce RTX 3090, ONNX Runtime 1.23.2, 480x640, one frame at a time, host array to host array. Peak VRAM is the whole process (NVML).

Model Runtime Latency (ms/frame) FPS Peak VRAM (MB) Disk (MB) AbsRel vs PyTorch FP32
DepthPro2-S PyTorch BF16 (Apple default) 10.3 96.9 622 - 0.0030
DepthPro2-S PyTorch FP32 10.5 95.1 588 - 0.0000
DepthPro2-S ORT CUDA FP32 11.6 86.2 798 99 0.0004
DepthPro2-S ORT CUDA FP16 8.8 114.1 548 51 0.0008
DepthPro2-S ORT CUDA INT8 11.8 84.8 796 38 0.0049
DepthPro2-S TensorRT FP16 engine 3.6 277.8 480 56 0.0014
DepthPro2-S TensorRT FP16 engine, stream window 4 15.1 66.2 546 57 0.0011
DepthPro2-S ORT CUDA FP16 stream, window 4 11.5 87.0 1378 51 0.0005
DepthPro2-S ORT CUDA FP16 clip of 4 8.8 113.2 2352 53 0.0005
DepthPro2-S ORT CPU FP32 259.4 3.9 - 99 0.0000
DepthPro2-S ORT CPU INT8 303.1 3.3 - 38 0.0049

TensorRT rows are engines from build_tensorrt.sh on that GPU. The stream engine copies its state through host memory each frame; ONNX Runtime keeps it on the GPU.

TensorRT and Jetson

On the device: ./build_tensorrt.sh onnx/model_fp16.onnx (script, trtexec --stronglyTyped). Do not use --fp16 or ONNX Runtime's TensorRT provider: both may run block 0's attention (logits up to 2e6) or B's LayerNorms in FP16 and return NaN.

ROS 2

Node: subscribes to Image and CameraInfo, takes f_px from k[0], publishes 32FC1 depth in meters, matching CameraInfo, optional PointCloud2.

Changes from the original

  • Focal-length network (apple/DepthPro2-focal) removed; f_px is an input.
  • ImageNet normalization and depth = f_px / (W * inverse_depth) are in the graph.
  • No padding: sides must be multiples of 32.
  • FP16: FP16 weights; block 0 attention, RoPE tables, depth head and final division stay FP32.
  • INT8: weight-only 8-bit MatMul weights (symmetric, blocks of 32). Dynamic activation INT8 collapses this backbone (delta1 < 0.5).
  • Stream graphs follow the unbuffered step of Apple's Core ML exporter, with the cache as explicit tensors.

Source: apple/DepthPro2-S@30e6471, 24.3M parameters. Export code: https://github.com/protortyp/depthpro2-onnx (Apache-2.0).

Citation

@article{DepthPro2:2026:arxiv,
  title   = {Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth},
  author  = {Peiyun Hu and Marko Mihajlovic and Lyndon Duong and Lahav Lipson and Wei Dong and Ama\"{e}l Delaunoy and Stephan R. Richter and Vladlen Koltun},
  journal = {arXiv preprint arXiv:2611.26603},
  year    = {2026},
}
Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for christianengel/DepthPro2-S-ONNX

Quantized
(1)
this model

Space using christianengel/DepthPro2-S-ONNX 1

Paper for christianengel/DepthPro2-S-ONNX