zerochocobo/DepthAnything3_ONNX

Depth Anything 3 ONNX

This document describes the Depth Anything 3 (DA3) ONNX export used by PTMediaServer for image, video, and 2D-to-3D/VR depth estimation.

The exporter is examples/da3_to_onnx.py. It converts the ONNX-friendly depth branch of DA3 Small/Base into fixed-resolution ONNX Runtime graphs so inference does not need PyTorch at runtime.

Upstream

• Official project: https://github.com/ByteDance-Seed/depth-anything-3
• This exporter targets the DA3 Main Series DA3-SMALL and DA3-BASE checkpoints.
• The upstream model cards list DA3-SMALL and DA3-BASE as Apache 2.0.
• These ONNX files are derived from Depth Anything 3 weights and should be used according to the upstream license, model cards, and citation requirements.

Files And Paths

examples/da3_to_onnx.py
examples/da3_README.md
models/DA3/da3_small.onnx
models/DA3/da3_base.onnx

The conversion script defaults are currently:

Purpose	Default
DA3 source tree	G:/GIT/debug/VR_Video_Toolbox_NE/tool_2dvr/_vendor/da3
PyTorch weights	G:/GIT/debug/VR_Video_Toolbox_NE/models/DA3/Small and Base
ONNX output	models/DA3 in this repository

Canonical output names at the default size:

File	Variant	Notes
da3_small.onnx	Depth Anything 3 Small	Faster and lighter.
da3_base.onnx	Depth Anything 3 Base	Larger model, usually better depth quality.

For non-default export sizes, the script appends the side length:

models/DA3/da3_small_700.onnx
models/DA3/da3_base_1036.onnx

Exported Graph

The converter wraps DepthAnything3Net.forward(...) with a singleton view dimension and these fixed branch settings:

skip_camera=True
skip_sky=True
infer_gs=False
use_ray_pose=False
ref_view_strategy="middle"

Only the depth-only sub-graph is exported:

• DINOv2 encoder, ViT-S for Small and ViT-B for Base.
• DualDPT depth head.
• Single-view input, S=1, one independent view per frame.
• Dynamic batch axis only.

Camera, sky, and Gaussian-splatting branches are intentionally excluded because their torch.quantile, random sampling, .item(), and boolean-mask control flow are not suitable for this static ONNX trace.

The script also patches DA3 RoPE's PositionGetter during export so torch.cartesian_prod is replaced with an ONNX-exportable meshgrid equivalent.

Fixed Square Input

The default input size is 518 x 518, matching DA3's native 37 x 37 patch grid with patch size 14. At this size the learned positional embedding uses the clean npatch == N and w == h path, avoiding bicubic position interpolation in the exported graph.

Rules:

• Height and width are fixed in each ONNX file.
• Only the batch dimension is dynamic.
• --size must be a multiple of 14.
• Re-export when you need another input side length.

Input And Output

Default export:

input name : image
input dtype: float32
input shape: [batch, 3, size, size]
layout     : RGB, CHW
range      : ImageNet-normalized float32

output name : depth
output dtype: float32
output shape: [batch, height, width]

Folded preprocessing export (--fold-preprocess):

input name : image
input dtype: uint8
input shape: [batch, size, size, 3]
layout     : RGB, HWC
range      : 0..255

output name : depth
output dtype: float32
output shape: [batch, height, width]

The output is raw DA3 depth, which is distance-like in this project. The 2D-to-3D path treats smaller values as nearer, then resizes, inverts, clips, and normalizes depth at runtime.

Preprocessing

For the default float32 ONNX export:

1. Load the image or video frame as RGB.
2. Resize or letterbox to the fixed square export size.
3. Convert to float32 in the 0.0 .. 1.0 range.
4. Apply ImageNet normalization with mean [0.485, 0.456, 0.406] and std [0.229, 0.224, 0.225].
5. Transpose from HWC to CHW.
6. Add the batch dimension.

For --fold-preprocess, the ONNX graph performs steps 3 to 5 internally. The application still needs to resize or letterbox to [size, size] and pass RGB uint8 input.

PTMediaServer's offline.da3_depth.Da3DepthEngine detects the ONNX input dtype and supports both contracts. The PyNv hot path requires the folded-preprocess model because it uploads a uint8 letterbox canvas.

Runtime Dependencies

CPU:

pip install onnxruntime numpy opencv-python

GPU:

pip install onnxruntime-gpu numpy opencv-python

Quick Inference Example

This example is for the default float32 export.

import cv2
import numpy as np
import onnxruntime as ort

model = "models/DA3/da3_small.onnx"
size = 518

mean = np.array([0.485, 0.456, 0.406], dtype=np.float32)
std = np.array([0.229, 0.224, 0.225], dtype=np.float32)

image_bgr = cv2.imread("input.jpg", cv2.IMREAD_COLOR)
image_rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB)
resized = cv2.resize(image_rgb, (size, size), interpolation=cv2.INTER_CUBIC)

x = resized.astype(np.float32) / 255.0
x = (x - mean) / std
x = np.transpose(x, (2, 0, 1))[None, ...]

session = ort.InferenceSession(
    model,
    providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)
depth = session.run(["depth"], {"image": x})[0][0]

depth = cv2.resize(depth, (image_bgr.shape[1], image_bgr.shape[0]))
depth_u8 = cv2.normalize(depth, None, 0, 255, cv2.NORM_MINMAX).astype(np.uint8)
cv2.imwrite("depth.png", depth_u8)

For a folded-preprocess model, feed resized[None, ...] directly as uint8 RGB [batch, size, size, 3].

Convert From PyTorch Weights

Run the exporter from this repository root. The VR_Video_Toolbox_NE virtual environment already contains the DA3 PyTorch dependencies:

G:/GIT/debug/VR_Video_Toolbox_NE/.venv/Scripts/python.exe \
  examples/da3_to_onnx.py --variant both --validate

Export one variant:

python examples/da3_to_onnx.py --variant small --validate
python examples/da3_to_onnx.py --variant base --validate

Export folded-preprocess models for PTMediaServer's fast video paths:

python examples/da3_to_onnx.py --variant both --validate --fold-preprocess

Export another fixed input size:

python examples/da3_to_onnx.py --variant base --size 700 --validate --fold-preprocess

Useful options:

--variant small|base|both
--src-root PATH       Folder containing Small/ and Base/ weight directories.
--vendor PATH         Vendored DA3 source root containing depth_anything_3/.
--out-dir PATH        Output folder for da3_*.onnx.
--size 518            Fixed square input side. Must be a multiple of 14.
--opset 18            ONNX opset version.
--device cpu|cuda     Device used for tracing. CPU is the default.
--no-validate         Skip ONNX Runtime validation.
--fold-preprocess     Export uint8 NHWC input with ImageNet normalize inside ONNX.

Expected weight layout:

DA3/
  Small/
    model.safetensors
  Base/
    model.safetensors

Validation

When validation is enabled, the converter compares PyTorch output with ONNX Runtime output and reports:

• output shape
• max absolute error
• mean absolute error
• relative mean error
• active ONNX Runtime providers

The script warns if relative error is higher than 1e-2.

Notes

• The ONNX graph is depth-only; it does not export pose, camera, confidence, sky segmentation, or Gaussian outputs.
• These exports are fixed-shape models. Re-export for another input side.
• Use input sizes that are multiples of 14; 518 is the safest default.
• --fold-preprocess changes the ONNX input contract and overwrites the same output filename unless you also change --out-dir or --size.
• The output is raw model depth, not metric depth.
• Smaller depth values are treated as nearer in PTMediaServer's 2D-to-3D path.

Citation

If you use Depth Anything 3 or ONNX exports derived from it in research or a published project, cite the upstream work:

@article{depthanything3,
  title={Depth Anything 3: Recovering the visual space from any views},
  author={Haotong Lin and Sili Chen and Jun Hao Liew and Donny Y. Chen and Zhenyu Li and Guang Shi and Jiashi Feng and Bingyi Kang},
  journal={arXiv preprint arXiv:2511.10647},
  year={2025}
}