DepthPro2 / README.md
markomih's picture amael-apple's picture patricejaton's picture
init
b1c7c98
|
Raw History Blame
6.88 kB
---
license: other
license_name: apple-amlr
license_link: https://github.com/apple-aiml-research/ml-depth-pro-2/blob/main/LICENSE_MODEL
pipeline_tag: depth-estimation
library_name: depth-pro-2
tags:
- depth-estimation
- metric-depth
- monocular-depth
- video-depth
---
<h1 align="center">Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth</h1>
<p align="center">
<a href="https://github.com/apple-aiml-research/ml-depth-pro-2"><img src="https://img.shields.io/badge/GitHub-181717?style=flat-square&amp;logo=github&amp;logoColor=white" alt="GitHub"/></a>
<a href="https://arxiv.org/abs/2611.26603"><img src="https://img.shields.io/badge/Technical%20Report-B31B1B?style=flat-square&amp;logo=arxiv&amp;logoColor=white" alt="Technical Report"/></a>
<a href="https://apple-aiml-research.github.io/ml-depth-pro-2/"><img src="https://img.shields.io/badge/Project%20Page-666666?style=flat-square&amp;logo=data%3Aimage%2Fsvg%2Bxml%3Bbase64%2CPHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAyNCAyNCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSJ3aGl0ZSIgc3Ryb2tlLXdpZHRoPSIyIj48Y2lyY2xlIGN4PSIxMiIgY3k9IjEyIiByPSIxMCIvPjxwYXRoIGQ9Ik0yIDEyaDIwTTEyIDJhMTUgMTUgMCAwIDEgMCAyME0xMiAyYTE1IDE1IDAgMCAwIDAgMjAiLz48L3N2Zz4%3D" alt="Project Page"/></a>
</p>
<!-- Hidden until the Space is public:
<h3 align="center">
🚀 <a href="https://huggingface.co/spaces/apple/DepthPro2">Try the Interactive Demo →</a>
</h3>
-->
Depth Pro 2 estimates metric depth for every pixel from an ordinary camera: a single photo, a
whole video, or a live feed, at the input's full resolution. Its depth is sharp at object edges
and steady from frame to frame.
<video src="https://huggingface.co/apple/DepthPro2/resolve/main/assets/hero.mp4" poster="https://huggingface.co/apple/DepthPro2/resolve/main/assets/hero-poster.jpg" autoplay muted loop playsinline width="100%"></video>
## Models
Each size is its own repository with its weights. Larger sizes give sharper, steadier depth;
smaller ones run faster.
| Model | Repository | Parameters | Download size |
|:---:|:---:|:---:|:---:|
| S | [apple/DepthPro2-S](https://huggingface.co/apple/DepthPro2-S) | 24M | 97 MB |
| B | [apple/DepthPro2-B](https://huggingface.co/apple/DepthPro2-B) | 97M | 386 MB |
| L | [apple/DepthPro2-L](https://huggingface.co/apple/DepthPro2-L) | 334M | 1.3 GB |
| H+ | [apple/DepthPro2-H](https://huggingface.co/apple/DepthPro2-H) | 872M | 3.5 GB |
All sizes share a focal length estimator,
[apple/DepthPro2-focal](https://huggingface.co/apple/DepthPro2-focal) (23.5M parameters, 94 MB),
which loads with each size and predicts the focal length when it is unknown.
## Quick start
```bash
pip install "git+https://github.com/apple-aiml-research/ml-depth-pro-2"
```
```python
from functools import partial
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from depth_pro_2 import DepthPro2, load_rgb, upright
from depth_pro_2.model import to_normalized_tensor
from depth_pro_2.network.fov import FOVEstimator
# apple/DepthPro2-S, -B, -L or -H; bfloat16 is how the models run on a GPU.
model = DepthPro2.from_pretrained("apple/DepthPro2-H", precision=torch.bfloat16)
# The focal length estimator all sizes share, for photos and videos without a focal length.
weights = hf_hub_download("apple/DepthPro2-focal", "model.safetensors")
model.fov = FOVEstimator(torch.bfloat16)
model.fov.load_state_dict(load_file(weights))
model = model.to("cuda").eval()
transform = partial(to_normalized_tensor, device=torch.device("cuda"))
```
### Image
```python
# f_px is the focal length recorded in the photo, or None.
image, f_px = load_rgb("photo.jpg")
depth = model.infer(transform(image), f_px=f_px)["depth"] # meters, (H, W)
```
### Video
```python
import av
# 16 frames at a time: one prediction per frame, in order.
video = av.open("video.mp4")
frames = (transform(upright(f)) for f in video.decode(video=0))
for prediction in model.infer_video(frames):
depth = prediction["depth"] # meters, (H, W)
```
Depth is infinite where the scene is infinitely far, such as the sky. The
[code](https://github.com/apple-aiml-research/ml-depth-pro-2) has the command line and a live-feed
example.
## Results
### Sharpness and temporal consistency
<img class="block dark:hidden" src="https://huggingface.co/apple/DepthPro2/resolve/main/assets/tradeoff-light.png" alt="Boundary sharpness against temporal consistency for Depth Pro 2 and prior single-frame and multi-frame methods"/>
<img class="hidden dark:block" src="https://huggingface.co/apple/DepthPro2/resolve/main/assets/tradeoff-dark.png" alt="Boundary sharpness against temporal consistency for Depth Pro 2 and prior single-frame and multi-frame methods"/>
Depth Pro 2 outperforms prior work operating on a single frame or multiple frames on sharpness
(Boundary F1 score) and temporal consistency (Jitter). Higher F1 and lower jitter are better, so the
top right is best. Results averaged over Sintel and Spring datasets.
### Photos
![Depth Pro 2 on photos: the inputs on top, their predicted depth below](assets/teaser.jpg)
Top row: the input photos. Bottom row: Depth Pro 2's depth, red near and blue far.
## Method
<a href="https://huggingface.co/apple/DepthPro2/blob/main/assets/method-light.png"><img class="block dark:hidden" src="https://huggingface.co/apple/DepthPro2/resolve/main/assets/method-light.png" alt="Depth Pro 2's architecture: a multi-view transformer over the input frames, then a DPT decoder per frame"/></a>
<a href="https://huggingface.co/apple/DepthPro2/blob/main/assets/method-dark.png"><img class="hidden dark:block" src="https://huggingface.co/apple/DepthPro2/resolve/main/assets/method-dark.png" alt="Depth Pro 2's architecture: a multi-view transformer over the input frames, then a DPT decoder per frame"/></a>
A multi-view transformer processes one or more frames jointly, and a DPT decoder predicts an inverse
depth map for each frame. ① After the first third of its blocks, the transformer alternates
attention within each frame and attention across frames. ② The decoder assembles tokens from
several blocks into feature maps at 1/32 to 1/4 of the input resolution.
## License
The weights are released under the [Apple Machine Learning Research Model license](https://github.com/apple-aiml-research/ml-depth-pro-2/blob/main/LICENSE_MODEL), for
research use only: non-commercial scientific research and academic development.
## Citation
```bibtex
@article{DepthPro2:2026:arxiv,
title = {Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth},
author = {Peiyun Hu and Marko Mihajlovic and Lyndon Duong and Lahav Lipson and Wei Dong and Ama\"{e}l Delaunoy and Stephan R. Richter and Vladlen Koltun},
journal = {arXiv preprint arXiv:2611.26603},
year = {2026},
url = {https://arxiv.org/abs/2611.26603},
}
```