|
Download README.md from apple/DepthPro2: direct link, hf CLI and curl.
- Browser
- Download file 6.88 kB
-
https://huggingface.co/apple/DepthPro2/resolve/main/README.md
- Command line
-
hf download hf://apple/DepthPro2/README.md
-
curl -L -o README.md https://huggingface.co/apple/DepthPro2/resolve/main/README.md
6.88 kB
| license: other | |
| license_name: apple-amlr | |
| license_link: https://github.com/apple-aiml-research/ml-depth-pro-2/blob/main/LICENSE_MODEL | |
| pipeline_tag: depth-estimation | |
| library_name: depth-pro-2 | |
| tags: | |
| - depth-estimation | |
| - metric-depth | |
| - monocular-depth | |
| - video-depth | |
| <h1 align="center">Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth</h1> | |
| <p align="center"> | |
| <a href="https://github.com/apple-aiml-research/ml-depth-pro-2"><img src="https://img.shields.io/badge/GitHub-181717?style=flat-square&logo=github&logoColor=white" alt="GitHub"/></a> | |
| <a href="https://arxiv.org/abs/2611.26603"><img src="https://img.shields.io/badge/Technical%20Report-B31B1B?style=flat-square&logo=arxiv&logoColor=white" alt="Technical Report"/></a> | |
| <a href="https://apple-aiml-research.github.io/ml-depth-pro-2/"><img src="https://img.shields.io/badge/Project%20Page-666666?style=flat-square&logo=data%3Aimage%2Fsvg%2Bxml%3Bbase64%2CPHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAyNCAyNCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSJ3aGl0ZSIgc3Ryb2tlLXdpZHRoPSIyIj48Y2lyY2xlIGN4PSIxMiIgY3k9IjEyIiByPSIxMCIvPjxwYXRoIGQ9Ik0yIDEyaDIwTTEyIDJhMTUgMTUgMCAwIDEgMCAyME0xMiAyYTE1IDE1IDAgMCAwIDAgMjAiLz48L3N2Zz4%3D" alt="Project Page"/></a> | |
| </p> | |
| <!-- Hidden until the Space is public: | |
| <h3 align="center"> | |
| 🚀 <a href="https://huggingface.co/spaces/apple/DepthPro2">Try the Interactive Demo →</a> | |
| </h3> | |
| --> | |
| Depth Pro 2 estimates metric depth for every pixel from an ordinary camera: a single photo, a | |
| whole video, or a live feed, at the input's full resolution. Its depth is sharp at object edges | |
| and steady from frame to frame. | |
| <video src="https://huggingface.co/apple/DepthPro2/resolve/main/assets/hero.mp4" poster="https://huggingface.co/apple/DepthPro2/resolve/main/assets/hero-poster.jpg" autoplay muted loop playsinline width="100%"></video> | |
| ## Models | |
| Each size is its own repository with its weights. Larger sizes give sharper, steadier depth; | |
| smaller ones run faster. | |
| | Model | Repository | Parameters | Download size | | |
| |:---:|:---:|:---:|:---:| | |
| | S | [apple/DepthPro2-S](https://huggingface.co/apple/DepthPro2-S) | 24M | 97 MB | | |
| | B | [apple/DepthPro2-B](https://huggingface.co/apple/DepthPro2-B) | 97M | 386 MB | | |
| | L | [apple/DepthPro2-L](https://huggingface.co/apple/DepthPro2-L) | 334M | 1.3 GB | | |
| | H+ | [apple/DepthPro2-H](https://huggingface.co/apple/DepthPro2-H) | 872M | 3.5 GB | | |
| All sizes share a focal length estimator, | |
| [apple/DepthPro2-focal](https://huggingface.co/apple/DepthPro2-focal) (23.5M parameters, 94 MB), | |
| which loads with each size and predicts the focal length when it is unknown. | |
| ## Quick start | |
| ```bash | |
| pip install "git+https://github.com/apple-aiml-research/ml-depth-pro-2" | |
| ``` | |
| ```python | |
| from functools import partial | |
| import torch | |
| from huggingface_hub import hf_hub_download | |
| from safetensors.torch import load_file | |
| from depth_pro_2 import DepthPro2, load_rgb, upright | |
| from depth_pro_2.model import to_normalized_tensor | |
| from depth_pro_2.network.fov import FOVEstimator | |
| # apple/DepthPro2-S, -B, -L or -H; bfloat16 is how the models run on a GPU. | |
| model = DepthPro2.from_pretrained("apple/DepthPro2-H", precision=torch.bfloat16) | |
| # The focal length estimator all sizes share, for photos and videos without a focal length. | |
| weights = hf_hub_download("apple/DepthPro2-focal", "model.safetensors") | |
| model.fov = FOVEstimator(torch.bfloat16) | |
| model.fov.load_state_dict(load_file(weights)) | |
| model = model.to("cuda").eval() | |
| transform = partial(to_normalized_tensor, device=torch.device("cuda")) | |
| ``` | |
| ### Image | |
| ```python | |
| # f_px is the focal length recorded in the photo, or None. | |
| image, f_px = load_rgb("photo.jpg") | |
| depth = model.infer(transform(image), f_px=f_px)["depth"] # meters, (H, W) | |
| ``` | |
| ### Video | |
| ```python | |
| import av | |
| # 16 frames at a time: one prediction per frame, in order. | |
| video = av.open("video.mp4") | |
| frames = (transform(upright(f)) for f in video.decode(video=0)) | |
| for prediction in model.infer_video(frames): | |
| depth = prediction["depth"] # meters, (H, W) | |
| ``` | |
| Depth is infinite where the scene is infinitely far, such as the sky. The | |
| [code](https://github.com/apple-aiml-research/ml-depth-pro-2) has the command line and a live-feed | |
| example. | |
| ## Results | |
| ### Sharpness and temporal consistency | |
| <img class="block dark:hidden" src="https://huggingface.co/apple/DepthPro2/resolve/main/assets/tradeoff-light.png" alt="Boundary sharpness against temporal consistency for Depth Pro 2 and prior single-frame and multi-frame methods"/> | |
| <img class="hidden dark:block" src="https://huggingface.co/apple/DepthPro2/resolve/main/assets/tradeoff-dark.png" alt="Boundary sharpness against temporal consistency for Depth Pro 2 and prior single-frame and multi-frame methods"/> | |
| Depth Pro 2 outperforms prior work operating on a single frame or multiple frames on sharpness | |
| (Boundary F1 score) and temporal consistency (Jitter). Higher F1 and lower jitter are better, so the | |
| top right is best. Results averaged over Sintel and Spring datasets. | |
| ### Photos | |
|  | |
| Top row: the input photos. Bottom row: Depth Pro 2's depth, red near and blue far. | |
| ## Method | |
| <a href="https://huggingface.co/apple/DepthPro2/blob/main/assets/method-light.png"><img class="block dark:hidden" src="https://huggingface.co/apple/DepthPro2/resolve/main/assets/method-light.png" alt="Depth Pro 2's architecture: a multi-view transformer over the input frames, then a DPT decoder per frame"/></a> | |
| <a href="https://huggingface.co/apple/DepthPro2/blob/main/assets/method-dark.png"><img class="hidden dark:block" src="https://huggingface.co/apple/DepthPro2/resolve/main/assets/method-dark.png" alt="Depth Pro 2's architecture: a multi-view transformer over the input frames, then a DPT decoder per frame"/></a> | |
| A multi-view transformer processes one or more frames jointly, and a DPT decoder predicts an inverse | |
| depth map for each frame. ① After the first third of its blocks, the transformer alternates | |
| attention within each frame and attention across frames. ② The decoder assembles tokens from | |
| several blocks into feature maps at 1/32 to 1/4 of the input resolution. | |
| ## License | |
| The weights are released under the [Apple Machine Learning Research Model license](https://github.com/apple-aiml-research/ml-depth-pro-2/blob/main/LICENSE_MODEL), for | |
| research use only: non-commercial scientific research and academic development. | |
| ## Citation | |
| ```bibtex | |
| @article{DepthPro2:2026:arxiv, | |
| title = {Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth}, | |
| author = {Peiyun Hu and Marko Mihajlovic and Lyndon Duong and Lahav Lipson and Wei Dong and Ama\"{e}l Delaunoy and Stephan R. Richter and Vladlen Koltun}, | |
| journal = {arXiv preprint arXiv:2611.26603}, | |
| year = {2026}, | |
| url = {https://arxiv.org/abs/2611.26603}, | |
| } | |
| ``` | |