|
Download README.md from pointcept-bot/L-DPT: direct link, hf CLI and curl.
- Browser
- Download file 5.23 kB
-
https://huggingface.co/pointcept-bot/L-DPT/resolve/main/README.md
- Command line
-
hf download hf://pointcept-bot/L-DPT/README.md
-
curl -L -o README.md https://huggingface.co/pointcept-bot/L-DPT/resolve/main/README.md
5.23 kB
| license: cc-by-nc-4.0 | |
| pipeline_tag: depth-estimation | |
| library_name: pytorch | |
| tags: | |
| - depth-estimation | |
| - relative-depth | |
| - dinov3 | |
| - high-resolution | |
| # L-DPT: Scaling Dense Prediction with Latent Decoding | |
| [Project page](https://xywu.me/l-dpt) · Code (coming) · Demo (coming) | |
| Relative depth from a frozen DINOv3 ViT-7B encoder and a latent DPT decoder. The | |
| decoder is trained on 768 px crops only, with no high-resolution training data, and runs | |
| at a photo's native resolution, up to 100 MP. | |
| ## Files | |
| | File | Model | Training data | Size | sha256 | | |
| |---|---|---|---|---| | |
| | `l-dpt_large_depth_zeroshot.pth` | L-DPT-L zero-shot | 8 synthetic / multi-view sets (below) | 1.08 GB | `b4e7e95301fda7557760f787e7992f4ccca3c03cb70e46eadeb48e227b1b31ca` | | |
| | `l-dpt_large_depth_mixed.pth` | L-DPT-L mixed | the same + NYUv2, KITTI, DIODE train | 1.08 GB | `4852b823c68fea30b3360a249e71daf72a9c6c2c74f747602a117b42f35d36fa` | | |
| Names: `l-dpt_<decoder size>_<task>_<training data>.pth`; the training recipes are | |
| `configs/hybrid/depth-l-dpt-v1m1-0-large-zeroshot.py` and | |
| `depth-l-dpt-v1m1-1-large-mixed.py` (+ `1a-large-mixed-anneal`) of the code release. | |
| Each file holds the decoder only (fp32 weights, 271M parameters) and its model | |
| configuration. **The DINOv3 encoder is not included**: it is Meta's, under the DINOv3 | |
| License. Get the ViT-7B/16 (LVD-1689M) weights from Meta: the original file | |
| `dinov3_vit7b16_pretrain_lvd1689m-a955f4ea.pth` of the DINOv3 release, or the gated | |
| Hugging Face repo | |
| [facebook/dinov3-vit7b16-pretrain-lvd1689m](https://huggingface.co/facebook/dinov3-vit7b16-pretrain-lvd1689m). | |
| ## Architecture | |
| - Encoder: DINOv3 ViT-7B/16, frozen, tapped after blocks 10, 20, 30 and 40 (the last). | |
| - Decoder: latent DPT, width 1024, 16 heads, 4 stages of 4 transformer blocks with 2D | |
| RoPE on the patch grid, 3x3 readout, exponential output head; bf16, with the attention | |
| of the last stage in fp32 and a bound of 1000 on the attention logits. | |
| - Output: positive relative depth (larger = farther), defined up to a scale and shift per | |
| image. | |
| ## Usage | |
| With the `ldpt` package of the L-DPT code release: | |
| ```python | |
| from huggingface_hub import hf_hub_download | |
| from ldpt.models.l_dpt import api | |
| decoder = hf_hub_download("pointcept-bot/L-DPT", "l-dpt_large_depth_zeroshot.pth") | |
| dinov3 = "dinov3_vit7b16_pretrain_lvd1689m-a955f4ea.pth" # Meta's file (or the HF folder) | |
| model = api.load(decoder, encoder_weights=dinov3) # cuda if available | |
| depth = api.predict(model, image) # HxWx3 uint8 RGB -> HxW float32 | |
| depth = api.predict(model, image, size=None) # at the native resolution | |
| ``` | |
| `api.predict` resizes the short side to 768 by default (`size=` another short side, or | |
| `None` for the native resolution). For tens of megapixels load with the memory options | |
| `api.load(..., selection_impl="stream", chunk_tokens=32768)`: peak GPU memory is then | |
| 15 GiB at 2 MP, 19 GiB at 8 MP, 24 GiB at 32 MP and 43 GiB at 100 MP (bf16; one NVIDIA | |
| GB200 takes about 5 minutes at 100 MP). | |
| ## Training data | |
| - Zero-shot (506,790 samples): Hypersim, Virtual KITTI 2, Structured3D, TartanAir, | |
| BlendedMVS (three releases), IRS. | |
| - Mixed (579,637 samples): the above plus the NYUv2, KITTI and DIODE (indoor and | |
| outdoor) training sets. The released weights are a uniform average of two checkpoints | |
| of this recipe. | |
| ## Results | |
| Relative depth, AbsRel (%, lower is better) / delta1 (%, higher is better): short side | |
| 768, horizontal-flip TTA, a least-squares scale and shift per image fitted to the ground | |
| truth in depth. Crops: NYUv2 rows 159:438, cols 23:616 of the 480x640 frame; KITTI the | |
| benchmark and Garg crops; DIODE indoor / outdoor the Eigen / Garg crop; ETH3D none. | |
| DIODE is the combined indoor + outdoor validation set. | |
| | Model | NYUv2 | KITTI | ETH3D | DIODE | | |
| |---|---|---|---|---| | |
| | `l-dpt_large_depth_zeroshot.pth` | 3.55 / 98.30 | 5.98 / 97.85 | 3.75 / 99.01 | 19.34 / 85.65 | | |
| | `l-dpt_large_depth_mixed.pth` | 2.89 / 99.08 | 5.39 / 98.38 | 3.57 / 98.94 | 14.98 / 88.17 | | |
| NYUv2, KITTI and DIODE are zero-shot for the zero-shot model and in-domain for the mixed | |
| one. On photos in the wild, the zero-shot model separates thin foreground (foliage, | |
| branches) from the background more cleanly. | |
| ## Limitations | |
| - Relative depth only: no metric scale, no camera intrinsics; a scale and shift per image | |
| must be fitted for metric use. | |
| - The 7B encoder dominates the cost: 13.5 GiB of GPU memory for the weights alone and a | |
| time that grows roughly with the square of the pixel count. | |
| - Trained on synthetic and multi-view data (plus three real sets for the mixed model): | |
| reflective, transparent and sky regions and unusual image statistics can fail. | |
| ## License | |
| The L-DPT weights are released under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/). The DINOv3 encoder they need is Meta's, under the DINOv3 License, and is not included here. | |
| ## Citation | |
| ```bibtex | |
| @inproceedings{wu2026latent, | |
| title = {Scaling Dense Prediction with Latent Decoding}, | |
| author = {Wu, Xiaoyang and Lao, Yixing and Wang, Chengyao and Yang, Senqiao and Zhang, Yujia and Zhao, Hengshuang}, | |
| booktitle = {European Conference on Computer Vision (ECCV)}, | |
| year = {2026} | |
| } | |
| ``` | |