L-DPT: Scaling Dense Prediction with Latent Decoding

Project page ยท Code (coming) ยท Demo (coming)

Relative depth from a frozen DINOv3 ViT-7B encoder and a latent DPT decoder. The decoder is trained on 768 px crops only, with no high-resolution training data, and runs at a photo's native resolution, up to 100 MP.

Files

File Model Training data Size sha256
l-dpt_large_depth_zeroshot.pth L-DPT-L zero-shot 8 synthetic / multi-view sets (below) 1.08 GB b4e7e95301fda7557760f787e7992f4ccca3c03cb70e46eadeb48e227b1b31ca
l-dpt_large_depth_mixed.pth L-DPT-L mixed the same + NYUv2, KITTI, DIODE train 1.08 GB 4852b823c68fea30b3360a249e71daf72a9c6c2c74f747602a117b42f35d36fa

Names: l-dpt_<decoder size>_<task>_<training data>.pth; the training recipes are configs/hybrid/depth-l-dpt-v1m1-0-large-zeroshot.py and depth-l-dpt-v1m1-1-large-mixed.py (+ 1a-large-mixed-anneal) of the code release. Each file holds the decoder only (fp32 weights, 271M parameters) and its model configuration. The DINOv3 encoder is not included: it is Meta's, under the DINOv3 License. Get the ViT-7B/16 (LVD-1689M) weights from Meta: the original file dinov3_vit7b16_pretrain_lvd1689m-a955f4ea.pth of the DINOv3 release, or the gated Hugging Face repo facebook/dinov3-vit7b16-pretrain-lvd1689m.

Architecture

  • Encoder: DINOv3 ViT-7B/16, frozen, tapped after blocks 10, 20, 30 and 40 (the last).
  • Decoder: latent DPT, width 1024, 16 heads, 4 stages of 4 transformer blocks with 2D RoPE on the patch grid, 3x3 readout, exponential output head; bf16, with the attention of the last stage in fp32 and a bound of 1000 on the attention logits.
  • Output: positive relative depth (larger = farther), defined up to a scale and shift per image.

Usage

With the ldpt package of the L-DPT code release:

from huggingface_hub import hf_hub_download
from ldpt.models.l_dpt import api

decoder = hf_hub_download("pointcept-bot/L-DPT", "l-dpt_large_depth_zeroshot.pth")
dinov3 = "dinov3_vit7b16_pretrain_lvd1689m-a955f4ea.pth"  # Meta's file (or the HF folder)
model = api.load(decoder, encoder_weights=dinov3)  # cuda if available
depth = api.predict(model, image)                   # HxWx3 uint8 RGB -> HxW float32
depth = api.predict(model, image, size=None)        # at the native resolution

api.predict resizes the short side to 768 by default (size= another short side, or None for the native resolution). For tens of megapixels load with the memory options api.load(..., selection_impl="stream", chunk_tokens=32768): peak GPU memory is then 15 GiB at 2 MP, 19 GiB at 8 MP, 24 GiB at 32 MP and 43 GiB at 100 MP (bf16; one NVIDIA GB200 takes about 5 minutes at 100 MP).

Training data

  • Zero-shot (506,790 samples): Hypersim, Virtual KITTI 2, Structured3D, TartanAir, BlendedMVS (three releases), IRS.
  • Mixed (579,637 samples): the above plus the NYUv2, KITTI and DIODE (indoor and outdoor) training sets. The released weights are a uniform average of two checkpoints of this recipe.

Results

Relative depth, AbsRel (%, lower is better) / delta1 (%, higher is better): short side 768, horizontal-flip TTA, a least-squares scale and shift per image fitted to the ground truth in depth. Crops: NYUv2 rows 159:438, cols 23:616 of the 480x640 frame; KITTI the benchmark and Garg crops; DIODE indoor / outdoor the Eigen / Garg crop; ETH3D none. DIODE is the combined indoor + outdoor validation set.

Model NYUv2 KITTI ETH3D DIODE
l-dpt_large_depth_zeroshot.pth 3.55 / 98.30 5.98 / 97.85 3.75 / 99.01 19.34 / 85.65
l-dpt_large_depth_mixed.pth 2.89 / 99.08 5.39 / 98.38 3.57 / 98.94 14.98 / 88.17

NYUv2, KITTI and DIODE are zero-shot for the zero-shot model and in-domain for the mixed one. On photos in the wild, the zero-shot model separates thin foreground (foliage, branches) from the background more cleanly.

Limitations

  • Relative depth only: no metric scale, no camera intrinsics; a scale and shift per image must be fitted for metric use.
  • The 7B encoder dominates the cost: 13.5 GiB of GPU memory for the weights alone and a time that grows roughly with the square of the pixel count.
  • Trained on synthetic and multi-view data (plus three real sets for the mixed model): reflective, transparent and sky regions and unusual image statistics can fail.

License

The L-DPT weights are released under CC BY-NC 4.0. The DINOv3 encoder they need is Meta's, under the DINOv3 License, and is not included here.

Citation

@inproceedings{wu2026latent,
  title     = {Scaling Dense Prediction with Latent Decoding},
  author    = {Wu, Xiaoyang and Lao, Yixing and Wang, Chengyao and Yang, Senqiao and Zhang, Yujia and Zhao, Hengshuang},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support