Model card (draft)
Browse files
README.md
ADDED
|
@@ -0,0 +1,114 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: other
|
| 3 |
+
license_name: tbd
|
| 4 |
+
pipeline_tag: depth-estimation
|
| 5 |
+
library_name: pytorch
|
| 6 |
+
tags:
|
| 7 |
+
- depth-estimation
|
| 8 |
+
- relative-depth
|
| 9 |
+
- dinov3
|
| 10 |
+
- high-resolution
|
| 11 |
+
---
|
| 12 |
+
|
| 13 |
+
# L-DPT: Scaling Dense Prediction with Latent Decoding
|
| 14 |
+
|
| 15 |
+
[Project page](https://xywu.me/l-dpt) · [Demo](https://huggingface.co/spaces/Pointcept-Bot/L-DPT-demo)
|
| 16 |
+
|
| 17 |
+
Relative depth from a frozen DINOv3 ViT-7B encoder and a latent DPT decoder. The
|
| 18 |
+
decoder is trained on 768 px crops only, with no high-resolution training data, and runs
|
| 19 |
+
from 768 px up to 100 MP.
|
| 20 |
+
|
| 21 |
+
## Files
|
| 22 |
+
|
| 23 |
+
| File | Model | Training data | Size | sha256 |
|
| 24 |
+
|---|---|---|---|---|
|
| 25 |
+
| `l-dpt-l-zs.pth` | L-DPT-L zero-shot | 8 synthetic / multi-view sets (below) | 1.08 GB | `b4e7e95301fda7557760f787e7992f4ccca3c03cb70e46eadeb48e227b1b31ca` |
|
| 26 |
+
| `l-dpt-l-mix.pth` | L-DPT-L mixed | the same + NYUv2, KITTI, DIODE train | 1.08 GB | `4852b823c68fea30b3360a249e71daf72a9c6c2c74f747602a117b42f35d36fa` |
|
| 27 |
+
|
| 28 |
+
Each file holds the decoder only (fp32 weights, 270,995,728 parameters) and its model
|
| 29 |
+
configuration (format `l-dpt-release/2`). **The DINOv3 encoder is not included**: it is
|
| 30 |
+
Meta's, under the DINOv3 License. Get the ViT-7B/16 (LVD-1689M) weights file
|
| 31 |
+
`dinov3_vit7b16_pretrain_lvd1689m-a955f4ea.pth` from Meta's DINOv3 release.
|
| 32 |
+
|
| 33 |
+
## Architecture
|
| 34 |
+
|
| 35 |
+
- Encoder: DINOv3 ViT-7B/16, frozen, tapped after blocks 10, 20, 30 and 40 (the last).
|
| 36 |
+
- Decoder: latent DPT, width 1024, 16 heads, 4 stages of 4 transformer blocks with 2D
|
| 37 |
+
RoPE on the patch grid, 3x3 readout, exponential output head; bf16, with the attention
|
| 38 |
+
of the last stage in fp32 and a bound of 1000 on the attention logits.
|
| 39 |
+
- Output: positive relative depth (larger = farther), defined up to a scale and shift per
|
| 40 |
+
image.
|
| 41 |
+
|
| 42 |
+
## Usage
|
| 43 |
+
|
| 44 |
+
With the `ldpt` package of the L-DPT code release:
|
| 45 |
+
|
| 46 |
+
```python
|
| 47 |
+
from huggingface_hub import hf_hub_download
|
| 48 |
+
from ldpt.models.l_dpt import api
|
| 49 |
+
|
| 50 |
+
decoder = hf_hub_download("Pointcept-Bot/L-DPT", "l-dpt-l-zs.pth")
|
| 51 |
+
dinov3 = "dinov3_vit7b16_pretrain_lvd1689m-a955f4ea.pth" # Meta's official file
|
| 52 |
+
model = api.load(decoder, encoder_weights=dinov3) # cuda if available
|
| 53 |
+
depth = api.predict(model, image) # HxWx3 uint8 RGB -> HxW float32
|
| 54 |
+
depth = api.predict(model, image, size=None) # at the native resolution
|
| 55 |
+
```
|
| 56 |
+
|
| 57 |
+
`api.predict` resizes the short side to 768 by default (`size=` another short side, or
|
| 58 |
+
`None` for the native resolution). For tens of megapixels load with the memory options
|
| 59 |
+
`api.load(..., selection_impl="stream", chunk_tokens=32768)`: peak GPU memory is then
|
| 60 |
+
15 GiB at 2 MP, 19 GiB at 8 MP and 24 GiB at 32 MP (bf16; an RTX PRO 6000 Blackwell
|
| 61 |
+
takes 0.7 s, 4.3 s and 45 s).
|
| 62 |
+
|
| 63 |
+
## Training data
|
| 64 |
+
|
| 65 |
+
- Zero-shot (506,790 samples): Hypersim, Virtual KITTI 2, Structured3D, TartanAir,
|
| 66 |
+
BlendedMVS (three releases), IRS.
|
| 67 |
+
- Mixed (579,637 samples): the above plus the NYUv2, KITTI and DIODE (indoor and
|
| 68 |
+
outdoor) training sets. The released weights are a uniform average of two checkpoints
|
| 69 |
+
of this recipe.
|
| 70 |
+
|
| 71 |
+
## Results
|
| 72 |
+
|
| 73 |
+
Relative depth at short side 768 with flip TTA, per-image scale and shift fitted to the
|
| 74 |
+
ground truth (the evaluation of the paper). AbsRel (%, lower is better) / delta1 (%,
|
| 75 |
+
higher is better). These numbers use the paper's protocol: the model loaded with
|
| 76 |
+
`api.load(..., dec_pos_embed_rope_normalize_coords="separate")` (inference defaults to
|
| 77 |
+
`"min"`), a least-squares scale and shift per image in depth (the paper's fp32 solver),
|
| 78 |
+
the paper's NYUv2 crop (rows 159:438, cols 23:616 of the 480x640 frame), the KB and
|
| 79 |
+
Garg crops for KITTI, the Eigen / Garg crop for DIODE indoor / outdoor, none for ETH3D.
|
| 80 |
+
|
| 81 |
+
| Model | NYUv2 | KITTI | ETH3D | DIODE |
|
| 82 |
+
|---|---|---|---|---|
|
| 83 |
+
| L-DPT-L zero-shot (paper) | 4.3 / 98.2 | 7.4 / 96.5 | 5.2 / 97.6 | 22.0 / 84.5 |
|
| 84 |
+
| **`l-dpt-l-zs.pth`** | **3.66 / 98.30** | **7.18 / 96.78** | **3.96 / 98.95** | **19.58 / 85.47** |
|
| 85 |
+
| L-DPT-L mixed (paper) | 4.1 / 98.9 | 7.1 / 96.8 | 4.9 / 98.1 | 15.3 / 87.0 |
|
| 86 |
+
| **`l-dpt-l-mix.pth`** | **3.02 / 99.08** | **6.88 / 97.11** | **3.84 / 98.82** | **15.07 / 88.04** |
|
| 87 |
+
|
| 88 |
+
NYUv2, KITTI and DIODE are zero-shot for the zero-shot model and in-domain for the mixed
|
| 89 |
+
one. On photos in the wild, the zero-shot model separates thin foreground (foliage,
|
| 90 |
+
branches) from the background more cleanly; it is the demo's default.
|
| 91 |
+
|
| 92 |
+
## Limitations
|
| 93 |
+
|
| 94 |
+
- Relative depth only: no metric scale, no camera intrinsics; a scale and shift per image
|
| 95 |
+
must be fitted for metric use.
|
| 96 |
+
- The 7B encoder dominates the cost: 13.5 GiB of GPU memory for the weights alone and a
|
| 97 |
+
time that grows roughly with the square of the pixel count.
|
| 98 |
+
- Trained on synthetic and multi-view data (plus three real sets for the mixed model):
|
| 99 |
+
reflective, transparent and sky regions and unusual image statistics can fail.
|
| 100 |
+
|
| 101 |
+
## License
|
| 102 |
+
|
| 103 |
+
To be announced. The DINOv3 encoder weights are under Meta's DINOv3 License.
|
| 104 |
+
|
| 105 |
+
## Citation
|
| 106 |
+
|
| 107 |
+
```bibtex
|
| 108 |
+
@inproceedings{wu2026latent,
|
| 109 |
+
title = {Scaling Dense Prediction with Latent Decoding},
|
| 110 |
+
author = {Wu, Xiaoyang and Lao, Yixing and Wang, Chengyao and Yang, Senqiao and Zhang, Yujia and Zhao, Hengshuang},
|
| 111 |
+
booktitle = {European Conference on Computer Vision (ECCV)},
|
| 112 |
+
year = {2026}
|
| 113 |
+
}
|
| 114 |
+
```
|