pointcept-bot commited on
Commit
38be483
·
verified ·
1 Parent(s): 18f288c

Model card (draft)

Browse files
Files changed (1) hide show
  1. README.md +114 -0
README.md ADDED
@@ -0,0 +1,114 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ license_name: tbd
4
+ pipeline_tag: depth-estimation
5
+ library_name: pytorch
6
+ tags:
7
+ - depth-estimation
8
+ - relative-depth
9
+ - dinov3
10
+ - high-resolution
11
+ ---
12
+
13
+ # L-DPT: Scaling Dense Prediction with Latent Decoding
14
+
15
+ [Project page](https://xywu.me/l-dpt) · [Demo](https://huggingface.co/spaces/Pointcept-Bot/L-DPT-demo)
16
+
17
+ Relative depth from a frozen DINOv3 ViT-7B encoder and a latent DPT decoder. The
18
+ decoder is trained on 768 px crops only, with no high-resolution training data, and runs
19
+ from 768 px up to 100 MP.
20
+
21
+ ## Files
22
+
23
+ | File | Model | Training data | Size | sha256 |
24
+ |---|---|---|---|---|
25
+ | `l-dpt-l-zs.pth` | L-DPT-L zero-shot | 8 synthetic / multi-view sets (below) | 1.08 GB | `b4e7e95301fda7557760f787e7992f4ccca3c03cb70e46eadeb48e227b1b31ca` |
26
+ | `l-dpt-l-mix.pth` | L-DPT-L mixed | the same + NYUv2, KITTI, DIODE train | 1.08 GB | `4852b823c68fea30b3360a249e71daf72a9c6c2c74f747602a117b42f35d36fa` |
27
+
28
+ Each file holds the decoder only (fp32 weights, 270,995,728 parameters) and its model
29
+ configuration (format `l-dpt-release/2`). **The DINOv3 encoder is not included**: it is
30
+ Meta's, under the DINOv3 License. Get the ViT-7B/16 (LVD-1689M) weights file
31
+ `dinov3_vit7b16_pretrain_lvd1689m-a955f4ea.pth` from Meta's DINOv3 release.
32
+
33
+ ## Architecture
34
+
35
+ - Encoder: DINOv3 ViT-7B/16, frozen, tapped after blocks 10, 20, 30 and 40 (the last).
36
+ - Decoder: latent DPT, width 1024, 16 heads, 4 stages of 4 transformer blocks with 2D
37
+ RoPE on the patch grid, 3x3 readout, exponential output head; bf16, with the attention
38
+ of the last stage in fp32 and a bound of 1000 on the attention logits.
39
+ - Output: positive relative depth (larger = farther), defined up to a scale and shift per
40
+ image.
41
+
42
+ ## Usage
43
+
44
+ With the `ldpt` package of the L-DPT code release:
45
+
46
+ ```python
47
+ from huggingface_hub import hf_hub_download
48
+ from ldpt.models.l_dpt import api
49
+
50
+ decoder = hf_hub_download("Pointcept-Bot/L-DPT", "l-dpt-l-zs.pth")
51
+ dinov3 = "dinov3_vit7b16_pretrain_lvd1689m-a955f4ea.pth" # Meta's official file
52
+ model = api.load(decoder, encoder_weights=dinov3) # cuda if available
53
+ depth = api.predict(model, image) # HxWx3 uint8 RGB -> HxW float32
54
+ depth = api.predict(model, image, size=None) # at the native resolution
55
+ ```
56
+
57
+ `api.predict` resizes the short side to 768 by default (`size=` another short side, or
58
+ `None` for the native resolution). For tens of megapixels load with the memory options
59
+ `api.load(..., selection_impl="stream", chunk_tokens=32768)`: peak GPU memory is then
60
+ 15 GiB at 2 MP, 19 GiB at 8 MP and 24 GiB at 32 MP (bf16; an RTX PRO 6000 Blackwell
61
+ takes 0.7 s, 4.3 s and 45 s).
62
+
63
+ ## Training data
64
+
65
+ - Zero-shot (506,790 samples): Hypersim, Virtual KITTI 2, Structured3D, TartanAir,
66
+ BlendedMVS (three releases), IRS.
67
+ - Mixed (579,637 samples): the above plus the NYUv2, KITTI and DIODE (indoor and
68
+ outdoor) training sets. The released weights are a uniform average of two checkpoints
69
+ of this recipe.
70
+
71
+ ## Results
72
+
73
+ Relative depth at short side 768 with flip TTA, per-image scale and shift fitted to the
74
+ ground truth (the evaluation of the paper). AbsRel (%, lower is better) / delta1 (%,
75
+ higher is better). These numbers use the paper's protocol: the model loaded with
76
+ `api.load(..., dec_pos_embed_rope_normalize_coords="separate")` (inference defaults to
77
+ `"min"`), a least-squares scale and shift per image in depth (the paper's fp32 solver),
78
+ the paper's NYUv2 crop (rows 159:438, cols 23:616 of the 480x640 frame), the KB and
79
+ Garg crops for KITTI, the Eigen / Garg crop for DIODE indoor / outdoor, none for ETH3D.
80
+
81
+ | Model | NYUv2 | KITTI | ETH3D | DIODE |
82
+ |---|---|---|---|---|
83
+ | L-DPT-L zero-shot (paper) | 4.3 / 98.2 | 7.4 / 96.5 | 5.2 / 97.6 | 22.0 / 84.5 |
84
+ | **`l-dpt-l-zs.pth`** | **3.66 / 98.30** | **7.18 / 96.78** | **3.96 / 98.95** | **19.58 / 85.47** |
85
+ | L-DPT-L mixed (paper) | 4.1 / 98.9 | 7.1 / 96.8 | 4.9 / 98.1 | 15.3 / 87.0 |
86
+ | **`l-dpt-l-mix.pth`** | **3.02 / 99.08** | **6.88 / 97.11** | **3.84 / 98.82** | **15.07 / 88.04** |
87
+
88
+ NYUv2, KITTI and DIODE are zero-shot for the zero-shot model and in-domain for the mixed
89
+ one. On photos in the wild, the zero-shot model separates thin foreground (foliage,
90
+ branches) from the background more cleanly; it is the demo's default.
91
+
92
+ ## Limitations
93
+
94
+ - Relative depth only: no metric scale, no camera intrinsics; a scale and shift per image
95
+ must be fitted for metric use.
96
+ - The 7B encoder dominates the cost: 13.5 GiB of GPU memory for the weights alone and a
97
+ time that grows roughly with the square of the pixel count.
98
+ - Trained on synthetic and multi-view data (plus three real sets for the mixed model):
99
+ reflective, transparent and sky regions and unusual image statistics can fail.
100
+
101
+ ## License
102
+
103
+ To be announced. The DINOv3 encoder weights are under Meta's DINOv3 License.
104
+
105
+ ## Citation
106
+
107
+ ```bibtex
108
+ @inproceedings{wu2026latent,
109
+ title = {Scaling Dense Prediction with Latent Decoding},
110
+ author = {Wu, Xiaoyang and Lao, Yixing and Wang, Chengyao and Yang, Senqiao and Zhang, Yujia and Zhao, Hengshuang},
111
+ booktitle = {European Conference on Computer Vision (ECCV)},
112
+ year = {2026}
113
+ }
114
+ ```