File size: 2,310 Bytes
f0311b4 6ec8baa f0311b4 1321741 f0311b4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 | ---
license: mit
library_name: pytorch
pipeline_tag: depth-estimation
base_model: Ruicheng/moge-3-vitl
tags: [depth-estimation, metric-depth, lidar, depth-completion, coreml, metal, on-device]
---
# PromptMoGe
Metric depth from one RGB frame and a phone LiDAR: [MoGe-3](https://github.com/microsoft/MoGe) ViT-L prompted with the
256×192 LiDAR depth of an iPhone Pro / iPad Pro, plus two compressed variants that run end to end on the device.
Project page: **https://sergmister.github.io/PromptMoGe/** · code, usage, training and the iOS demo: **https://github.com/sergmister/PromptMoGe**
| file | model | point map | refiner | iPad Pro 11-inch (M5), K = 1 |
|---|---|---|---|---|
| `promptmoge_l.pt` | PromptMoGe-L, the teacher | any resolution | fp16 | — |
| `promptmoge_a.pt` | Model A | 480×640 | int8 QAT | 152 ms |
| `promptmoge_b.pt` | Model B | 240×320 | int8 QAT | 129 ms |
| `ios/models/` | ready-to-run device models: shared ViT (`vit/`, Core ML, split across Neural Engine and GPU) and, per model (`A/`, `B/`), the prompt, neck and head Core ML models and the int8 refiner weights for the Metal engine | | | |
Each checkpoint holds only the tensors that differ from MoGe-3 (prompt stem and pyramid, neck, heads, refiner); the
frozen DINOv2 backbone is loaded from [`Ruicheng/moge-3-vitl`](https://huggingface.co/Ruicheng/moge-3-vitl).
```python
from promptmoge import load_model, infer # pip install -e . from the GitHub repository
model = load_model("A", device="cuda") # "L", "A" or "B"; downloads from this repository
out = infer(model, rgb_uint8, lidar_metres, lidar_confidence, refine_steps=1)
depth, points = out["depth"], out["points"] # metres, camera space
```
AbsRel on ARKitScenes against laser-scan ground truth (3 held-out captures, 1 094 frames, 1 200 tokens):
| | all pixels | confident pixels |
|---|---|---|
| raw ARKit LiDAR | 0.0211 | 0.0192 |
| PromptDA-L, 420×560 (1 200 tokens) | 0.0149 | 0.0133 |
| **PromptMoGe-L** | **0.0137** | **0.0122** |
| **Model A** | 0.0138 | **0.0122** |
| **Model B** | 0.0142 | 0.0125 |
The iOS models can be regenerated from the checkpoints with `python -m promptmoge.export.coreml` and
`python -m promptmoge.export.refiner`.
MIT license. Built on MoGe (Microsoft, MIT) and DINOv2 (Meta AI, Apache 2.0).
|