| --- |
| license: mit |
| library_name: pytorch |
| pipeline_tag: depth-estimation |
| base_model: Ruicheng/moge-3-vitl |
| tags: [depth-estimation, metric-depth, lidar, depth-completion, coreml, metal, on-device] |
| --- |
| |
| # PromptMoGe |
|
|
| Metric depth from one RGB frame and a phone LiDAR: [MoGe-3](https://github.com/microsoft/MoGe) ViT-L prompted with the |
| 256×192 LiDAR depth of an iPhone Pro / iPad Pro, plus two compressed variants that run end to end on the device. |
| Project page: **https://sergmister.github.io/PromptMoGe/** · code, usage, training and the iOS demo: **https://github.com/sergmister/PromptMoGe** |
|
|
| | file | model | point map | refiner | iPad Pro 11-inch (M5), K = 1 | |
| |---|---|---|---|---| |
| | `promptmoge_l.pt` | PromptMoGe-L, the teacher | any resolution | fp16 | — | |
| | `promptmoge_a.pt` | Model A | 480×640 | int8 QAT | 152 ms | |
| | `promptmoge_b.pt` | Model B | 240×320 | int8 QAT | 129 ms | |
| | `ios/models/` | ready-to-run device models: shared ViT (`vit/`, Core ML, split across Neural Engine and GPU) and, per model (`A/`, `B/`), the prompt, neck and head Core ML models and the int8 refiner weights for the Metal engine | | | | |
|
|
| Each checkpoint holds only the tensors that differ from MoGe-3 (prompt stem and pyramid, neck, heads, refiner); the |
| frozen DINOv2 backbone is loaded from [`Ruicheng/moge-3-vitl`](https://huggingface.co/Ruicheng/moge-3-vitl). |
|
|
| ```python |
| from promptmoge import load_model, infer # pip install -e . from the GitHub repository |
| model = load_model("A", device="cuda") # "L", "A" or "B"; downloads from this repository |
| out = infer(model, rgb_uint8, lidar_metres, lidar_confidence, refine_steps=1) |
| depth, points = out["depth"], out["points"] # metres, camera space |
| ``` |
|
|
| AbsRel on ARKitScenes against laser-scan ground truth (3 held-out captures, 1 094 frames, 1 200 tokens): |
|
|
| | | all pixels | confident pixels | |
| |---|---|---| |
| | raw ARKit LiDAR | 0.0211 | 0.0192 | |
| | PromptDA-L, 420×560 (1 200 tokens) | 0.0149 | 0.0133 | |
| | **PromptMoGe-L** | **0.0137** | **0.0122** | |
| | **Model A** | 0.0138 | **0.0122** | |
| | **Model B** | 0.0142 | 0.0125 | |
|
|
| The iOS models can be regenerated from the checkpoints with `python -m promptmoge.export.coreml` and |
| `python -m promptmoge.export.refiner`. |
|
|
| MIT license. Built on MoGe (Microsoft, MIT) and DINOv2 (Meta AI, Apache 2.0). |
|
|
