Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -1,3 +1,181 @@
|
|
| 1 |
---
|
| 2 |
-
license:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
license: mit
|
| 3 |
+
tags:
|
| 4 |
+
- monocular-depth-estimation
|
| 5 |
+
- self-supervised
|
| 6 |
+
- autonomous-driving
|
| 7 |
+
- yolo
|
| 8 |
+
- pytorch
|
| 9 |
+
- kitti
|
| 10 |
+
- cityscapes
|
| 11 |
+
library_name: pytorch
|
| 12 |
+
pipeline_tag: depth-estimation
|
| 13 |
---
|
| 14 |
+
|
| 15 |
+
# FlexDepth: A Flexible Scale-Driven Family for Self-Supervised Monocular Depth Estimation
|
| 16 |
+
|
| 17 |
+
**Towards Robust Driving Perception: A Flexible Scale-Driven Family for Self-Supervised Monocular Depth Estimation**
|
| 18 |
+
|
| 19 |
+
[](https://eccv.ecva.net/)
|
| 20 |
+
[](https://arxiv.org/abs/2607.00736)
|
| 21 |
+
[](https://startnew.github.io/projects/flexdepth/)
|
| 22 |
+
[](https://github.com/StarNew/FlexDepth)
|
| 23 |
+
|
| 24 |
+
## Overview
|
| 25 |
+
|
| 26 |
+
FlexDepth is a family of self-supervised monocular depth estimation models designed for robust driving perception. It introduces a **Scale-Driven Decoder (SDD)** with adaptive component selection, enabling a single architecture to span five model scales β from ultra-lightweight (1.5M params, 0.7 GFLOPs) to high-accuracy (32.3M params, 24.6 GFLOPs).
|
| 27 |
+
|
| 28 |
+
### Key Features
|
| 29 |
+
|
| 30 |
+
- **Five model scales**: Nano (N), Small (S), Medium (M), Large (L), X-Large (X)
|
| 31 |
+
- **Scale-Driven Decoder (SDD)**: Adaptive component selection based on model scale
|
| 32 |
+
- **High-Efficiency Bottleneck (HEB)**: For small models (N, S) β maximizes efficiency
|
| 33 |
+
- **High-Performance Bottleneck (HPB)**: For large models (M, L, X) β maximizes accuracy
|
| 34 |
+
- **Dynamic upsampling**: Sharper depth boundaries via learned upsampling
|
| 35 |
+
- **Two-stage static-dynamic decoupled training**: Handles dynamic scenes in driving scenarios
|
| 36 |
+
- **YOLO11-based encoder**: Leverages rich visual representations from YOLO segmentation pretraining
|
| 37 |
+
|
| 38 |
+
## Model Zoo
|
| 39 |
+
|
| 40 |
+
### KITTI-trained Models
|
| 41 |
+
|
| 42 |
+
| Model | Params | GFLOPs | Abs Rel β | Sq Rel β | RMSE β | RMSE log β | Ξ΄<1.25 β | Ξ΄<1.25Β² β | Ξ΄<1.25Β³ β |
|
| 43 |
+
|-------|--------|--------|-----------|----------|--------|------------|----------|-----------|-----------|
|
| 44 |
+
| Flex-Nano | 1.5M | 0.7 | 0.110 | 0.794 | 4.678 | 0.184 | 0.878 | 0.961 | 0.983 |
|
| 45 |
+
| Flex-Small | 6.1M | 2.8 | 0.104 | 0.713 | 4.458 | 0.179 | 0.890 | 0.964 | 0.983 |
|
| 46 |
+
| Flex-Medium | 12.7M | 10.0 | 0.096 | 0.639 | 4.253 | 0.172 | 0.903 | 0.968 | 0.985 |
|
| 47 |
+
| Flex-Large | 15.2M | 11.5 | 0.095 | 0.642 | 4.199 | 0.171 | 0.906 | 0.968 | 0.984 |
|
| 48 |
+
| Flex-X-Large | 32.3M | 24.6 | **0.093** | **0.605** | **4.114** | **0.167** | **0.910** | **0.969** | **0.985** |
|
| 49 |
+
|
| 50 |
+
### Cityscapes-trained Models
|
| 51 |
+
|
| 52 |
+
| Model | Params | GFLOPs | Abs Rel β | Sq Rel β | RMSE β | RMSE log β | Ξ΄<1.25 β | Ξ΄<1.25Β² β | Ξ΄<1.25Β³ β |
|
| 53 |
+
|-------|--------|--------|-----------|----------|--------|------------|----------|-----------|-----------|
|
| 54 |
+
| Flex-Nano | 1.5M | 0.6 | 0.107 | 1.261 | 6.133 | 0.164 | 0.893 | 0.971 | 0.989 |
|
| 55 |
+
| Flex-Small | 6.1M | 2.2 | 0.100 | 1.078 | 5.813 | 0.153 | 0.904 | 0.975 | 0.991 |
|
| 56 |
+
| Flex-Medium | 12.7M | 8.0 | 0.089 | 0.885 | 5.358 | 0.143 | 0.917 | 0.979 | 0.993 |
|
| 57 |
+
| Flex-Large | 15.2M | 9.2 | 0.087 | 0.911 | 5.310 | 0.139 | 0.924 | 0.981 | 0.993 |
|
| 58 |
+
| Flex-X-Large | 32.3M | 19.7 | **0.086** | **0.877** | **5.268** | **0.137** | **0.926** | **0.982** | **0.993** |
|
| 59 |
+
|
| 60 |
+
### Efficiency
|
| 61 |
+
|
| 62 |
+
| Model | FPS (Snapdragon 8 Elite) | FPS (RTX 2080 Ti) |
|
| 63 |
+
|-------|--------------------------|---------------------|
|
| 64 |
+
| Flex-Nano | 37.6 | 180+ |
|
| 65 |
+
| Flex-Small | 18.6 | 120+ |
|
| 66 |
+
| Flex-Medium | 5.8 | 60+ |
|
| 67 |
+
| Flex-Large | 5.2 | 55+ |
|
| 68 |
+
| Flex-X-Large | 3.0 | 40+ |
|
| 69 |
+
|
| 70 |
+
## Model Files
|
| 71 |
+
|
| 72 |
+
Each model consists of two weight files:
|
| 73 |
+
|
| 74 |
+
```
|
| 75 |
+
βββ kitti/
|
| 76 |
+
β βββ flex_n/
|
| 77 |
+
β β βββ encoder.pth # YOLO11-based encoder weights
|
| 78 |
+
β β βββ depth.pth # Scale-Driven Decoder weights
|
| 79 |
+
β βββ flex_s/
|
| 80 |
+
β βββ flex_m/
|
| 81 |
+
β βββ flex_l/
|
| 82 |
+
β βββ flex_x/
|
| 83 |
+
βββ cs/
|
| 84 |
+
βββ flex_n/
|
| 85 |
+
βββ flex_s/
|
| 86 |
+
βββ flex_m/
|
| 87 |
+
βββ flex_l/
|
| 88 |
+
βββ flex_x/
|
| 89 |
+
```
|
| 90 |
+
|
| 91 |
+
## Usage
|
| 92 |
+
|
| 93 |
+
### Installation
|
| 94 |
+
|
| 95 |
+
```bash
|
| 96 |
+
conda create -n flexdepth python=3.10
|
| 97 |
+
conda activate flexdepth
|
| 98 |
+
|
| 99 |
+
pip install torch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cu118
|
| 100 |
+
pip install -r requirements.txt
|
| 101 |
+
```
|
| 102 |
+
|
| 103 |
+
### Download Weights
|
| 104 |
+
|
| 105 |
+
```python
|
| 106 |
+
from huggingface_hub import snapshot_download
|
| 107 |
+
|
| 108 |
+
# Download all models
|
| 109 |
+
snapshot_download(repo_id="StarNew/flexdepth", local_dir="./models")
|
| 110 |
+
|
| 111 |
+
# Or download a specific model
|
| 112 |
+
from huggingface_hub import hf_hub_download
|
| 113 |
+
|
| 114 |
+
hf_hub_download(
|
| 115 |
+
repo_id="StarNew/flexdepth",
|
| 116 |
+
filename="kitti/flex_n/encoder.pth",
|
| 117 |
+
local_dir="./models"
|
| 118 |
+
)
|
| 119 |
+
hf_hub_download(
|
| 120 |
+
repo_id="StarNew/flexdepth",
|
| 121 |
+
filename="kitti/flex_n/depth.pth",
|
| 122 |
+
local_dir="./models"
|
| 123 |
+
)
|
| 124 |
+
```
|
| 125 |
+
|
| 126 |
+
### Evaluation
|
| 127 |
+
|
| 128 |
+
```bash
|
| 129 |
+
# Flex-Nano on KITTI
|
| 130 |
+
python evaluate_depth.py --png --eval_mono --scale 4 \
|
| 131 |
+
--encoder_model_type yolo11n-seg --decoder_model_type flexn \
|
| 132 |
+
--load_weights_folder ./models/kitti/flex_n \
|
| 133 |
+
--data_path <kitti_data_path> --split_path <splits_path>
|
| 134 |
+
|
| 135 |
+
# Flex-X-Large on KITTI
|
| 136 |
+
python evaluate_depth.py --png --eval_mono --scale 4 \
|
| 137 |
+
--encoder_model_type yolo11x-seg --decoder_model_type flexx \
|
| 138 |
+
--load_weights_folder ./models/kitti/flex_x \
|
| 139 |
+
--data_path <kitti_data_path> --split_path <splits_path>
|
| 140 |
+
```
|
| 141 |
+
|
| 142 |
+
### ONNX Export
|
| 143 |
+
|
| 144 |
+
```bash
|
| 145 |
+
python export_onnx.py --encoder_model_type yolo11n-seg --decoder_model_type flexn \
|
| 146 |
+
--load_weights_folder ./models/kitti/flex_n --scales 4 --export_name flex-n
|
| 147 |
+
```
|
| 148 |
+
|
| 149 |
+
## Comparison with Depth Anything V2
|
| 150 |
+
|
| 151 |
+
On the KITTI Eigen benchmark with dense ground truth and least-squares alignment:
|
| 152 |
+
|
| 153 |
+
| Method | Type | Params | GFLOPs | Resolution | Abs Rel β | Ξ΄<1.25 β |
|
| 154 |
+
|--------|------|--------|--------|------------|-----------|----------|
|
| 155 |
+
| DA2 (ViT-L) | Zero-Shot | 335M | 1947 | 1722Γ518 | 0.070 | **0.956** |
|
| 156 |
+
| DA2 (ViT-S) | Zero-Shot | 25M | 137 | 1722Γ518 | 0.077 | 0.944 |
|
| 157 |
+
| DA2 (ViT-L) | Zero-Shot | 335M | 276 | 644Γ196 | 0.092 | 0.915 |
|
| 158 |
+
| DA2 (ViT-S) | Zero-Shot | 25M | 19 | 644Γ196 | 0.110 | 0.881 |
|
| 159 |
+
| **Flex-X-Large** | Self-Supervised | 32M | 25 | 640Γ192 | **0.063** | 0.952 |
|
| 160 |
+
|
| 161 |
+
FlexDepth achieves comparable or better accuracy than Depth Anything V2 with **~13Γ fewer parameters** and **~78Γ fewer GFLOPs** at similar resolution.
|
| 162 |
+
|
| 163 |
+
## Citation
|
| 164 |
+
|
| 165 |
+
```bibtex
|
| 166 |
+
@misc{zhu2026robustdrivingperceptionflexible,
|
| 167 |
+
title={Towards Robust Driving Perception: A Flexible Scale-Driven Family for Self-Supervised Monocular Depth Estimation},
|
| 168 |
+
author={Zhaowen Zhu and Li Zhang and Yujie Chen and Tian Zhang and Yingjie Wang and Mingxia Zhan},
|
| 169 |
+
year={2026},
|
| 170 |
+
eprint={2607.00736},
|
| 171 |
+
archivePrefix={arXiv},
|
| 172 |
+
primaryClass={cs.CV},
|
| 173 |
+
url={https://arxiv.org/abs/2607.00736}
|
| 174 |
+
}
|
| 175 |
+
```
|
| 176 |
+
|
| 177 |
+
## Acknowledgment
|
| 178 |
+
|
| 179 |
+
This work is supported by the National Natural Science Foundation of China under Grant 62332016.
|
| 180 |
+
|
| 181 |
+
Our code is built upon [Monodepth2](https://github.com/nianticlabs/monodepth2), [Manydepth](https://github.com/nianticlabs/manydepth), and [Ultralytics](https://github.com/ultralytics/ultralytics).
|