scene_3d / README.md
TranHuuNhatHuy's picture
init upload of Scene3D and Scene3DLite checkpoints, demo assets, plus model card
3d8b854 verified
|
Raw History Blame Contribute Delete
6.2 kB
---
license: apache-2.0
library_name: pytorch
pipeline_tag: depth-estimation
base_model:
- google/efficientnet-b0
- google/efficientnet-b1
tags:
- autonomous-driving
- adas
- monocular-depth-estimation
- depth-estimation
- efficientnet
- deeplabv3plus
- onnx
- autoware
- visionpilot
---
# Scene3D - Monocular Depth Estimation
<video src="https://huggingface.co/AutowareFoundation/scene_3d/resolve/main/assets/scene3d_demo_1.mp4"
poster="https://huggingface.co/AutowareFoundation/scene_3d/resolve/main/assets/thumbnail.jpg"
controls muted loop playsinline width="100%"></video>
Depth estimation is an essential technology for safe operation of self-driving cars, especially in challenging long-tail edge case scenarios. By sensing depth from a single front-view monocular camera, autonomous vehicles can detect obstacles in the scene irrespective of their object class, shape, size, or appearance. **Scene3D** processes monocular camera images to produce high-resolution per-pixel relative depth maps with sharp object boundaries — visible on foliage, thin structures such as poles, and foreground obstacles — enabling real-time scene geometry understanding on edge hardware.
Scene3D is distilled from **DepthAnythingV2 ViT-Large** pseudo-labels onto lightweight convolutional architectures specifically optimized for real-time on-road autonomous driving:
- **Scene3D**: Encoder-Context-Decoder CNN combining a frozen road-pretrained `EfficientNet-B0` backbone (`SceneSegNetwork`), a `DepthContext` residual attention module, a progressive upsampling decoder neck (`Scene3DNeck`), and a dense depth regression head (`Scene3DHead`).
- **Scene3DLite**: Compact variant combining an `EfficientNet-B1` encoder, a `DeepLabV3+` decoder, and a lightweight regression head, reducing compute by ~29× (`7.78` vs. `224` GOPs) while preserving edge sharpness, relative depth ordering, and scale consistency.
### Performance Results
Scene3D was trained on **488,535** street-scene images (**24,426** held out for validation) curated across BDD100K, Mapillary Vistas, ROADWork, Comma10K, KITTI, DDAD, Driving Stereo, Indian Driving Dataset, Zenesact Open Dataset, and Mapillary Planet Scale Dataset, optimizing a weighted sum of a **Scale Invariant robust MAE Loss** and a **Multi-Scale Edge Preservation Loss**.
#### Validation Set Performance (Scene3D, 640×320)
| Total Loss | Scale Invariant Loss | Edge Preservation Loss |
|------------|----------------------|------------------------|
| **0.08408** | 0.07004 | 0.01404 |
#### Embedded Inference Performance — Jetson Orin Nano (8GB)
*Forward pass includes Host→Device (H2D) + Inference + Device→Host (D2H) at `640×320` input resolution.*
| Model | GOPs | Backend | Forward [ms] | FPS |
|-------|------|---------|--------------|-----|
| Scene3D | 224 | FP32 ONNX | 168.4 | 5.9 |
| Scene3D | 224 | FP32 TensorRT | 99.7 | 10.0 |
| Scene3DLite | 7.78 | FP32 ONNX | 41.0 | 24.4 |
| Scene3DLite | 7.78 | FP32 TensorRT | 23.4 | 42.7 |
| Scene3DLite | 7.78 | INT8 TensorRT | 10.9 | 91.4 |
On a laptop GPU (**RTX 3060 Mobile** / AMD Ryzen 7 5800H, 238.41 GFLOPs), Scene3D achieves **15.1 FPS** in FP32 and **23.9 FPS** in FP16.
## Model variants
The repository provides both the original **Scene3D** network and the lightweight **Scene3DLite** variant. We recommend **Scene3DLite** for embedded/real-time deployment and **Scene3D** for legacy VisionPilot 0.9 compatibility.
| Variant | Input frame | PyTorch | Traced PyTorch | ONNX FP32 |
|---|---|---|---|---|
| **Scene3DLite** | 640x320 (2:1)\* | [model weights](models/scene3d_lite.pth) | | [model weights](models/scene3d_lite_fp32.onnx) |
| **Scene3D** | 640x320 (2:1) | [model weights](models/scene3d.pth) | [model weights](models/scene3d_traced.pt) | [model weights](models/scene3d_fp32.onnx) |
*\*The `Scene3DLite` PyTorch architecture supports dynamic input resolutions (e.g., `640×320` or `1024×512`); the pre-exported `scene3d_lite_fp32.onnx` file uses `[1, 3, 320, 640]`.*
## Quick start
### PyTorch (`Scene3D`)
Clone the [Scene3D GitHub repository](https://github.com/autowarefoundation/scene_3d), install dependencies (`pip install -r requirements.txt huggingface_hub`), and run:
```python
import sys
from PIL import Image
from huggingface_hub import hf_hub_download
sys.path.append("Models")
from Models.inference.scene_3d_infer import Scene3DNetworkInfer
repo_id = "AutowareFoundation/scene_3d"
ckpt = hf_hub_download(repo_id, "models/scene3d.pth")
model = Scene3DNetworkInfer(ckpt) # uses CUDA if available
test_img = hf_hub_download(repo_id, "assets/highway.jpg")
img = Image.open(test_img).convert("RGB").resize((640, 320))
depth_map = model.inference(img) # shape: (320, 640, 1)
print(depth_map.shape, depth_map.min(), depth_map.max())
```
### ONNX Runtime (`Scene3DLite` / `Scene3D`, standalone)
```python
import numpy as np
import onnxruntime as ort
from PIL import Image
from huggingface_hub import hf_hub_download
repo_id = "AutowareFoundation/scene_3d"
onnx_path = hf_hub_download(repo_id, "models/scene3d_lite_fp32.onnx")
test_img = hf_hub_download(repo_id, "assets/highway.jpg")
img = Image.open(test_img).convert("RGB").resize((640, 320))
x = np.asarray(img, dtype=np.float32) / 255.0
mean = np.array([0.485, 0.456, 0.406], dtype=np.float32)
std = np.array([0.229, 0.224, 0.225], dtype=np.float32)
x = ((x - mean) / std).transpose(2, 0, 1)[None, ...]
session = ort.InferenceSession(onnx_path, providers=["CUDAExecutionProvider", "CPUExecutionProvider"])
depth_map = session.run(None, {"input": x})[0] # shape: (1, 1, 320, 640)
print(depth_map.shape, depth_map.min(), depth_map.max())
```
<video src="https://huggingface.co/AutowareFoundation/scene_3d/resolve/main/assets/scene3d_demo_2.mp4"
controls muted loop playsinline width="100%"></video>
## Citation
```bibtex
@software{autoware_scene_3d_2026,
author = {The Autoware Foundation},
title = {Scene3D: Monocular relative depth estimation model},
year = {2026},
url = {https://github.com/autowarefoundation/scene_3d},
note = {Checkpoints: https://huggingface.co/AutowareFoundation/scene_3d}
}
```