Scene3D - Monocular Depth Estimation
Depth estimation is an essential technology for safe operation of self-driving cars, especially in challenging long-tail edge case scenarios. By sensing depth from a single front-view monocular camera, autonomous vehicles can detect obstacles in the scene irrespective of their object class, shape, size, or appearance. Scene3D processes monocular camera images to produce high-resolution per-pixel relative depth maps with sharp object boundaries — visible on foliage, thin structures such as poles, and foreground obstacles — enabling real-time scene geometry understanding on edge hardware.
Scene3D is distilled from DepthAnythingV2 ViT-Large pseudo-labels onto lightweight convolutional architectures specifically optimized for real-time on-road autonomous driving:
- Scene3D: Encoder-Context-Decoder CNN combining a frozen road-pretrained
EfficientNet-B0backbone (SceneSegNetwork), aDepthContextresidual attention module, a progressive upsampling decoder neck (Scene3DNeck), and a dense depth regression head (Scene3DHead). - Scene3DLite: Compact variant combining an
EfficientNet-B1encoder, aDeepLabV3+decoder, and a lightweight regression head, reducing compute by ~29× (7.78vs.224GOPs) while preserving edge sharpness, relative depth ordering, and scale consistency.
Performance Results
Scene3D was trained on 488,535 street-scene images (24,426 held out for validation) curated across BDD100K, Mapillary Vistas, ROADWork, Comma10K, KITTI, DDAD, Driving Stereo, Indian Driving Dataset, Zenesact Open Dataset, and Mapillary Planet Scale Dataset, optimizing a weighted sum of a Scale Invariant robust MAE Loss and a Multi-Scale Edge Preservation Loss.
Validation Set Performance (Scene3D, 640×320)
| Total Loss | Scale Invariant Loss | Edge Preservation Loss |
|---|---|---|
| 0.08408 | 0.07004 | 0.01404 |
Embedded Inference Performance — Jetson Orin Nano (8GB)
Forward pass includes Host→Device (H2D) + Inference + Device→Host (D2H) at 640×320 input resolution.
| Model | GOPs | Backend | Forward [ms] | FPS |
|---|---|---|---|---|
| Scene3D | 224 | FP32 ONNX | 168.4 | 5.9 |
| Scene3D | 224 | FP32 TensorRT | 99.7 | 10.0 |
| Scene3DLite | 7.78 | FP32 ONNX | 41.0 | 24.4 |
| Scene3DLite | 7.78 | FP32 TensorRT | 23.4 | 42.7 |
| Scene3DLite | 7.78 | INT8 TensorRT | 10.9 | 91.4 |
On a laptop GPU (RTX 3060 Mobile / AMD Ryzen 7 5800H, 238.41 GFLOPs), Scene3D achieves 15.1 FPS in FP32 and 23.9 FPS in FP16.
Model variants
The repository provides both the original Scene3D network and the lightweight Scene3DLite variant. We recommend Scene3DLite for embedded/real-time deployment and Scene3D for legacy VisionPilot 0.9 compatibility.
| Variant | Input frame | PyTorch | Traced PyTorch | ONNX FP32 |
|---|---|---|---|---|
| Scene3DLite | 640x320 (2:1)* | model weights | model weights | |
| Scene3D | 640x320 (2:1) | model weights | model weights | model weights |
*The Scene3DLite PyTorch architecture supports dynamic input resolutions (e.g., 640×320 or 1024×512); the pre-exported scene3d_lite_fp32.onnx file uses [1, 3, 320, 640].
Quick start
PyTorch (Scene3D)
Clone the Scene3D GitHub repository, install dependencies (pip install -r requirements.txt huggingface_hub), and run:
import sys
from PIL import Image
from huggingface_hub import hf_hub_download
sys.path.append("Models")
from Models.inference.scene_3d_infer import Scene3DNetworkInfer
repo_id = "AutowareFoundation/scene_3d"
ckpt = hf_hub_download(repo_id, "models/scene3d.pth")
model = Scene3DNetworkInfer(ckpt) # uses CUDA if available
test_img = hf_hub_download(repo_id, "assets/highway.jpg")
img = Image.open(test_img).convert("RGB").resize((640, 320))
depth_map = model.inference(img) # shape: (320, 640, 1)
print(depth_map.shape, depth_map.min(), depth_map.max())
ONNX Runtime (Scene3DLite / Scene3D, standalone)
import numpy as np
import onnxruntime as ort
from PIL import Image
from huggingface_hub import hf_hub_download
repo_id = "AutowareFoundation/scene_3d"
onnx_path = hf_hub_download(repo_id, "models/scene3d_lite_fp32.onnx")
test_img = hf_hub_download(repo_id, "assets/highway.jpg")
img = Image.open(test_img).convert("RGB").resize((640, 320))
x = np.asarray(img, dtype=np.float32) / 255.0
mean = np.array([0.485, 0.456, 0.406], dtype=np.float32)
std = np.array([0.229, 0.224, 0.225], dtype=np.float32)
x = ((x - mean) / std).transpose(2, 0, 1)[None, ...]
session = ort.InferenceSession(onnx_path, providers=["CUDAExecutionProvider", "CPUExecutionProvider"])
depth_map = session.run(None, {"input": x})[0] # shape: (1, 1, 320, 640)
print(depth_map.shape, depth_map.min(), depth_map.max())
Citation
@software{autoware_scene_3d_2026,
author = {The Autoware Foundation},
title = {Scene3D: Monocular relative depth estimation model},
year = {2026},
url = {https://github.com/autowarefoundation/scene_3d},
note = {Checkpoints: https://huggingface.co/AutowareFoundation/scene_3d}
}
Model tree for AutowareFoundation/scene_3d
Base model
google/efficientnet-b0