--- license: apache-2.0 library_name: pytorch pipeline_tag: depth-estimation base_model: - google/efficientnet-b0 - google/efficientnet-b1 tags: - autonomous-driving - adas - monocular-depth-estimation - depth-estimation - efficientnet - deeplabv3plus - onnx - autoware - visionpilot --- # Scene3D - Monocular Depth Estimation Depth estimation is an essential technology for safe operation of self-driving cars, especially in challenging long-tail edge case scenarios. By sensing depth from a single front-view monocular camera, autonomous vehicles can detect obstacles in the scene irrespective of their object class, shape, size, or appearance. **Scene3D** processes monocular camera images to produce high-resolution per-pixel relative depth maps with sharp object boundaries — visible on foliage, thin structures such as poles, and foreground obstacles — enabling real-time scene geometry understanding on edge hardware. Scene3D is distilled from **DepthAnythingV2 ViT-Large** pseudo-labels onto lightweight convolutional architectures specifically optimized for real-time on-road autonomous driving: - **Scene3D**: Encoder-Context-Decoder CNN combining a frozen road-pretrained `EfficientNet-B0` backbone (`SceneSegNetwork`), a `DepthContext` residual attention module, a progressive upsampling decoder neck (`Scene3DNeck`), and a dense depth regression head (`Scene3DHead`). - **Scene3DLite**: Compact variant combining an `EfficientNet-B1` encoder, a `DeepLabV3+` decoder, and a lightweight regression head, reducing compute by ~29× (`7.78` vs. `224` GOPs) while preserving edge sharpness, relative depth ordering, and scale consistency. ### Performance Results Scene3D was trained on **488,535** street-scene images (**24,426** held out for validation) curated across BDD100K, Mapillary Vistas, ROADWork, Comma10K, KITTI, DDAD, Driving Stereo, Indian Driving Dataset, Zenesact Open Dataset, and Mapillary Planet Scale Dataset, optimizing a weighted sum of a **Scale Invariant robust MAE Loss** and a **Multi-Scale Edge Preservation Loss**. #### Validation Set Performance (Scene3D, 640×320) | Total Loss | Scale Invariant Loss | Edge Preservation Loss | |------------|----------------------|------------------------| | **0.08408** | 0.07004 | 0.01404 | #### Embedded Inference Performance — Jetson Orin Nano (8GB) *Forward pass includes Host→Device (H2D) + Inference + Device→Host (D2H) at `640×320` input resolution.* | Model | GOPs | Backend | Forward [ms] | FPS | |-------|------|---------|--------------|-----| | Scene3D | 224 | FP32 ONNX | 168.4 | 5.9 | | Scene3D | 224 | FP32 TensorRT | 99.7 | 10.0 | | Scene3DLite | 7.78 | FP32 ONNX | 41.0 | 24.4 | | Scene3DLite | 7.78 | FP32 TensorRT | 23.4 | 42.7 | | Scene3DLite | 7.78 | INT8 TensorRT | 10.9 | 91.4 | On a laptop GPU (**RTX 3060 Mobile** / AMD Ryzen 7 5800H, 238.41 GFLOPs), Scene3D achieves **15.1 FPS** in FP32 and **23.9 FPS** in FP16. ## Model variants The repository provides both the original **Scene3D** network and the lightweight **Scene3DLite** variant. We recommend **Scene3DLite** for embedded/real-time deployment and **Scene3D** for legacy VisionPilot 0.9 compatibility. | Variant | Input frame | PyTorch | Traced PyTorch | ONNX FP32 | |---|---|---|---|---| | **Scene3DLite** | 640x320 (2:1)\* | [model weights](models/scene3d_lite.pth) | | [model weights](models/scene3d_lite_fp32.onnx) | | **Scene3D** | 640x320 (2:1) | [model weights](models/scene3d.pth) | [model weights](models/scene3d_traced.pt) | [model weights](models/scene3d_fp32.onnx) | *\*The `Scene3DLite` PyTorch architecture supports dynamic input resolutions (e.g., `640×320` or `1024×512`); the pre-exported `scene3d_lite_fp32.onnx` file uses `[1, 3, 320, 640]`.* ## Quick start ### PyTorch (`Scene3D`) Clone the [Scene3D GitHub repository](https://github.com/autowarefoundation/scene_3d), install dependencies (`pip install -r requirements.txt huggingface_hub`), and run: ```python import sys from PIL import Image from huggingface_hub import hf_hub_download sys.path.append("Models") from Models.inference.scene_3d_infer import Scene3DNetworkInfer repo_id = "AutowareFoundation/scene_3d" ckpt = hf_hub_download(repo_id, "models/scene3d.pth") model = Scene3DNetworkInfer(ckpt) # uses CUDA if available test_img = hf_hub_download(repo_id, "assets/highway.jpg") img = Image.open(test_img).convert("RGB").resize((640, 320)) depth_map = model.inference(img) # shape: (320, 640, 1) print(depth_map.shape, depth_map.min(), depth_map.max()) ``` ### ONNX Runtime (`Scene3DLite` / `Scene3D`, standalone) ```python import numpy as np import onnxruntime as ort from PIL import Image from huggingface_hub import hf_hub_download repo_id = "AutowareFoundation/scene_3d" onnx_path = hf_hub_download(repo_id, "models/scene3d_lite_fp32.onnx") test_img = hf_hub_download(repo_id, "assets/highway.jpg") img = Image.open(test_img).convert("RGB").resize((640, 320)) x = np.asarray(img, dtype=np.float32) / 255.0 mean = np.array([0.485, 0.456, 0.406], dtype=np.float32) std = np.array([0.229, 0.224, 0.225], dtype=np.float32) x = ((x - mean) / std).transpose(2, 0, 1)[None, ...] session = ort.InferenceSession(onnx_path, providers=["CUDAExecutionProvider", "CPUExecutionProvider"]) depth_map = session.run(None, {"input": x})[0] # shape: (1, 1, 320, 640) print(depth_map.shape, depth_map.min(), depth_map.max()) ``` ## Citation ```bibtex @software{autoware_scene_3d_2026, author = {The Autoware Foundation}, title = {Scene3D: Monocular relative depth estimation model}, year = {2026}, url = {https://github.com/autowarefoundation/scene_3d}, note = {Checkpoints: https://huggingface.co/AutowareFoundation/scene_3d} } ```