|
Download README.md from AutowareFoundation/scene_3d: direct link, hf CLI and curl.
- Browser
- Download file 6.2 kB
-
https://huggingface.co/AutowareFoundation/scene_3d/resolve/main/README.md
- Command line
-
hf download hf://AutowareFoundation/scene_3d/README.md
-
curl -L -o README.md https://huggingface.co/AutowareFoundation/scene_3d/resolve/main/README.md
6.2 kB
| license: apache-2.0 | |
| library_name: pytorch | |
| pipeline_tag: depth-estimation | |
| base_model: | |
| - google/efficientnet-b0 | |
| - google/efficientnet-b1 | |
| tags: | |
| - autonomous-driving | |
| - adas | |
| - monocular-depth-estimation | |
| - depth-estimation | |
| - efficientnet | |
| - deeplabv3plus | |
| - onnx | |
| - autoware | |
| - visionpilot | |
| # Scene3D - Monocular Depth Estimation | |
| <video src="https://huggingface.co/AutowareFoundation/scene_3d/resolve/main/assets/scene3d_demo_1.mp4" | |
| poster="https://huggingface.co/AutowareFoundation/scene_3d/resolve/main/assets/thumbnail.jpg" | |
| controls muted loop playsinline width="100%"></video> | |
| Depth estimation is an essential technology for safe operation of self-driving cars, especially in challenging long-tail edge case scenarios. By sensing depth from a single front-view monocular camera, autonomous vehicles can detect obstacles in the scene irrespective of their object class, shape, size, or appearance. **Scene3D** processes monocular camera images to produce high-resolution per-pixel relative depth maps with sharp object boundaries — visible on foliage, thin structures such as poles, and foreground obstacles — enabling real-time scene geometry understanding on edge hardware. | |
| Scene3D is distilled from **DepthAnythingV2 ViT-Large** pseudo-labels onto lightweight convolutional architectures specifically optimized for real-time on-road autonomous driving: | |
| - **Scene3D**: Encoder-Context-Decoder CNN combining a frozen road-pretrained `EfficientNet-B0` backbone (`SceneSegNetwork`), a `DepthContext` residual attention module, a progressive upsampling decoder neck (`Scene3DNeck`), and a dense depth regression head (`Scene3DHead`). | |
| - **Scene3DLite**: Compact variant combining an `EfficientNet-B1` encoder, a `DeepLabV3+` decoder, and a lightweight regression head, reducing compute by ~29× (`7.78` vs. `224` GOPs) while preserving edge sharpness, relative depth ordering, and scale consistency. | |
| ### Performance Results | |
| Scene3D was trained on **488,535** street-scene images (**24,426** held out for validation) curated across BDD100K, Mapillary Vistas, ROADWork, Comma10K, KITTI, DDAD, Driving Stereo, Indian Driving Dataset, Zenesact Open Dataset, and Mapillary Planet Scale Dataset, optimizing a weighted sum of a **Scale Invariant robust MAE Loss** and a **Multi-Scale Edge Preservation Loss**. | |
| #### Validation Set Performance (Scene3D, 640×320) | |
| | Total Loss | Scale Invariant Loss | Edge Preservation Loss | | |
| |------------|----------------------|------------------------| | |
| | **0.08408** | 0.07004 | 0.01404 | | |
| #### Embedded Inference Performance — Jetson Orin Nano (8GB) | |
| *Forward pass includes Host→Device (H2D) + Inference + Device→Host (D2H) at `640×320` input resolution.* | |
| | Model | GOPs | Backend | Forward [ms] | FPS | | |
| |-------|------|---------|--------------|-----| | |
| | Scene3D | 224 | FP32 ONNX | 168.4 | 5.9 | | |
| | Scene3D | 224 | FP32 TensorRT | 99.7 | 10.0 | | |
| | Scene3DLite | 7.78 | FP32 ONNX | 41.0 | 24.4 | | |
| | Scene3DLite | 7.78 | FP32 TensorRT | 23.4 | 42.7 | | |
| | Scene3DLite | 7.78 | INT8 TensorRT | 10.9 | 91.4 | | |
| On a laptop GPU (**RTX 3060 Mobile** / AMD Ryzen 7 5800H, 238.41 GFLOPs), Scene3D achieves **15.1 FPS** in FP32 and **23.9 FPS** in FP16. | |
| ## Model variants | |
| The repository provides both the original **Scene3D** network and the lightweight **Scene3DLite** variant. We recommend **Scene3DLite** for embedded/real-time deployment and **Scene3D** for legacy VisionPilot 0.9 compatibility. | |
| | Variant | Input frame | PyTorch | Traced PyTorch | ONNX FP32 | | |
| |---|---|---|---|---| | |
| | **Scene3DLite** | 640x320 (2:1)\* | [model weights](models/scene3d_lite.pth) | | [model weights](models/scene3d_lite_fp32.onnx) | | |
| | **Scene3D** | 640x320 (2:1) | [model weights](models/scene3d.pth) | [model weights](models/scene3d_traced.pt) | [model weights](models/scene3d_fp32.onnx) | | |
| *\*The `Scene3DLite` PyTorch architecture supports dynamic input resolutions (e.g., `640×320` or `1024×512`); the pre-exported `scene3d_lite_fp32.onnx` file uses `[1, 3, 320, 640]`.* | |
| ## Quick start | |
| ### PyTorch (`Scene3D`) | |
| Clone the [Scene3D GitHub repository](https://github.com/autowarefoundation/scene_3d), install dependencies (`pip install -r requirements.txt huggingface_hub`), and run: | |
| ```python | |
| import sys | |
| from PIL import Image | |
| from huggingface_hub import hf_hub_download | |
| sys.path.append("Models") | |
| from Models.inference.scene_3d_infer import Scene3DNetworkInfer | |
| repo_id = "AutowareFoundation/scene_3d" | |
| ckpt = hf_hub_download(repo_id, "models/scene3d.pth") | |
| model = Scene3DNetworkInfer(ckpt) # uses CUDA if available | |
| test_img = hf_hub_download(repo_id, "assets/highway.jpg") | |
| img = Image.open(test_img).convert("RGB").resize((640, 320)) | |
| depth_map = model.inference(img) # shape: (320, 640, 1) | |
| print(depth_map.shape, depth_map.min(), depth_map.max()) | |
| ``` | |
| ### ONNX Runtime (`Scene3DLite` / `Scene3D`, standalone) | |
| ```python | |
| import numpy as np | |
| import onnxruntime as ort | |
| from PIL import Image | |
| from huggingface_hub import hf_hub_download | |
| repo_id = "AutowareFoundation/scene_3d" | |
| onnx_path = hf_hub_download(repo_id, "models/scene3d_lite_fp32.onnx") | |
| test_img = hf_hub_download(repo_id, "assets/highway.jpg") | |
| img = Image.open(test_img).convert("RGB").resize((640, 320)) | |
| x = np.asarray(img, dtype=np.float32) / 255.0 | |
| mean = np.array([0.485, 0.456, 0.406], dtype=np.float32) | |
| std = np.array([0.229, 0.224, 0.225], dtype=np.float32) | |
| x = ((x - mean) / std).transpose(2, 0, 1)[None, ...] | |
| session = ort.InferenceSession(onnx_path, providers=["CUDAExecutionProvider", "CPUExecutionProvider"]) | |
| depth_map = session.run(None, {"input": x})[0] # shape: (1, 1, 320, 640) | |
| print(depth_map.shape, depth_map.min(), depth_map.max()) | |
| ``` | |
| <video src="https://huggingface.co/AutowareFoundation/scene_3d/resolve/main/assets/scene3d_demo_2.mp4" | |
| controls muted loop playsinline width="100%"></video> | |
| ## Citation | |
| ```bibtex | |
| @software{autoware_scene_3d_2026, | |
| author = {The Autoware Foundation}, | |
| title = {Scene3D: Monocular relative depth estimation model}, | |
| year = {2026}, | |
| url = {https://github.com/autowarefoundation/scene_3d}, | |
| note = {Checkpoints: https://huggingface.co/AutowareFoundation/scene_3d} | |
| } | |
| ``` | |