2.87 TB
3,219 files
Updated about 2 months ago
Name
Size
sample_set
.gitattributes366 kB
xet
.gitignore10 Bytes
xet
LICENSE.pdf89.6 kB
xet
README.md28.6 kB
xet
clip_ratings_26.04.csv88.4 kB
xet
README.md

task_categories: - robotics tags: - physicalAI

Find the 1500+ scenes in the sample_set/26.04_release folder.

Dataset Description:

Neural reconstructed dataset that carries 3D reconstructed driving scenes. The scenes are about 20 second long and stored in form of usdz files, along with respective xodr map files, surface mesh. The reconstructions were generated using 6 camera views (front-wide 120 deg, front-tele 30 deg, cross right/left 120 deg and rear right/left 70 deg). Users can use these 3D reconstructed driving scenes for training and testing their autonomous vehicle (AV) systems. This dataset is ready for commercial/non-commercial AV only use.

Dataset Owner(s):

NVIDIA Corporation

Dataset Creation Date:

06/09/2025

Version History

Version Notes
26.04
  • NuRec scenes have been constructed with NuRec 26.04 version.
  • Clips from Physical AI Dataset, count 1607
26.02
  • NuRec scenes have been constructed with NuRec 26.02 version.
  • Clips from Physical AI Dataset
26.01
  • NuRec scenes have been constructed with NuRec 26.01 version.
  • Reduced processed count to 918
25.05
  • NuRec scenes have been constructed with NuRec 25.05
  • Added labels.json file for helping users who want to search by types of scenes based on: behavior, layout, lighting, road types, surface conditions, traffic density, vrus presence, and weather. (Note this is only available for files under Batch0002 and onwards)
  • A front camera video file for each clips so that users can assess the scene before opening the usdz. (Note this is only available for files under Batch0002 and onwards)

License/Terms of Use:

NVIDIA Autonomous Vehicle Dataset License Agreement

Intended Usage:

This dataset is intended to provide AV developers with the experience of NuRec capability and try out 3DGUT. Users can use this dataset to run a set of tests / experiments for an AV system and train AI models that use camera and reconstructed data. The scenes in this dataset are generated by and can be rendered by using NVIDIA NuRec. CARLA users can also utilize this dataset by which leveraging the NVIDIA NuRec integration in CARLA.

Dataset Characterization

Data Collection Method

  • [Automatic/Sensors] - [Machine-derived]

Labeling Method

  • [Automatic/Sensors] - [Machine-derived]

Dataset

clip_ratings_XX.XX.csv is now included from version 26.02 onward. This spreadsheet contains the following:

Column Description
clip_id The ID of the clip
quality_score A perception based quality score (beta)
time_of_day day, night, evening
ego_speed High, medium, slow

These scores are meant to simplify navigating the datasets and highlight the range of outcomes a user can expect when reconstructing AV scenes. Known challenges include night, fog, high ego speed, pedestrian legs, and certain trailer types.

Dataset Format

The scenes are stored by batches containing a number of clip folders listed by their UUIDs. Each UUID folder contains:

  • usdz file (always)
  • camera_front_wide_120fov.mp4 (in most cases)

Each reconstructed scene is stored as a USDZ file containing the following:

Files Description
checkpoint.ckpt Trained neural network weights
clipgt/association.parquet ClipGT association data
clipgt/buffer_zone.parquet ClipGT buffer zone data
clipgt/calibration_estimate.parquet ClipGT calibration estimate data
clipgt/clip.parquet ClipGT clip metadata
clipgt/crosswalk.parquet ClipGT crosswalk data
clipgt/drivable_space.parquet ClipGT drivable space data
clipgt/egomotion_estimate.parquet ClipGT egomotion estimate data
clipgt/gore_area.parquet ClipGT gore area data
clipgt/intersection_area.parquet ClipGT intersection area data
clipgt/lane.parquet ClipGT lane data
clipgt/lane_line.parquet ClipGT lane line data
clipgt/mads_trace.parquet ClipGT MADS trace data
clipgt/obstacle.parquet ClipGT obstacle data
clipgt/pole.parquet ClipGT pole data
clipgt/road_boundary.parquet ClipGT road boundary data
clipgt/road_island.parquet ClipGT road island data
clipgt/road_marking.parquet ClipGT road marking data
clipgt/traffic_light.parquet ClipGT traffic light data
clipgt/traffic_sign.parquet ClipGT traffic sign data
clipgt/wait_line.parquet ClipGT wait line data
data_info.json Timestamp and frame range detail per sensors
datasource_summary.json Sensor track and poses summary
default.usda Main scene file referencing all assets and configurations
frames/camera_cross_left_120fov/<timestamp>.jpeg First frame image for the cross-left camera
frames/camera_cross_right_120fov/<timestamp>.jpeg First frame image for the cross-right camera
frames/camera_front_tele_30fov/<timestamp>.jpeg First frame image for the front tele camera
frames/camera_front_wide_120fov/<timestamp>.jpeg First frame image for the front wide camera
map.pb Protobuf map file, present in a small subset of scanned USDZs
map.xodr OpenDRIVE map file
mesh.ply Polygon mesh file for 3D geometry
mesh.usd USD file for 3D mesh
mesh_ground.ply Polygon mesh file for ground surface geometry
mesh_ground.usd USD file for ground mesh
metadata.yaml YAML file with scene metadata
parsed_config.yaml YAML configuration file
pose_record.json JSON file containing pose record data
rig_trajectories.json JSON file containing sensor rig trajectory data
rig_trajectories.usda Rig trajectories in the USD scene
sequence_tracks.json JSON file with object tracking information
sequence_tracks.usda Object sequence tracks in the USD scene
volume.nurec Volumetric data file for neural reconstruction
volume.usda USD ASCII file describing volumetric data in the scene

Dataset Quantification

Record Count: 1500+ usdz files

Measurement of Total Data Storage: approx 2.9 TB

Downloading Data

Please see https://huggingface.co/docs/huggingface_hub/v1.0.0.rc5/en/guides/download for complete documentation on how download dataset files. The code below is just an example only.

from huggingface_hub import login, snapshot_download

def main():
    hf_api_token = os.getenv("HF_TOKEN")

    login(token=hf_api_token)

    # Download an entire repository
    snapshot_download(repo_id="nvidia/PhysicalAI-Autonomous-Vehicles-NuRec", repo_type="dataset")

    # Download all the files in a folder
    snapshot_download(repo_id="nvidia/PhysicalAI-Autonomous-Vehicles-NuRec", repo_type="dataset", allow_patterns="sample_set/25.07_release/Batch0002/001b28cb-b8f7-4627-ae65-fda88612d5bf/*")

    # Download an individual file
    snapshot_download(repo_id="nvidia/PhysicalAI-Autonomous-Vehicles-NuRec", repo_type="dataset", allow_patterns="sample_set/25.07_release/Batch0002/001b28cb-b8f7-4627-ae65-fda88612d5bf/001b28cb-b8f7-4627-ae65-fda88612d5bf.usdz")

if __name__ == "__main__":
    main()

Downloading usdz based upon categories in labels.json

import argparse
from pathlib import Path
import json
import os

from huggingface_hub import login, snapshot_download


def string_to_boolean(s):
    s = s.strip().lower()  # Normalize the string
    if s in ('true', '1', 'yes', 'on'):
        return True

    return False

def main():
    valid_categories = ["behavior", "layout", "lighting", "road_types", "surface_conditions", "traffic_density", "vrus", "weather"]

    parser = argparse.ArgumentParser(
        description="Downloads usdz clips based upon criteria specified in the labels.json"
    )
    parser.add_argument(
        "--local-dir", type=str, required=True, help="The path to store the usdz"
    )
    parser.add_argument(
        "--category",
        type=str,
        required=True,
        choices=valid_categories,
        help="The specified category in the labels.json. Must be one of: %(choices)",
    )
    parser.add_argument(
        "--value",
        type=str,
        required=True,
        help="The specified value in the category",
    )

    args = parser.parse_args()

    hf_api_token = os.getenv("HF_TOKEN")

    login(token=hf_api_token)

    # First download all the labels.json files
    print(f"Downloading dataset labels.json to {args.local_dir}.")

    snapshot_download(repo_id="nvidia/PhysicalAI-Autonomous-Vehicles-NuRec", repo_type="dataset", allow_patterns="*.json", local_dir=args.local_dir)

    category = args.category
    value = args.value

    # Find all of the labels.json files that have been downloaded
    local_dir = Path(args.local_dir)
    label_paths = local_dir.rglob("labels.json")

    # Filter through the labels.json and find all usdz that match our criteria
    paths_to_download = {}

    print(f"Filtering usdz downloads based upon labels.json downloaded with criteria {category} and {value}.")

    for label_path in label_paths:
        with open(label_path, "r", encoding="utf-8") as f:
            metadata = json.load(f)

            if category in metadata:
                if category == "vrus":
                    if string_to_boolean(value) == metadata["vrus"]:
                        paths_to_download[label_path.parent] = True
                else:
                    if value in metadata[args.category]:
                        paths_to_download[label_path.parent] = True

    print(f"Found {len(paths_to_download)} that matched criteria.")

    # Download the selected usdz and front camera mp4
    for path in paths_to_download.keys():
        relative_path = path.relative_to(local_dir)

        print(f"Downloading usdz and front camera at path {relative_path}")
        snapshot_download(repo_id="nvidia/PhysicalAI-Autonomous-Vehicles-NuRec", repo_type="dataset", allow_patterns=f"{relative_path}/*", local_dir=args.local_dir)


if __name__ == "__main__":
    main()

Reference(s):

@article{wu20253dgut, title={3DGUT: Enabling Distorted Cameras and Secondary Rays in Gaussian Splatting}, author={Wu, Qi and Martinez Esturo, Janick and Mirzaei, Ashkan and Moenne-Loccoz, Nicolas and Gojcic, Zan}, journal={Conference on Computer Vision and Pattern Recognition (CVPR)}, year={2025} }

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please report security vulnerabilities or NVIDIA AI Concerns here.

Total size
2.87 TB
Files
3,219
Last updated
Sep 26
Pre-warmed CDN
US EU US EU

Contributors