| --- |
| license: openmdw-1.1 |
| --- |
| |
| <h1 align="center">Into the LidarVerse:<br>A Real-Time Unified LiDAR World Model Across Tasks and Sensors</h1> |
|
|
| <p align="center"><strong> |
| Henry Che<sup>1,2</sup>   |
| Xuanchi Ren<sup>1</sup>   |
| Sanja Fidler<sup>1</sup>   |
| Shenlong Wang<sup>2</sup>   |
| Laura Leal-Taixé<sup>1</sup>   |
| Shengyu Huang<sup>1</sup> |
| </strong></p> |
| <p align="center"><sup>1</sup>NVIDIA <sup>2</sup>University of Illinois Urbana-Champaign</p> |
|
|
| <p align="center"> |
| <a href="https://research.nvidia.com/labs/sil/projects/lidarverse/"><img src="https://img.shields.io/badge/Project-Page-76b900" alt="Project Page"></a> |
| |
| <a href="https://arxiv.org/abs/2610.19201"><img src="https://img.shields.io/badge/arXiv-2610.19201-b31b1b" alt="arXiv"></a> |
| |
| <a href="https://huggingface.co/nvidia/lidarverse"><img src="https://img.shields.io/badge/Hugging_Face-Models-ffbd45" alt="Hugging Face Models"></a> |
| |
| <a href="https://huggingface.co/datasets/nvidia/Cosmos-Transfer-LidarGen-Example"><img src="https://img.shields.io/badge/Hugging_Face-Examples-ffbd45" alt="Hugging Face Examples"></a> |
| |
| <a href="https://github.com/nv-tlabs/LidarVerse#quickstart"><img src="https://img.shields.io/badge/Quickstart-Code-0071bc" alt="Quickstart"></a> |
| </p> |
|
|
| <video src="https://huggingface.co/nvidia/lidarverse/resolve/main/docs/media/teaser.mp4" autoplay loop muted playsinline controls width="100%"> |
| <a href="https://huggingface.co/nvidia/lidarverse/resolve/main/docs/media/teaser.mp4">Watch the LidarVerse teaser</a> |
| </video> |
|
|
| [Quickstart](https://github.com/nv-tlabs/LidarVerse#quickstart) | [Fine-tuning](https://github.com/nv-tlabs/LidarVerse/blob/main/docs/finetuning.md) | [Model card](https://github.com/nv-tlabs/LidarVerse/blob/main/model-cards/lidarverse.md) |
|
|
| ## Abstract |
|
|
| **LidarVerse** is a real-time unified LiDAR world model for generation, |
| point-cloud forecasting, and generalizable resimulation. It combines the |
| flexibility of generative modeling with reconstruction-grounded simulation |
| without per-scene optimization. An attention-based tokenizer maps different |
| LiDAR scanning patterns into a shared latent space, while flexible conditioning |
| on observed LiDAR, HDMaps, and accumulated geometry enables one model to perform |
| all three tasks. Self-forcing distillation produces a four-step causal model; |
| a rolling temporal K/V cache and a persistent spatial cache support long-horizon |
| generation at 11 FPS on a single H100 GPU. LidarVerse achieves state-of-the-art |
| performance on most reported metrics across the three evaluation protocols. |
| The framework also supports cross-sensor generation with intensity, and its |
| tokenizer generalizes to an unseen scanning pattern. |
|
|
|
|
| ## Stage 3 Quickstart |
|
|
| `lidarverse-sfdmd-pai/` contains the EMA-only SF-DMD student. Use the current |
| [LidarVerse code](https://github.com/nv-tlabs/LidarVerse) and run from its repository root: |
|
|
| ```bash |
| python scripts/download_checkpoints.py |
| CUDA_VISIBLE_DEVICES=0 STAGE=3 MODE=all NUM_SAMPLES=1 \ |
| NUM_STEPS=4 SAMPLER=sde SDE_ETA=1 GUIDANCE=1 SEED=0 \ |
| bash scripts/diffusion/getting_started.sh |
| ``` |
|
|
| The single-sample quickstart uses the second official NVIDIA PAI clip. |
| Sampling uses four-frame blocks, KV9 including the sink, FP32 tokenizer |
| encoding/decoding, and a persistent spatial cache with 0.10 m voxels. |
| HDMap generation rasterizes raw lanes and actor edges with calibrated rays |
| and rolling-shutter poses, then widens only bbox returns to 3x3 pixels before |
| encoding. Set `RAW_HDMAP_CONDITION_RASTERIZE=0` for the original cached |
| condition; forecasting/resimulation do not use this HDMap override. |
| Set `PREPARE_DATA=0` to reuse prepared examples. |
|
|
| The package contains no regular-model, fake-score, optimizer, or training state. |
| Fine-tuning from it starts a new run, not an exact full-state continuation. |
| Its teacher came from the same causal-training lineage as the released Stage 2, |
| but not from the exact Stage-2 snapshot packaged here. |
|
|
| # LidarVerse Overview |
|
|
| > **Current checkpoint scope:** the quickstarts use the range-only NVIDIA PAI |
| > tokenizer and PAI bidirectional, causal, and SF-DMD EMA |
| > packages, with 128x1024 inputs at 10 Hz and 2.5--100 m support. They do not |
| > include intensity or the paper's cross-sensor checkpoints. The broader |
| > architecture and dataset descriptions below describe the research project, |
| > not a claim that these particular packages support every sensor. Quickstart |
| > throughput is not a paper benchmark. |
|
|
| ## Description: |
| LidarVerse is a real-time unified LiDAR world model for sensor simulation across tasks and sensors. LidarVerse bridges both generative and reconstruction-based simulation into a single, unified conditional generative framework. It is composed of a universal, fully attention-based tokenizer maps LiDAR range sequences with different scanning patterns into a shared latent space; a bidirectional diffusion teacher learns a common interface for conditioning on a LiDAR prefix, an HD-Map, a persistent spatial cache, and optionally, text; and a four-step autoregressive causal student and maintain both a rolling temporal K/V cache and a world-aligned spatial cache, enabling persistent long-horizon roll-outs with real-time inference at 11 FPS on a single H100 GPU. |
| LidarVerse was developed by NVIDIA. |
| _This model is for research and development only._ |
|
|
|
|
| ### License/Terms of Use: |
| [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/) |
| ### Deployment Geography: |
| Global |
|
|
| ### Use Case: |
| This model is intended for research community to study, ablate, and build upon of. |
| |
|
|
| ### Release Date: |
| **GitHub:** 09/11/2026 via https://github.com/nv-tlabs/lidarverse |
|
|
| |
|
|
| ## Reference(s): |
| [Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models](https://research.nvidia.com/labs/toronto-ai/cosmos_drive_dreams/) |
|
|
| ## Model Architecture: |
| **Architecture Type:** Transformer |
| **Network Architecture:** Diffusion Transformer (DiT) |
| **This model was developed based on Cosmos-Predict2-T2V-2B.** |
| **Number of model parameters:** 2B |
|
|
| ## Input: |
| **Input Type(s):** Video |
| **Input Format(s):** Lidar |
| **Input Parameters:** Three-Dimensional (3D) |
| **Other Properties Related to Input:** Flexible resolution and length |
| |
|
|
| ## Output: |
| **Output Type(s):** Video |
| **Output Format:** Lidar |
| **Output Parameters:** Three-Dimensional (3D) |
| **Other Properties Related to Output:** Flexible resolution and length, streaming supported |
| |
|
|
| Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions. |
|
|
| ## Software Integration: |
| **Runtime Engine(s):** Not Applicable (N/A) |
| **Supported Hardware Microarchitecture Compatibility:** |
| * NVIDIA Ampere |
| * NVIDIA Blackwell |
| * NVIDIA Hopper |
| * NVIDIA Lovelace |
|
|
| **Supported Operating System(s):** Linux |
|
|
| The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment. |
|
|
|
|
| ## Model Version(s): |
| LidarVerse v1.0 — 2B-parameter trained research model |
|
|
| ## Training, Testing, and Evaluation Datasets: |
|
|
|
|
| ## Training Dataset: |
|
|
| **Data Modality:** |
| * Video |
| * Text |
| * Image |
|
|
|
|
| **Image Training Data Size:** 1 Million to 1 Billion Images |
| **Text Training Data Size:** Less than a Billion Tokens |
| **Video Training Data Size:** 10,000 to 1 Million Hours |
| **Data Collection Method by dataset:** Automatic/Sensors |
| **Labeling Method by dataset:** Hybrid: Automatic/Sensors/Manually-Labeled |
| **Properties:** 112,803 sensor-generated LiDAR range-sequence entries: NVIDIA PAI (111,988), PandaSet (93), and nuScenes (722). Sensor grids: NVIDIA PAI 128×1200 at 10 Hz, PandaSet 64×1024 at 10 Hz, and nuScenes 32×1024 at 20 Hz. No linguistic content. |
|
|
| ### Testing Dataset: |
|
|
| **Data Collection Method by dataset:** Automatic/Sensors |
| **Labeling Method by dataset:** Hybrid: Automatic/Sensors/Manually-Labeled |
| **Properties:** 1,319 sensor-generated LiDAR range-sequence entries: NVIDIA PAI (1,180), PandaSet (10), nuScenes (128), and KITTI (1). Sensor grids: NVIDIA PAI 128×1200 at 10 Hz, PandaSet 64×1024 at 10 Hz, nuScenes 32×1024 at 20 Hz, and KITTI 64×1024 at 10 Hz. No linguistic content. |
|
|
| ### Evaluation Dataset: |
| **Benchmark Score:** Refer to Technical Report |
|
|
| **Data Collection Method by dataset:** Automatic/Sensors |
| **Labeling Method by dataset:** Hybrid: Automatic/Sensors/Manually-Labeled |
| **Properties:** 202 sensor-generated LiDAR range-sequence entries from the Waymo Open Dataset validation split. Sensor grid: 64×1024 at 10 Hz. No linguistic content. |
|
|
|
|
|
|
| ## Inference: |
| **Acceleration Engine:** PyTorch |
| **Hardware Requirements (GPU Architecture, Model):** |
| * "NVIDIA H100" |
| * "NVIDIA A100 80GB" |
|
|
|
|
| ## Ethical Considerations: |
| NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. |
| Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included. |
| For more detailed information on ethical considerations for this model, please see the Model Card++ [Bias](lidarverse-subcards/bias.md), [Explainability](lidarverse-subcards/explainability.md), [Safety & Security](lidarverse-subcards/safety_security.md), and [Privacy](lidarverse-subcards/privacy.md) Subcards. |
| Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/). |
|
|
| **Generated by NVIDIA Model Card Generator Toolkit.** |
|
|
| ## Citation |
|
|
| If you find LidarVerse useful for your research, please cite: |
|
|
| ```bibtex |
| @misc{che2026lidarverse, |
| title = {Into the {LidarVerse}: A Real-Time Unified {LiDAR} World Model Across Tasks and Sensors}, |
| author = {Henry Che and Xuanchi Ren and Sanja Fidler and Shenlong Wang and Laura Leal-Taix{\'e} and Shengyu Huang}, |
| year = {2026}, |
| eprint = {2610.19201}, |
| archivePrefix = {arXiv}, |
| primaryClass = {cs.CV} |
| } |
| ``` |
|
|
