Commit ·
2992437
1
Parent(s): c2b114b
Update README.md (#1)
Browse files- Update README.md (c99608298afc98961f07b02034be10c3e6b2b831)
- Update README.md (b60813ec5d1af6075a7788d7734e2515d216038b)
README.md
CHANGED
|
@@ -1,47 +1,123 @@
|
|
| 1 |
---
|
| 2 |
-
license:
|
| 3 |
---
|
| 4 |
|
| 5 |
-
# LidarVerse
|
| 6 |
|
| 7 |
-
**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
|
| 9 |
-
|
|
|
|
|
|
|
|
|
|
| 10 |
|
| 11 |
-
| Package | Contents |
|
| 12 |
-
| --- | --- |
|
| 13 |
-
| `lidarverse-tokenizer-pai` | EMA tokenizer, matching latent statistics, configuration |
|
| 14 |
-
| `lidarverse-bidir-pai` (Stage 1) | EMA bidirectional model and configuration |
|
| 15 |
-
| `lidarverse-causal-pai` (Stage 2) | EMA causal model and configuration |
|
| 16 |
|
| 17 |
-
|
| 18 |
-
|
|
|
|
|
|
|
| 19 |
|
| 20 |
-
##
|
|
|
|
|
|
|
| 21 |
|
| 22 |
-
|
|
|
|
| 23 |
|
| 24 |
-
|
| 25 |
-
hf download nvidia/lidarverse --local-dir checkpoints
|
| 26 |
-
```
|
| 27 |
|
| 28 |
-
|
| 29 |
-
|
| 30 |
|
| 31 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 42 |
|
| 43 |
-
These packages use the NVIDIA PAI 128x1024 sensor grid and a 2.5--100 m range.
|
| 44 |
-
Use the paired PAI tokenizer and latent statistics for both diffusion models.
|
| 45 |
-
Both stages use 35-step SDE sampling by default, with shift 5, eta 1, and CFG 1.
|
| 46 |
-
These checkpoints support starting post-training with a fresh optimizer;
|
| 47 |
-
they do not contain the state needed to resume the original training run.
|
|
|
|
| 1 |
---
|
| 2 |
+
license: openmdw-1.1
|
| 3 |
---
|
| 4 |
|
| 5 |
+
# LidarVerse Overview
|
| 6 |
|
| 7 |
+
> **Current checkpoint scope:** the quickstarts use the range-only NVIDIA PAI
|
| 8 |
+
> tokenizer and PAI bidirectional, causal, and SF-DMD EMA
|
| 9 |
+
> packages, with 128x1024 inputs at 10 Hz and 2.5--100 m support. They do not
|
| 10 |
+
> include intensity or the paper's cross-sensor checkpoints. The broader
|
| 11 |
+
> architecture and dataset descriptions below describe the research project,
|
| 12 |
+
> not a claim that these particular packages support every sensor. Quickstart
|
| 13 |
+
> throughput is not a paper benchmark.
|
| 14 |
|
| 15 |
+
## Description:
|
| 16 |
+
LidarVerse is a real-time unified LiDAR world model for sensor simulation across tasks and sensors. LidarVerse bridges both generative and reconstruction-based simulation into a single, unified conditional generative framework. It is composed of a universal, fully attention-based tokenizer maps LiDAR range sequences with different scanning patterns into a shared latent space; a bidirectional diffusion teacher learns a common interface for conditioning on a LiDAR prefix, an HD-Map, a persistent spatial cache, and optionally, text; and a four-step autoregressive causal student and maintain both a rolling temporal K/V cache and a world-aligned spatial cache, enabling persistent long-horizon roll-outs with real-time inference at 11 FPS on a single H100 GPU.
|
| 17 |
+
LidarVerse was developed by NVIDIA.
|
| 18 |
+
_This model is for research and development only._
|
| 19 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
|
| 21 |
+
### License/Terms of Use:
|
| 22 |
+
[NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/)
|
| 23 |
+
### Deployment Geography:
|
| 24 |
+
Global
|
| 25 |
|
| 26 |
+
### Use Case:
|
| 27 |
+
This model is intended for research community to study, ablate, and build upon of.
|
| 28 |
+
|
| 29 |
|
| 30 |
+
### Release Date:
|
| 31 |
+
**GitHub:** 09/11/2026 via https://github.com/nv-tlabs/lidarverse
|
| 32 |
|
| 33 |
+
|
|
|
|
|
|
|
| 34 |
|
| 35 |
+
## Reference(s):
|
| 36 |
+
[Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models](https://research.nvidia.com/labs/toronto-ai/cosmos_drive_dreams/)
|
| 37 |
|
| 38 |
+
## Model Architecture:
|
| 39 |
+
**Architecture Type:** Transformer
|
| 40 |
+
**Network Architecture:** Diffusion Transformer (DiT)
|
| 41 |
+
**This model was developed based on Cosmos-Predict2-T2V-2B.**
|
| 42 |
+
**Number of model parameters:** 2B
|
| 43 |
|
| 44 |
+
## Input:
|
| 45 |
+
**Input Type(s):** Video
|
| 46 |
+
**Input Format(s):** Lidar
|
| 47 |
+
**Input Parameters:** Three-Dimensional (3D)
|
| 48 |
+
**Other Properties Related to Input:** Flexible resolution and length
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
## Output:
|
| 52 |
+
**Output Type(s):** Video
|
| 53 |
+
**Output Format:** Lidar
|
| 54 |
+
**Output Parameters:** Three-Dimensional (3D)
|
| 55 |
+
**Other Properties Related to Output:** Flexible resolution and length, streaming supported
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
|
| 59 |
+
|
| 60 |
+
## Software Integration:
|
| 61 |
+
**Runtime Engine(s):** Not Applicable (N/A)
|
| 62 |
+
**Supported Hardware Microarchitecture Compatibility:**
|
| 63 |
+
* NVIDIA Ampere
|
| 64 |
+
* NVIDIA Blackwell
|
| 65 |
+
* NVIDIA Hopper
|
| 66 |
+
* NVIDIA Lovelace
|
| 67 |
+
|
| 68 |
+
**Supported Operating System(s):** Linux
|
| 69 |
+
|
| 70 |
+
The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
|
| 71 |
+
|
| 72 |
+
|
| 73 |
+
## Model Version(s):
|
| 74 |
+
LidarVerse v1.0 — 2B-parameter trained research model
|
| 75 |
+
|
| 76 |
+
## Training, Testing, and Evaluation Datasets:
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
## Training Dataset:
|
| 80 |
+
|
| 81 |
+
**Data Modality:**
|
| 82 |
+
* Video
|
| 83 |
+
* Text
|
| 84 |
+
* Image
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
**Image Training Data Size:** 1 Million to 1 Billion Images
|
| 88 |
+
**Text Training Data Size:** Less than a Billion Tokens
|
| 89 |
+
**Video Training Data Size:** 10,000 to 1 Million Hours
|
| 90 |
+
**Data Collection Method by dataset:** Automatic/Sensors
|
| 91 |
+
**Labeling Method by dataset:** Hybrid: Automatic/Sensors/Manually-Labeled
|
| 92 |
+
**Properties:** 112,803 sensor-generated LiDAR range-sequence entries: NVIDIA PAI (111,988), PandaSet (93), and nuScenes (722). Sensor grids: NVIDIA PAI 128×1200 at 10 Hz, PandaSet 64×1024 at 10 Hz, and nuScenes 32×1024 at 20 Hz. No linguistic content.
|
| 93 |
+
|
| 94 |
+
### Testing Dataset:
|
| 95 |
+
|
| 96 |
+
**Data Collection Method by dataset:** Automatic/Sensors
|
| 97 |
+
**Labeling Method by dataset:** Hybrid: Automatic/Sensors/Manually-Labeled
|
| 98 |
+
**Properties:** 1,319 sensor-generated LiDAR range-sequence entries: NVIDIA PAI (1,180), PandaSet (10), nuScenes (128), and KITTI (1). Sensor grids: NVIDIA PAI 128×1200 at 10 Hz, PandaSet 64×1024 at 10 Hz, nuScenes 32×1024 at 20 Hz, and KITTI 64×1024 at 10 Hz. No linguistic content.
|
| 99 |
+
|
| 100 |
+
### Evaluation Dataset:
|
| 101 |
+
**Benchmark Score:** Refer to Technical Report
|
| 102 |
+
|
| 103 |
+
**Data Collection Method by dataset:** Automatic/Sensors
|
| 104 |
+
**Labeling Method by dataset:** Hybrid: Automatic/Sensors/Manually-Labeled
|
| 105 |
+
**Properties:** 202 sensor-generated LiDAR range-sequence entries from the Waymo Open Dataset validation split. Sensor grid: 64×1024 at 10 Hz. No linguistic content.
|
| 106 |
+
|
| 107 |
+
|
| 108 |
+
|
| 109 |
+
## Inference:
|
| 110 |
+
**Acceleration Engine:** PyTorch
|
| 111 |
+
**Hardware Requirements (GPU Architecture, Model):**
|
| 112 |
+
* "NVIDIA H100"
|
| 113 |
+
* "NVIDIA A100 80GB"
|
| 114 |
+
|
| 115 |
+
|
| 116 |
+
## Ethical Considerations:
|
| 117 |
+
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
|
| 118 |
+
Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.
|
| 119 |
+
For more detailed information on ethical considerations for this model, please see the Model Card++ [Bias](lidarverse-subcards/bias.md), [Explainability](lidarverse-subcards/explainability.md), [Safety & Security](lidarverse-subcards/safety_security.md), and [Privacy](lidarverse-subcards/privacy.md) Subcards.
|
| 120 |
+
Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
|
| 121 |
+
|
| 122 |
+
**Generated by NVIDIA Model Card Generator Toolkit.**
|
| 123 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|