hche-nvidia commited on
Commit
2992437
·
1 Parent(s): c2b114b

Update README.md (#1)

Browse files

- Update README.md (c99608298afc98961f07b02034be10c3e6b2b831)
- Update README.md (b60813ec5d1af6075a7788d7734e2515d216038b)

Files changed (1) hide show
  1. README.md +109 -33
README.md CHANGED
@@ -1,47 +1,123 @@
1
  ---
2
- license: apache-2.0
3
  ---
4
 
5
- # LidarVerse
6
 
7
- **Into the LidarVerse: A Real-Time Unified LiDAR World Model Across Tasks and Sensors**
 
 
 
 
 
 
8
 
9
- NVIDIA PAI checkpoint packages for the LidarVerse tokenizer and world model.
 
 
 
10
 
11
- | Package | Contents |
12
- | --- | --- |
13
- | `lidarverse-tokenizer-pai` | EMA tokenizer, matching latent statistics, configuration |
14
- | `lidarverse-bidir-pai` (Stage 1) | EMA bidirectional model and configuration |
15
- | `lidarverse-causal-pai` (Stage 2) | EMA causal model and configuration |
16
 
17
- Stage 3 (few-step SF-DMD) is not included in this release. Stage 2 is the
18
- causal teacher, not the four-step distilled model.
 
 
19
 
20
- ## Download
 
 
21
 
22
- Run from the LidarVerse code repository:
 
23
 
24
- ```bash
25
- hf download nvidia/lidarverse --local-dir checkpoints
26
- ```
27
 
28
- Authenticate with `hf auth login` if the repository requires access.
29
- See the [code repository](https://github.com/nv-tlabs/LidarVerse) for setup and quickstarts.
30
 
31
- ## Checkpoint Format
 
 
 
 
32
 
33
- - Tokenizer: `ckpt.pt` is a tensor-only `network.*` state dictionary containing
34
- EMA parameters and required inference buffers. `latent_stats.pt` contains
35
- only the matching 64-channel `mean` and `std` tensors.
36
- - Stages 1 and 2: `ema_model.pt` contains a single `model` state dictionary
37
- with EMA model tensors, including required model buffers.
38
- - No optimizer, scheduler, training-step state, fake-score network, or
39
- separate non-EMA model weights are included.
40
- - `pai_manifest.json` lists only these three packages, their dependencies,
41
- and SHA-256 checksums.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
42
 
43
- These packages use the NVIDIA PAI 128x1024 sensor grid and a 2.5--100 m range.
44
- Use the paired PAI tokenizer and latent statistics for both diffusion models.
45
- Both stages use 35-step SDE sampling by default, with shift 5, eta 1, and CFG 1.
46
- These checkpoints support starting post-training with a fresh optimizer;
47
- they do not contain the state needed to resume the original training run.
 
1
  ---
2
+ license: openmdw-1.1
3
  ---
4
 
5
+ # LidarVerse Overview
6
 
7
+ > **Current checkpoint scope:** the quickstarts use the range-only NVIDIA PAI
8
+ > tokenizer and PAI bidirectional, causal, and SF-DMD EMA
9
+ > packages, with 128x1024 inputs at 10 Hz and 2.5--100 m support. They do not
10
+ > include intensity or the paper's cross-sensor checkpoints. The broader
11
+ > architecture and dataset descriptions below describe the research project,
12
+ > not a claim that these particular packages support every sensor. Quickstart
13
+ > throughput is not a paper benchmark.
14
 
15
+ ## Description:
16
+ LidarVerse is a real-time unified LiDAR world model for sensor simulation across tasks and sensors. LidarVerse bridges both generative and reconstruction-based simulation into a single, unified conditional generative framework. It is composed of a universal, fully attention-based tokenizer maps LiDAR range sequences with different scanning patterns into a shared latent space; a bidirectional diffusion teacher learns a common interface for conditioning on a LiDAR prefix, an HD-Map, a persistent spatial cache, and optionally, text; and a four-step autoregressive causal student and maintain both a rolling temporal K/V cache and a world-aligned spatial cache, enabling persistent long-horizon roll-outs with real-time inference at 11 FPS on a single H100 GPU.
17
+ LidarVerse was developed by NVIDIA.
18
+ _This model is for research and development only._
19
 
 
 
 
 
 
20
 
21
+ ### License/Terms of Use:
22
+ [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/)
23
+ ### Deployment Geography:
24
+ Global
25
 
26
+ ### Use Case:
27
+ This model is intended for research community to study, ablate, and build upon of.
28
+
29
 
30
+ ### Release Date:
31
+ **GitHub:** 09/11/2026 via https://github.com/nv-tlabs/lidarverse
32
 
33
+
 
 
34
 
35
+ ## Reference(s):
36
+ [Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models](https://research.nvidia.com/labs/toronto-ai/cosmos_drive_dreams/)
37
 
38
+ ## Model Architecture:
39
+ **Architecture Type:** Transformer
40
+ **Network Architecture:** Diffusion Transformer (DiT)
41
+ **This model was developed based on Cosmos-Predict2-T2V-2B.**
42
+ **Number of model parameters:** 2B
43
 
44
+ ## Input:
45
+ **Input Type(s):** Video
46
+ **Input Format(s):** Lidar
47
+ **Input Parameters:** Three-Dimensional (3D)
48
+ **Other Properties Related to Input:** Flexible resolution and length
49
+
50
+
51
+ ## Output:
52
+ **Output Type(s):** Video
53
+ **Output Format:** Lidar
54
+ **Output Parameters:** Three-Dimensional (3D)
55
+ **Other Properties Related to Output:** Flexible resolution and length, streaming supported
56
+
57
+
58
+ Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
59
+
60
+ ## Software Integration:
61
+ **Runtime Engine(s):** Not Applicable (N/A)
62
+ **Supported Hardware Microarchitecture Compatibility:**
63
+ * NVIDIA Ampere
64
+ * NVIDIA Blackwell
65
+ * NVIDIA Hopper
66
+ * NVIDIA Lovelace
67
+
68
+ **Supported Operating System(s):** Linux
69
+
70
+ The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
71
+
72
+
73
+ ## Model Version(s):
74
+ LidarVerse v1.0 — 2B-parameter trained research model
75
+
76
+ ## Training, Testing, and Evaluation Datasets:
77
+
78
+
79
+ ## Training Dataset:
80
+
81
+ **Data Modality:**
82
+ * Video
83
+ * Text
84
+ * Image
85
+
86
+
87
+ **Image Training Data Size:** 1 Million to 1 Billion Images
88
+ **Text Training Data Size:** Less than a Billion Tokens
89
+ **Video Training Data Size:** 10,000 to 1 Million Hours
90
+ **Data Collection Method by dataset:** Automatic/Sensors
91
+ **Labeling Method by dataset:** Hybrid: Automatic/Sensors/Manually-Labeled
92
+ **Properties:** 112,803 sensor-generated LiDAR range-sequence entries: NVIDIA PAI (111,988), PandaSet (93), and nuScenes (722). Sensor grids: NVIDIA PAI 128×1200 at 10 Hz, PandaSet 64×1024 at 10 Hz, and nuScenes 32×1024 at 20 Hz. No linguistic content.
93
+
94
+ ### Testing Dataset:
95
+
96
+ **Data Collection Method by dataset:** Automatic/Sensors
97
+ **Labeling Method by dataset:** Hybrid: Automatic/Sensors/Manually-Labeled
98
+ **Properties:** 1,319 sensor-generated LiDAR range-sequence entries: NVIDIA PAI (1,180), PandaSet (10), nuScenes (128), and KITTI (1). Sensor grids: NVIDIA PAI 128×1200 at 10 Hz, PandaSet 64×1024 at 10 Hz, nuScenes 32×1024 at 20 Hz, and KITTI 64×1024 at 10 Hz. No linguistic content.
99
+
100
+ ### Evaluation Dataset:
101
+ **Benchmark Score:** Refer to Technical Report
102
+
103
+ **Data Collection Method by dataset:** Automatic/Sensors
104
+ **Labeling Method by dataset:** Hybrid: Automatic/Sensors/Manually-Labeled
105
+ **Properties:** 202 sensor-generated LiDAR range-sequence entries from the Waymo Open Dataset validation split. Sensor grid: 64×1024 at 10 Hz. No linguistic content.
106
+
107
+
108
+
109
+ ## Inference:
110
+ **Acceleration Engine:** PyTorch
111
+ **Hardware Requirements (GPU Architecture, Model):**
112
+ * "NVIDIA H100"
113
+ * "NVIDIA A100 80GB"
114
+
115
+
116
+ ## Ethical Considerations:
117
+ NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
118
+ Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.
119
+ For more detailed information on ethical considerations for this model, please see the Model Card++ [Bias](lidarverse-subcards/bias.md), [Explainability](lidarverse-subcards/explainability.md), [Safety & Security](lidarverse-subcards/safety_security.md), and [Privacy](lidarverse-subcards/privacy.md) Subcards.
120
+ Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
121
+
122
+ **Generated by NVIDIA Model Card Generator Toolkit.**
123