Cosmos3-Nano-Sim-Auto is designed for real-time, closed-loop driving simulation by generating synchronized multi-view camera videos and LiDAR observations conditioned on a text prompt and World Scenario Map (WSM) controls, including aligned LiDAR controls when LiDAR output is requested. A WSM renderer converts driving inputs from an autonomous vehicle (AV) policy or a human-operated steering-wheel interface into scene controls. The model generates video in small latent-frame chunks conditioned on cached history and incoming WSM controls, enabling subsequent observations to reflect updated driving inputs. It supports up to 11 standard and fisheye camera views, with spatially and temporally aligned LiDAR output converted to 3D point clouds. The joint sensor outputs support evaluation of multi-sensor perception stacks and driving policies within the same rollout. Inference throughput depends on the hardware and generation configuration. Cosmos3-Nano-Sim-Auto was developed by NVIDIA as part of the Cosmos example model family. For a quick start, see the [Cosmos3-Nano-Sim-Auto skill](https://github.com/NVIDIA/cosmos-framework/tree/main/.agents/skills/cosmos3-nano-sim-auto). This model is ready for commercial or non-commercial use. ## License This model is released under [OpenMDW1.1](https://openmdw.ai/license/1-1/). ## Deployment Geography Global ## Use Case Physical AI for autonomous vehicles: closed-loop driving policy evaluation and verification, and synchronized multi-sensor synthetic data generation. ## Release Date - Hugging Face 10/20/2026 via [https://huggingface.co/collections/nvidia/cosmos3-examples](https://huggingface.co/collections/nvidia/cosmos3-examples) - GitHub 10/20/2026 via [https://github.com/nvidia/cosmos](https://github.com/nvidia/cosmos) ## Model Architecture **Architecture Type:** Transformer **Network Architecture:** Mixture-of-Transformers (MoT) Cosmos3-Nano-Sim-Auto is derived from [Cosmos3-Nano](https://huggingface.co/nvidia/Cosmos3-Nano) in two stages: first, post-training on multi-sensor driving data, including LiDAR, to produce [Cosmos3-Nano-Transfer-Auto](https://huggingface.co/nvidia/Cosmos3-Nano-Transfer-Auto); then, causal distillation to produce a few-step model that generates video in small latent-frame chunks conditioned on cached history and the incoming WSM controls. Cached history provides temporal context across generation chunks. Camera views and LiDAR are generated jointly to support spatial and temporal consistency. This model was developed using the [Cosmos Framework](https://github.com/NVIDIA/cosmos-framework). **Number of trainable model parameters:** 16B ## Inputs - **Input Type(s):** Text; Video (World Scenario Map (WSM) camera controls); Image (optional initial RGB conditioning); Other: numerical LiDAR controls and optional measured LiDAR conditioning. - **Input Format(s):** - **Text:** String (scene description: weather, lighting, time of day; also carries the camera configuration) - **World Scenario Map (WSM) camera controls:** Synchronized MP4 videos of rendered scene structure, including lane markings and object cuboids, one per selected camera view. - **LiDAR controls:** Prepared HD-map range-map sequences aligned with the camera controls, supplied in the supported range-map archive or tensor format. Required when generating LiDAR. - **Optional RGB conditioning:** Synchronized initial camera images, supplied directly or extracted from the first frame of each camera video. - **Optional LiDAR conditioning:** An initial measured LiDAR sweep in the supported range/intensity representation, aligned with the start of the camera and control sequences. - **Input Parameters:** - **Text:** One-dimensional (1D) - **WSM camera controls:** Three-dimensional (3D), per view - **Optional RGB conditioning:** Two-dimensional (2D), per image - **HD-map/LiDAR controls:** Three-dimensional (3D) sequences of projected range maps: time × height × width. - **Optional LiDAR conditioning:** Two-dimensional (2D) range-view map per initial sweep: height × width, with range, intensity, and validity channels. - **Other Properties Related to Input:** - For measured LiDAR conditioning, the current causal inference path supports one initial sweep only; longer measured prefixes are not supported. - Supported camera rigs contain 1–11 views (Hyperion 8.1 camera set: front-wide 120°, front-tele 30°, cross-left/right 120°, rear-left/right 70°, rear-tele 30°, and front/left/right/rear fisheye 200°) and 128-beam rotating LiDAR. - Users can select up to 11 supported camera views through the input configuration. To adapt to a custom rig, first post-train Transfer-Auto using the [Transfer-Auto skill](https://github.com/NVIDIA/cosmos-framework/tree/main/.agents/skills/cosmos3-nano-transfer-auto), then perform causal distillation using the [Sim-Auto skill](https://github.com/NVIDIA/cosmos-framework/tree/main/.agents/skills/cosmos3-nano-sim-auto). - WSM controls are produced by a WSM renderer from an AV policy's planned trajectory or a human-operated steering-wheel interface. See the [Cosmos3-Nano-Sim-Auto skill](https://github.com/NVIDIA/cosmos-framework/tree/main/.agents/skills/cosmos3-nano-sim-auto) for details. ## Outputs - **Output Type(s):** Video (multi-view, synchronized), LiDAR - **Output Format(s):** - **Video:** MP4 or streamed video frames, one stream per camera view (standard and fisheye) - **LiDAR:** Range and intensity maps, converted to 3D point clouds with per-point intensity - **Output Parameters:** - **Video:** Three-dimensional (3D), per view - **LiDAR:** Three-dimensional (3D) - **Other Properties Related to Output:** - Camera videos and LiDAR observations are spatially and temporally synchronized. - Generates video at 480p or 720p with output video frame rates of 10–30 frames per second (fps) across all selected views. - Supported inference hardware includes, but is not limited to, H series (H100/H20/H200) and B/GB series (B200/300, GB200/300) GPUs. - Output video frame rates describe playback timing; inference throughput is reported separately. Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g., GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions. ## Software Integration **Runtime Engine(s):** - [PyTorch](https://pytorch.org/) - [vLLM-Omni](https://github.com/vllm-project/vllm-omni) **Supported Hardware Microarchitecture Compatibility:** - NVIDIA Blackwell - NVIDIA Hopper **Operating System(s):** - Linux (other operating systems have not been tested). The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment. ## Model Version(s) **Cosmos3-Nano-Sim-Auto — V1.0 (10/20/2026)** Cosmos3-Nano-Sim-Auto is a causal-distilled variant of [Cosmos3-Nano-Transfer-Auto](https://huggingface.co/nvidia/Cosmos3-Nano-Transfer-Auto), which is a post-trained variant of [Cosmos3-Nano](https://huggingface.co/nvidia/Cosmos3-Nano). ## Training and Evaluation Datasets ### Post-Training Datasets **Data Modality:** Video, Text, and LiDAR. **Post-Training Data Size:** - **Text:** 1 Billion to 10 Trillion Tokens - **Video:** 10,000 to 1 Million Hours - **LiDAR:** Less than 10,000 Hours of LiDAR range/intensity-map sequences **Data Collection Method:** Hybrid: Automatic/Sensors, Synthetic, Automated. **Labeling Method:** Hybrid: Manually-Labeled, Automated. **Properties:** The post-training datasets consist of multi-camera driving videos, associated English captions, and LiDAR observations sourced from NVIDIA-created driving datasets. ### Dataset Overview - **Total Size:** 1.3B data points - **Total Number of Datasets:** 393 dataset entries - **Dataset partition:** Training [100%], Testing [N/A — evaluation benchmarks used separately], Validation [N/A — evaluation benchmarks used separately] - **Time period for training data collection:** 2024–2026 - **Time period for validation data collection:** N/A (standard public benchmarks) Raw data from internal and external sources is transformed into training-ready data through multiple stages of curation, filtering, and quality review. Data acquisition spans diverse multimodal sources — robotics, autonomous driving, industrial environments, indoor and outdoor scenes, varied lighting and weather conditions, camera viewpoints, object categories, and human activities — to broaden coverage across Physical AI operating environments. Automated filtering pipelines remove corrupted, duplicate, low-quality, and restricted content. Metadata analysis, heuristic rules, and model-assisted classifiers are applied during preprocessing to flag anomalous distributions and low-diversity subsets. Human review supplements automated filtering for selected datasets, benchmark construction, and targeted quality analysis. Datasets are balanced across modalities and task categories — visual reasoning, text-to-image, text-to-video, image-to-video, audio generation, video transfer, action-conditioned generation, and action command generation — to reduce overrepresentation of narrow domains. Synthetic and simulation-based augmentation supplements coverage of rare physical interactions and edge-case scenarios. Deduplication and provenance tracking are applied across the corpus. The resulting processed data is converted into model-ready tokenized or encoded representations through modality-specific preprocessors before training begins. Training datasets passed through multiple layers of automated and manual safeguards designed to reduce the presence of harmful or policy-violating content across categories including weapons and weapons-related instructional content, criminal planning, child sexual abuse material (CSAM), non-consensual intimate imagery (NCII), sexual content involving minors, harassment, hate speech, profanity, threats and incitement to violence, self-harm or suicide-related content, and graphic violence. Data sources are reviewed for licensing compatibility, provenance, and alignment with internal data governance and safety policies before admission into training corpora. Automated filtering pipelines combine multiple detection strategies: hash-matching against known CSAM and NCII reference databases; classifier-based moderation models trained for explicit sexual content, hate speech, violence, weapons imagery, and other restricted categories; keyword and regex-based screening for criminal-planning, threats, and self-harm phrases in text data; metadata and provenance heuristics for source-level risk signals; and embedding-based anomaly detection to surface samples that fall outside expected distributions. Human review and targeted audits supplement automated filtering for selected datasets, benchmark construction, and safety-sensitive evaluation. For multimodal Physical AI data (robotics, autonomous driving, industrial scenes), additional filtering targets invalid action trajectories, physically implausible interactions, and unsafe control sequences. Synthetic and simulation-generated data are evaluated through internal validation before inclusion. Benchmark evaluations and red-team testing are applied post-training to surface remaining safety gaps across world generation, reasoning, audio, and action tasks. No large-scale data-filtering process can guarantee complete removal of all harmful content; residual risks may remain, particularly in rare edge cases or open-world deployment settings. Ongoing monitoring and dataset review continue post-release. For more information about the datasets used to train this model, please see the [Public Summary of Training Content](https://docs.nvidia.com/cosmos/latest/cosmos3/index.html). **Data Modality and Training Data Size** | Modality | Reasoning Data Sample Count | Generation Data Sample Count | | -------- | ------------------- | -------------------- | | Text | 22M | Not Applicable | | Image | 19M | 767M | | Video | 1M | 348M | | Audio | Not Applicable | 139M | | Action | Not Applicable | 7M | **Data Collection Method by dataset** - Hybrid: Automatic/Sensors, Synthetic, Automated **Labeling Method by dataset** - Hybrid: Manually-Labeled, Automated **Properties:** The training, testing, and evaluation datasets consist of diverse multimodal video, image, audio, action, synthetic, and sensor-conditioned data sourced from NVIDIA-owned data and publicly available, commercially permissive datasets. These datasets are curated to exclude known restricted content and to support building an Omni model that learns to generate and reason about dynamic physical environments across world reasoning and generation tasks. ### Public Datasets | Dataset | Samples | |---------------------------|-------------| | OpenImage | 1.2M | | Coyo700M | 100M | | YouTube Video | 340M | | UMI | 4.5M | ### Private Datasets | Dataset | Samples | |---------------------------|-------------| | Egocentric | 7M | | Nexar | 0.6M | | AgiBot | 0.2M | | HOI | 0.3M | ### Synthetic Datasets | Dataset | Samples | |-----------------------------------------|-------------| | synthetic images generated using HiDream-I1 | 15M | | synthetic images generated using Qwen-Image-2512 | 14M | | synthetic captions generated using Qwen3-VL | 1115M | ## Evaluation Datasets **Data Collection Method by dataset** - Hybrid: Automatic/Sensors, Synthetic, Automated **Labeling Method by dataset** - Hybrid: Manually-Labeled, Automated **Properties:** The training, testing, and evaluation datasets consist of diverse multimodal video, image, audio, action, synthetic, and sensor-conditioned data sourced from NVIDIA-owned data and publicly available, commercially permissive datasets. These datasets are curated to exclude known restricted content and to support building an Omni model that learns to generate and reason about dynamic physical environments across world reasoning and generation tasks. ## Quality Benchmark(s) We provide control inputs and text prompts in the [Cosmos3 Driving Simulation Evaluation Dataset](https://huggingface.co/datasets/nvidia/Cosmos3-Driving-Simulation-Evaluation) to support evaluation. **Cosmos3-Nano-Sim-Auto** results use these text prompts; the Cosmos-Transfer2.5-2B Multiview baseline uses different prompts. ### Multi-view RGB quality We evaluate **Cosmos3-Nano-Sim-Auto** at 720p in text-to-video (T2V) and image-to-video (I2V) settings, using a guidance scale of 1. The model generates 297 frames across seven cameras with four denoising steps. Cosmos-Transfer2.5-2B Multiview uses a guidance scale of 3. [Fréchet Inception Distance (FID)](https://arxiv.org/abs/1706.08500) compares generated and reference frame-feature distributions; [Fréchet Video Distance (FVD)](https://arxiv.org/abs/1812.01717) compares video-feature distributions, with I3D and StyleGAN-V results reported separately. [DOVER](https://github.com/VQAssessment/DOVER) evaluates video quality. Temporal Sampson error measures geometric agreement between matched image points across frames. Adjacent Cosine Mean (ACM) measures within-view visual consistency by averaging CLIP-feature cosine similarities between each frame after the first and both its preceding frame and the first frame, with negative similarities clamped to zero. Scores are computed per camera video, then averaged across views and samples. The score ranges from 0 to 1; higher is better. ACM uses no ground-truth video and does not measure cross-view consistency. A frozen video can score highly, so ACM alone does not establish realistic motion. Cross-view Sampson error measures agreement across camera views, evaluated over two or five view pairs.
| Model | Frame FID ↓ | FVD (I3D) ↓ | FVD (StyleGAN-V) ↓ | DOVER technical ↑ | Temporal Sampson error ↓ | Adjacent Cosine Mean (ACM) ↑ | Cross-view Sampson error, 2 view pairs (CSE2) ↓ | Cross-view Sampson error, 5 view pairs (CSE5) ↓ |
|---|---|---|---|---|---|---|---|---|
| Cosmos-Transfer2.5-2B Multiview (Transfer-T2V, guidance 3) |
61.9 | 772 | 770 | 8.56 | 0.530 | 0.885 | 2.51 | 2.07 |
| Cosmos3-Nano-Sim-Auto (T2V, guidance 1) |
48.6 | 480 | 479 | 7.30 | 0.503 | 0.906 | 2.60 | 2.24 |
| Cosmos-Transfer2.5-2B Multiview (Transfer-I2V, guidance 3) |
41.9 | 383 | 382 | 8.22 | 0.524 | 0.878 | 2.45 | 1.98 |
| Cosmos3-Nano-Sim-Auto (I2V, guidance 1) |
33.6 | 325 | 324 | 7.51 | 0.505 | 0.900 | 2.36 | 2.09 |
| Model | BEV JSD, global ↓ | BEV JSD, local ↓ | BEV MMD^2 ↓ | Frechet Utonia dist ↓ | Utonia MMD^2 ↓ | Temporal CTC / real, k=1 (closer to 1) | Vehicle CTC / real (closer to 1) | Object support recall (real >=30, generated >=10 points) ↑ | Object detection F1: generated LiDAR vs GT @3m ↑ |
|---|---|---|---|---|---|---|---|---|---|
| LidarVerse (21 scans) | 0.151 | 0.400 | 0.0120 | 33.6 | 0.0636 | 1.20 | 2.51 | 0.843 | 0.201 |
| LidarVerse (116 scans) | 0.116 | 0.300 | 0.00644 | 44.8 | 0.0907 | 1.35 | 3.73 | 0.788 | 0.200 |
| Cosmos3-Nano-Sim-Auto (T2V, guidance 1) |
0.0552 | 0.277 | 0.00593 | 20.1 | 0.0354 | 1.28 | 3.59 | 0.816 | 0.188 |