Cosmos3-Nano-Transfer-Auto generates synchronized multi-view camera videos and LiDAR observations. It supports up to 11 standard and fisheye camera views. Generation is conditioned on a text prompt, World Scenario Map (WSM) controls for each camera view, and aligned LiDAR controls when LiDAR output is requested. The WSM inputs encode driving-scene structure, including traffic signs, traffic lights, lane markings, and object cuboids, while the text prompt specifies appearance attributes such as weather, lighting, and time of day. This enables the same driving scenario to be regenerated at scale with controlled variations in scene appearance. The model uses bidirectional generation for batch synthetic data generation and serves as the source checkpoint for causal distillation into Cosmos3-Nano-Sim-Auto, which provides closed-loop simulation with the same multi-view, multi-sensor output modalities. Cosmos3-Nano-Transfer-Auto was developed by NVIDIA as part of the Cosmos example model family. For a quick start, see the [Cosmos3-Nano-Transfer-Auto skill](https://github.com/NVIDIA/cosmos-framework/tree/main/.agents/skills/cosmos3-nano-transfer-auto). This model is ready for commercial or non-commercial use. ## License This model is released under [OpenMDW1.1](https://openmdw.ai/license/1-1/). ## Deployment Geography Global ## Use Case Physical AI for autonomous vehicles: synchronized multi-sensor synthetic data generation at scale for training and evaluating perception and driving models. ## Release Date - Hugging Face 10/20/2026 via [https://huggingface.co/collections/nvidia/cosmos3-examples](https://huggingface.co/collections/nvidia/cosmos3-examples) - GitHub 10/20/2026 via [https://github.com/nvidia/cosmos](https://github.com/nvidia/cosmos) ## Model Architecture **Architecture Type:** Transformer **Network Architecture:** Mixture-of-Transformers (MoT) Cosmos3-Nano-Transfer-Auto is derived from [Cosmos3-Nano](https://huggingface.co/nvidia/Cosmos3-Nano) through post-training on multi-sensor driving data, including LiDAR. The model jointly denoises the full multi-view sequence using bidirectional temporal context, conditioned on the text prompt, WSM camera controls, and aligned LiDAR controls when generating LiDAR. It produces synchronized camera videos for up to 11 views and corresponding LiDAR observations. This model was developed using the [Cosmos Framework](https://github.com/NVIDIA/cosmos-framework). **Number of trainable model parameters:** 16B ## Inputs - **Input Type(s):** Text; Video (World Scenario Map (WSM) camera controls); Image (optional initial RGB conditioning); Other: numerical LiDAR controls and optional measured LiDAR conditioning. - **Input Format(s):** - **Text:** String (scene description: weather, lighting, time of day; also carries the camera configuration) - **World Scenario Map (WSM) camera controls:** Synchronized MP4 videos of rendered scene structure, including lane markings and object cuboids, one per selected camera view. - **LiDAR controls:** Prepared HD-map range-map sequences aligned with the camera controls, supplied in the supported range-map archive or tensor format. Required when generating LiDAR. - **Optional RGB conditioning:** Synchronized initial camera images, supplied directly or extracted from the first frame of each camera video. - **Optional LiDAR conditioning:** An initial measured LiDAR sweep in the supported range/intensity representation, aligned with the start of the camera and control sequences. - **Input Parameters:** - **Text:** One-dimensional (1D) - **WSM camera controls:** Three-dimensional (3D), per view - **Optional RGB conditioning:** Two-dimensional (2D), per image - **HD-map/LiDAR controls:** Three-dimensional (3D) sequences of projected range maps: time × height × width. - **Optional LiDAR conditioning:** Two-dimensional (2D) range-view map per initial sweep: height × width, with range, intensity, and validity channels. - **Other Properties Related to Input:** - Supported camera rigs contain 1–11 views (Hyperion 8.1 camera set: including front-wide 120°, front-tele 30°, cross-left/right 120°, rear-left/right 70°, rear-tele 30°, and front/left/right/rear fisheye 200°) and 128-beam rotating LiDAR. Adaptation to other rigs requires additional post-training. - WSM controls are produced by a WSM renderer; see the [Cosmos3-Nano-Transfer-Auto skill](https://github.com/NVIDIA/cosmos-framework/tree/main/.agents/skills/cosmos3-nano-transfer-auto) for details. ## Outputs - **Output Type(s):** Video (multi-view, synchronized), LiDAR - **Output Format(s):** - **Video:** MP4, one stream per camera view (standard and fisheye) - **LiDAR:** Range and intensity maps, converted to 3D point clouds with per-point intensity - **Output Parameters:** - **Video:** Three-dimensional (3D), per view - **LiDAR:** Three-dimensional (3D) - **Other Properties Related to Output:** - Camera videos and LiDAR observations are spatially and temporally synchronized. - Generates up to 300 frames per view at 480p or 720p, with output video frame rates of 10–30 frames per second (fps). - Output video frame rates describe playback timing; inference throughput is reported separately. Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g., GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions. ## Software Integration **Runtime Engine(s):** - [PyTorch](https://pytorch.org/) - [vLLM-Omni](https://github.com/vllm-project/vllm-omni) **Supported Hardware Microarchitecture Compatibility:** - NVIDIA Blackwell - NVIDIA Hopper **Operating System(s):** - Linux (other operating systems have not been tested). The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment. ## Model Version(s) **Cosmos3-Nano-Transfer-Auto — V1.0 (10/20/2026)** Cosmos3-Nano-Transfer-Auto is a post-trained variant of [Cosmos3-Nano](https://huggingface.co/nvidia/Cosmos3-Nano). ## Training and Evaluation Datasets ### Post-Training Datasets **Data Modality:** Video, Text, and LiDAR. **Post-Training Data Size:** - **Text:** 1 Billion to 10 Trillion Tokens - **Video:** 10,000 to 1 Million Hours - **LiDAR:** Less than 10,000 Hours of LiDAR range/intensity-map sequences **Data Collection Method:** Hybrid: Automatic/Sensors, Synthetic, Automated. **Labeling Method:** Hybrid: Manually-Labeled, Automated. **Properties:** The post-training datasets consist of multi-camera driving videos, associated English captions, and LiDAR observations sourced from NVIDIA-created driving datasets. ### Dataset Overview - **Total Size:** 1.3B data points - **Total Number of Datasets:** 393 dataset entries - **Dataset partition:** Training [100%], Testing [N/A — evaluation benchmarks used separately], Validation [N/A — evaluation benchmarks used separately] - **Time period for training data collection:** 2024–2026 - **Time period for validation data collection:** N/A (standard public benchmarks) Raw data from internal and external sources is transformed into training-ready data through multiple stages of curation, filtering, and quality review. Data acquisition spans diverse multimodal sources — robotics, autonomous driving, industrial environments, indoor and outdoor scenes, varied lighting and weather conditions, camera viewpoints, object categories, and human activities — to broaden coverage across Physical AI operating environments. Automated filtering pipelines remove corrupted, duplicate, low-quality, and restricted content. Metadata analysis, heuristic rules, and model-assisted classifiers are applied during preprocessing to flag anomalous distributions and low-diversity subsets. Human review supplements automated filtering for selected datasets, benchmark construction, and targeted quality analysis. Datasets are balanced across modalities and task categories — visual reasoning, text-to-image, text-to-video, image-to-video, audio generation, video transfer, action-conditioned generation, and action command generation — to reduce overrepresentation of narrow domains. Synthetic and simulation-based augmentation supplements coverage of rare physical interactions and edge-case scenarios. Deduplication and provenance tracking are applied across the corpus. The resulting processed data is converted into model-ready tokenized or encoded representations through modality-specific preprocessors before training begins. Training datasets passed through multiple layers of automated and manual safeguards designed to reduce the presence of harmful or policy-violating content across categories including weapons and weapons-related instructional content, criminal planning, child sexual abuse material (CSAM), non-consensual intimate imagery (NCII), sexual content involving minors, harassment, hate speech, profanity, threats and incitement to violence, self-harm or suicide-related content, and graphic violence. Data sources are reviewed for licensing compatibility, provenance, and alignment with internal data governance and safety policies before admission into training corpora. Automated filtering pipelines combine multiple detection strategies: hash-matching against known CSAM and NCII reference databases; classifier-based moderation models trained for explicit sexual content, hate speech, violence, weapons imagery, and other restricted categories; keyword and regex-based screening for criminal-planning, threats, and self-harm phrases in text data; metadata and provenance heuristics for source-level risk signals; and embedding-based anomaly detection to surface samples that fall outside expected distributions. Human review and targeted audits supplement automated filtering for selected datasets, benchmark construction, and safety-sensitive evaluation. For multimodal Physical AI data (robotics, autonomous driving, industrial scenes), additional filtering targets invalid action trajectories, physically implausible interactions, and unsafe control sequences. Synthetic and simulation-generated data are evaluated through internal validation before inclusion. Benchmark evaluations and red-team testing are applied post-training to surface remaining safety gaps across world generation, reasoning, audio, and action tasks. No large-scale data-filtering process can guarantee complete removal of all harmful content; residual risks may remain, particularly in rare edge cases or open-world deployment settings. Ongoing monitoring and dataset review continue post-release. For more information about the datasets used to train this model, please see the [Public Summary of Training Content](https://docs.nvidia.com/cosmos/latest/cosmos3/index.html). **Data Modality and Training Data Size** | Modality | Reasoning Data Sample Count | Generation Data Sample Count | | -------- | ------------------- | -------------------- | | Text | 22M | Not Applicable | | Image | 19M | 767M | | Video | 1M | 348M | | Audio | Not Applicable | 139M | | Action | Not Applicable | 7M | **Data Collection Method by dataset** - Hybrid: Automatic/Sensors, Synthetic, Automated **Labeling Method by dataset** - Hybrid: Manually-Labeled, Automated **Properties:** The training, testing, and evaluation datasets consist of diverse multimodal video, image, audio, action, synthetic, and sensor-conditioned data sourced from NVIDIA-owned data and publicly available, commercially permissive datasets. These datasets are curated to exclude known restricted content and to support building an Omni model that learns to generate and reason about dynamic physical environments across world reasoning and generation tasks. ### Public Datasets | Dataset | Samples | |---------------------------|-------------| | OpenImage | 1.2M | | Coyo700M | 100M | | YouTube Video | 340M | | UMI | 4.5M | ### Private Datasets | Dataset | Samples | |---------------------------|-------------| | Egocentric | 7M | | Nexar | 0.6M | | AgiBot | 0.2M | | HOI | 0.3M | ### Synthetic Datasets | Dataset | Samples | |-----------------------------------------|-------------| | synthetic images generated using HiDream-I1 | 15M | | synthetic images generated using Qwen-Image-2512 | 14M | | synthetic captions generated using Qwen3-VL | 1115M | ## Evaluation Datasets **Data Collection Method by dataset** - Hybrid: Automatic/Sensors, Synthetic, Automated **Labeling Method by dataset** - Hybrid: Manually-Labeled, Automated **Properties:** The training, testing, and evaluation datasets consist of diverse multimodal video, image, audio, action, synthetic, and sensor-conditioned data sourced from NVIDIA-owned data and publicly available, commercially permissive datasets. These datasets are curated to exclude known restricted content and to support building an Omni model that learns to generate and reason about dynamic physical environments across world reasoning and generation tasks. ## Quality Benchmark(s) We provide control inputs and text prompts in the [Cosmos3 Driving Simulation Evaluation Dataset](https://huggingface.co/datasets/nvidia/Cosmos3-Driving-Simulation-Evaluation) to support evaluation. **Cosmos3-Nano-Transfer-Auto** results use these text prompts; the Cosmos-Transfer2.5-2B Multiview baseline uses different prompts. ### Multi-view RGB quality We evaluate **Cosmos3-Nano-Transfer-Auto** at 720p in text-to-video (T2V) and image-to-video (I2V) settings, using a guidance scale of 3. [Fréchet Inception Distance (FID)](https://arxiv.org/abs/1706.08500) compares generated and reference frame-feature distributions; [Fréchet Video Distance (FVD)](https://arxiv.org/abs/1812.01717) compares video-feature distributions, with I3D and StyleGAN-V results reported separately. [DOVER](https://github.com/VQAssessment/DOVER) evaluates video quality. Temporal Sampson error measures geometric agreement between matched image points across frames. Adjacent Cosine Mean (ACM) measures within-view visual consistency by averaging CLIP-feature cosine similarities between each frame after the first and both its preceding frame and the first frame, with negative similarities clamped to zero. Scores are computed per camera video, then averaged across views and samples. The score ranges from 0 to 1; higher is better. ACM uses no ground-truth video and does not measure cross-view consistency. A frozen video can score highly, so ACM alone does not establish realistic motion. Cross-view Sampson error measures agreement across camera views, evaluated over two or five view pairs.
| Model | Frame FID ↓ | FVD (I3D) ↓ | FVD (StyleGAN-V) ↓ | DOVER technical ↑ | Temporal Sampson error ↓ | Adjacent Cosine Mean (ACM) ↑ | Cross-view Sampson error, 2 view pairs (CSE2) ↓ | Cross-view Sampson error, 5 view pairs (CSE5) ↓ |
|---|---|---|---|---|---|---|---|---|
| Cosmos-Transfer2.5-2B Multiview (Transfer-T2V, guidance 3) |
61.9 | 772 | 770 | 8.56 | 0.530 | 0.885 | 2.51 | 2.07 |
| Cosmos3-Nano-Transfer-Auto (T2V, guidance 3) |
45.5 | 488 | 487 | 9.01 | 0.473 | 0.906 | 2.01 | 1.63 |
| Cosmos-Transfer2.5-2B Multiview (Transfer-I2V, guidance 3) |
41.9 | 383 | 382 | 8.22 | 0.524 | 0.878 | 2.45 | 1.98 |
| Cosmos3-Nano-Transfer-Auto (I2V, guidance 3) |
27.8 | 238 | 237 | 8.27 | 0.472 | 0.902 | 2.25 | 1.80 |
| Model | BEV JSD, global ↓ | BEV JSD, local ↓ | BEV MMD^2 ↓ | Frechet Utonia dist ↓ | Utonia MMD^2 ↓ | Temporal CTC / real, k=1 (closer to 1) | Vehicle CTC / real (closer to 1) | Object support recall (real >=30, generated >=10 points) ↑ | Object detection F1: generated LiDAR vs GT @3m ↑ |
|---|---|---|---|---|---|---|---|---|---|
| LidarVerse (21 scans) | 0.151 | 0.400 | 0.0120 | 33.6 | 0.0636 | 1.20 | 2.51 | 0.843 | 0.201 |
| LidarVerse (116 scans) | 0.116 | 0.300 | 0.00644 | 44.8 | 0.0907 | 1.35 | 3.73 | 0.788 | 0.200 |
| Cosmos3-Nano-Transfer-Auto (T2V, guidance 3) |
0.0401 | 0.220 | 0.00347 | 12.8 | 0.0208 | 1.17 | 2.53 | 0.885 | 0.237 |