| --- |
| license: other |
| license_name: openmdw1.1-license |
| license_link: >- |
| https://openmdw.ai/license/1-1/ |
| library_name: cosmos |
| tags: |
| - nvidia |
| - cosmos |
| - cosmos3 |
| - omnimodel |
| --- |
| |
| # **Interactive World Simulation with Cosmos for Physical AI** |
|
|
| **[Cosmos3 Examples Collection](https://huggingface.co/collections/nvidia/cosmos3-examples)** | **[Code](https://github.com/nvidia/cosmos)** | **[Technical Report](https://research.nvidia.com/labs/cosmos-lab/cosmos-simulation/technical-report.pdf)** (coming soon) | **[Website](https://research.nvidia.com/labs/cosmos-lab/cosmos-simulation/)** |
|
|
| # Model Overview: Cosmos3-Nano-Sim-Transfer |
|
|
|  |
|
|
| The highlighted checkpoint, Cosmos3-Nano-Sim-Transfer, is a post-trained, causal-distilled few-step variant of the Cosmos3-Nano base model. |
|
|
| ## Description |
|
|
| **Depth and segmentation:** Controls on top, corresponding generated RGB videos below. |
|
|
| <p align="center"> |
| <img src="images/cosmos3-nano-sim-transfer-depth-seg.gif" alt="Depth control (top left), segmentation control (top right), and corresponding generated RGB videos below"> |
| </p> |
|
|
| **Edge and blur:** Controls on top, corresponding generated RGB videos below. |
|
|
| <p align="center"> |
| <img src="images/cosmos3-nano-sim-transfer-edge-blur.gif" alt="Edge control (top left), blur control (top right), and corresponding generated RGB videos below"> |
| </p> |
|
|
| Cosmos3-Nano-Sim-Transfer generates photorealistic RGB video conditioned on simulator-rendered control sequences, such as depth maps, segmentation masks, edge maps, or blurred images, and a text prompt. It is designed for real-time integration with simulators such as NVIDIA Isaac Lab, generating visual observations that preserve the scene geometry and dynamics specified by the simulator controls. Policies developed in these simulators can be trained and evaluated using the generated photorealistic observations while the underlying simulation and its ground-truth state remain unchanged. The simulator remains responsible for physics, scene state, and ground-truth annotations, while the model generates visual appearance conditioned on the simulator controls. Inference throughput depends on the hardware and generation configuration. |
|
|
| Cosmos3-Nano-Sim-Transfer was developed by NVIDIA as part of the Cosmos example model family. |
|
|
| For a quick start, see the [Cosmos3-Nano-Sim-Transfer skill](https://github.com/NVIDIA/cosmos-framework/tree/main/.agents/skills/cosmos3-nano-sim-transfer). |
|
|
| This model is ready for commercial or non-commercial use. |
|
|
| ## License |
|
|
| This model is released under [OpenMDW1.1](https://openmdw.ai/license/1-1/). |
|
|
| ## Deployment Geography |
|
|
| Global |
|
|
| ## Use Case |
|
|
| Physical AI for robotics: real-time sim-to-real visual transfer through photorealistic rendering of simulated environments for reinforcement learning (RL) training and policy evaluation in classical simulators such as NVIDIA Isaac Lab. |
|
|
| ## Release Date |
|
|
| - Hugging Face 10/20/2026 via [https://huggingface.co/collections/nvidia/cosmos3-examples](https://huggingface.co/collections/nvidia/cosmos3-examples) |
| - GitHub 10/20/2026 via [https://github.com/nvidia/cosmos](https://github.com/nvidia/cosmos) |
|
|
| ## Model Architecture |
|
|
| **Architecture Type:** Transformer |
|
|
| **Network Architecture:** Mixture-of-Transformers (MoT) |
|
|
| Cosmos3-Nano-Sim-Transfer is derived from [Cosmos3-Nano](https://huggingface.co/nvidia/Cosmos3-Nano) through post-training for controlled video generation and causal distillation. The model generates video in small latent-frame chunks conditioned on cached history, incoming simulator-rendered controls, and a text prompt. Supported controls include depth maps, segmentation masks, edge maps, and blurred images. This generation process supports closed-loop policy training and evaluation while the simulator maintains the physical state and dynamics. |
|
|
| This model was developed using the [Cosmos Framework](https://github.com/NVIDIA/cosmos-framework). |
|
|
| **Number of trainable model parameters:** 16B |
|
|
| ## Inputs |
|
|
| - **Input Type(s):** Video (control video), Text |
| - **Input Format(s):** |
| - **Control video:** Simulator-rendered depth maps, segmentation masks, edge maps, or blurred images for each frame |
| - **Text:** String |
| - **Input Parameters:** |
| - **Control video:** Three-dimensional (3D: time, height, width) |
| - **Text:** One-dimensional (1D) |
| - **Other Properties Related to Input:** |
| - Designed for integration into a simulator rendering loop (for example, NVIDIA Isaac Lab), with geometry and physics maintained by the simulator and visual appearance generated by the model. |
| - Designed for real-time batched inference across up to eight parallel environments on a single GB200 GPU. Inference throughput depends on the batch size and generation configuration. |
|
|
| ## Outputs |
|
|
| - **Output Type(s):** Video |
| - **Output Format(s):** |
| - **Video:** MP4 or streamed video frames |
| - **Output Parameters:** |
| - **Video:** Three-dimensional (3D) |
| - **Other Properties Related to Output:** |
| - Generated frames are conditioned on the input scene geometry. Simulator annotations describe the underlying simulated scene; their alignment with generated imagery should be validated for the intended use case. |
| - Generates up to 600 frames at 480p with an output video frame rate of 30 frames per second (fps). |
| - Output video frame rates describe playback timing; inference throughput is reported separately. |
|
|
| Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g., GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions. |
|
|
| ## Software Integration |
|
|
| **Runtime Engine(s):** |
|
|
| - [PyTorch](https://pytorch.org/) |
| - [vLLM-Omni](https://github.com/vllm-project/vllm-omni) |
|
|
| **Supported Hardware Microarchitecture Compatibility:** |
|
|
| - NVIDIA Blackwell |
| - NVIDIA Hopper |
|
|
| **Operating System(s):** |
|
|
| - Linux (other operating systems have not been tested). |
|
|
| The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment. |
|
|
| ## Model Version(s) |
|
|
| **Cosmos3-Nano-Sim-Transfer — V1.0 (10/20/2026)** |
|
|
| Cosmos3-Nano-Sim-Transfer is a post-trained, causal-distilled variant of [Cosmos3-Nano](https://huggingface.co/nvidia/Cosmos3-Nano) that supports few-step generation. |
|
|
| It can be integrated into a classical simulation pipeline by taking simulator-rendered control sequences as input and generating photorealistic video observations for reinforcement learning and policy evaluation, while the simulator remains responsible for physics and ground-truth state. |
|
|
| ## Training and Evaluation Datasets |
|
|
| ### Post-Training Datasets |
|
|
| **Data Modality:** Video and Text. |
|
|
| **Post-Training Data Size:** |
| - **Text:** 1 Billion to 10 Trillion Tokens |
| - **Video:** Less than 10,000 Hours |
|
|
| **Data Collection Method:** Hybrid: Automatic/Sensors, Synthetic, Automated. |
|
|
| **Labeling Method:** Hybrid: Manually-Labeled, Automated. |
|
|
| **Properties:** The post-training dataset consists of video clips with English captions and associated visual control annotations. The data includes real and synthetic content from an NVIDIA-curated dataset. |
|
|
| ### Dataset Overview |
|
|
| - **Total Size:** 1.3B data points |
| - **Total Number of Datasets:** 393 dataset entries |
| - **Dataset partition:** Training [100%], Testing [N/A — evaluation benchmarks used separately], Validation [N/A — evaluation benchmarks used separately] |
| - **Time period for training data collection:** 2024–2026 |
| - **Time period for validation data collection:** N/A (standard public benchmarks) |
|
|
| Raw data from internal and external sources is transformed into training-ready data through multiple stages of curation, filtering, and quality review. Data acquisition spans diverse multimodal sources — robotics, autonomous driving, industrial environments, indoor and outdoor scenes, varied lighting and weather conditions, camera viewpoints, object categories, and human activities — to broaden coverage across Physical AI operating environments. Automated filtering pipelines remove corrupted, duplicate, low-quality, and restricted content. Metadata analysis, heuristic rules, and model-assisted classifiers are applied during preprocessing to flag anomalous distributions and low-diversity subsets. Human review supplements automated filtering for selected datasets, benchmark construction, and targeted quality analysis. Datasets are balanced across modalities and task categories — visual reasoning, text-to-image, text-to-video, image-to-video, audio generation, video transfer, action-conditioned generation, and action command generation — to reduce overrepresentation of narrow domains. Synthetic and simulation-based augmentation supplements coverage of rare physical interactions and edge-case scenarios. Deduplication and provenance tracking are applied across the corpus. The resulting processed data is converted into model-ready tokenized or encoded representations through modality-specific preprocessors before training begins. |
|
|
| Training datasets passed through multiple layers of automated and manual safeguards designed to reduce the presence of harmful or policy-violating content across categories including weapons and weapons-related instructional content, criminal planning, child sexual abuse material (CSAM), non-consensual intimate imagery (NCII), sexual content involving minors, harassment, hate speech, profanity, threats and incitement to violence, self-harm or suicide-related content, and graphic violence. Data sources are reviewed for licensing compatibility, provenance, and alignment with internal data governance and safety policies before admission into training corpora. Automated filtering pipelines combine multiple detection strategies: hash-matching against known CSAM and NCII reference databases; classifier-based moderation models trained for explicit sexual content, hate speech, violence, weapons imagery, and other restricted categories; keyword and regex-based screening for criminal-planning, threats, and self-harm phrases in text data; metadata and provenance heuristics for source-level risk signals; and embedding-based anomaly detection to surface samples that fall outside expected distributions. Human review and targeted audits supplement automated filtering for selected datasets, benchmark construction, and safety-sensitive evaluation. For multimodal Physical AI data (robotics, autonomous driving, industrial scenes), additional filtering targets invalid action trajectories, physically implausible interactions, and unsafe control sequences. Synthetic and simulation-generated data are evaluated through internal validation before inclusion. Benchmark evaluations and red-team testing are applied post-training to surface remaining safety gaps across world generation, reasoning, audio, and action tasks. No large-scale data-filtering process can guarantee complete removal of all harmful content; residual risks may remain, particularly in rare edge cases or open-world deployment settings. Ongoing monitoring and dataset review continue post-release. |
|
|
| For more information about the datasets used to train this model, please see the [Public Summary of Training Content](https://docs.nvidia.com/cosmos/latest/cosmos3/index.html). |
|
|
| **Data Modality and Training Data Size** |
|
|
| | Modality | Reasoning Data Sample Count | Generation Data Sample Count | |
| | -------- | ------------------- | -------------------- | |
| | Text | 22M | Not Applicable | |
| | Image | 19M | 767M | |
| | Video | 1M | 348M | |
| | Audio | Not Applicable | 139M | |
| | Action | Not Applicable | 7M | |
|
|
| **Data Collection Method by dataset** |
|
|
| - Hybrid: Automatic/Sensors, Synthetic, Automated |
|
|
| **Labeling Method by dataset** |
|
|
| - Hybrid: Manually-Labeled, Automated |
|
|
| **Properties:** The training, testing, and evaluation datasets consist of diverse multimodal video, image, audio, action, synthetic, and sensor-conditioned data sourced from NVIDIA-owned data and publicly available, commercially permissive datasets. These datasets are curated to exclude known restricted content and to support building an Omni model that learns to generate and reason about dynamic physical environments across world reasoning and generation tasks. |
|
|
| ### Public Datasets |
|
|
| | Dataset | Samples | |
| |---------------------------|-------------| |
| | OpenImage | 1.2M | |
| | Coyo700M | 100M | |
| | YouTube Video | 340M | |
| | UMI | 4.5M | |
|
|
| ### Private Datasets |
|
|
| | Dataset | Samples | |
| |---------------------------|-------------| |
| | Egocentric | 7M | |
| | Nexar | 0.6M | |
| | AgiBot | 0.2M | |
| | HOI | 0.3M | |
|
|
| ### Synthetic Datasets |
|
|
| | Dataset | Samples | |
| |-----------------------------------------|-------------| |
| | synthetic images generated using HiDream-I1 | 15M | |
| | synthetic images generated using Qwen-Image-2512 | 14M | |
| | synthetic captions generated using Qwen3-VL | 1115M | |
|
|
| ## Evaluation Datasets |
|
|
| **Data Collection Method by dataset** |
|
|
| - Hybrid: Automatic/Sensors, Synthetic, Automated |
|
|
| **Labeling Method by dataset** |
|
|
| - Hybrid: Manually-Labeled, Automated |
|
|
| **Properties:** The training, testing, and evaluation datasets consist of diverse multimodal video, image, audio, action, synthetic, and sensor-conditioned data sourced from NVIDIA-owned data and publicly available, commercially permissive datasets. These datasets are curated to exclude known restricted content and to support building an Omni model that learns to generate and reason about dynamic physical environments across world reasoning and generation tasks. |
|
|
| ## Quality Benchmark(s) |
|
|
| ### [PAI-Bench-C](https://arxiv.org/abs/2512.01989) (TransferBench) |
|
|
| We compare **Cosmos3-Nano-Sim-Transfer** with Cosmos3-Nano on [PAI-Bench-C](https://arxiv.org/abs/2512.01989) (TransferBench) using [DOVER](https://github.com/VQAssessment/DOVER) to estimate video quality and four control-following metrics to evaluate control fidelity. We map generated videos into each control modality and compare the resulting representations with the supplied controls. Segmentation mean intersection over union (mIoU) measures mask overlap; blur structural similarity (SSIM) measures agreement after blurring; edge F1 combines precision and recall of edge agreement; and depth scale-invariant root mean squared error (SI-RMSE) measures depth differences after scale normalization. |
|
|
| <table> |
| <thead> |
| <tr> |
| <th scope="col" style="text-align: left;">Model</th> |
| <th scope="col" style="text-align: right;">DOVER ↑</th> |
| <th scope="col" style="text-align: right;">Segmentation mIoU ↑</th> |
| <th scope="col" style="text-align: right;">Blur SSIM ↑</th> |
| <th scope="col" style="text-align: right;">Edge F1 ↑</th> |
| <th scope="col" style="text-align: right;">Depth SI-RMSE ↓</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td style="text-align: left;">Cosmos3-Nano</td> |
| <td style="text-align: right;"><strong>10.359606</strong></td> |
| <td style="text-align: right;">0.643263</td> |
| <td style="text-align: right;">0.855906</td> |
| <td style="text-align: right;">0.214024</td> |
| <td style="text-align: right;">0.842346</td> |
| </tr> |
| <tr> |
| <td style="text-align: left; background-color: #e8f4df; color: #18202b;"><strong style="color: inherit;">Cosmos3-Nano-Sim-Transfer</strong></td> |
| <td style="text-align: right; background-color: #e8f4df; color: #18202b;">10.175942</td> |
| <td style="text-align: right; background-color: #e8f4df; color: #18202b;"><strong style="color: inherit;">0.73776</strong></td> |
| <td style="text-align: right; background-color: #e8f4df; color: #18202b;"><strong style="color: inherit;">0.88665</strong></td> |
| <td style="text-align: right; background-color: #e8f4df; color: #18202b;"><strong style="color: inherit;">0.340696</strong></td> |
| <td style="text-align: right; background-color: #e8f4df; color: #18202b;"><strong style="color: inherit;">0.619426</strong></td> |
| </tr> |
| </tbody> |
| </table> |
| |
| ### Prompt following and temporal consistency |
|
|
| We report VLM-judge scores for prompt following and video quality. Perceptual drift evaluates changes in video quality over time ([PAI-Bench-G](https://arxiv.org/abs/2512.01989)). Temporal Sampson error evaluates temporal consistency by measuring geometric agreement between matched image points across frames. |
|
|
| <table> |
| <thead> |
| <tr> |
| <th scope="col" style="text-align: left;">Model</th> |
| <th scope="col" style="text-align: right;">Prompt Following ↑</th> |
| <th scope="col" style="text-align: right;">Video Quality ↑</th> |
| <th scope="col" style="text-align: right;">Perceptual Drift ↓</th> |
| <th scope="col" style="text-align: right;">Temporal Sampson error ↓</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td style="text-align: left;">Cosmos3-Nano</td> |
| <td style="text-align: right;"><strong>7.184583</strong></td> |
| <td style="text-align: right;">4.929583</td> |
| <td style="text-align: right;"><strong>0.006349</strong></td> |
| <td style="text-align: right;"><strong>0.081045</strong></td> |
| </tr> |
| <tr> |
| <td style="text-align: left; background-color: #e8f4df; color: #18202b;"><strong style="color: inherit;">Cosmos3-Nano-Sim-Transfer</strong></td> |
| <td style="text-align: right; background-color: #e8f4df; color: #18202b;">7.025417</td> |
| <td style="text-align: right; background-color: #e8f4df; color: #18202b;"><strong style="color: inherit;">4.950833</strong></td> |
| <td style="text-align: right; background-color: #e8f4df; color: #18202b;">0.006504</td> |
| <td style="text-align: right; background-color: #e8f4df; color: #18202b;">0.096544</td> |
| </tr> |
| </tbody> |
| </table> |
| |
| ## Inference |
|
|
| **Acceleration Engine:** [PyTorch](https://pytorch.org/), [vLLM-Omni](https://github.com/vllm-project/vllm-omni) |
|
|
| **Test Hardware:** NVIDIA Blackwell GB200, NVIDIA Hopper H100, H200, H20, RTX PRO 6000 |
|
|
| ### Inference Performance |
|
|
| Numbers are based on vLLM-Omni, using a single GPU. Empty table cells indicate that measurements are not reported. |
|
|
| | | 480p inference throughput (frames/s) | |
| | --- | --- | |
| | GB200 | 24.5 | |
| | H100 | 14.6 | |
| | H200 | 14.9 | |
| | H20 | 4 | |
| | RTX Pro 6000 | 8.3 | |
|
|
| ## Usage Example |
|
|
| ### Quick Start |
|
|
| This example serves the model with [vLLM-Omni](https://github.com/vllm-project/vllm-omni) |
| and generates a 600-frame depth-controlled video at 30 FPS and 640 × 640. The |
| control video and prompt are published in [`assets/sample_data/example1`](https://huggingface.co/nvidia/Cosmos3-Nano-Sim-Transfer/tree/main/assets/sample_data/example1). |
|
|
| #### 1. Start the server |
|
|
| On a Linux host with an NVIDIA GPU (tested on one GB200), [Docker](https://docs.docker.com/engine/install/), and the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html): |
|
|
| ```bash |
| docker run --runtime nvidia --gpus all --ipc=host \ |
| -v ~/.cache/huggingface:/root/.cache/huggingface \ |
| -v "$PWD:$PWD:ro" \ |
| --env "HF_TOKEN=$HF_TOKEN" \ |
| --env VLLM_OMNI_VIDEO_SYNC_TIMEOUT=7200 \ |
| -p 127.0.0.1:8000:8000 \ |
| vllm/vllm-omni:cosmos3 \ |
| vllm serve nvidia/Cosmos3-Nano-Sim-Transfer \ |
| --omni --model-class-name Cosmos3NanoSimTransferPipeline \ |
| --deploy-config /app/vllm-omni/vllm_omni/deploy/cosmos3_nano_sim_transfer.yaml \ |
| --no-guardrails \ |
| --host 0.0.0.0 --port 8000 --init-timeout 1800 |
| ``` |
|
|
| The container publishes the server on `127.0.0.1:8000` of the host. |
| `--deploy-config` selects the Sim-Transfer pipeline. |
| `--no-guardrails` disables the Cosmos3 text and video safety checks. The first |
| start downloads the checkpoint. |
|
|
| Default serving is eager. For faster repeated runs, enable regional compilation |
| and `use_cuda_graphs`, and set `vae_fast_path` and `vae_encode_fast_path` to |
| `channels_last`. The first request will be slower while these warm up. |
|
|
| For optimized server settings and more examples with depth, edge, blur, and |
| segmentation controls, see the |
| [Sim-Transfer vLLM-Omni notebook](https://github.com/NVIDIA/cosmos-framework/blob/main/.agents/skills/cookbooks/cosmos3/simulation/transfer/run_transfer_with_vllm_omni.ipynb). |
|
|
| #### 2. Generate the depth video |
|
|
| Generate a 600-frame depth-controlled video using the supplied prompt and seed. |
| The example uses a 30-frame history window with three attention sinks. |
|
|
| ```bash |
| pip install requests huggingface_hub |
| ``` |
|
|
| ```python |
| import json |
| from pathlib import Path |
| from uuid import uuid4 |
| |
| import requests |
| from huggingface_hub import snapshot_download |
| |
| MODEL = "nvidia/Cosmos3-Nano-Sim-Transfer" |
| example_dir = Path(snapshot_download( |
| MODEL, |
| allow_patterns=[ |
| "assets/sample_data/example1/example1_depth.mp4", |
| "assets/sample_data/example1/example1_upsampled_prompt.json", |
| ], |
| local_dir=Path("Cosmos3-Nano-Sim-Transfer").resolve(), |
| )) / "assets/sample_data/example1" |
| metadata = json.loads((example_dir / "example1_upsampled_prompt.json").read_text()) |
| |
| extra_params = { |
| "session_id": uuid4().hex, |
| "reset": True, |
| "close_session": True, |
| "resolution": "480", |
| "max_prompt_tokens": 4096, |
| "kv_cache_inference_size": 30, |
| "attention_sink_size": 3, |
| "emphasize_control_in_prompt": False, |
| "depth": {"control_path": str(example_dir / "example1_depth.mp4")}, |
| } |
| response = requests.post( |
| "http://127.0.0.1:8000/v1/videos/sync", |
| data={ |
| "model": MODEL, |
| "prompt": metadata["prompt"], |
| "size": "640x640", |
| "num_frames": "600", |
| "fps": "30", |
| "seed": str(metadata["seed"]), |
| "extra_params": json.dumps(extra_params), |
| }, |
| headers={"Accept": "video/mp4"}, |
| timeout=7260, |
| ) |
| response.raise_for_status() |
| Path("robot_depth.mp4").write_bytes(response.content) |
| ``` |
|
|
| With the supplied depth control, the generated output looks like this: |
|
|
| <video src="https://huggingface.co/nvidia/Cosmos3-Nano-Sim-Transfer/resolve/main/assets/output/cosmos3-nano-sim-transfer-quickstart.mp4" controls autoplay loop muted playsinline width="640"></video> |
|
|
| ## Ethical Considerations |
|
|
| NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. |
|
|
| Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included. |
|
|
| Users are responsible for model inputs and outputs. Users are responsible for ensuring safe integration of this model, including implementing guardrails as well as other safety mechanisms, prior to deployment. |
|
|
| For more detailed information on ethical considerations for this model, please see the Model Card++ [Explainability](explainability-Cosmos3-Nano-Sim-Transfer.md), [Bias](bias.md), [Safety & Security](safety.md), and [Privacy](privacy.md) subcards. Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/). |
|
|
