README.md

metadata

license: other
license_name: openmdw1.1-license
license_link: https://openmdw.ai/license/1-1/
library_name: cosmos
tags:
  - nvidia
  - cosmos
  - cosmos3
  - omnimodel

Interactive World Simulation with Cosmos for Physical AI

Cosmos3 Examples Collection | Code | Technical Report (coming soon) | Website

Model Overview: Cosmos3-Nano-Sim-Transfer

The highlighted checkpoint, Cosmos3-Nano-Sim-Transfer, is a post-trained, causal-distilled few-step variant of the Cosmos3-Nano base model.

Description

Depth and segmentation: Controls on top, corresponding generated RGB videos below.

Edge and blur: Controls on top, corresponding generated RGB videos below.

Cosmos3-Nano-Sim-Transfer generates photorealistic RGB video conditioned on simulator-rendered control sequences, such as depth maps, segmentation masks, edge maps, or blurred images, and a text prompt. It is designed for real-time integration with simulators such as NVIDIA Isaac Lab, generating visual observations that preserve the scene geometry and dynamics specified by the simulator controls. Policies developed in these simulators can be trained and evaluated using the generated photorealistic observations while the underlying simulation and its ground-truth state remain unchanged. The simulator remains responsible for physics, scene state, and ground-truth annotations, while the model generates visual appearance conditioned on the simulator controls. Inference throughput depends on the hardware and generation configuration.

Cosmos3-Nano-Sim-Transfer was developed by NVIDIA as part of the Cosmos example model family.

For a quick start, see the Cosmos3-Nano-Sim-Transfer skill.

This model is ready for commercial or non-commercial use.

License

This model is released under OpenMDW1.1.

Deployment Geography

Global

Use Case

Physical AI for robotics: real-time sim-to-real visual transfer through photorealistic rendering of simulated environments for reinforcement learning (RL) training and policy evaluation in classical simulators such as NVIDIA Isaac Lab.

Release Date

• Hugging Face 10/20/2026 via https://huggingface.co/collections/nvidia/cosmos3-examples
• GitHub 10/20/2026 via https://github.com/nvidia/cosmos

Model Architecture

Architecture Type: Transformer

Network Architecture: Mixture-of-Transformers (MoT)

Cosmos3-Nano-Sim-Transfer is derived from Cosmos3-Nano through post-training for controlled video generation and causal distillation. The model generates video in small latent-frame chunks conditioned on cached history, incoming simulator-rendered controls, and a text prompt. Supported controls include depth maps, segmentation masks, edge maps, and blurred images. This generation process supports closed-loop policy training and evaluation while the simulator maintains the physical state and dynamics.

This model was developed using the Cosmos Framework.

Number of trainable model parameters: 16B

Inputs

• Input Type(s): Video (control video), Text
• Input Format(s):
  • Control video: Simulator-rendered depth maps, segmentation masks, edge maps, or blurred images for each frame
  • Text: String
• Input Parameters:
  • Control video: Three-dimensional (3D: time, height, width)
  • Text: One-dimensional (1D)
• Other Properties Related to Input:
  • Designed for integration into a simulator rendering loop (for example, NVIDIA Isaac Lab), with geometry and physics maintained by the simulator and visual appearance generated by the model.
  • Designed for real-time batched inference across up to eight parallel environments on a single GB200 GPU. Inference throughput depends on the batch size and generation configuration.

Outputs

• Output Type(s): Video
• Output Format(s):
  • Video: MP4 or streamed video frames
• Output Parameters:
  • Video: Three-dimensional (3D)
• Other Properties Related to Output:
  • Generated frames are conditioned on the input scene geometry. Simulator annotations describe the underlying simulated scene; their alignment with generated imagery should be validated for the intended use case.
  • Generates up to 600 frames at 480p with an output video frame rate of 30 frames per second (fps).
  • Output video frame rates describe playback timing; inference throughput is reported separately.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g., GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration

Runtime Engine(s):

• PyTorch
• vLLM-Omni

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Blackwell
• NVIDIA Hopper

Operating System(s):

• Linux (other operating systems have not been tested).

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s)

Cosmos3-Nano-Sim-Transfer — V1.0 (10/20/2026)

Cosmos3-Nano-Sim-Transfer is a post-trained, causal-distilled variant of Cosmos3-Nano that supports few-step generation.

It can be integrated into a classical simulation pipeline by taking simulator-rendered control sequences as input and generating photorealistic video observations for reinforcement learning and policy evaluation, while the simulator remains responsible for physics and ground-truth state.

Training and Evaluation Datasets

Post-Training Datasets

Data Modality: Video and Text.

Post-Training Data Size:

• Text: 1 Billion to 10 Trillion Tokens
• Video: Less than 10,000 Hours

Data Collection Method: Hybrid: Automatic/Sensors, Synthetic, Automated.

Labeling Method: Hybrid: Manually-Labeled, Automated.

Properties: The post-training dataset consists of video clips with English captions and associated visual control annotations. The data includes real and synthetic content from an NVIDIA-curated dataset.

Dataset Overview

• Total Size: 1.3B data points
• Total Number of Datasets: 393 dataset entries
• Dataset partition: Training [100%], Testing [N/A — evaluation benchmarks used separately], Validation [N/A — evaluation benchmarks used separately]
• Time period for training data collection: 2024–2026
• Time period for validation data collection: N/A (standard public benchmarks)

Raw data from internal and external sources is transformed into training-ready data through multiple stages of curation, filtering, and quality review. Data acquisition spans diverse multimodal sources — robotics, autonomous driving, industrial environments, indoor and outdoor scenes, varied lighting and weather conditions, camera viewpoints, object categories, and human activities — to broaden coverage across Physical AI operating environments. Automated filtering pipelines remove corrupted, duplicate, low-quality, and restricted content. Metadata analysis, heuristic rules, and model-assisted classifiers are applied during preprocessing to flag anomalous distributions and low-diversity subsets. Human review supplements automated filtering for selected datasets, benchmark construction, and targeted quality analysis. Datasets are balanced across modalities and task categories — visual reasoning, text-to-image, text-to-video, image-to-video, audio generation, video transfer, action-conditioned generation, and action command generation — to reduce overrepresentation of narrow domains. Synthetic and simulation-based augmentation supplements coverage of rare physical interactions and edge-case scenarios. Deduplication and provenance tracking are applied across the corpus. The resulting processed data is converted into model-ready tokenized or encoded representations through modality-specific preprocessors before training begins.

Training datasets passed through multiple layers of automated and manual safeguards designed to reduce the presence of harmful or policy-violating content across categories including weapons and weapons-related instructional content, criminal planning, child sexual abuse material (CSAM), non-consensual intimate imagery (NCII), sexual content involving minors, harassment, hate speech, profanity, threats and incitement to violence, self-harm or suicide-related content, and graphic violence. Data sources are reviewed for licensing compatibility, provenance, and alignment with internal data governance and safety policies before admission into training corpora. Automated filtering pipelines combine multiple detection strategies: hash-matching against known CSAM and NCII reference databases; classifier-based moderation models trained for explicit sexual content, hate speech, violence, weapons imagery, and other restricted categories; keyword and regex-based screening for criminal-planning, threats, and self-harm phrases in text data; metadata and provenance heuristics for source-level risk signals; and embedding-based anomaly detection to surface samples that fall outside expected distributions. Human review and targeted audits supplement automated filtering for selected datasets, benchmark construction, and safety-sensitive evaluation. For multimodal Physical AI data (robotics, autonomous driving, industrial scenes), additional filtering targets invalid action trajectories, physically implausible interactions, and unsafe control sequences. Synthetic and simulation-generated data are evaluated through internal validation before inclusion. Benchmark evaluations and red-team testing are applied post-training to surface remaining safety gaps across world generation, reasoning, audio, and action tasks. No large-scale data-filtering process can guarantee complete removal of all harmful content; residual risks may remain, particularly in rare edge cases or open-world deployment settings. Ongoing monitoring and dataset review continue post-release.

For more information about the datasets used to train this model, please see the Public Summary of Training Content.

Data Modality and Training Data Size

Modality	Reasoning Data Sample Count	Generation Data Sample Count
Text	22M	Not Applicable
Image	19M	767M
Video	1M	348M
Audio	Not Applicable	139M
Action	Not Applicable	7M

Data Collection Method by dataset

• Hybrid: Automatic/Sensors, Synthetic, Automated

Labeling Method by dataset

• Hybrid: Manually-Labeled, Automated

Properties: The training, testing, and evaluation datasets consist of diverse multimodal video, image, audio, action, synthetic, and sensor-conditioned data sourced from NVIDIA-owned data and publicly available, commercially permissive datasets. These datasets are curated to exclude known restricted content and to support building an Omni model that learns to generate and reason about dynamic physical environments across world reasoning and generation tasks.

Public Datasets

Dataset	Samples
OpenImage	1.2M
Coyo700M	100M
YouTube Video	340M
UMI	4.5M

Private Datasets

Dataset	Samples
Egocentric	7M
Nexar	0.6M
AgiBot	0.2M
HOI	0.3M

Synthetic Datasets

Dataset	Samples
synthetic images generated using HiDream-I1	15M
synthetic images generated using Qwen-Image-2512	14M
synthetic captions generated using Qwen3-VL	1115M

Evaluation Datasets

Data Collection Method by dataset

• Hybrid: Automatic/Sensors, Synthetic, Automated

Labeling Method by dataset

• Hybrid: Manually-Labeled, Automated

Properties: The training, testing, and evaluation datasets consist of diverse multimodal video, image, audio, action, synthetic, and sensor-conditioned data sourced from NVIDIA-owned data and publicly available, commercially permissive datasets. These datasets are curated to exclude known restricted content and to support building an Omni model that learns to generate and reason about dynamic physical environments across world reasoning and generation tasks.

Quality Benchmark(s)

PAI-Bench-C (TransferBench)

We compare Cosmos3-Nano-Sim-Transfer with Cosmos3-Nano on PAI-Bench-C (TransferBench) using DOVER to estimate video quality and four control-following metrics to evaluate control fidelity. We map generated videos into each control modality and compare the resulting representations with the supplied controls. Segmentation mean intersection over union (mIoU) measures mask overlap; blur structural similarity (SSIM) measures agreement after blurring; edge F1 combines precision and recall of edge agreement; and depth scale-invariant root mean squared error (SI-RMSE) measures depth differences after scale normalization.

Model	DOVER ↑	Segmentation mIoU ↑	Blur SSIM ↑	Edge F1 ↑	Depth SI-RMSE ↓
Cosmos3-Nano	10.359606	0.643263	0.855906	0.214024	0.842346
Cosmos3-Nano-Sim-Transfer	10.175942	0.73776	0.88665	0.340696	0.619426

Prompt following and temporal consistency

We report VLM-judge scores for prompt following and video quality. Perceptual drift evaluates changes in video quality over time (PAI-Bench-G). Temporal Sampson error evaluates temporal consistency by measuring geometric agreement between matched image points across frames.

Model	Prompt Following ↑	Video Quality ↑	Perceptual Drift ↓	Temporal Sampson error ↓
Cosmos3-Nano	7.184583	4.929583	0.006349	0.081045
Cosmos3-Nano-Sim-Transfer	7.025417	4.950833	0.006504	0.096544

Inference

Acceleration Engine: PyTorch, vLLM-Omni

Test Hardware: NVIDIA Blackwell GB200, NVIDIA Hopper H100, H200, H20, RTX PRO 6000

Inference Performance

Numbers are based on vLLM-Omni, using a single GPU. Empty table cells indicate that measurements are not reported.

	480p inference throughput (frames/s)
GB200	24.5
H100	14.6
H200	14.9
H20	4
RTX Pro 6000	8.3

Usage Example

Quick Start

This example serves the model with vLLM-Omni and generates a 600-frame depth-controlled video at 30 FPS and 640 × 640. The control video and prompt are published in assets/sample_data/example1.

1. Start the server

On a Linux host with an NVIDIA GPU (tested on one GB200), Docker, and the NVIDIA Container Toolkit:

docker run --runtime nvidia --gpus all --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v "$PWD:$PWD:ro" \
  --env "HF_TOKEN=$HF_TOKEN" \
  --env VLLM_OMNI_VIDEO_SYNC_TIMEOUT=7200 \
  -p 127.0.0.1:8000:8000 \
  vllm/vllm-omni:cosmos3 \
  vllm serve nvidia/Cosmos3-Nano-Sim-Transfer \
    --omni --model-class-name Cosmos3NanoSimTransferPipeline \
    --deploy-config /app/vllm-omni/vllm_omni/deploy/cosmos3_nano_sim_transfer.yaml \
    --no-guardrails \
    --host 0.0.0.0 --port 8000 --init-timeout 1800

The container publishes the server on 127.0.0.1:8000 of the host. --deploy-config selects the Sim-Transfer pipeline. --no-guardrails disables the Cosmos3 text and video safety checks. The first start downloads the checkpoint.

Default serving is eager. For faster repeated runs, enable regional compilation and use_cuda_graphs, and set vae_fast_path and vae_encode_fast_path to channels_last. The first request will be slower while these warm up.

For optimized server settings and more examples with depth, edge, blur, and segmentation controls, see the Sim-Transfer vLLM-Omni notebook.

2. Generate the depth video

Generate a 600-frame depth-controlled video using the supplied prompt and seed. The example uses a 30-frame history window with three attention sinks.

pip install requests huggingface_hub

import json
from pathlib import Path
from uuid import uuid4

import requests
from huggingface_hub import snapshot_download

MODEL = "nvidia/Cosmos3-Nano-Sim-Transfer"
example_dir = Path(snapshot_download(
    MODEL,
    allow_patterns=[
        "assets/sample_data/example1/example1_depth.mp4",
        "assets/sample_data/example1/example1_upsampled_prompt.json",
    ],
    local_dir=Path("Cosmos3-Nano-Sim-Transfer").resolve(),
)) / "assets/sample_data/example1"
metadata = json.loads((example_dir / "example1_upsampled_prompt.json").read_text())

extra_params = {
    "session_id": uuid4().hex,
    "reset": True,
    "close_session": True,
    "resolution": "480",
    "max_prompt_tokens": 4096,
    "kv_cache_inference_size": 30,
    "attention_sink_size": 3,
    "emphasize_control_in_prompt": False,
    "depth": {"control_path": str(example_dir / "example1_depth.mp4")},
}
response = requests.post(
    "http://127.0.0.1:8000/v1/videos/sync",
    data={
        "model": MODEL,
        "prompt": metadata["prompt"],
        "size": "640x640",
        "num_frames": "600",
        "fps": "30",
        "seed": str(metadata["seed"]),
        "extra_params": json.dumps(extra_params),
    },
    headers={"Accept": "video/mp4"},
    timeout=7260,
)
response.raise_for_status()
Path("robot_depth.mp4").write_bytes(response.content)

With the supplied depth control, the generated output looks like this:

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.

Users are responsible for model inputs and outputs. Users are responsible for ensuring safe integration of this model, including implementing guardrails as well as other safety mechanisms, prior to deployment.

For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy subcards. Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.