Add any-to-any pipeline tag, update library metadata and link paper

#16
by nielsr HF Staff - opened
README.md CHANGED
@@ -1,27 +1,62 @@
1
  ---
 
2
  license: other
3
  license_name: openmdw1.1-license
4
- license_link: >-
5
- https://openmdw.ai/license/1-1/
6
- library_name: cosmos
7
  tags:
8
- - nvidia
9
- - cosmos
10
- - cosmos3
11
- - vllm
12
- - vllm-omni
13
- - sglang
14
- - sglang-diffusion
15
- - diffusers
16
- - text, image, video, audio, and action generation
17
- - omnimodel
18
  ---
19
 
20
  # **Cosmos 3: Omnimodal World Models for Physical AI**
21
- **[Model Collection](https://huggingface.co/collections/nvidia/cosmos3)** | **[Code](https://github.com/nvidia/cosmos)** | **[White Paper](https://research.nvidia.com/labs/cosmos-lab/cosmos3/technical-report.pdf)** | **[Website](https://research.nvidia.com/labs/cosmos-lab/cosmos3/)**
22
 
23
  [NVIDIA Cosmos™](https://github.com/nvidia/cosmos) is a world foundation model platform designed to accelerate the development of Physical AI by enabling machines to understand, simulate, and interact with the physical world across robotics, autonomous driving, and smart space environments, including industrial and factory-scale applications.
24
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
25
  # Model Overview: Cosmos3-Nano
26
 
27
  ## Description
@@ -36,155 +71,52 @@ This model is ready for commercial and non-commercial use.
36
  - Cosmos3-Nano:
37
  - Given multimodal inputs including text, images, video, audio, and action trajectories, generate coherent text, images, video, audio, and action outputs for multimodal understanding, world simulation, future prediction, action reasoning, and Physical AI applications.
38
 
39
- ### License
40
-
41
- This model is released under the [OpenMDW1.1](https://openmdw.ai/license/1-1/)
42
-
43
- ### Deployment Geography
44
 
45
- Global
 
46
 
47
- ### Use Case
 
48
 
49
- Physical AI: Encompassing robotics, autonomous vehicles (AV), and smart space environments, including industrial and factory-scale applications.
 
50
 
51
- ### Release Date
52
 
53
- Hugging Face 05/31/2026 via [https://huggingface.co/collections/nvidia/cosmos3](https://huggingface.co/collections/nvidia/cosmos3)
54
- GitHub 05/31/2026 via [https://github.com/nvidia/cosmos](https://github.com/nvidia/cosmos)
55
 
56
  ## Model Architecture
57
 
58
  **Architecture Type:** Transformer
59
-
60
  **Network Architecture:** Mixture-of-Transformers (MoT)
61
 
62
- Cosmos3 is an Omni-modal foundation model built on a Mixture-of-Transformers (MoT) architecture consisting of two complementary transformer towers: an autoregressive transformer for discrete token generation and a diffusion transformer for continuous multimodal generation. During inference, text is generated through standard next-token autoregressive decoding, while non-text modalities, such as images, video, audio, and actions, are synthesized through iterative denoising. This unified architecture enables Cosmos3 to model heterogeneous modalities within a single framework while preserving generation mechanisms best suited to each modality.
63
-
64
- **This model was developed based on:** [Cosmos Framework](https://github.com/nvidia/cosmos-framework)
65
 
66
  **Number of trainable model parameters:**
67
-
68
  - Cosmos3-Nano: 16B
69
-
70
- ## Input/Output Specifications
71
-
72
- - **Generator Input**
73
- - **Input Type(s)**: Text, Image, Video (with audio or without audio), Action Trajectory
74
- - **Input Format(s)**:
75
- - Text: String
76
- - Image: jpg, png, jpeg, webp
77
- - Video (with or without audio): mp4
78
- - Action: json (1D list)
79
- - **Input Parameters**:
80
- - Text: One-dimensional (1D)
81
- - Image: Two-dimensional (2D)
82
- - Video: Three-dimensional (3D)
83
- - Audio: One-dimensional (1D)
84
- - Action trajectory: One-dimensional (1D)
85
- - **Other Properties Related to Input**:
86
- - For video inputs, we accept various resolutions, including 720p, 480p, and 256p.
87
- - When using input video with audio muxed into the video MP4 file, the audio should have 2 channels (stereo) and a 48 kHz sample rate.
88
- - Image and video inputs are RGB color (8 bits per channel, sRGB color space); grayscale inputs are not supported.
89
- - Action input is a per-frame sequence of robot/agent state or control values (e.g., joint positions, gripper state, camera pose). The full input is a 2D array shaped (T, D), where T is the number of frames and D is the embodiment-specific dimensionality listed below.
90
- - Input action is only supported for compatible embodiments, including general camera motion (9D), autonomous vehicle (9D), egocentric motion (57D), single Franka Panda arm with RobotiQ gripper (10D), dual Franka Panda arm with RobotiQ gripper (20D), Agibot (29D), UR (10D), Google robot (10D), WidowX 250 (10D), UMI (9D).
91
- - **Input Size and Length limits:**
92
- - **Text:** 4096 tokens
93
- - **Image:** 256p, 480p, and 720p resolution at one of these aspect ratios (16:9, 4:3, 1:1, 3:4, 9:16)
94
- - **Video:** 256p, 480p, and 720p resolution at one of these aspect ratios (16:9, 4:3, 1:1, 3:4, 9:16). Max number of frames = 5.
95
- - **Audio:** Max 0.5 second
96
- - **Action:** 16 – 400 video frames
97
- - **Generator Output**
98
- - **Output Type(s)**: Image, video, audio, action, text
99
- - **Output Format(s)**:
100
- - Image: JPG
101
- - Video: MP4
102
- - Audio: Advanced Audio Coding (AAC) stream (muxed within the MP4)
103
- - Action: 1D list (.json)
104
- - Text: string
105
- - **Output Parameters**:
106
- - Image: Two-dimensional (2D)
107
- - Video: Three-dimensional (3D)
108
- - Audio: One-dimensional (1D)
109
- - Action: One-dimensional (1D)
110
- - Text: One-dimensional (1D)
111
- - **Other Properties Related to Output**:
112
- - The generated video is an MP4 file, with the resolution, frame rate, and duration specified in the input. The generated audio is encoded in AAC format, muxed into the video MP4 file with 2 channels (stereo) and a 48 kHz sample rate.
113
- - Video generation supports durations from 5 to 400 frames, with 189 frames as the default generation duration.
114
- - The generated action is only supported for compatible embodiments, including general camera motion (9D), autonomous vehicle (9D), egocentric motion (57D), single Franka Panda arm with RobotiQ gripper (10D), dual Franka Panda arm with RobotiQ gripper (20D), Agibot (29D), UR (10D), Google robot (10D), WidowX 250 (10D), UMI (9D).
115
- - Audio: 48 kHz stereo AAC stream muxed into video mp4
116
- - Video: mp4 at the FPS specified in input
117
- - Image: JPEG
118
- - **Reasoner Input**
119
- - **Input Type(s)**: Text, Text+Image, Text+Video
120
- - **Input Format(s)**:
121
- - Text: String
122
- - Image: jpg, png, jpeg, webp
123
- - Video: mp4
124
- - **Input Parameters**:
125
- - Text: One-dimensional (1D)
126
- - Image: Two-dimensional (2D)
127
- - Video: Three-dimensional (3D)
128
- - **Other Properties Related to Input**:
129
- - Video inputs are recommended at a frame rate of 4 fps.
130
- - Long-context inputs supported up to 256K tokens.
131
- - **Input Size and Length limits:**
132
- - **Text:** Up to 256K tokens (context window).
133
- - **Image:** Standard input image formats; passed as file or URL.
134
- - **Video:** mp4 at the recommended 4 fps.
135
- - **Reasoner Output**
136
- - **Output Type(s)**: Text
137
- - **Output Format(s)**:
138
- - Text: string
139
- - **Output Parameters**:
140
- - Text: One-dimensional (1D)
141
- - **Other Properties Related to Output**:
142
- - Default `max_tokens=4096+` is recommended for reasoning outputs; longer outputs may be requested.
143
- - Reasoning outputs may include structured chain-of-thought, 2D/3D point localization, and bounding-box coordinates for vision-based tasks.
144
-
145
- The video content visualizes the input text description as a short animated scene, capturing key elements within the specified time constraints.
146
-
147
- Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g., GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
148
 
149
  ## Software Integration
150
 
151
  **Runtime Engine(s):**
152
-
153
  - [PyTorch](https://github.com/nvidia/cosmos3)
154
  - [vLLM-Omni](https://github.com/vllm-project/vllm-omni)
155
  - [Hugging Face Diffusers](https://huggingface.co/docs/diffusers/en/index)
156
- - [SGLang](https://github.com/sgl-project/sglang)
157
 
158
  **Supported Hardware Microarchitecture Compatibility:**
159
-
160
  - NVIDIA Ampere
161
  - NVIDIA Blackwell
162
  - NVIDIA Hopper
163
 
164
- **Operating System(s):**
165
-
166
- - Linux
167
-
168
- **Note:** Only BF16 precision is tested. Other precisions like FP4, FP8, and FP16 are not officially supported.
169
-
170
- The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
171
-
172
  ## Training, Testing, and Evaluation Datasets
173
 
174
  ### Dataset Overview
175
-
176
  - **Total Size:** 1.3B data points
177
  - **Total Number of Datasets:** 393 dataset entries
178
- - **Dataset partition:** Training [100%], Testing [N/A — evaluation benchmarks used separately], Validation [N/A — evaluation benchmarks used separately]
179
- - **Time period for training data collection:** 2024–2026
180
- - **Time period for testing data collection:** N/A (standard public benchmarks)
181
- - **Time period for validation data collection:** N/A (standard public benchmarks)
182
-
183
- Raw data from internal and external sources is transformed into training-ready data through multiple stages of curation, filtering, and quality review. Data acquisition spans diverse multimodal sources — robotics, autonomous driving, industrial environments, indoor and outdoor scenes, varied lighting and weather conditions, camera viewpoints, object categories, and human activities — to broaden coverage across Physical AI operating environments. Automated filtering pipelines remove corrupted, duplicate, low-quality, and restricted content. Metadata analysis, heuristic rules, and model-assisted classifiers are applied during preprocessing to flag anomalous distributions and low-diversity subsets. Human review supplements automated filtering for selected datasets, benchmark construction, and targeted quality analysis. Datasets are balanced across modalities and task categories — visual reasoning, text-to-image, text-to-video, image-to-video, audio generation, video transfer, action-conditioned generation, and action command generation — to reduce overrepresentation of narrow domains. Synthetic and simulation-based augmentation supplements coverage of rare physical interactions and edge-case scenarios. Deduplication and provenance tracking are applied across the corpus. The resulting processed data is converted into model-ready tokenized or encoded representations through modality-specific preprocessors before training begins.
184
-
185
- Training datasets passed through multiple layers of automated and manual safeguards designed to reduce the presence of harmful or policy-violating content across categories including weapons and weapons-related instructional content, criminal planning, child sexual abuse material (CSAM), non-consensual intimate imagery (NCII), sexual content involving minors, harassment, hate speech, profanity, threats and incitement to violence, self-harm or suicide-related content, and graphic violence. Data sources are reviewed for licensing compatibility, provenance, and alignment with internal data governance and safety policies before admission into training corpora. Automated filtering pipelines combine multiple detection strategies: hash-matching against known CSAM and NCII reference databases; classifier-based moderation models trained for explicit sexual content, hate speech, violence, weapons imagery, and other restricted categories; keyword and regex-based screening for criminal-planning, threats, and self-harm phrases in text data; metadata and provenance heuristics for source-level risk signals; and embedding-based anomaly detection to surface samples that fall outside expected distributions. Human review and targeted audits supplement automated filtering for selected datasets, benchmark construction, and safety-sensitive evaluation. For multimodal Physical AI data (robotics, autonomous driving, industrial scenes), additional filtering targets invalid action trajectories, physically implausible interactions, and unsafe control sequences. Synthetic and simulation-generated data are evaluated through internal validation before inclusion. Benchmark evaluations and red-team testing are applied post-training to surface remaining safety gaps across world generation, reasoning, audio, and action tasks. No large-scale data-filtering process can guarantee complete removal of all harmful content; residual risks may remain, particularly in rare edge cases or open-world deployment settings. Ongoing monitoring and dataset review continue post-release.
186
 
187
- - For more information about the datasets used to train this model, please see the [Public Summary of Training Content](https://docs.nvidia.com/cosmos/latest/_downloads/e482b7114ce8dbfbb07d2d4b42cafe4e/training-content-Cosmos-3.pdf).
188
 
189
  **Data Modality and Training Data Size**
190
 
@@ -196,796 +128,16 @@ Training datasets passed through multiple layers of automated and manual safegua
196
  | Audio | Not Applicable | 139M |
197
  | Action | Not Applicable | 8M |
198
 
199
- **Data Collection Method by dataset**
200
-
201
- - Hybrid: Automatic/Sensors, Synthetic, Automated
202
-
203
- **Labeling Method by dataset**
204
-
205
- - Hybrid: Human, Automated
206
-
207
- **Properties:** The training, testing, and evaluation datasets consist of diverse multimodal video, image, audio, action, synthetic, and sensor-conditioned data sourced from NVIDIA-owned data and publicly available, commercially permissive datasets. These datasets are curated to exclude known restricted content and to support building an Omni model that learns to generate and reason about dynamic physical environments across world reasoning and generation tasks.
208
-
209
- ### Public Datasets
210
-
211
- | Dataset                                                             | Samples           |
212
- |---|---|
213
- | OpenImage | 1.2M |
214
- | Coyo700M | 100M |
215
- | YouTube Video | 340M |
216
- | UMI | 4.5M |
217
-
218
- ### Private Datasets
219
-
220
- | Dataset                                                             | Samples           |
221
- |---|---|
222
- | Egocentric | 7M |
223
- | Nexar | 0.6M |
224
- | AgiBot | 0.2M |
225
- | HOI | 0.3M |
226
-
227
- ### Synthetic Datasets
228
-
229
- | Dataset | Samples |
230
- |---|---|
231
- | synthetic images generated using HiDream-I1 | 15M |
232
- | synthetic images generated using Qwen-Image-2512 | 14M |
233
- | synthetic captions generated using Qwen3-VL | 1115M |
234
-
235
- ## Evaluation Datasets
236
-
237
- **Data Collection Method by dataset**
238
-
239
- - Hybrid: Automatic/Sensors, Synthetic, Automated
240
-
241
- **Labeling Method by dataset**
242
-
243
- - Hybrid: Human, Automated
244
-
245
- **Properties:** The training, testing, and evaluation datasets consist of diverse multimodal video, image, audio, action, synthetic, and sensor-conditioned data sourced from NVIDIA-owned data and publicly available, commercially permissive datasets. These datasets are curated to exclude known restricted content and to support building an Omni model that learns to generate and reason about dynamic physical environments across world reasoning and generation tasks.
246
-
247
  ## Benchmarks
248
 
249
- Please see our [technical paper](https://research.nvidia.com/labs/cosmos-lab/cosmos3/technical-report.pdf) for detailed evaluations of the base model.
250
-
251
- ### Overall
252
-
253
- ![Overall benchmark results](images/benchmark-overall.png)
254
-
255
- ### Reasoning Benchmarks
256
-
257
- ![Reasoning benchmarks](images/benchmark-reasoning.png)
258
-
259
- ### Generation Benchmarks
260
-
261
- #### Visual-Audio Generation
262
-
263
- ![Visual & audio generation benchmarks](images/benchmark-visual-audio.png)
264
-
265
- #### Action
266
-
267
- ![Action benchmarks — forward and inverse dynamics](images/benchmark-action-1.png)
268
-
269
- ## Usage
270
-
271
- - See [Cosmos](https://github.com/nvidia/cosmos) for details.
272
-
273
- ### Prompt upsampling
274
-
275
- For optimal quality, text prompts should be upsampled into a specific JSON structure. Description and code can be found [here](https://github.com/nvidia/cosmos-framework/blob/main/docs/prompt_upsampling.md).
276
-
277
- For example, for text-to-video upsampling using Opus-4.6:
278
-
279
- ```bash
280
- git clone https://github.com/NVIDIA/cosmos-framework.git packages/cosmos-framework
281
- pip install -e packages/cosmos-framework
282
-
283
- export PROMPT_UPSAMPLER_ENDPOINT_URL="https://api.anthropic.com/v1/"
284
- export PROMPT_UPSAMPLER_MODEL_NAME="claude-opus-4-6"
285
- export PROMPT_UPSAMPLER_API_TOKEN="<you_token>"
286
-
287
- python -m cosmos_framework.inference.prompt_upsampling \
288
- --input assets/example_t2v_prompt_short.txt \
289
- --output /tmp/upsampled_t2v_opus/ \
290
- --mode text2video \
291
- --endpoint-url "${PROMPT_UPSAMPLER_ENDPOINT_URL}" \
292
- --model "${PROMPT_UPSAMPLER_MODEL_NAME}" \
293
- --api-token "${PROMPT_UPSAMPLER_API_TOKEN}" \
294
- --resolution 720 \
295
- --aspect-ratio "16,9"
296
- ```
297
-
298
- The JSON-upsampled version of `assets/example_t2v_prompt_short.txt` is saved in `assets/example_t2v_prompt.json` for convenience, and is used for the video generation examples below.
299
-
300
- ### vLLM-Omni
301
-
302
- #### Container
303
-
304
- ```
305
- docker pull vllm/vllm-omni:cosmos3
306
- ```
307
-
308
- #### General Invocation
309
-
310
- You can use the release-tested `vllm-omni` package for deploying an OpenAI-compatible API inference endpoint.
311
- The recommended vLLM-Omni serving configuration for nvidia/Cosmos3-Nano on H200 is:
312
-
313
- ```bash
314
- vllm serve nvidia/Cosmos3-Nano \
315
- --omni \
316
- --host 0.0.0.0 \
317
- --port 8000 \
318
- --init-timeout 1800
319
- ```
320
-
321
- To speed up inference with additional GPUs, enable context parallelism with `--ulysses-degree` or switch to tensor parallelism with `--tensor-parallel-size`. Setting `--enable-layerwise-offload` can help reduce memory usage on GPUs with less available memory.
322
-
323
- #### Examples
324
-
325
- ##### Download example prompts
326
-
327
- The example inputs (`assets/`) live in this model repo. Download just this folder with the Hugging Face CLI:
328
-
329
- ```bash
330
- pip install -U "huggingface_hub[cli]"
331
- hf download nvidia/Cosmos3-Nano assets/ --local-dir Cosmos3-Nano
332
- cd Cosmos3-Nano
333
- ```
334
-
335
- Run all commands below from the downloaded repo root.
336
-
337
- ---
338
-
339
- ##### Image to video generation
340
-
341
- ```python
342
- import json
343
- import mimetypes
344
- from pathlib import Path
345
-
346
- import requests
347
-
348
- # 1. Read JSON-upsampled prompt and negative prompt
349
- json_prompt = json.load(open("assets/example_i2v_prompt.json"))
350
- negative_prompt = json.load(open("assets/negative_prompt.json"))
351
-
352
- # 2. Build and send the multipart API request
353
- url = "http://localhost:8000/v1/videos/sync"
354
- image_path = Path("assets/example_i2v_input.jpg")
355
- mime_type = mimetypes.guess_type(image_path)[0] or "image/png"
356
- data = {
357
- "prompt": json.dumps(json_prompt),
358
- "negative_prompt": json.dumps(negative_prompt),
359
- "size": "1280x720",
360
- "num_frames": "189",
361
- "fps": "24",
362
- "num_inference_steps": "35",
363
- "guidance_scale": "6.0",
364
- "max_sequence_length": "4096",
365
- "flow_shift": "10.0",
366
- "extra_params": json.dumps(
367
- {
368
- "use_resolution_template": False,
369
- "use_duration_template": False,
370
- "guardrails": True,
371
- }
372
- ),
373
- "seed": "1111",
374
- }
375
-
376
- with image_path.open("rb") as image_file:
377
- files = {
378
- "input_reference": (image_path.name, image_file, mime_type),
379
- }
380
- print("Sending request to server...")
381
- response = requests.post(
382
- url,
383
- data=data,
384
- files=files,
385
- headers={"Accept": "video/mp4"},
386
- )
387
- response.raise_for_status()
388
-
389
- # 3. Save the generated video
390
- output_path = Path("/tmp/cosmos3_nano_i2v.mp4")
391
- output_path.write_bytes(response.content)
392
- print(f"Saved video to {output_path}")
393
- ```
394
-
395
- Example output:
396
-
397
- <video controls width="1280" height="720" src="https://huggingface.co/nvidia/Cosmos3-Nano/resolve/main/assets/example_i2v_output.mp4"></video>
398
-
399
- ---
400
-
401
- ##### Text to video generation
402
-
403
- ```python
404
- import json
405
- from pathlib import Path
406
-
407
- import requests
408
-
409
- # 1. Read JSON-upsampled prompt and negative prompt
410
- json_prompt = json.load(open("assets/example_t2v_prompt.json"))
411
- negative_prompt = json.load(open("assets/negative_prompt.json"))
412
-
413
- # 2. Build your API payload
414
- data = {
415
- "prompt": json.dumps(json_prompt),
416
- "negative_prompt": json.dumps(negative_prompt),
417
- "size": "1280x720",
418
- "num_frames": "189",
419
- "fps": "24",
420
- "num_inference_steps": "35",
421
- "guidance_scale": "6.0",
422
- "max_sequence_length": "4096",
423
- "flow_shift": "10.0",
424
- "extra_params": json.dumps(
425
- {
426
- "use_resolution_template": False,
427
- "use_duration_template": False,
428
- "guardrails": True,
429
- }
430
- ),
431
- "seed": "123",
432
- }
433
-
434
- # 3. Send the POST request
435
- url = "http://localhost:8000/v1/videos/sync"
436
- print("Sending request to server...")
437
- response = requests.post(
438
- url,
439
- data=data,
440
- headers={"Accept": "video/mp4"},
441
- )
442
- response.raise_for_status()
443
-
444
- # 4. Save the generated video
445
- output_path = Path("/tmp/cosmos3_nano_t2v.mp4")
446
- output_path.write_bytes(response.content)
447
- print(f"Saved video to {output_path}")
448
- ```
449
-
450
- Example output:
451
-
452
- <video controls width="1280" height="720" src="https://huggingface.co/nvidia/Cosmos3-Nano/resolve/main/assets/example_t2v_output.mp4"></video>
453
-
454
- ---
455
-
456
- ##### Image to Video + Audio generation
457
-
458
- ```python
459
- import json
460
- import mimetypes
461
- from pathlib import Path
462
-
463
- import requests
464
-
465
- # 1. Read JSON-upsampled prompt and negative prompt
466
- json_prompt = json.load(open("assets/example_i2v_prompt.json"))
467
- negative_prompt = json.load(open("assets/negative_prompt.json"))
468
-
469
- # 2. Build and send the multipart API request
470
- url = "http://localhost:8000/v1/videos/sync"
471
- image_path = Path("assets/example_i2v_input.jpg")
472
- mime_type = mimetypes.guess_type(image_path)[0] or "image/png"
473
- data = {
474
- "prompt": json.dumps(json_prompt),
475
- "negative_prompt": json.dumps(negative_prompt),
476
- "size": "1280x720",
477
- "num_frames": "189",
478
- "fps": "24",
479
- "num_inference_steps": "35",
480
- "guidance_scale": "6.0",
481
- "max_sequence_length": "4096",
482
- "generate_sound": "true",
483
- "sound_duration": "7.875",
484
- "flow_shift": "10.0",
485
- "extra_params": json.dumps(
486
- {
487
- "use_resolution_template": False,
488
- "use_duration_template": False,
489
- "guardrails": True,
490
- }
491
- ),
492
- "seed": "0",
493
- }
494
-
495
- with image_path.open("rb") as image_file:
496
- files = {
497
- "input_reference": (image_path.name, image_file, mime_type),
498
- }
499
- print("Sending request to server...")
500
- response = requests.post(
501
- url,
502
- data=data,
503
- files=files,
504
- headers={"Accept": "video/mp4"},
505
- )
506
- response.raise_for_status()
507
-
508
- # 3. Save the generated video
509
- output_path = Path("/tmp/cosmos3_nano_i2vs.mp4")
510
- output_path.write_bytes(response.content)
511
- print(f"Saved video to {output_path}")
512
- ```
513
-
514
- Example output:
515
-
516
- <video controls width="1280" height="720" src="https://huggingface.co/nvidia/Cosmos3-Nano/resolve/main/assets/example_i2vs_output.mp4"></video>
517
-
518
- ---
519
-
520
- ##### Text to Video + Audio generation
521
-
522
- ```python
523
- import json
524
- from pathlib import Path
525
-
526
- import requests
527
-
528
- # 1. Read JSON-upsampled prompt and negative prompt
529
- json_prompt = json.load(open("assets/example_t2vs_prompt.json"))
530
- negative_prompt = json.load(open("assets/negative_prompt.json"))
531
-
532
- # 2. Build your API payload
533
- data = {
534
- "prompt": json.dumps(json_prompt),
535
- "negative_prompt": json.dumps(negative_prompt),
536
- "size": "1280x720",
537
- "num_frames": "189",
538
- "fps": "24",
539
- "num_inference_steps": "35",
540
- "guidance_scale": "6.0",
541
- "max_sequence_length": "4096",
542
- "generate_sound": "true",
543
- "sound_duration": "7.875",
544
- "flow_shift": "10.0",
545
- "extra_params": json.dumps(
546
- {
547
- "use_resolution_template": False,
548
- "use_duration_template": False,
549
- "guardrails": True,
550
- }
551
- ),
552
- "seed": "0",
553
- }
554
-
555
- # 3. Send the POST request
556
- url = "http://localhost:8000/v1/videos/sync"
557
- print("Sending request to server...")
558
- response = requests.post(
559
- url,
560
- data=data,
561
- headers={"Accept": "video/mp4"},
562
- )
563
- response.raise_for_status()
564
-
565
- # 4. Save the generated video
566
- output_path = Path("/tmp/cosmos3_nano_t2vs.mp4")
567
- output_path.write_bytes(response.content)
568
- print(f"Saved video to {output_path}")
569
- ```
570
-
571
- Example output:
572
-
573
- <video controls width="1280" height="720" src="https://huggingface.co/nvidia/Cosmos3-Nano/resolve/main/assets/example_t2vs_output.mp4"></video>
574
-
575
- ---
576
-
577
- ##### Action generation
578
-
579
- The forward-dynamics example uses AgiBotWorld-Beta robotics action trajectories, and the inverse-dynamics examples use autonomous-vehicle (AV) action trajectories. Source files:
580
-
581
- - Forward dynamics first frame: `assets/example_action_fd_agibotworld_first_frame.png`
582
- - Forward dynamics action chunks: `assets/example_action_fd_agibotworld_action_chunks.json`
583
- - Forward dynamics output video: `assets/example_action_fd_agibotworld_4chunk_output.mp4`
584
- - Inverse dynamics source videos: `assets/example_action_id_av_0_input.mp4`, `assets/example_action_id_av_1_input.mp4`
585
- - Inverse dynamics predicted actions: `assets/example_action_id_av_0_output.json`, `assets/example_action_id_av_1_output.json`
586
-
587
- ###### Action forward dynamics
588
-
589
- The example below performs a 4-chunk AgiBotWorld-Beta robotics rollout with the vLLM-Omni `/v1/videos/sync` inference endpoint. Each request sends one conditioning frame through `input_reference` and one 16-step normalized 29-D action chunk through `extra_params["action"]`. The request also sets the top-level `size` field to the input image resolution, so vLLM-Omni returns each chunk at the same resolution as the conditioning image without reflection padding. The stitched output drops each chunk's conditioning frame, producing 64 generated frames. The script extracts the last generated frame from each chunk and uses it as the next chunk's conditioning frame.
590
-
591
- ```python
592
- import json
593
- import mimetypes
594
- from pathlib import Path
595
-
596
- import imageio.v3 as iio
597
- import numpy as np
598
- import requests
599
- from PIL import Image
600
-
601
- url = "http://localhost:8000/v1/videos/sync"
602
- first_frame_path = Path("assets/example_action_fd_agibotworld_first_frame.png")
603
- action_spec = json.loads(Path("assets/example_action_fd_agibotworld_action_chunks.json").read_text())
604
- action_chunks = action_spec["action_chunks"]
605
-
606
- prompt = action_spec.get("prompt", "Pickup items in the supermarket")
607
- fps = int(action_spec.get("fps", 10))
608
- action_chunk_size = int(action_spec.get("action_chunk_size", 16))
609
- current_frame_path = first_frame_path
610
- input_width, input_height = Image.open(first_frame_path).size
611
- chunk_video_paths = []
612
- stitch_frames = []
613
-
614
- for chunk_idx, action_chunk in enumerate(action_chunks):
615
- mime_type = mimetypes.guess_type(current_frame_path)[0] or "image/png"
616
- extra_params = {
617
- "action_mode": "forward_dynamics",
618
- "domain_name": action_spec.get("domain_name", "agibotworld"),
619
- "action_chunk_size": action_chunk_size,
620
- "image_size": action_spec.get("image_size", 480),
621
- "view_point": action_spec.get("view_point", "concat_view"),
622
- "action": action_chunk,
623
- "guardrails": True,
624
- }
625
- data = {
626
- "prompt": prompt,
627
- "num_frames": str(action_chunk_size + 1), # conditioning frame + generated frames
628
- "fps": str(fps),
629
- "size": f"{input_width}x{input_height}", # return chunks at input resolution
630
- "num_inference_steps": "30",
631
- "guidance_scale": "1.0",
632
- "flow_shift": "10.0",
633
- "seed": "0",
634
- "extra_params": json.dumps(extra_params),
635
- }
636
-
637
- with current_frame_path.open("rb") as image_file:
638
- files = {"input_reference": (current_frame_path.name, image_file, mime_type)}
639
- print(f"Sending action FD chunk {chunk_idx} to vLLM-Omni...")
640
- response = requests.post(
641
- url,
642
- data=data,
643
- files=files,
644
- headers={"Accept": "video/mp4"},
645
- timeout=600,
646
- )
647
- response.raise_for_status()
648
-
649
- chunk_video_path = Path(f"/tmp/cosmos3_nano_action_fd_chunk_{chunk_idx:02d}.mp4")
650
- chunk_video_path.write_bytes(response.content)
651
- chunk_video_paths.append(chunk_video_path)
652
-
653
- # The returned chunk contains the conditioning frame followed by generated frames.
654
- # Drop the conditioning frame when stitching the generated-only rollout.
655
- frames = iio.imread(chunk_video_path)
656
- stitch_frames.extend(frames[1:])
657
-
658
- # Autoregressive conditioning: use the final generated frame from this chunk
659
- # as the input image for the next vLLM-Omni request.
660
- if chunk_idx + 1 < len(action_chunks):
661
- current_frame_path = Path(f"/tmp/cosmos3_nano_action_fd_ar_frame_{chunk_idx + 1:02d}.png")
662
- iio.imwrite(current_frame_path, frames[-1])
663
-
664
- stitched_path = Path("/tmp/cosmos3_nano_action_fd_agibotworld_4chunk.mp4")
665
- iio.imwrite(stitched_path, np.asarray(stitch_frames), fps=fps)
666
- print("Generated chunk videos:", chunk_video_paths)
667
- print("Saved stitched rollout:", stitched_path)
668
- print("stitched resolution:", f"{input_width}x{input_height}")
669
- ```
670
-
671
-
672
- Example output:
673
-
674
- <video width="640" controls src="https://huggingface.co/nvidia/Cosmos3-Nano/resolve/main/assets/example_action_fd_agibotworld_4chunk_output.mp4"></video>
675
-
676
- ###### Action inverse dynamics
677
-
678
- ```python
679
- import json
680
- import time
681
- from pathlib import Path
682
-
683
- import requests
684
-
685
- base_url = "http://localhost:8000"
686
- input_videos = {
687
- "av_inverse_0": Path("assets/example_action_id_av_0_input.mp4"),
688
- "av_inverse_1": Path("assets/example_action_id_av_1_input.mp4"),
689
- }
690
-
691
- for name, video_path in input_videos.items():
692
- extra_params = {
693
- "action_mode": "inverse_dynamics",
694
- "domain_name": "av",
695
- "action_chunk_size": 60,
696
- "image_size": 480,
697
- "view_point": "ego_view",
698
- "raw_action_dim": 9,
699
- "guardrails": True,
700
- }
701
- data = {
702
- "prompt": "You are an autonomous vehicle planning system.",
703
- "num_frames": "61",
704
- "fps": "10",
705
- "num_inference_steps": "30",
706
- "guidance_scale": "1.0",
707
- "flow_shift": "10.0",
708
- "seed": "0",
709
- "extra_params": json.dumps(extra_params),
710
- }
711
-
712
- with video_path.open("rb") as video_file:
713
- files = {
714
- "input_reference": (video_path.name, video_file, "video/mp4"),
715
- }
716
- print(f"Submitting {name} request to server...")
717
- response = requests.post(f"{base_url}/v1/videos", data=data, files=files)
718
- response.raise_for_status()
719
- initial = response.json()
720
-
721
- while True:
722
- response = requests.get(f"{base_url}/v1/videos/{initial['id']}", timeout=30)
723
- response.raise_for_status()
724
- final = response.json()
725
- print(initial["id"], final.get("status"), f"{final.get('progress', 0)}%")
726
- if final.get("status") == "completed":
727
- break
728
- if final.get("status") in {"failed", "cancelled"}:
729
- raise RuntimeError(json.dumps(final, indent=2))
730
- time.sleep(2)
731
-
732
- action = final.get("action")
733
- if not action or "data" not in action:
734
- raise RuntimeError(f"Response did not include action data: {json.dumps(final, indent=2)}")
735
-
736
- output_path = Path(f"/tmp/cosmos3_nano_action_id_{name}.json")
737
- output_path.write_text(json.dumps(action, indent=2))
738
- print(f"Saved predicted action to {output_path}")
739
- print("action shape:", action.get("shape"), "dtype:", action.get("dtype"))
740
- ```
741
-
742
- Example outputs:
743
-
744
- - [av_inverse_0 predicted action JSON](https://huggingface.co/nvidia/Cosmos3-Nano/blob/main/assets/example_action_id_av_0_output.json)
745
- - [av_inverse_1 predicted action JSON](https://huggingface.co/nvidia/Cosmos3-Nano/blob/main/assets/example_action_id_av_1_output.json)
746
-
747
- <img width="1280" src="https://huggingface.co/nvidia/Cosmos3-Nano/resolve/main/assets/example_action_id_av_0_output.png">
748
-
749
- <img width="1280" src="https://huggingface.co/nvidia/Cosmos3-Nano/resolve/main/assets/example_action_id_av_1_output.png">
750
-
751
- ### vLLM
752
-
753
- #### General Invocation
754
-
755
- You can use the release-tested `vllm` package for deploying an OpenAI-compatible API endpoint:
756
-
757
- ```shell
758
- uv venv --python 3.13 --seed --managed-python
759
- source .venv/bin/activate
760
- uv pip install --torch-backend=cu130 "vllm==0.21.0" \
761
- "vllm-cosmos3 @ git+https://github.com/NVIDIA/cosmos-framework.git#subdirectory=packages/vllm-cosmos3" \
762
- openai
763
- ```
764
-
765
- Use `--torch-backend=cu130 "vllm==0.21.0"` for CUDA 13 drivers. For CUDA 12.8 drivers, use `--torch-backend=cu128 "vllm==0.19.1"`.
766
-
767
- Start the Reasoner server:
768
-
769
- ```shell
770
- CUDA_VISIBLE_DEVICES=0 \
771
- vllm serve nvidia/Cosmos3-Nano \
772
- --hf-overrides '{"architectures": ["Cosmos3ReasonerForConditionalGeneration"]}' \
773
- --tensor-parallel-size 1 \
774
- --mm-encoder-tp-mode data \
775
- --async-scheduling \
776
- --allowed-local-media-path / \
777
- --media-io-kwargs '{"video": {"num_frames": -1}}' \
778
- --port 8000
779
- ```
780
-
781
- #### Examples
782
-
783
- ##### Reasoning
784
-
785
- Run this example from the model repository root. It reads the robot planning prompt from `assets/example_reasoning_prompt.json` and sends `assets/example_reasoning_input.png` to the local vLLM server.
786
-
787
- ```python
788
- import json
789
- from pathlib import Path
790
-
791
- import openai
792
-
793
- # 1. Read the image reasoning prompt
794
- example = json.load(open("assets/example_reasoning_prompt.json"))
795
- image_path = Path("assets/example_reasoning_input.png").resolve()
796
- image_url = image_path.as_uri()
797
-
798
- # 2. Query the OpenAI-compatible vLLM server
799
- client = openai.OpenAI(
800
- api_key="EMPTY",
801
- base_url="http://localhost:8000/v1",
802
- )
803
-
804
- response = client.chat.completions.create(
805
- model=client.models.list().data[0].id,
806
- messages=[
807
- {
808
- "role": "user",
809
- "content": [
810
- {"type": "image_url", "image_url": {"url": image_url}},
811
- {"type": "text", "text": example["prompt"]},
812
- ],
813
- },
814
- ],
815
- max_tokens=example["max_tokens"],
816
- seed=0,
817
- )
818
-
819
- # 3. Print the generated reasoning output
820
- print(response.choices[0].message.content)
821
- ```
822
-
823
- Example input:
824
-
825
- <img src="assets/example_reasoning_input.png" width="640">
826
-
827
- Prompt:
828
-
829
- ```text
830
- The task is to put flower into the red bottle. Generate a plan consisting of subtasks for accomplish the task.
831
- ```
832
-
833
- Example output from the command above:
834
-
835
- ```text
836
- Move your arm to the flower. Grasp the flower. Move your arm to the red bottle. Place the flower in the red bottle.
837
- ```
838
-
839
- ### Diffusers
840
-
841
- Cosmos3 is fully supported within the popular HuggingFace Diffusers package. This integration makes it a supported inference backend, allowing developers to easily incorporate Cosmos3's capabilities — such as text-to-video generation — into their pipelines using the `Cosmos3OmniPipeline` class, as demonstrated by the provided code examples (see examples for other modalities on the HuggingFace Cosmos3 page).
842
-
843
- #### Container
844
-
845
- To install diffusers with Cosmos3OmniPipeline:
846
-
847
- ```
848
- uv venv --python 3.13 --seed --managed-python
849
- source .venv/bin/activate
850
- uv pip install \
851
- "diffusers @ git+https://github.com/huggingface/diffusers.git" \
852
- accelerate \
853
- av \
854
- cosmos_guardrail \
855
- huggingface_hub \
856
- imageio \
857
- imageio-ffmpeg \
858
- torch \
859
- torchvision \
860
- transformers
861
-
862
- ```
863
-
864
- #### Examples
865
-
866
- ##### Text to video generation
867
-
868
- Run this example from the model repository root. It reads the JSON-upsampled prompt from `assets/example_t2v_prompt.json` and the negative prompt from `assets/negative_prompt.json`. It then loads the pipeline and generate the video, then save it to an MP4 file.
869
-
870
- ```python
871
- import json
872
- import torch
873
- from diffusers import Cosmos3OmniPipeline
874
- from diffusers.schedulers.scheduling_unipc_multistep import UniPCMultistepScheduler
875
- from diffusers.utils import export_to_video
876
-
877
- # Read JSON-upsampled prompt and negative prompt
878
- json_prompt = json.load(open("assets/example_t2v_prompt.json"))
879
- negative_prompt = json.load(open("assets/negative_prompt.json"))
880
-
881
- pipe = Cosmos3OmniPipeline.from_pretrained(
882
- "nvidia/Cosmos3-Nano",
883
- torch_dtype=torch.bfloat16,
884
- device_map="cuda",
885
- enable_safety_checker=True,
886
- )
887
- pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=10.0)
888
-
889
- result = pipe(
890
- prompt=json.dumps(json_prompt),
891
- negative_prompt=json.dumps(negative_prompt),
892
- num_frames=189,
893
- height=720,
894
- width=1280,
895
- num_inference_steps=35,
896
- guidance_scale=6.0,
897
- generator=torch.Generator(device="cuda").manual_seed(123),
898
- )
899
-
900
- export_to_video(result.video, "/tmp/cosmos3_nano_t2v_diffusers.mp4", fps=24)
901
- print("Saved video to /tmp/cosmos3_nano_t2v_diffusers.mp4")
902
- ```
903
-
904
- Example output:
905
-
906
- <video controls width="1280" height="720" src="https://huggingface.co/nvidia/Cosmos3-Nano/resolve/main/assets/example_t2v_diffusers_output.mp4"></video>
907
-
908
- ### SGLang
909
-
910
- [SGLang Diffusion](https://docs.sglang.io/docs/sglang-diffusion/index) can serve `nvidia/Cosmos3-Nano` through OpenAI-compatible image and video generation endpoints. Install SGLang from the main branch with diffusion dependencies, then start a server:
911
-
912
- ```shell
913
- git clone --branch main https://github.com/sgl-project/sglang.git
914
- cd sglang
915
- pip install -e "python[diffusion]"
916
- pip install "cosmos-guardrail==0.3.1"
917
-
918
- sglang serve --model-path nvidia/Cosmos3-Nano
919
- ```
920
-
921
- Cosmos 3 support in SGLang Diffusion currently requires the SGLang main branch. Switch to a stable SGLang release once Cosmos 3 support is included there.
922
-
923
- For a video-specialized checkpoint, use `Cosmos3-Super-Image2Video` with multiple GPUs:
924
-
925
- ```shell
926
- sglang serve \
927
- --model-path nvidia/Cosmos3-Super-Image2Video \
928
- --num-gpus 4
929
- ```
930
-
931
- Supported SGLang endpoints:
932
-
933
- | Mode | Endpoint | Notes |
934
- | --- | --- | --- |
935
- | Text to image | `POST /v1/images/generations` | Returns base64 image data by default |
936
- | Text to video | `POST /v1/videos` | Creates an async job; poll `GET /v1/videos/{id}` and download `/content` |
937
- | Image to video | `POST /v1/videos` | Upload the conditioning image with `input_reference` |
938
-
939
- Example text-to-video request:
940
-
941
- ```shell
942
- job_id=$(curl -sS -X POST http://localhost:30000/v1/videos \
943
- --form-string "prompt=A small warehouse robot moves a blue box across a clean floor." \
944
- --form-string "negative_prompt=blurry, distorted, low quality" \
945
- --form-string "size=1280x720" \
946
- --form-string "num_frames=81" \
947
- --form-string "fps=24" \
948
- --form-string "num_inference_steps=35" \
949
- --form-string "guidance_scale=4.0" \
950
- --form-string "flow_shift=10.0" \
951
- --form-string "seed=42" \
952
- --form-string 'extra_params={"guardrails":true,"use_resolution_template":false,"use_duration_template":false}' \
953
- | python -c 'import json, sys; print(json.load(sys.stdin)["id"])')
954
-
955
- while true; do
956
- status=$(curl -sS "http://localhost:30000/v1/videos/${job_id}" \
957
- | python -c 'import json, sys; print(json.load(sys.stdin)["status"])')
958
- [ "$status" = "completed" ] && break
959
- [ "$status" = "failed" ] && exit 1
960
- sleep 1
961
- done
962
-
963
- curl -sS -L "http://localhost:30000/v1/videos/${job_id}/content" \
964
- -o cosmos3_t2v_output.mp4
965
- ```
966
-
967
- SGLang accepts Cosmos 3 request options including `max_sequence_length`, `flow_shift`, `extra_params.guardrails`, `extra_params.use_resolution_template`, and `extra_params.use_duration_template`. Video-to-video, video-with-sound, and action generation are not supported by SGLang yet.
968
-
969
- For complete serving instructions and request examples, see the [Cosmos3 SGLang cookbook](https://docs.sglang.io/cookbook/diffusion/Cosmos/Cosmos3).
970
 
971
  ## Limitations
972
 
973
- Cosmos3 may produce imperfect outputs in challenging scenarios. Generation artifacts include temporal inconsistency, unstable camera or object motion, imprecise physical interactions, inaccurate audio-video synchronization, and action-state drift — especially in long-horizon or high-resolution outputs. Reasoning may also be incorrect: object states, causal relationships, spatial geometry, temporal ordering, agent intent, and future outcomes can be misinferred, and complex or long-context inputs may yield hallucinated entities, inconsistent interpretations, or implausible predictions. Because the model lacks an explicit physics simulator, 3D geometry, 4D space-time evolution, object permanence, contact dynamics, and physical laws are only approximated — producing artifacts such as disappearing or morphing objects, unrealistic collisions, and physically implausible motions. Quality further degrades in out-of-distribution environments, safety-critical edge cases, and domains underrepresented in training.
974
-
975
- Cosmos3 outputs should not be treated as physically accurate simulation, reliable ground-truth reasoning, or safety-certified decision making. Applications involving robotics control, autonomous systems, scientific simulation, or safety-critical planning require additional validation, external constraints, system-level safety analysis, and domain-specific guardrails before deployment.
976
 
977
  ## Inference
978
 
979
- **Acceleration Engine:** [PyTorch](https://pytorch.org/), [vLLM](https://github.com/vllm-project/vllm), [vLLM-Omni](https://github.com/vllm-project/vllm-omni), [Hugging Face Diffusers](https://github.com/huggingface/diffusers), [SGLang](https://github.com/sgl-project/sglang), [SGLang Diffusion](https://docs.sglang.io/docs/sglang-diffusion/index)
980
-
981
- **Test Hardware:** GB200 and H100
982
-
983
- ## Ethical Considerations
984
-
985
- NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
986
-
987
- Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.
988
-
989
- Users are responsible for model inputs and outputs. Users are responsible for ensuring safe integration of this model, including implementing guardrails as well as other safety mechanisms, prior to deployment.
990
 
991
- For more detailed information on ethical considerations for this model, please see the Model Card++ [Explainability](EXPLAINABILITY.md), [Bias](BIAS.md), [Safety & Security](SAFETY.md), and [Privacy](PRIVACY.md) subcards. Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
 
1
  ---
2
+ library_name: diffusers
3
  license: other
4
  license_name: openmdw1.1-license
5
+ license_link: https://openmdw.ai/license/1-1/
6
+ pipeline_tag: any-to-any
 
7
  tags:
8
+ - nvidia
9
+ - cosmos
10
+ - cosmos3
11
+ - vllm
12
+ - vllm-omni
13
+ - text, image, video, audio, and action generation
14
+ - omnimodel
 
 
 
15
  ---
16
 
17
  # **Cosmos 3: Omnimodal World Models for Physical AI**
18
+ **[Model Collection](https://huggingface.co/collections/nvidia/cosmos3)** | **[Code](https://github.com/nvidia/cosmos)** | **[Paper](https://huggingface.co/papers/2606.02800)** | **[Website](https://research.nvidia.com/labs/cosmos-lab/cosmos3/)**
19
 
20
  [NVIDIA Cosmos™](https://github.com/nvidia/cosmos) is a world foundation model platform designed to accelerate the development of Physical AI by enabling machines to understand, simulate, and interact with the physical world across robotics, autonomous driving, and smart space environments, including industrial and factory-scale applications.
21
 
22
+ Cosmos 3 is a family of omnimodal world models designed to jointly process and generate language, image, video, audio, and action sequences within a unified mixture-of-transformers architecture.
23
+
24
+ ## Sample Usage (Diffusers)
25
+
26
+ Cosmos 3 is fully supported within the Hugging Face `diffusers` library.
27
+
28
+ ```python
29
+ import torch
30
+ from diffusers import Cosmos3OmniPipeline
31
+ from diffusers.schedulers.scheduling_unipc_multistep import UniPCMultistepScheduler
32
+ from diffusers.utils import export_to_video
33
+
34
+ pipe = Cosmos3OmniPipeline.from_pretrained(
35
+ "nvidia/Cosmos3-Nano",
36
+ torch_dtype=torch.bfloat16,
37
+ device_map="cuda",
38
+ )
39
+ pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=10.0)
40
+
41
+ result = pipe(
42
+ prompt="A mobile robot navigates a warehouse aisle and stops at a shelf.",
43
+ negative_prompt="",
44
+ image=None,
45
+ num_frames=189,
46
+ height=720,
47
+ width=1280,
48
+ fps=24,
49
+ num_inference_steps=35,
50
+ guidance_scale=6.0,
51
+ enable_sound=False,
52
+ add_resolution_template=False,
53
+ add_duration_template=False,
54
+ generator=torch.Generator(device="cuda").manual_seed(1234),
55
+ )
56
+
57
+ export_to_video(result.video, "cosmos3_t2v.mp4", fps=24, macro_block_size=1)
58
+ ```
59
+
60
  # Model Overview: Cosmos3-Nano
61
 
62
  ## Description
 
71
  - Cosmos3-Nano:
72
  - Given multimodal inputs including text, images, video, audio, and action trajectories, generate coherent text, images, video, audio, and action outputs for multimodal understanding, world simulation, future prediction, action reasoning, and Physical AI applications.
73
 
74
+ - Cosmos3-Super:
75
+ - Given multimodal inputs including text, images, video, audio, and action trajectories, generate coherent text, images, video, audio, and action outputs for multimodal understanding, world simulation, future prediction, action reasoning, and Physical AI applications.
 
 
 
76
 
77
+ - Cosmos3-Nano-Policy-DROID:
78
+ - Given language instructions and visual observations from the DROID robot platform, generate robot action trajectories for manipulation and control tasks.
79
 
80
+ - Cosmos3-Super-Image2Video:
81
+ - Given one input image and text instructions, generate temporally coherent video sequences that are consistent with the provided visual content.
82
 
83
+ - Cosmos3-Super-Text2Image:
84
+ - Given text input, generate high-fidelity images that are consistent with the provided description.
85
 
86
+ ### License
87
 
88
+ This model is released under the [OpenMDW1.1](https://openmdw.ai/license/1-1/)
 
89
 
90
  ## Model Architecture
91
 
92
  **Architecture Type:** Transformer
 
93
  **Network Architecture:** Mixture-of-Transformers (MoT)
94
 
95
+ Cosmos3 is an Omni-modal foundation model built on a Mixture-of-Transformers (MoT) architecture consisting of two complementary transformer towers: an autoregressive transformer for discrete token generation and a diffusion transformer for continuous multimodal generation. During inference, text is generated through standard next-token autoregressive decoding, while non-text modalities, such as images, video, audio, and actions, are synthesized through iterative denoising.
 
 
96
 
97
  **Number of trainable model parameters:**
 
98
  - Cosmos3-Nano: 16B
99
+ - Cosmos3-Super: 64B
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
100
 
101
  ## Software Integration
102
 
103
  **Runtime Engine(s):**
 
104
  - [PyTorch](https://github.com/nvidia/cosmos3)
105
  - [vLLM-Omni](https://github.com/vllm-project/vllm-omni)
106
  - [Hugging Face Diffusers](https://huggingface.co/docs/diffusers/en/index)
 
107
 
108
  **Supported Hardware Microarchitecture Compatibility:**
 
109
  - NVIDIA Ampere
110
  - NVIDIA Blackwell
111
  - NVIDIA Hopper
112
 
 
 
 
 
 
 
 
 
113
  ## Training, Testing, and Evaluation Datasets
114
 
115
  ### Dataset Overview
 
116
  - **Total Size:** 1.3B data points
117
  - **Total Number of Datasets:** 393 dataset entries
 
 
 
 
 
 
 
 
118
 
119
+ The training, testing, and evaluation datasets consist of diverse multimodal video, image, audio, action, synthetic, and sensor-conditioned data sourced from NVIDIA-owned data and publicly available, commercially permissive datasets.
120
 
121
  **Data Modality and Training Data Size**
122
 
 
128
  | Audio | Not Applicable | 139M |
129
  | Action | Not Applicable | 8M |
130
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
131
  ## Benchmarks
132
 
133
+ Please see our [technical paper](https://huggingface.co/papers/2606.02800) for detailed evaluations of the model.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
134
 
135
  ## Limitations
136
 
137
+ Cosmos3 may produce artifacts in long, high-resolution, or physically complex outputs. Common failure modes include temporal inconsistency, unstable camera or object motion, inaccurate sound-video alignment, imperfect action-state consistency, and physically implausible dynamics. It does not have an explicit physics engine and approximates physical laws.
 
 
138
 
139
  ## Inference
140
 
141
+ **Acceleration Engine:** [PyTorch](https://pytorch.org/), [vLLM](https://github.com/vllm-project/vllm), [vLLM-Omni](https://github.com/vllm-project/vllm-omni), [Hugging Face Diffusers](https://github.com/huggingface/diffusers)
 
 
 
 
 
 
 
 
 
 
142
 
143
+ **Test Hardware:** GB200 and H100
modular_model_index.json DELETED
@@ -1,75 +0,0 @@
1
- {
2
- "_blocks_class_name": "Cosmos3OmniBlocks",
3
- "_class_name": "Cosmos3OmniModularPipeline",
4
- "_diffusers_version": "0.39.0.dev0",
5
- "text_tokenizer": [
6
- "transformers",
7
- "Qwen2TokenizerFast",
8
- {
9
- "pretrained_model_name_or_path": "nvidia/Cosmos3-Nano",
10
- "revision": null,
11
- "subfolder": "text_tokenizer",
12
- "type_hint": [
13
- "transformers",
14
- "Qwen2TokenizerFast"
15
- ],
16
- "variant": null
17
- }
18
- ],
19
- "vae": [
20
- "diffusers",
21
- "AutoencoderKLWan",
22
- {
23
- "pretrained_model_name_or_path": "nvidia/Cosmos3-Nano",
24
- "revision": null,
25
- "subfolder": "vae",
26
- "type_hint": [
27
- "diffusers",
28
- "AutoencoderKLWan"
29
- ],
30
- "variant": null
31
- }
32
- ],
33
- "transformer": [
34
- "diffusers",
35
- "Cosmos3OmniTransformer",
36
- {
37
- "pretrained_model_name_or_path": "nvidia/Cosmos3-Nano",
38
- "revision": null,
39
- "subfolder": "transformer",
40
- "type_hint": [
41
- "diffusers",
42
- "Cosmos3OmniTransformer"
43
- ],
44
- "variant": null
45
- }
46
- ],
47
- "scheduler": [
48
- "diffusers",
49
- "UniPCMultistepScheduler",
50
- {
51
- "pretrained_model_name_or_path": "nvidia/Cosmos3-Nano",
52
- "revision": null,
53
- "subfolder": "scheduler",
54
- "type_hint": [
55
- "diffusers",
56
- "UniPCMultistepScheduler"
57
- ],
58
- "variant": null
59
- }
60
- ],
61
- "sound_tokenizer": [
62
- "diffusers",
63
- "Cosmos3AVAEAudioTokenizer",
64
- {
65
- "pretrained_model_name_or_path": "nvidia/Cosmos3-Nano",
66
- "revision": null,
67
- "subfolder": "sound_tokenizer",
68
- "type_hint": [
69
- "diffusers",
70
- "Cosmos3AVAEAudioTokenizer"
71
- ],
72
- "variant": null
73
- }
74
- ]
75
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
sound_tokenizer/diffusion_pytorch_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:a4b6da5975bf89f6853fe589ad4752d281ac79fbdfad52ea90537fa080b4b9c2
3
- size 1985176840
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9d4c61cde38acfb0cad9048a140c3533750277a8462b19dc08450d9fe1ad9879
3
+ size 1892409600