Add pipeline tag, library name, and paper link

README.md

@@ -1,463 +1,52 @@
 ---
+library_name: diffusers
 license: other
 license_name: openmdw1.1-license
-license_link: >-
-  https://openmdw.ai/license/1-1/
-library_name: cosmos
+license_link: https://openmdw.ai/license/1-1/
+pipeline_tag: text-to-image
 tags:
-  - nvidia
-  - cosmos
-  - cosmos3
-  - vllm-omni
-  - sglang
-  - sglang-diffusion
-  - diffusers
-  - text-to-image
-  - image-generation
+- nvidia
+- cosmos
+- cosmos3
+- vllm-omni
+- image-generation
 ---
 
 # **Cosmos 3: Omnimodal World Models for Physical AI**
-**[Model Collection](https://huggingface.co/collections/nvidia/cosmos3)** | **[Code](https://github.com/nvidia/cosmos)** | **[White Paper](https://research.nvidia.com/labs/cosmos-lab/cosmos3/technical-report.pdf)** | **[Website](https://research.nvidia.com/labs/cosmos-lab/cosmos3/)**
+**[Model Collection](https://huggingface.co/collections/nvidia/cosmos3)** | **[Code](https://github.com/nvidia/cosmos)** | **[Paper](https://huggingface.co/papers/2606.02800)** | **[Website](https://research.nvidia.com/labs/cosmos-lab/cosmos3/)**
 
-[NVIDIA Cosmos™](https://github.com/nvidia/cosmos) is a world foundation model platform designed to accelerate the development of Physical AI by enabling machines to understand, simulate, and interact with the physical world across robotics, autonomous driving, and smart space environments, including industrial and factory-scale applications.
+[NVIDIA Cosmos™](https://github.com/nvidia/cosmos) is a world foundation model platform designed to accelerate the development of Physical AI by enabling machines to understand, simulate, and interact with the physical world across robotics, autonomous driving, and smart space environments.
+
+Cosmos 3 is a family of omnimodal world models designed to jointly process and generate language, image, video, audio, and action sequences within a unified mixture-of-transformers architecture.
 
 # Model Overview: Cosmos3-Super-Text2Image
 
 ## Description
 
-Cosmos3 is a collection of Omnimodal world models capable of generating dynamic, high-quality video, image, audio, and action commands from combinations of text, image, video, and action trajectory inputs. It serves as a foundational building block for a broad range of Physical AI applications and research spanning world understanding, world generation, simulation, and embodied policy learning.
-
-This model is ready for commercial and non-commercial use.
-
-**Model Developer:** NVIDIA
-
-### Model Versions
-- Cosmos3-Nano:
-  - Given multimodal inputs including text, images, video, audio, and action trajectories, generate coherent text, images, video, audio, and action outputs for multimodal understanding, world simulation, future prediction, action reasoning, and Physical AI applications.
-
-- Cosmos3-Super:
-  - Given multimodal inputs including text, images, video, audio, and action trajectories, generate coherent text, images, video, audio, and action outputs for multimodal understanding, world simulation, future prediction, action reasoning, and Physical AI applications.
-
-- Cosmos3-Nano-Policy-DROID:
-  - Given language instructions and visual observations from the DROID robot platform, generate robot action trajectories for manipulation and control tasks.
-
-- Cosmos3-Super-Image2Video:
-  - Given one input image and text instructions, generate temporally coherent video sequences that are consistent with the provided visual content.
-
-- Cosmos3-Super-Text2Image:
-  - Given text input, generate high-fidelity images that are consistent with the provided description.
-
-### License
-
-This model is released under the [OpenMDW1.1](https://openmdw.ai/license/1-1/)
-
-### Deployment Geography
-
-Global
-
-### Use Case
-
-Physical AI: Encompassing robotics, autonomous vehicles (AV), and smart space environments, including industrial and factory-scale applications.
-
-### Release Date
-
-Hugging Face 05/31/2026 via [https://huggingface.co/collections/nvidia/cosmos3](https://huggingface.co/collections/nvidia/cosmos3)
-GitHub 05/31/2026 via [https://github.com/nvidia/cosmos](https://github.com/nvidia/cosmos)
-
-## Model Architecture
-
-**Architecture Type:** Transformer
-
-**Network Architecture:** Mixture-of-Transformers (MoT)
-
-Cosmos3 is an Omni-modal foundation model built on a Mixture-of-Transformers (MoT) architecture consisting of two complementary transformer towers: an autoregressive transformer for discrete token generation and a diffusion transformer for continuous multimodal generation. During inference, text is generated through standard next-token autoregressive decoding, while non-text modalities, such as images, video, audio, and actions, are synthesized through iterative denoising. This unified architecture enables Cosmos3 to model heterogeneous modalities within a single framework while preserving generation mechanisms best suited to each modality.
-
-**This model was developed based on:**  [Cosmos Framework](https://github.com/nvidia/cosmos-framework)
-
-**Number of trainable model parameters:**
-
-- Cosmos3-Nano: 16B
-- Cosmos3-Super: 64B
-- Cosmos3-Nano-Policy-DROID: 16B
-- Cosmos3-Super-Image2Video: 64B
-- Cosmos3-Super-Text2Image: 64B
-
-## Input/Output Specifications
-
-- **Generator Input**
-  - **Input Type(s)**: Text, Image, Video (with audio or without audio), Action Trajectory
-  - **Input Format(s)**:
-    - Text: String
-    - Image: jpg, png, jpeg, webp
-    - Video (with or without audio): mp4
-    - Action: json (1D list)
-  - **Input Parameters**:
-    - Text: One-dimensional (1D)
-    - Image: Two-dimensional (2D)
-    - Video: Three-dimensional (3D)
-    - Audio: One-dimensional (1D)
-    - Action trajectory: One-dimensional (1D)
-  - **Other Properties Related to Input**:
-    - For video inputs, we accept various resolutions, including 720p, 480p, and 256p.
-    - When using input video with audio muxed into the video MP4 file, the audio should have 2 channels (stereo) and a 48 kHz sample rate.
-    - Image and video inputs are RGB color (8 bits per channel, sRGB color space); grayscale inputs are not supported.
-    - Action input is a per-frame sequence of robot/agent state or control values (e.g., joint positions, gripper state, camera pose). The full input is a 2D array shaped (T, D), where T is the number of frames and D is the embodiment-specific dimensionality listed below.
-    - Input action is only supported for compatible embodiments, including general camera motion (9D), autonomous vehicle (9D), egocentric motion (57D), single Franka Panda arm with RobotiQ gripper (10D), dual Franka Panda arm with RobotiQ gripper (20D), Agibot (29D), UR (10D), Google robot (10D), WidowX 250 (10D), UMI (9D).
-  - **Input Size and Length limits:**
-    - **Text:** 4096 tokens
-    - **Image:** 256p, 480p, and 720p resolution at one of these aspect ratios (16:9, 4:3, 1:1, 3:4, 9:16)
-    - **Video:** 256p, 480p, and 720p resolution at one of these aspect ratios (16:9, 4:3, 1:1, 3:4, 9:16). Max number of frames = 5.
-    - **Audio:** Max 0.5 second
-    - **Action:** 16 – 400 video frames
-- **Generator Output**
-  - **Output Type(s)**: Image, video, audio, action, text
-  - **Output Format(s)**:
-    - Image: JPG
-    - Video: MP4
-    - Audio: Advanced Audio Coding (AAC) stream (muxed within the MP4)
-    - Action: 1D list (.json)
-    - Text: string
-  - **Output Parameters**:
-    - Image: Two-dimensional (2D)
-    - Video: Three-dimensional (3D)
-    - Audio: One-dimensional (1D)
-    - Action: One-dimensional (1D)
-    - Text: One-dimensional (1D)
-  - **Other Properties Related to Output**:
-    - The generated video is an MP4 file, with the resolution, frame rate, and duration specified in the input. The generated audio is encoded in AAC format, muxed into the video MP4 file with 2 channels (stereo) and a 48 kHz sample rate.
-    - Video generation supports durations from 5 to 400 frames, with 189 frames as the default generation duration.
-    - The generated action is only supported for compatible embodiments, including general camera motion (9D), autonomous vehicle (9D), egocentric motion (57D), single Franka Panda arm with RobotiQ gripper (10D), dual Franka Panda arm with RobotiQ gripper (20D), Agibot (29D), UR (10D), Google robot (10D), WidowX 250 (10D), UMI (9D).
-    - Audio: 48 kHz stereo AAC stream muxed into video mp4
-    - Video: mp4 at the FPS specified in input
-    - Image: JPEG
-- **Reasoner Input**
-  - **Input Type(s)**: Text, Text+Image, Text+Video
-  - **Input Format(s)**:
-    - Text: String
-    - Image: jpg, png, jpeg, webp
-    - Video: mp4
-  - **Input Parameters**:
-    - Text: One-dimensional (1D)
-    - Image: Two-dimensional (2D)
-    - Video: Three-dimensional (3D)
-  - **Other Properties Related to Input**:
-    - Video inputs are recommended at a frame rate of 4 fps.
-    - Long-context inputs supported up to 256K tokens.
-  - **Input Size and Length limits:**
-    - **Text:** Up to 256K tokens (context window).
-    - **Image:** Standard input image formats; passed as file or URL.
-    - **Video:** mp4 at the recommended 4 fps.
-- **Reasoner Output**
-  - **Output Type(s)**: Text
-  - **Output Format(s)**:
-    - Text: string
-  - **Output Parameters**:
-    - Text: One-dimensional (1D)
-  - **Other Properties Related to Output**:
-    - Default `max_tokens=4096+` is recommended for reasoning outputs; longer outputs may be requested.
-    - Reasoning outputs may include structured chain-of-thought, 2D/3D point localization, and bounding-box coordinates for vision-based tasks.
-
-The video content visualizes the input text description as a short animated scene, capturing key elements within the specified time constraints.
-
-Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g., GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
-
-## Software Integration
-
-**Runtime Engine(s):**
-
-- [PyTorch](https://github.com/nvidia/cosmos3)
-- [vLLM-Omni](https://github.com/vllm-project/vllm-omni)
-- [Hugging Face Diffusers](https://huggingface.co/docs/diffusers/en/index)
-- [SGLang](https://github.com/sgl-project/sglang)
-
-**Supported Hardware Microarchitecture Compatibility:**
-
-- NVIDIA Ampere
-- NVIDIA Blackwell
-- NVIDIA Hopper
-
-**Operating System(s):**
-
-- Linux (We have not tested on other operating systems.)
-
-**Note:** Only BF16 precision is tested. Other precisions like FP4, FP8, and FP16 are not officially supported.
-
-The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
-
-## Training, Testing, and Evaluation Datasets
-
-### Dataset Overview
+Cosmos3-Super-Text2Image is a 64B parameter model designed to generate high-fidelity images from text input. It serves as a foundational building block for Physical AI applications spanning world understanding, simulation, and embodied policy learning.
 
-- **Total Size:** 1.3B data points
-- **Total Number of Datasets:** 393 dataset entries
-- **Dataset partition:** Training [100%], Testing [N/A — evaluation benchmarks used separately], Validation [N/A — evaluation benchmarks used separately]
-- **Time period for training data collection:** 2024–2026
-- **Time period for testing data collection:** N/A (standard public benchmarks)
-- **Time period for validation data collection:** N/A (standard public benchmarks)
-
-Raw data from internal and external sources is transformed into training-ready data through multiple stages of curation, filtering, and quality review. Data acquisition spans diverse multimodal sources — robotics, autonomous driving, industrial environments, indoor and outdoor scenes, varied lighting and weather conditions, camera viewpoints, object categories, and human activities — to broaden coverage across Physical AI operating environments. Automated filtering pipelines remove corrupted, duplicate, low-quality, and restricted content. Metadata analysis, heuristic rules, and model-assisted classifiers are applied during preprocessing to flag anomalous distributions and low-diversity subsets. Human review supplements automated filtering for selected datasets, benchmark construction, and targeted quality analysis. Datasets are balanced across modalities and task categories — visual reasoning, text-to-image, text-to-video, image-to-video, audio generation, video transfer, action-conditioned generation, and action command generation — to reduce overrepresentation of narrow domains. Synthetic and simulation-based augmentation supplements coverage of rare physical interactions and edge-case scenarios. Deduplication and provenance tracking are applied across the corpus. The resulting processed data is converted into model-ready tokenized or encoded representations through modality-specific preprocessors before training begins.
-
-Training datasets passed through multiple layers of automated and manual safeguards designed to reduce the presence of harmful or policy-violating content across categories including weapons and weapons-related instructional content, criminal planning, child sexual abuse material (CSAM), non-consensual intimate imagery (NCII), sexual content involving minors, harassment, hate speech, profanity, threats and incitement to violence, self-harm or suicide-related content, and graphic violence. Data sources are reviewed for licensing compatibility, provenance, and alignment with internal data governance and safety policies before admission into training corpora. Automated filtering pipelines combine multiple detection strategies: hash-matching against known CSAM and NCII reference databases; classifier-based moderation models trained for explicit sexual content, hate speech, violence, weapons imagery, and other restricted categories; keyword and regex-based screening for criminal-planning, threats, and self-harm phrases in text data; metadata and provenance heuristics for source-level risk signals; and embedding-based anomaly detection to surface samples that fall outside expected distributions. Human review and targeted audits supplement automated filtering for selected datasets, benchmark construction, and safety-sensitive evaluation. For multimodal Physical AI data (robotics, autonomous driving, industrial scenes), additional filtering targets invalid action trajectories, physically implausible interactions, and unsafe control sequences. Synthetic and simulation-generated data are evaluated through internal validation before inclusion. Benchmark evaluations and red-team testing are applied post-training to surface remaining safety gaps across world generation, reasoning, audio, and action tasks. No large-scale data-filtering process can guarantee complete removal of all harmful content; residual risks may remain, particularly in rare edge cases or open-world deployment settings. Ongoing monitoring and dataset review continue post-release.
-
-**Data Modality and Training Data Size**
-
-| Modality | Reasoning Data Sample Count | Generation Data Sample Count |
-| -------- | ------------------- | -------------------- |
-| Text     | 22M                 | Not Applicable       |
-| Image    | 19M                 | 767M                 |
-| Video    | 1M                  | 348M                 |
-| Audio    | Not Applicable      | 139M                 |
-| Action   | Not Applicable      | 8M                   |
-
-**Data Collection Method by dataset**
-
-- Hybrid: Automatic/Sensors, Synthetic, Automated
-
-**Labeling Method by dataset**
-
-- Hybrid: Human, Automated
-
-**Properties:** The training, testing, and evaluation datasets consist of diverse multimodal video, image, audio, action, synthetic, and sensor-conditioned data sourced from NVIDIA-owned data and publicly available, commercially permissive datasets. These datasets are curated to exclude known restricted content and to support building an Omni model that learns to generate and reason about dynamic physical environments across world reasoning and generation tasks.
-
-### Public Datasets
-
-| Dataset                                                             | Samples           |
-|---|---|
-| OpenImage | 1.2M |
-| Coyo700M | 100M |
-| YouTube Video | 340M |
-| UMI | 4.5M |
-
-### Private Datasets
-
-| Dataset                                                             | Samples           |
-|---|---|
-| Egocentric | 7M |
-| Nexar | 0.6M |
-| AgiBot | 0.2M |
-| HOI | 0.3M |
-
-### Synthetic Datasets
-
-| Dataset | Samples |
-|---|---|
-| synthetic images generated using HiDream-I1 | 15M |
-| synthetic images generated using Qwen-Image-2512 | 14M |
-| synthetic captions generated using Qwen3-VL | 1115M |
-
-## Evaluation Datasets
-
-**Data Collection Method by dataset**
-
-- Hybrid: Automatic/Sensors, Synthetic, Automated
-
-**Labeling Method by dataset**
-
-- Hybrid: Human, Automated
-
-**Properties:** The training, testing, and evaluation datasets consist of diverse multimodal video, image, audio, action, synthetic, and sensor-conditioned data sourced from NVIDIA-owned data and publicly available, commercially permissive datasets. These datasets are curated to exclude known restricted content and to support building an Omni model that learns to generate and reason about dynamic physical environments across world reasoning and generation tasks.
-
-## Benchmarks
-
-Please see our [technical paper](https://research.nvidia.com/labs/cosmos-lab/cosmos3/technical-report.pdf) for detailed evaluations of the base model.
-
-### Text-to-image benchmark results
-
-![benchmark results](assets/benchmark-text2image.png)
-
-### Artificial Analysis Leaderboard
-
-#### Open-Source Models [2026/05/28/]
-
-![Artificial Analysis Text-to-Image leaderboard — open-source models](assets/benchmark-text2image-leaderboard.png)
-
-#### All Models [2026/05/28/] (Including Closed-Source)
-
-![Artificial Analysis Text-to-Image leaderboard — all models including closed-source](assets/benchmark-text2image-leaderboard-all-models.jpg)
-
-## Qualitative examples
-
-![Qualitative examples](assets/more_images.jpg)
+- **Model Developer:** NVIDIA
+- **Architecture:** Mixture-of-Transformers (MoT)
+- **Trainable Parameters:** 64B
+- **License:** [OpenMDW1.1](https://openmdw.ai/license/1-1/)
 
 ## Usage
 
-- See [Cosmos](https://github.com/nvidia/cosmos) for details.
-
-### Prompt upsampling
-
-For optimal quality, text prompts should be upsampled into a specific JSON structure. Description and code can be found [here](https://github.com/nvidia/cosmos-framework/blob/main/docs/prompt_upsampling.md).
-
-For example, for text-to-image upsampling using Opus-4.7:
-
-```bash
-git clone https://github.com/NVIDIA/cosmos-framework.git packages/cosmos-framework
-pip install -e packages/cosmos-framework
-
-export PROMPT_UPSAMPLER_ENDPOINT_URL="https://api.anthropic.com/v1/"
-export PROMPT_UPSAMPLER_MODEL_NAME="claude-opus-4-7"
-export PROMPT_UPSAMPLER_API_TOKEN="<your_token>"
-
-python -m cosmos_framework.inference.prompt_upsampling \
-    --input assets/original_prompt.txt \
-    --output /tmp/upsampled_t2i_opus/ \
-    --mode text2image \
-    --endpoint-url "${PROMPT_UPSAMPLER_ENDPOINT_URL}" \
-    --model "${PROMPT_UPSAMPLER_MODEL_NAME}" \
-    --api-token "${PROMPT_UPSAMPLER_API_TOKEN}" \
-    --resolution 768 \
-    --aspect-ratio "1,1"
-```
-
-The JSON-upsampled version of `assets/original_prompt.txt` is saved in `assets/example_caption.json` for convenience, and is used for the image generation examples below.
-
-### vLLM-Omni
-
-#### Container
-
-```
-docker pull vllm/vllm-omni:cosmos3
-```
-
-#### General Invocation
-
-You can use the release-tested `vllm-omni` package for deploying an OpenAI-compatible API inference endpoint.
-The recommended vLLM-Omni serving configuration for `nvidia/Cosmos3-Super-Text2Image` on a 8xH100 node is:
-
-```bash
-vllm serve nvidia/Cosmos3-Super-Text2Image \
-  --omni \
-  --host 0.0.0.0 \
-  --port 8000 \
-  --cfg-parallel-size 2 \
-  --ulysses-degree 4 \
-  --tensor-parallel-size 1 \
-  --use-hsdp \
-  --hsdp-shard-size 8 \
-  --init-timeout 1800
-```
-
-Setting `--enable-layerwise-offload` can help with memory usage on GPUs with less available memory; however, please note that for text2image generation, this may incur a significant performance penalty. For 4xH200 or 4xGB200 one can simply use `--cfg-parallel-size 2 --ulysses-degree 2 --tensor-parallel-size 1`.
-
-#### Examples
-
-##### Text to image generation
-```python
-import base64
-import json
-import requests
-
-# 1. Read JSON-upsampled prompt
-json_prompt = json.load(open("assets/example_caption.json"))
-
-# 2. Build your API payload
-payload = {
-    "prompt": json.dumps(json_prompt),
-    "size": "1024x1024",
-    "n": 1,  # single frame generation
-    "num_inference_steps": 50,
-    "guidance_scale": 4.0,
-    "flow_shift": 3.0,
-    "negative_prompt": "",
-    "seed": 1143,
-    "extra_args": {
-      "use_resolution_template": False,
-      "guardrails": True,
-    },
-}
-
-# 3. Send the POST request
-url = "http://localhost:8000/v1/images/generations"
-print("Sending request to server...")
-response = requests.post(url, json=payload, headers={"Content-Type": "application/json"})
-response.raise_for_status()
-
-# 4. Extract the base64 data and decode it into an image
-response_json = response.json()
-b64_data = response_json["data"][0]["b64_json"]
-image_bytes = base64.b64decode(b64_data)
-
-# 5. Save the final PNG file
-with open("/tmp/cosmos3_t2i.png", "wb") as image_file:
-    image_file.write(image_bytes)
-print("Saved image to /tmp/cosmos3_t2i.png")
-```
-
-![example_image](assets/example_image.png)
-
-### SGLang
-
-[SGLang Diffusion](https://docs.sglang.io/docs/sglang-diffusion/index) can serve `nvidia/Cosmos3-Super-Text2Image` through OpenAI-compatible image generation endpoints. Install SGLang from the main branch with diffusion dependencies, then start the server:
-
-```bash
-git clone --branch main https://github.com/sgl-project/sglang.git
-cd sglang
-pip install -e "python[diffusion]"
-pip install "cosmos-guardrail==0.3.1"
-
-sglang serve \
-  --model-path nvidia/Cosmos3-Super-Text2Image \
-  --num-gpus 4
-```
-
-Cosmos 3 support in SGLang Diffusion currently requires the SGLang main branch. Switch to a stable SGLang release once Cosmos 3 support is included there.
-
-Example text-to-image request:
-
-```bash
-curl -sS -X POST http://localhost:30000/v1/images/generations \
-  -H "Content-Type: application/json" \
-  -d '{
-    "prompt": "A warehouse robot folds a blue cloth on a clean workbench.",
-    "size": "1280x720",
-    "n": 1,
-    "num_inference_steps": 35,
-    "guidance_scale": 6.0,
-    "flow_shift": 10.0,
-    "seed": 0,
-    "extra_args": {
-      "use_resolution_template": false,
-      "guardrails": true
-    }
-  }'
-```
-
-For complete serving instructions and request examples, see the [Cosmos3 SGLang cookbook](https://docs.sglang.io/cookbook/diffusion/Cosmos/Cosmos3).
-
 ### Diffusers
 
-Cosmos3 is fully supported within the popular HuggingFace Diffusers package. This integration makes it a supported inference backend, allowing developers to easily incorporate Cosmos3's capabilities - such as text-to-image generation - into their pipelines using the Cosmos3OmniPipeline class, as demonstrated by the provided code examples (see examples for other modalities on the HuggingFace Cosmos3 page).
-
-**Note:** This example is tested on GB200. For H100, use the [vLLM-Omni serving recipe](#vllm-omni) above, which supports multi-GPU deployment via HSDP.
-
-#### Installation
+Cosmos3 is fully supported within the Hugging Face [Diffusers](https://huggingface.co/docs/diffusers/en/index) library.
 
-To install diffusers with Cosmos3OmniPipeline:
-```
-uv venv --python 3.13 --seed --managed-python
-source .venv/bin/activate
-uv pip install \
-  "diffusers @ git+https://github.com/huggingface/diffusers.git" \
-  accelerate \
-  av \
-  cosmos_guardrail \
-  huggingface_hub \
-  imageio \
-  imageio-ffmpeg \
-  torch \
-  torchvision \
-  transformers
-```
-
-#### Examples
-
-##### Text to image generation
 ```python
 import json
 import torch
 from diffusers import Cosmos3OmniPipeline
 from diffusers.schedulers.scheduling_unipc_multistep import UniPCMultistepScheduler
 
-json_prompt = json.load(open("assets/example_caption.json"))
+# Cosmos 3 works best with structured, upsampled prompts.
+# See https://github.com/nvidia/cosmos for prompt upsampling details.
+json_prompt = {
+    "comprehensive_t2i_caption": "A photorealistic studio photograph of two hands working a spinning cylinder of wet gray clay on a pottery wheel, fingers gently pinching and pulling the walls upward to form a narrow neck and rounded belly."
+}
 
 pipe = Cosmos3OmniPipeline.from_pretrained(
     "nvidia/Cosmos3-Super-Text2Image",
@@ -478,28 +67,37 @@ result = pipe(
     generator=torch.Generator(device="cuda").manual_seed(1143),
 )
 
-result.video[0].save("/tmp/cosmos3_t2i.png")
-print("Saved image to /tmp/cosmos3_t2i.png")
+result.video[0].save("cosmos3_t2i.png")
 ```
 
-## Limitations
-
-Cosmos3 may produce imperfect outputs in challenging scenarios. Generation artifacts include temporal inconsistency, unstable camera or object motion, imprecise physical interactions, inaccurate audio-video synchronization, and action-state drift — especially in long-horizon or high-resolution outputs. Reasoning may also be incorrect: object states, causal relationships, spatial geometry, temporal ordering, agent intent, and future outcomes can be misinferred, and complex or long-context inputs may yield hallucinated entities, inconsistent interpretations, or implausible predictions. Because the model lacks an explicit physics simulator, 3D geometry, 4D space-time evolution, object permanence, contact dynamics, and physical laws are only approximated — producing artifacts such as disappearing or morphing objects, unrealistic collisions, and physically implausible motions. Quality further degrades in out-of-distribution environments, safety-critical edge cases, and domains underrepresented in training.
-
-Cosmos3 outputs should not be treated as physically accurate simulation, reliable ground-truth reasoning, or safety-certified decision making. Applications involving robotics control, autonomous systems, scientific simulation, or safety-critical planning require additional validation, external constraints, system-level safety analysis, and domain-specific guardrails before deployment.
+### vLLM-Omni
 
-## Inference
+You can use the `vllm-omni` package for deploying an OpenAI-compatible API inference endpoint.
 
-**Acceleration Engine:** [PyTorch](https://pytorch.org/), [vLLM](https://github.com/vllm-project/vllm), [vLLM-Omni](https://github.com/vllm-project/vllm-omni), [Hugging Face Diffusers](https://github.com/huggingface/diffusers), [SGLang](https://github.com/sgl-project/sglang), [SGLang Diffusion](https://docs.sglang.io/docs/sglang-diffusion/index)
+```bash
+vllm serve nvidia/Cosmos3-Super-Text2Image \
+  --omni \
+  --host 0.0.0.0 \
+  --port 8000 \
+  --cfg-parallel-size 2 \
+  --ulysses-degree 4 \
+  --tensor-parallel-size 1 \
+  --use-hsdp \
+  --hsdp-shard-size 8 \
+  --init-timeout 1800
+```
 
-**Test Hardware:** GB200 and H100
+## Input/Output Specifications
 
-## Ethical Considerations
+- **Input Type**: Text
+- **Output Type**: Image (JPEG)
+- **Input Size**: Up to 4096 tokens
+- **Supported Resolutions**: 256p, 480p, and 720p resolution at aspect ratios 16:9, 4:3, 1:1, 3:4, 9:16.
 
-NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications.  Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+## Limitations
 
-Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.
+Cosmos3 may produce imperfect outputs in challenging scenarios, such as imprecise physical interactions or object morphing. Because the model lacks an explicit physics simulator, physical laws are only approximated. Quality further degrades in out-of-distribution environments or safety-critical edge cases. Outputs should not be treated as physically accurate simulations or safety-certified decision-making tools.
 
-Users are responsible for model inputs and outputs. Users are responsible for ensuring safe integration of this model, including implementing guardrails as well as other safety mechanisms, prior to deployment.
+## Ethical Considerations
 
-For more detailed information on ethical considerations for this model, please see the Model Card++ [Explainability](EXPLAINABILITY.md), [Bias](BIAS.md), [Safety & Security](SAFETY.md), and [Privacy](PRIVACY.md) subcards. Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
+NVIDIA believes Trustworthy AI is a shared responsibility. Users are responsible for ensuring safe integration of this model, including implementing guardrails and other safety mechanisms prior to deployment. Please ensure you have proper rights and permissions for all input content.

agentic_upsampling/clients.py

@@ -477,9 +477,8 @@ class VLMQualityJudge:
         api_token: str,
         endpoint_url: str = DEFAULT_CRITIC_ENDPOINT_URL,
         model: str = DEFAULT_CRITIC_MODEL,
-        max_tokens: int = 16384,
+        max_tokens: int = 8192,
         image_jpeg_quality: int | None = DEFAULT_JPEG_QUALITY,
-        extra_body: dict[str, Any] | None = None,
     ) -> None:
         self.chat_client = OpenAIChatClient(
             ChatClientConfig(
@@ -488,7 +487,6 @@ class VLMQualityJudge:
                 api_token=api_token,
                 max_tokens=max_tokens,
                 max_retries=3,
-                extra_body=extra_body,
             )
         )
         self.image_jpeg_quality = image_jpeg_quality

agentic_upsampling/runner.py

@@ -3,7 +3,6 @@
 from __future__ import annotations
 
 import json
-import random
 import traceback
 from concurrent.futures import ThreadPoolExecutor, as_completed
 from dataclasses import dataclass
@@ -113,12 +112,6 @@ class AgenticUpsamplerRunner:
         (item_dir / "failure.json").unlink(missing_ok=True)
         (item_dir / "incomplete.json").unlink(missing_ok=True)
         self._log(f"[prompt {item.prompt_id}] start")
-        # One base per item, reused by every iteration (sample slot j gets base + j), so a
-        # rewrite is compared against the same noise. A resumed run draws a new base.
-        seed_base = self.config.seed_base
-        if seed_base is None:
-            seed_base = random.randint(0, 2**31 - 1 - self.config.samples_per_iteration)
-            self._log(f"[prompt {item.prompt_id}] seed_base drawn: {seed_base}")
         candidates: list[dict[str, Any]] = []
         previous_prompt: dict[str, Any] | None = None
         previous_analysis: dict[str, Any] | None = None
@@ -138,7 +131,6 @@ class AgenticUpsamplerRunner:
                         previous_analysis,
                         previous_negative_prompt,
                         candidates,
-                        seed_base,
                     )
                 except Exception as exc:
                     if not candidates:
@@ -160,9 +152,7 @@ class AgenticUpsamplerRunner:
                 self._log(f"[prompt {item.prompt_id}] early stop at iter={iteration}")
                 break
 
-        result = self.finalize_item(item, candidates, incomplete_error=incomplete_error)
-        result["seed_base"] = seed_base
-        return result
+        return self.finalize_item(item, candidates, incomplete_error=incomplete_error)
 
     def run_item_safely(self, item: PromptItem) -> dict[str, Any]:
         """Run one item and convert failures into structured records."""
@@ -189,7 +179,6 @@ class AgenticUpsamplerRunner:
         previous_analysis: dict[str, Any] | None,
         previous_negative_prompt: str,
         candidates: list[dict[str, Any]],
-        seed_base: int,
     ) -> dict[str, Any]:
         prepared = self.prepare_iteration_prompt(
             item,
@@ -206,7 +195,6 @@ class AgenticUpsamplerRunner:
             iteration,
             prepared.prompt_json,
             prepared.negative_prompt,
-            seed_base,
         )
         return self.finalize_iteration(item, iteration_dir, iteration, sample_candidates, sample_errors)
 
@@ -217,7 +205,6 @@ class AgenticUpsamplerRunner:
         iteration: int,
         prompt_json: dict[str, Any],
         negative_prompt: str,
-        seed_base: int,
     ) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
         """Generate seed samples concurrently, then judge successful images in sample order."""
         generation_outputs: dict[int, GenerationOutput] = {}
@@ -231,7 +218,6 @@ class AgenticUpsamplerRunner:
                     sample_index,
                     prompt_json,
                     negative_prompt,
-                    seed_base,
                 ): sample_index
                 for sample_index in range(self.config.samples_per_iteration)
             }
@@ -314,11 +300,8 @@ class AgenticUpsamplerRunner:
         sample_index: int,
         prompt_json: dict[str, Any],
         negative_prompt: str,
-        seed_base: int,
     ) -> dict[str, Any]:
-        generation = self.run_generation_sample(
-            item, iteration_dir, sample_index, prompt_json, negative_prompt, seed_base
-        )
+        generation = self.run_generation_sample(item, iteration_dir, sample_index, prompt_json, negative_prompt)
         return self.judge_iteration_sample(
             item,
             iteration_dir,
@@ -336,7 +319,6 @@ class AgenticUpsamplerRunner:
         sample_index: int,
         prompt_json: dict[str, Any],
         negative_prompt: str,
-        seed_base: int,
     ) -> GenerationOutput:
         """Generate one sample image for an iteration."""
         sample_dir = self._sample_dir(iteration_dir, sample_index)
@@ -346,7 +328,7 @@ class AgenticUpsamplerRunner:
             prompt_json=prompt_json,
             prompt_id=item.prompt_id,
             output_dir=sample_dir,
-            seed=seed_base + sample_index,
+            seed=self._sample_seed(sample_index),
             negative_prompt=negative_prompt,
             jpeg_quality=self.config.jpeg_quality,
         )
@@ -440,6 +422,11 @@ class AgenticUpsamplerRunner:
         if self.config.verbose:
             print(message, flush=True)
 
+    def _sample_seed(self, sample_index: int) -> int | None:
+        if self.config.seed_base is None:
+            return None
+        return self.config.seed_base + sample_index
+
     def _sample_dir(self, iteration_dir: Path, sample_index: int) -> Path:
         if self.config.samples_per_iteration == 1:
             return iteration_dir

modular_model_index.json

@@ -1,75 +0,0 @@
-{
-  "_blocks_class_name": "Cosmos3OmniBlocks",
-  "_class_name": "Cosmos3OmniModularPipeline",
-  "_diffusers_version": "0.39.0.dev0",
-  "text_tokenizer": [
-    "transformers",
-    "Qwen2TokenizerFast",
-    {
-      "pretrained_model_name_or_path": "nvidia/Cosmos3-Super-Text2Image",
-      "revision": null,
-      "subfolder": "text_tokenizer",
-      "type_hint": [
-        "transformers",
-        "Qwen2TokenizerFast"
-      ],
-      "variant": null
-    }
-  ],
-  "vae": [
-    "diffusers",
-    "AutoencoderKLWan",
-    {
-      "pretrained_model_name_or_path": "nvidia/Cosmos3-Super-Text2Image",
-      "revision": null,
-      "subfolder": "vae",
-      "type_hint": [
-        "diffusers",
-        "AutoencoderKLWan"
-      ],
-      "variant": null
-    }
-  ],
-  "transformer": [
-    "diffusers",
-    "Cosmos3OmniTransformer",
-    {
-      "pretrained_model_name_or_path": "nvidia/Cosmos3-Super-Text2Image",
-      "revision": null,
-      "subfolder": "transformer",
-      "type_hint": [
-        "diffusers",
-        "Cosmos3OmniTransformer"
-      ],
-      "variant": null
-    }
-  ],
-  "scheduler": [
-    "diffusers",
-    "UniPCMultistepScheduler",
-    {
-      "pretrained_model_name_or_path": "nvidia/Cosmos3-Super-Text2Image",
-      "revision": null,
-      "subfolder": "scheduler",
-      "type_hint": [
-        "diffusers",
-        "UniPCMultistepScheduler"
-      ],
-      "variant": null
-    }
-  ],
-  "sound_tokenizer": [
-    "diffusers",
-    "Cosmos3AVAEAudioTokenizer",
-    {
-      "pretrained_model_name_or_path": "nvidia/Cosmos3-Super-Text2Image",
-      "revision": null,
-      "subfolder": "sound_tokenizer",
-      "type_hint": [
-        "diffusers",
-        "Cosmos3AVAEAudioTokenizer"
-      ],
-      "variant": null
-    }
-  ]
-}

tests/test_agentic_upsampling.py

@@ -10,7 +10,7 @@ from typing import Any
 
 from PIL import Image
 
-from agentic_upsampling.clients import ImageGenerationClient, PromptRewriterClient, VLMQualityJudge
+from agentic_upsampling.clients import ImageGenerationClient, PromptRewriterClient
 from agentic_upsampling.constants import (
     DEFAULT_CRITIC_ENDPOINT_URL,
     DEFAULT_CRITIC_MODEL,
@@ -413,32 +413,7 @@ def test_runner_early_stops_by_default(tmp_path: Path) -> None:
     assert result["best_iteration"] == 0
     assert rewriter.initial_calls == 1
     assert rewriter.joint_rewrite_calls == 0
-    assert generator.seeds == [result["seed_base"]]
-
-
-def test_runner_draws_one_seed_base_when_unset(tmp_path: Path) -> None:
-    """An unset seed_base is drawn once and reused, so slot j gets base + j every iteration."""
-    rewriter = FakeRewriter()
-    generator = FakeGenerator()
-    runner = AgenticUpsamplerRunner(
-        rewriter=rewriter,
-        generator=generator,  # type: ignore[arg-type]
-        judge=FakeJudge([5.0, 6.0, 5.5, 6.5]),
-        config=RunnerConfig(
-            output_dir=tmp_path,
-            max_iterations=2,
-            samples_per_iteration=2,
-            early_stop=False,
-        ),
-    )
-
-    result = runner.run_item(_item())
-
-    seed_base = result["seed_base"]
-    assert seed_base is not None
-    # Samples are generated concurrently, so compare each iteration's seeds as a set.
-    assert sorted(generator.seeds[:2]) == [seed_base, seed_base + 1]
-    assert sorted(generator.seeds[2:]) == [seed_base, seed_base + 1]
+    assert generator.seeds == [None]
 
 
 def test_runner_can_disable_early_stop_and_select_best_sample(tmp_path: Path) -> None:
@@ -459,9 +434,7 @@ def test_runner_can_disable_early_stop_and_select_best_sample(tmp_path: Path) ->
 
     result = runner.run_item(_item("8", "exactly 12 balloons with exact color counts"))
 
-    # Samples are generated concurrently, so compare each iteration's seeds as a set.
-    assert sorted(generator.seeds[:3]) == [1000, 1001, 1002]
-    assert sorted(generator.seeds[3:]) == [1000, 1001, 1002]
+    assert generator.seeds == [1000, 1001, 1002, 1000, 1001, 1002]
     assert rewriter.previous_scores == [9.0]
     assert result["best_iteration"] == 1
     assert result["best"]["selected_sample_index"] == 1
@@ -521,14 +494,3 @@ def test_extract_best_images_copies_images_and_writes_manifests(tmp_path: Path)
     assert (tmp_path / "export" / "best_generations.jsonl").exists()
     assert (tmp_path / "export" / "best_generations.csv").exists()
 
-
-def test_vlm_quality_judge_sends_no_extra_body_by_default() -> None:
-    judge = VLMQualityJudge(api_token="token")
-
-    assert judge.chat_client.config.extra_body is None
-
-
-def test_vlm_quality_judge_forwards_extra_body() -> None:
-    judge = VLMQualityJudge(api_token="token", extra_body={"reasoning_effort": "high"})
-
-    assert judge.chat_client.config.extra_body == {"reasoning_effort": "high"}