assets/sample_data/README.md

Shared daytime sample for Cosmos3 Auto

One synchronized, seven-camera scene for Cosmos3-Nano-Transfer-Auto and Cosmos3-Nano-Sim-Auto. The bundle contains real RGB and LiDAR targets, rendered controls, and camera-local captions. Model checkpoints, RGB and LiDAR VAEs, and the matching text tokenizer are supplied separately.

This sample and its original captions in daytime_scene/captions/ come from the released NVIDIA PhysicalAI Autonomous Vehicles dataset. They are packaged here for use with both Auto models.

Contents

sample_data/
  README.md
  FILES.tsv
  SHA256SUMS
  input-transfer-auto.jsonl
  input-sim-auto.jsonl
  training-index.jsonl
  daytime_scene/
    rgb/<camera_key>.mp4       # seven synchronized real videos
    wsm/<camera_key>.mp4       # seven shared WSM control videos
    captions/<camera_key>.json # original captions from the dataset release
    captions_with_timestamp/
      <camera_key>.json        # seven timestamped alternatives
      README.md
    lidar/control.tar         # HD-map range controls
    lidar/target.tar          # recorded range/intensity targets

Download the entire directory and set AUTO_SAMPLE_ROOT to its local path. All asset references are relative; keep the directory structure together. Verify the download with sha256sum -c SHA256SUMS from that directory.

Sensor and caption contract

Camera key	Learned camera ID
camera_front_wide_120fov	0
camera_cross_right_120fov	1
camera_rear_right_70fov	2
camera_rear_tele_30fov	3
camera_rear_left_70fov	4
camera_cross_left_120fov	5
camera_front_tele_30fov	6

The filenames and camera_key fields identify physical cameras. Do not replace these IDs with a view's position in a list.

• RGB and WSM: 1280 × 720, 300 frames at 30 FPS.
• Captions: caption_structured.chunk_0_300, covering the end-exclusive interval [0,300) for the corresponding camera.
• WSM: v4.0.1, shared by both models. The renderer's configuration value remains version=v4; the sample bundle version is unrelated to WSM version.
• LiDAR: 100 Pandar128 sweeps at 10 Hz, with camera-clock IDs 000000,000003,...,000297. Every archive member uses the clip key daytime_scene.
• LiDAR archives store sparse native coordinates on a 128 × 3600 grid. The matching runtime pools the native columns to 1800 semantic columns and applies four circular-padding columns at each side (1808 model columns). Do not pre-pool these archive coordinates to 1800 columns: that would discard half the azimuth coverage when the native reader pools again.
• Each control sweep contains lidar_row, lidar_col, and lidar_range NPZ members. Targets additionally contain lidar_intensity. Range is in meters; intensity is in [0,1]. The matching projection uses a 5–100 m range.

Caption variants

The original captions/ files use a single descriptive paragraph per camera. We also provide captions_with_timestamp/, which better matches our models' training caption style and has empirically produced better inference results. Both formats are supported; prefer timestamped captions for post-training to reduce the caption-format gap.

The supplied requests and training index select the original captions by default. See timestamped captions to select the alternative for inference or training.

Inference

Use the native Framework inference entry point with either request:

torchrun --standalone --nproc-per-node=4 \
  -m cosmos_framework.scripts.inference \
  -i "${AUTO_SAMPLE_ROOT:?}/input-transfer-auto.jsonl" \
  -o outputs/transfer-auto-daytime \
  --checkpoint-path "${TRANSFER_AUTO_CHECKPOINT:?}" \
  --cp-size 4 --cfgp-size 1 --dp-shard-size 1 --dp-replicate-size 1 \
  --no-use-torch-compile --no-use-cuda-graphs --no-diffusion-cache

For Sim-Auto, use input-sim-auto.jsonl, the matching Sim-Auto checkpoint, and a separate output directory. Follow Framework's installation and model inference guides, including Sim-Auto's FlashAttention-4 dependencies. Set COSMOS_TRAINING=0 and COSMOS_INTERNAL=0 for inference.

The requests select seven cameras, the first RGB frame as image conditioning, per-camera captions, and joint LiDAR generation from HD-map controls. They do not supply an initial measured LiDAR sweep. Framework determines joint generation from these inputs; joint_generation is not a supported JSONL field.

Both request 297 output frames at 30 FPS and seed 0. Transfer-Auto uses guidance 3, 35 steps, and shift 10. Sim-Auto uses guidance 1 and the checkpoint's fixed four-step sampler. Expected outputs are seven 1280 × 720 videos and lidar.safetensors with range_m and intensity of shape [99,128,1800], plus frame_ids equal to 0,3,...,294. Check sample_outputs.json for success.

Training with Framework's existing local Auto loader

Both models use the same training-index.jsonl. It maps existing files to native training member names; no sample-specific dataset adapter is needed. From an installed Framework environment, package the media with the native builder:

python -m cosmos_framework.data.generator.local_simulation.auto_bundle \
  --index "${AUTO_SAMPLE_ROOT:?}/training-index.jsonl" \
  --tokenizer "${LOCAL_QWEN_TOKENIZER:?set the matching local tokenizer}" \
  --output "${AUTO_BUNDLE:?set a new output directory}" \
  --camera-keys \
    camera_front_wide_120fov camera_cross_right_120fov \
    camera_rear_right_70fov camera_rear_tele_30fov \
    camera_rear_left_70fov camera_cross_left_120fov camera_front_tele_30fov

The builder writes manifest.json, samples.json, assets/, and resources/tokenizer/. It copies the supplied media and tokenizer; it does not render controls, generate captions, or train a model. Use the resulting bundle with COSMOS_TRANSFER_AUTO_LOCAL_BUNDLE or COSMOS_SIM_AUTO_LOCAL_BUNDLE, then follow the corresponding training skill for the selected training stage.

This sample provides only the 720 source stratum. Select that stream and remove the absent 480 stream when wiring the training loader. Use a window that fits within the 300 source frames; the default full-length training recipe may request more frames. A short smoke run does not reproduce the full 480/720 mixture, eleven-camera rig, or long-window training curriculum.

One scene demonstrates the format; it is not a training corpus or an independent evaluation set. Supply additional recordings and separate training/validation drives. File integrity and loader checks do not establish training convergence, sensor calibration accuracy, or generation quality.