assets/sample_data/README.md

# Shared daytime sample for Cosmos3 Auto

One synchronized, seven-camera scene for **Cosmos3-Nano-Transfer-Auto** and
**Cosmos3-Nano-Sim-Auto**. The bundle contains real RGB and LiDAR targets,
rendered controls, and camera-local captions. Model checkpoints, RGB and LiDAR
VAEs, and the matching text tokenizer are supplied separately.

This sample and its original captions in `daytime_scene/captions/` come from
the released [NVIDIA PhysicalAI Autonomous Vehicles dataset](https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles).
They are packaged here for use with both Auto models.

## Contents

```text
sample_data/
  README.md
  FILES.tsv
  SHA256SUMS
  input-transfer-auto.jsonl
  input-sim-auto.jsonl
  training-index.jsonl
  daytime_scene/
    rgb/<camera_key>.mp4       # seven synchronized real videos
    wsm/<camera_key>.mp4       # seven shared WSM control videos
    captions/<camera_key>.json # original captions from the dataset release
    captions_with_timestamp/
      <camera_key>.json        # seven timestamped alternatives
      README.md
    lidar/control.tar         # HD-map range controls
    lidar/target.tar          # recorded range/intensity targets
```

Download the entire directory and set `AUTO_SAMPLE_ROOT` to its local path.
All asset references are relative; keep the directory structure together.
Verify the download with `sha256sum -c SHA256SUMS` from that directory.

## Sensor and caption contract

| Camera key | Learned camera ID |
| --- | --- |
| camera_front_wide_120fov | 0 |
| camera_cross_right_120fov | 1 |
| camera_rear_right_70fov | 2 |
| camera_rear_tele_30fov | 3 |
| camera_rear_left_70fov | 4 |
| camera_cross_left_120fov | 5 |
| camera_front_tele_30fov | 6 |

The filenames and `camera_key` fields identify physical cameras. Do not replace
these IDs with a view's position in a list.

- RGB and WSM: 1280 × 720, 300 frames at 30 FPS.
- Captions: `caption_structured.chunk_0_300`, covering the end-exclusive
  interval `[0,300)` for the corresponding camera.
- WSM: **v4.0.1**, shared by both models. The renderer's configuration value
  remains `version=v4`; the sample bundle version is unrelated to WSM version.
- LiDAR: 100 Pandar128 sweeps at 10 Hz, with camera-clock IDs
  `000000,000003,...,000297`. Every archive member uses the clip key
  `daytime_scene`.
- LiDAR archives store sparse native coordinates on a **128 × 3600** grid.
  The matching runtime pools the native columns to 1800 semantic columns and
  applies four circular-padding columns at each side (1808 model columns).
  Do not pre-pool these archive coordinates to 1800 columns: that would discard
  half the azimuth coverage when the native reader pools again.
- Each control sweep contains `lidar_row`, `lidar_col`, and `lidar_range` NPZ
  members. Targets additionally contain `lidar_intensity`. Range is in meters;
  intensity is in `[0,1]`. The matching projection uses a 5–100 m range.

## Caption variants

The original `captions/` files use a single descriptive paragraph per camera.
We also provide `captions_with_timestamp/`, which better matches our models'
training caption style and has empirically produced better inference results.
Both formats are supported; prefer timestamped captions for post-training to
reduce the caption-format gap.

The supplied requests and training index select the original captions by default.
See [timestamped captions](daytime_scene/captions_with_timestamp/README.md) to
select the alternative for inference or training.

## Inference

Use the native Framework inference entry point with either request:

```bash
torchrun --standalone --nproc-per-node=4 \
  -m cosmos_framework.scripts.inference \
  -i "${AUTO_SAMPLE_ROOT:?}/input-transfer-auto.jsonl" \
  -o outputs/transfer-auto-daytime \
  --checkpoint-path "${TRANSFER_AUTO_CHECKPOINT:?}" \
  --cp-size 4 --cfgp-size 1 --dp-shard-size 1 --dp-replicate-size 1 \
  --no-use-torch-compile --no-use-cuda-graphs --no-diffusion-cache
```

For Sim-Auto, use `input-sim-auto.jsonl`, the matching Sim-Auto checkpoint,
and a separate output directory. Follow Framework's installation and model
inference guides, including Sim-Auto's FlashAttention-4 dependencies. Set
`COSMOS_TRAINING=0` and `COSMOS_INTERNAL=0` for inference.

The requests select seven cameras, the first RGB frame as image conditioning,
per-camera captions, and joint LiDAR generation from HD-map controls. They do
not supply an initial measured LiDAR sweep. Framework determines joint
generation from these inputs; `joint_generation` is not a supported JSONL field.

Both request 297 output frames at 30 FPS and seed 0. Transfer-Auto uses guidance
3, 35 steps, and shift 10. Sim-Auto uses guidance 1 and the checkpoint's fixed
four-step sampler. Expected outputs are seven 1280 × 720 videos and
`lidar.safetensors` with `range_m` and `intensity` of shape `[99,128,1800]`,
plus `frame_ids` equal to `0,3,...,294`. Check `sample_outputs.json` for success.

## Training with Framework's existing local Auto loader

Both models use the same `training-index.jsonl`. It maps existing files to
native training member names; no sample-specific dataset adapter is needed.
From an installed Framework environment, package the media with the native builder:

```bash
python -m cosmos_framework.data.generator.local_simulation.auto_bundle \
  --index "${AUTO_SAMPLE_ROOT:?}/training-index.jsonl" \
  --tokenizer "${LOCAL_QWEN_TOKENIZER:?set the matching local tokenizer}" \
  --output "${AUTO_BUNDLE:?set a new output directory}" \
  --camera-keys \
    camera_front_wide_120fov camera_cross_right_120fov \
    camera_rear_right_70fov camera_rear_tele_30fov \
    camera_rear_left_70fov camera_cross_left_120fov camera_front_tele_30fov
```

The builder writes `manifest.json`, `samples.json`, `assets/`, and
`resources/tokenizer/`. It copies the supplied media and tokenizer; it does not
render controls, generate captions, or train a model. Use the resulting bundle
with `COSMOS_TRANSFER_AUTO_LOCAL_BUNDLE` or `COSMOS_SIM_AUTO_LOCAL_BUNDLE`, then
follow the corresponding training skill for the selected training stage.

This sample provides only the `720` source stratum. Select that stream and
remove the absent `480` stream when wiring the training loader. Use a window
that fits within the 300 source frames; the default full-length training
recipe may request more frames. A short smoke run does not reproduce the
full 480/720 mixture, eleven-camera rig, or long-window training curriculum.

One scene demonstrates the format; it is not a training corpus or an independent
evaluation set. Supply additional recordings and separate training/validation
drives. File integrity and loader checks do not establish training convergence,
sensor calibration accuracy, or generation quality.