| # Shared daytime sample for Cosmos3 Auto |
|
|
| One synchronized, seven-camera scene for **Cosmos3-Nano-Transfer-Auto** and |
| **Cosmos3-Nano-Sim-Auto**. The bundle contains real RGB and LiDAR targets, |
| rendered controls, and camera-local captions. Model checkpoints, RGB and LiDAR |
| VAEs, and the matching text tokenizer are supplied separately. |
|
|
| This sample and its original captions in `daytime_scene/captions/` come from |
| the released [NVIDIA PhysicalAI Autonomous Vehicles dataset](https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles). |
| They are packaged here for use with both Auto models. |
|
|
| ## Contents |
|
|
| ```text |
| sample_data/ |
| README.md |
| FILES.tsv |
| SHA256SUMS |
| input-transfer-auto.jsonl |
| input-sim-auto.jsonl |
| training-index.jsonl |
| daytime_scene/ |
| rgb/<camera_key>.mp4 # seven synchronized real videos |
| wsm/<camera_key>.mp4 # seven shared WSM control videos |
| captions/<camera_key>.json # original captions from the dataset release |
| captions_with_timestamp/ |
| <camera_key>.json # seven timestamped alternatives |
| README.md |
| lidar/control.tar # HD-map range controls |
| lidar/target.tar # recorded range/intensity targets |
| ``` |
|
|
| Download the entire directory and set `AUTO_SAMPLE_ROOT` to its local path. |
| All asset references are relative; keep the directory structure together. |
| Verify the download with `sha256sum -c SHA256SUMS` from that directory. |
|
|
| ## Sensor and caption contract |
|
|
| | Camera key | Learned camera ID | |
| | --- | --- | |
| | camera_front_wide_120fov | 0 | |
| | camera_cross_right_120fov | 1 | |
| | camera_rear_right_70fov | 2 | |
| | camera_rear_tele_30fov | 3 | |
| | camera_rear_left_70fov | 4 | |
| | camera_cross_left_120fov | 5 | |
| | camera_front_tele_30fov | 6 | |
| |
| The filenames and `camera_key` fields identify physical cameras. Do not replace |
| these IDs with a view's position in a list. |
|
|
| - RGB and WSM: 1280 × 720, 300 frames at 30 FPS. |
| - Captions: `caption_structured.chunk_0_300`, covering the end-exclusive |
| interval `[0,300)` for the corresponding camera. |
| - WSM: **v4.0.1**, shared by both models. The renderer's configuration value |
| remains `version=v4`; the sample bundle version is unrelated to WSM version. |
| - LiDAR: 100 Pandar128 sweeps at 10 Hz, with camera-clock IDs |
| `000000,000003,...,000297`. Every archive member uses the clip key |
| `daytime_scene`. |
| - LiDAR archives store sparse native coordinates on a **128 × 3600** grid. |
| The matching runtime pools the native columns to 1800 semantic columns and |
| applies four circular-padding columns at each side (1808 model columns). |
| Do not pre-pool these archive coordinates to 1800 columns: that would discard |
| half the azimuth coverage when the native reader pools again. |
| - Each control sweep contains `lidar_row`, `lidar_col`, and `lidar_range` NPZ |
| members. Targets additionally contain `lidar_intensity`. Range is in meters; |
| intensity is in `[0,1]`. The matching projection uses a 5–100 m range. |
|
|
| ## Caption variants |
|
|
| The original `captions/` files use a single descriptive paragraph per camera. |
| We also provide `captions_with_timestamp/`, which better matches our models' |
| training caption style and has empirically produced better inference results. |
| Both formats are supported; prefer timestamped captions for post-training to |
| reduce the caption-format gap. |
|
|
| The supplied requests and training index select the original captions by default. |
| See [timestamped captions](daytime_scene/captions_with_timestamp/README.md) to |
| select the alternative for inference or training. |
|
|
| ## Inference |
|
|
| Use the native Framework inference entry point with either request: |
|
|
| ```bash |
| torchrun --standalone --nproc-per-node=4 \ |
| -m cosmos_framework.scripts.inference \ |
| -i "${AUTO_SAMPLE_ROOT:?}/input-transfer-auto.jsonl" \ |
| -o outputs/transfer-auto-daytime \ |
| --checkpoint-path "${TRANSFER_AUTO_CHECKPOINT:?}" \ |
| --cp-size 4 --cfgp-size 1 --dp-shard-size 1 --dp-replicate-size 1 \ |
| --no-use-torch-compile --no-use-cuda-graphs --no-diffusion-cache |
| ``` |
|
|
| For Sim-Auto, use `input-sim-auto.jsonl`, the matching Sim-Auto checkpoint, |
| and a separate output directory. Follow Framework's installation and model |
| inference guides, including Sim-Auto's FlashAttention-4 dependencies. Set |
| `COSMOS_TRAINING=0` and `COSMOS_INTERNAL=0` for inference. |
|
|
| The requests select seven cameras, the first RGB frame as image conditioning, |
| per-camera captions, and joint LiDAR generation from HD-map controls. They do |
| not supply an initial measured LiDAR sweep. Framework determines joint |
| generation from these inputs; `joint_generation` is not a supported JSONL field. |
|
|
| Both request 297 output frames at 30 FPS and seed 0. Transfer-Auto uses guidance |
| 3, 35 steps, and shift 10. Sim-Auto uses guidance 1 and the checkpoint's fixed |
| four-step sampler. Expected outputs are seven 1280 × 720 videos and |
| `lidar.safetensors` with `range_m` and `intensity` of shape `[99,128,1800]`, |
| plus `frame_ids` equal to `0,3,...,294`. Check `sample_outputs.json` for success. |
|
|
| ## Training with Framework's existing local Auto loader |
|
|
| Both models use the same `training-index.jsonl`. It maps existing files to |
| native training member names; no sample-specific dataset adapter is needed. |
| From an installed Framework environment, package the media with the native builder: |
|
|
| ```bash |
| python -m cosmos_framework.data.generator.local_simulation.auto_bundle \ |
| --index "${AUTO_SAMPLE_ROOT:?}/training-index.jsonl" \ |
| --tokenizer "${LOCAL_QWEN_TOKENIZER:?set the matching local tokenizer}" \ |
| --output "${AUTO_BUNDLE:?set a new output directory}" \ |
| --camera-keys \ |
| camera_front_wide_120fov camera_cross_right_120fov \ |
| camera_rear_right_70fov camera_rear_tele_30fov \ |
| camera_rear_left_70fov camera_cross_left_120fov camera_front_tele_30fov |
| ``` |
|
|
| The builder writes `manifest.json`, `samples.json`, `assets/`, and |
| `resources/tokenizer/`. It copies the supplied media and tokenizer; it does not |
| render controls, generate captions, or train a model. Use the resulting bundle |
| with `COSMOS_TRANSFER_AUTO_LOCAL_BUNDLE` or `COSMOS_SIM_AUTO_LOCAL_BUNDLE`, then |
| follow the corresponding training skill for the selected training stage. |
|
|
| This sample provides only the `720` source stratum. Select that stream and |
| remove the absent `480` stream when wiring the training loader. Use a window |
| that fits within the 300 source frames; the default full-length training |
| recipe may request more frames. A short smoke run does not reproduce the |
| full 480/720 mixture, eleven-camera rig, or long-window training curriculum. |
|
|
| One scene demonstrates the format; it is not a training corpus or an independent |
| evaluation set. Supply additional recordings and separate training/validation |
| drives. File integrity and loader checks do not establish training convergence, |
| sensor calibration accuracy, or generation quality. |
|
|
