Albertchen96/Driving-with-DINO-PCA8

Driving with DINO: PCA8 ×1

Driving with DINO: Vision Foundation Features as a Unified Bridge for Sim-to-Real Generation in Autonomous Driving — accepted to ACM MM 2026.

Code and inference instructions

This release contains the full BF16 DwD checkpoint pca8_iter16000, exported at iteration 16,000, and its matching PCA8 basis. The model adapts Cosmos Predict2.5 through the Cosmos Transfer2.5 codebase for DINO-feature-conditioned driving-video generation.

Architecture and files

• Frozen DINOv3 ViT-L/16, final-layer features with pretrained LayerNorm retained.
• DINO input: 704×1280 (×1), feature grid: 44×80.
• Fixed 8-dimensional centered PCA, no whitening; this is not the PCA32 random-tail-drop checkpoint.
• Temporal sampling every fourth frame; no temporal convolution or feature upsampling.
• Bias-free linear control projection [2048, 130]: 8 PCA features padded to 128 channels plus two mask channels.
• model.pt: full BF16 inference checkpoint (796 state-dict entries).
• pca/patch_pca_mean_8.npy, pca/patch_pca_components_8.npy: exact PCA files used in the validated inference run.
• manifest.json and SHA256SUMS: configuration and file checksums.

PCA8 and the first eight components of a separately fitted PCA32 basis are not interchangeable. Use the supplied files without refitting. DINOv3, Cosmos VAE, and Cosmos-Reason1 text encoder are obtained separately from their upstream repositories; no separate diffusion backbone is needed.

Inference

Install the code repository's environment, then run python scripts/infer_pca8.py --help. The entry point automatically downloads this checkpoint and PCA files and validates their checksums. See the code README for complete commands and dependencies.

The validated example uses 93 input frames, 35 steps, guidance 3, control weight 1, no RGB reference frame, and a locally generated Qwen3 caption. Sampling and decoding completed on a single RTX PRO 6000 Blackwell with 96 GB VRAM. The packaged entry point also passed a 1-step GPU smoke test. These checks establish execution, not reproduction of the paper's metrics. Full training and minimum-memory requirements have not been validated in the release environment.

License and attribution

Licensed by NVIDIA Corporation under the NVIDIA Open Model License.

This is a modified model: Driving with DINO adds DINO feature conditioning and fine-tuned parameters to the Cosmos backbone. The weights are distributed under the included NVIDIA Open Model License; code in the linked repository is Apache-2.0. Upstream DINOv3 and text-encoder assets retain their own terms.