PRG-Fusion

Orchestrating Generative Priors with Reconstruction Evidence for Driving View Synthesis

Sipeng He¹ · Jialei Chen² · Zhen Fang¹ · Dongchun Ren² · Feng Zhao¹

¹University of Science and Technology of China · ²Yootta

Project page · Paper · Code

PRG-Fusion synthesizes driving videos along novel trajectories using dense 3D Gaussian Splatting (3DGS) renderings and sparse colored LiDAR projections. This repository provides the PRG-Fusion model checkpoint; inference also requires the code, component checkpoints, and aligned control inputs described below.

Region-wise reconstruction evidence produces Preserve, Repair, and Generate (PRG) labels that route appearance, geometry, and generative priors:

• Preserve reliable appearance from dense 3DGS renderings.
• Repair uncertain geometry using sparse LiDAR projections.
• Generate missing content with video priors.

Training proceeds in two stages: geometric control is learned from sparse LiDAR projections, followed by appearance control from dense 3DGS renderings.

Model Details

Model	PRG-Fusion (2B base with two control branches)
Base model	NVIDIA Cosmos-Transfer2.5-2B
Conditioning	Aligned 3DGS rendering, colored LiDAR projection, PRG region labels, and a text prompt
Checkpoint format	PyTorch Distributed Checkpoint (DCP)
Demo output	One 93-frame, 960 × 640 driving video chunk
Inference hardware	Approximately 70 GB GPU memory; use an 80 GB-class NVIDIA GPU
Evaluation domain	Waymo driving scenes

Set Up the Code

Use a Linux x86-64 host with an 80 GB-class NVIDIA GPU, a compatible CUDA driver, FFmpeg, and uv. Clone the PRG-Fusion code and install the CUDA 13.0 environment:

git clone https://github.com/Pre-cici/PRG-Fusion.git
cd PRG-Fusion

uv venv --python 3.10
uv sync --extra cu130
uv run --extra cu130 python examples/check_environment.py

Use cu128 instead of cu130 on a CUDA 12.8 system.

Download the Checkpoint

From the code repository root, download the model checkpoint into the expected layout:

uv tool install -U huggingface_hub
hf download Post-pone/PRG-Fusion \
  --revision 19af7dc45c73a07bcee5f3a4a92bfc276bd99175 \
  --include '.metadata' '*.distcp' \
  --local-dir checkpoints/PRG-Fusion/model

The complete checkpoint is approximately 15.95 GB:

checkpoints/PRG-Fusion/
├── model/
│   ├── .metadata
│   ├── __0_0.distcp
│   └── __1_0.distcp

All three files are required. Keep the original filenames: .metadata references the two DCP shards. The pinned revision matches the code repository's checkpoint guide.

Run the Demo

In addition to the PRG-Fusion checkpoint above, download the Cosmos tokenizer, text encoder, and initialization weights using the checkpoint guide. Weights are not bundled with the code repository.

The demo inputs ship with the repository in assets/demo/:

assets/demo/
├── vis.mp4                  # Dense 3DGS rendering for appearance control
├── lidar.mp4                # Sparse colored LiDAR projection for geometric control
└── prg/                     # PRG region labels used by the dense gate

The two control videos and PRG labels must be aligned frame by frame on the target trajectory. The demo uses 93 frames at 960 × 640; labels encode preserve (0), repair (1), and generate (2). For other trajectories, prepare inputs in the same layout, then run from the code repository root:

uv run --extra cu130 python examples/inference.py \
  --config configs/inference/demo.yaml

Generated videos and run metadata are written to outputs/debug/<timestamp>_demo/ by default. See the inference guide for input and path semantics.

Checkpoint Integrity

File	SHA-256
.metadata	93f8707b53b7f16bb84ddb6a9c7a231feb8f5c4892261ebdc2f4eab9895a13bd
__0_0.distcp	efbee0f62f9a0ad02efc2859483972d33161a018e4f37549ab664aa3a6db15c7
__1_0.distcp	d3846b51b9c4691083fa90d6cf3106add1aa6227584a9896e22e7873dcf1c4f6

Limitations

The model is intended for research on driving view synthesis, not safety-critical driving decisions. It was evaluated on Waymo driving scenes; performance on other domains has not been established here. Large viewpoint changes or weak reconstruction evidence may cause temporal, geometric, or appearance artifacts. The checkpoint does not generate the required 3DGS, LiDAR, or PRG-label inputs for you.

Acknowledgements

We thank the NVIDIA Cosmos team for making their work available. We also thank the DriveStudio and FreeVS teams for sharing their work.

License

The model weights are a derivative of NVIDIA Cosmos-Transfer2.5-2B and are distributed under the NVIDIA Open Model License. Built on NVIDIA Cosmos. See LICENSE.pdf and NOTICE for details. The separate code repository uses Apache-2.0; dataset and media rights are separate from the model license.

Citation

@misc{he2026prgfusion,
  title         = {{PRG-Fusion}: Orchestrating Generative Priors with Reconstruction Evidence for Driving View Synthesis},
  author        = {Sipeng He and Jialei Chen and Zhen Fang and Dongchun Ren and Feng Zhao},
  year          = {2026},
  eprint        = {2609.06948},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2609.06948}
}