README.md

---
license: other
license_name: nvidia-open-model-license
license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/
base_model: nvidia/Cosmos-Transfer2.5-2B
base_model_relation: finetune
pipeline_tag: video-to-video
language:
  - en
tags:
  - video-generation
  - driving-view-synthesis
  - novel-view-synthesis
  - 3d-gaussian-splatting
  - lidar
  - prg-fusion
  - arxiv:2609.06948
---

# PRG-Fusion

**Orchestrating Generative Priors with Reconstruction Evidence for Driving View Synthesis**

Sipeng He<sup>1</sup> · Jialei Chen<sup>2</sup> · Zhen Fang<sup>1</sup> · Dongchun Ren<sup>2</sup> · Feng Zhao<sup>1</sup>

<sup>1</sup>University of Science and Technology of China · <sup>2</sup>Yootta

[Project page](https://pre-cici.github.io/PRG-Fusion/) · [Paper](https://arxiv.org/abs/2609.06948) · [Code](https://github.com/Pre-cici/PRG-Fusion)

PRG-Fusion synthesizes driving videos along novel trajectories using dense 3D Gaussian Splatting (3DGS) renderings and sparse colored LiDAR projections. This repository provides the **PRG-Fusion model checkpoint**; inference also requires the code, component checkpoints, and aligned control inputs described below.

![PRG-Fusion pipeline](assets/pipeline.png)

Region-wise reconstruction evidence produces **Preserve, Repair, and Generate (PRG)** labels that route appearance, geometry, and generative priors:

- **Preserve** reliable appearance from dense 3DGS renderings.
- **Repair** uncertain geometry using sparse LiDAR projections.
- **Generate** missing content with video priors.

Training proceeds in two stages: geometric control is learned from sparse LiDAR projections, followed by appearance control from dense 3DGS renderings.

## Model Details

| | |
| --- | --- |
| **Model** | PRG-Fusion (2B base with two control branches) |
| **Base model** | [NVIDIA Cosmos-Transfer2.5-2B](https://huggingface.co/nvidia/Cosmos-Transfer2.5-2B) |
| **Conditioning** | Aligned 3DGS rendering, colored LiDAR projection, PRG region labels, and a text prompt |
| **Checkpoint format** | PyTorch Distributed Checkpoint (DCP) |
| **Demo output** | One 93-frame, 960 × 640 driving video chunk |
| **Inference hardware** | Approximately 70 GB GPU memory; use an 80 GB-class NVIDIA GPU |
| **Evaluation domain** | Waymo driving scenes |

## Set Up the Code

Use a Linux x86-64 host with an 80 GB-class NVIDIA GPU, a compatible CUDA driver, FFmpeg, and [uv](https://docs.astral.sh/uv/). Clone the [PRG-Fusion code](https://github.com/Pre-cici/PRG-Fusion) and install the CUDA 13.0 environment:

```bash
git clone https://github.com/Pre-cici/PRG-Fusion.git
cd PRG-Fusion

uv venv --python 3.10
uv sync --extra cu130
uv run --extra cu130 python examples/check_environment.py
```

Use `cu128` instead of `cu130` on a CUDA 12.8 system.

## Download the Checkpoint

From the code repository root, download the model checkpoint into the expected layout:

```bash
uv tool install -U huggingface_hub
hf download Post-pone/PRG-Fusion \
  --revision 19af7dc45c73a07bcee5f3a4a92bfc276bd99175 \
  --include '.metadata' '*.distcp' \
  --local-dir checkpoints/PRG-Fusion/model
```

The complete checkpoint is approximately 15.95 GB:

```text
checkpoints/PRG-Fusion/
├── model/
│   ├── .metadata
│   ├── __0_0.distcp
│   └── __1_0.distcp
```

All three files are required. Keep the original filenames: `.metadata` references the two DCP shards. The pinned revision matches the [code repository's checkpoint guide](https://github.com/Pre-cici/PRG-Fusion/blob/main/docs/checkpoints.md).

## Run the Demo

In addition to the PRG-Fusion checkpoint above, download the Cosmos tokenizer, text encoder, and initialization weights using the [checkpoint guide](https://github.com/Pre-cici/PRG-Fusion/blob/main/docs/checkpoints.md). Weights are not bundled with the code repository.

The demo inputs ship with the repository in `assets/demo/`:

```text
assets/demo/
├── vis.mp4                  # Dense 3DGS rendering for appearance control
├── lidar.mp4                # Sparse colored LiDAR projection for geometric control
└── prg/                     # PRG region labels used by the dense gate
```

The two control videos and PRG labels must be aligned frame by frame on the target trajectory. The demo uses 93 frames at 960 × 640; labels encode preserve (0), repair (1), and generate (2). For other trajectories, prepare inputs in the same layout, then run from the code repository root:

```bash
uv run --extra cu130 python examples/inference.py \
  --config configs/inference/demo.yaml
```

Generated videos and run metadata are written to `outputs/debug/<timestamp>_demo/` by default. See the [inference guide](https://github.com/Pre-cici/PRG-Fusion/blob/main/docs/inference.md) for input and path semantics.

## Checkpoint Integrity

| File | SHA-256 |
| --- | --- |
| `.metadata` | `93f8707b53b7f16bb84ddb6a9c7a231feb8f5c4892261ebdc2f4eab9895a13bd` |
| `__0_0.distcp` | `efbee0f62f9a0ad02efc2859483972d33161a018e4f37549ab664aa3a6db15c7` |
| `__1_0.distcp` | `d3846b51b9c4691083fa90d6cf3106add1aa6227584a9896e22e7873dcf1c4f6` |

## Limitations

The model is intended for research on driving view synthesis, not safety-critical driving decisions. It was evaluated on Waymo driving scenes; performance on other domains has not been established here. Large viewpoint changes or weak reconstruction evidence may cause temporal, geometric, or appearance artifacts. The checkpoint does not generate the required 3DGS, LiDAR, or PRG-label inputs for you.

## Acknowledgements

We thank the NVIDIA Cosmos team for making their work available. We also thank the [DriveStudio](https://github.com/ziyc/drivestudio) and [FreeVS](https://github.com/esdolo/FreeVS) teams for sharing their work.

## License

The **model weights** are a derivative of NVIDIA Cosmos-Transfer2.5-2B and are distributed under the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). **Built on NVIDIA Cosmos.** See [LICENSE.pdf](LICENSE.pdf) and [NOTICE](NOTICE) for details. The separate code repository uses Apache-2.0; dataset and media rights are separate from the model license.

## Citation

```bibtex
@misc{he2026prgfusion,
  title         = {{PRG-Fusion}: Orchestrating Generative Priors with Reconstruction Evidence for Driving View Synthesis},
  author        = {Sipeng He and Jialei Chen and Zhen Fang and Dongchun Ren and Feng Zhao},
  year          = {2026},
  eprint        = {2609.06948},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2609.06948}
}
```