license: other
pipeline_tag: image-to-image
DVSM Overview
Description:
DVSM is a decoder-only Transformer for novel-view synthesis. It takes user input images and reconstruct the scene as KV cache. The user then can do arbitrary novel-view synthesis within that scene.
The model was developed by NVIDIA CORPORATION as a part of DVSM.
This model is for research and development only.
Third-Party Community Consideration
This model is not owned or developed by NVIDIA. This model has been developed and built to a third-party's requirements for this application and use case; see link to Non-NVIDIA [Not Applicable (N/A) Model Card](Not Applicable (N/A)).
License/Terms of Use:
NSCLv1
Deployment Geography:
Global
Use Case:
Researchers and developers working on computer vision and graphics who need to generate novel views of 3D scenes from sparse input images. The model supports view synthesis tasks on datasets like DL3DV and Re10k, enabling applications in virtual reality, augmented reality, and 3D content creation.
Release Date:
HuggingFace: 30 June, 2026 via https://huggingface.co/nvidia/DVSM
Reference(s):
Technical report
TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models
Project page
Code
Model Architecture:
Architecture Type: Transformer
Network Architecture: Each block consists of a intral-view attention and a cross-view attention layer, each followed by a MLP layer. We stack 12 blocks with 768 hidden dimensions.
This model was developed based on Not Applicable (N/A).
Number of model parameters: 163M
Input:
Input Type(s): Image
Input Format(s): RGB (Red, Green, Blue)
Input Parameters: Two-Dimensional (2D)
Other Properties Related to Input: We provide models working on 340p, 480p, 720p, and 960p resolutions. Cameras are normalized and embedded as plucker ray map. Linear patchfiers are applied with 8x8 patch size.
Output:
Output Type(s): Image
Output Format: Other: RGB (Red, Green, Blue)
Output Parameters: Two-Dimensional (2D)
Other Properties Related to Output: The output is in the same resolution as the input images.
Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
Software Integration:
Runtime Engine(s): Transformers
Supported Hardware Microarchitecture Compatibility:
- NVIDIA Ampere
- NVIDIA Blackwell
- NVIDIA Lovelace
- NVIDIA Hopper
Supported Operating System(s): Linux
The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
Model Version(s):
DVSM
To integrate DVSM, start from the NVIDIA PyTorch container (nvcr.io/nvidia/pytorch:25.10-py3) and install the required Python packages including PyTorch, Transformers, Diffusers, Accelerate, and additional libraries such as OpenCV, Einops, XFormers, and others listed in the requirements. Then mount the dataset and checkpoint directories, and launch training or inference using the provided SLURM scripts or direct python calls.
Training, Testing, and Evaluation Datasets:
Training Dataset:
Data Modality: Image
Image Training Data Size: 1 Million to 1 Billion Images
Data Collection Method by dataset: Automatic/Sensors
Labeling Method by dataset: Automated
Properties (Quantity, Dataset Descriptions, Sensor(s)): DL3DV: 10K image sequences, camera sensor
Re10K: 10K image sequences, camera sensor
Testing Dataset:
Data Collection Method by dataset: Automatic/Sensors
Labeling Method by dataset: Automated
Properties (Quantity, Dataset Descriptions, Sensor(s)): DL3DV: 10K image sequences, camera sensor
Re10K: 10K image sequences, camera sensor
Evaluation Dataset:
Benchmark Score: None
| Benchmark | Metric | Score |
|---|---|---|
| DL3DV 64 views 960p | PSNR | 29.71 |
| DL3DV 32 views 960p | PSNR | 27.96 |
| DL3DV 16 views 960p | PSNR | 25.88 |
| Re10K 2 views 512x512 | PSNR | 30.17 |
| MipNerf360 64 views | PSNR | 24.76 |
| Free 64 views | PSNR | 25.57 |
| Hike 64 views | PSNR | 23.10 |
Data Collection Method by dataset: Automatic/Sensors
Labeling Method by dataset: Automated
Properties (Quantity, Dataset Descriptions, Sensor(s)): DL3DV: 10K image sequences, camera sensor
Re10K: 10K image sequences, camera sensor
Inference:
Acceleration Engine: PyTorch, Other: flash-attention Test Hardware: NVIDIA A100 80GB
Ethical Considerations:
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.
For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards.
Generated by NVIDIA Model Card Generator Toolkit.