File size: 5,261 Bytes

---
library_name: pytorch
pipeline_tag: image-to-3d
license: other
base_model:
- facebook/dinov2-with-registers-base
- google/siglip2-base-patch16-256
datasets:
- tracywei/scannet_3drae
---

# 3DRAE and 3DDiT

Pretrained weights for **Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale** (NeurIPS 2026).

3DRAE encodes an arbitrary number of calibrated multi-view images into a fixed-length 3D-grounded representation. 3DDiT performs diffusion modeling directly in this latent space for efficient and spatially consistent 3D scene generation.

- [Paper](https://arxiv.org/abs/2604.11331)
- [Project page](https://wswdx.github.io/3DRAE)
- [Code](https://github.com/WU-CVGL/3DRAE)
- [Processed ScanNet data](https://huggingface.co/datasets/tracywei/scannet_3drae)

https://github.com/user-attachments/assets/9c584612-5fa0-4cfb-b711-046b8c0a2446

## Model Variants

| Variant | Latent shape | Reconstruction / NVS | Point maps | 3D generation |
| --- | ---: | :---: | :---: | :---: |
| `dino2` | `1024 x 768` | Yes | Yes | Yes |
| `dino2-256` | `256 x 768` | Yes | No | No |
| `siglip2` | `256 x 768` | Yes | No | No |

The released 3DDiT checkpoint is paired with the 1024-token DINOv2 representation.

## Quick Start

Clone the official implementation and create the recommended Python 3.10 environment:

```bash
git clone https://github.com/WU-CVGL/3DRAE.git
cd 3DRAE
uv sync --python 3.10
source .venv/bin/activate
```

Download the complete checkpoint bundle into `checkpoints/`:

```bash
hf download tracywei/3drae \
  --exclude "README.md" ".gitattributes" \
  --local-dir checkpoints
```

Verify the downloaded files:

```bash
python tools/verify_checkpoints.py
```

Run novel-view synthesis on a bundled calibrated example:

```bash
python scripts/reconstruct.py \
  --model dino2 \
  --scene examples/scene_000 \
  --condition-frame-ids 0 5 \
  --target-frame-ids 6 7 8 9 \
  --output-dir outputs/example_nvs_dino2
```

Run single-view conditioned 3D scene generation with a synthetic camera trajectory:

```bash
python scripts/generate.py \
  --scene examples/scene_000 \
  --appearance-condition-frame-ids 0 \
  --trajectory-mode synthetic-orbit \
  --output-dir outputs/example_generation_1view_synthetic
```

See the [official repository](https://github.com/WU-CVGL/3DRAE) for Python APIs, additional conditioning settings, training, feature extraction, dense-view reconstruction, and the interactive Viser viewer.

## Checkpoint Contents

```text
backbones/
├── dinov2-with-registers-base/
└── siglip2-base-patch16-256/
dino2/
├── neck.safetensors
├── decoder.safetensors
├── point_decoder.safetensors
├── stats.safetensors
└── 3ddit.safetensors
dino2_256/
├── neck.safetensors
├── decoder.safetensors
└── stats.safetensors
siglip2/
├── neck.safetensors
├── decoder.safetensors
└── stats.safetensors
discriminator/
perceptual/
manifest.json
```

Files under `discriminator/` and `perceptual/` are required only for Stage 1 training. File provenance, sizes, and SHA256 checksums are recorded in `manifest.json`.

## Input and Output

Inference expects calibrated multi-view RGB images with camera intrinsics and OpenCV world-to-camera extrinsics. The repository includes ready-to-run examples and documents the exact scene format.

Depending on the selected checkpoint, 3DRAE can output fixed-length 3D latent tokens, novel-view RGB images, world-space point maps, confidence maps, and fused RGB point clouds. 3DDiT generates a 3D latent that can be decoded along arbitrary camera trajectories.

## Limitations

- The released models require calibrated camera intrinsics and poses; they are not pose-free reconstruction models.
- The released 3DDiT checkpoint supports DINOv2-based 1024-token generation. The compact DINOv2 and SigLIP2 variants are reconstruction/NVS models only.
- The compact DINOv2 and SigLIP2 variants predict RGB images but do not include point-map decoders.
- Generated content may be inaccurate or inconsistent for sparse observations, large viewpoint extrapolation, or domains far from the training distribution.
- The model produces implicit latent representations, rendered images, and optional point maps rather than explicit watertight meshes.

## Sources and Terms

The bundle includes the [DINOv2 with registers base](https://huggingface.co/facebook/dinov2-with-registers-base) and [SigLIP2 base patch16 256](https://huggingface.co/google/siglip2-base-patch16-256) foundation backbones. It also includes the auxiliary DINO discriminator and VGG/LPIPS weights required by the released Stage 1 training configuration. Third-party files remain subject to their original licenses and terms.

The released code is available under the [MIT License](https://github.com/WU-CVGL/3DRAE/blob/main/LICENSE).

## Citation

```bibtex
@inproceedings{wei20263drae,
  title     = {Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale},
  author    = {Wei, Dongxu and Xu, Qi and Li, Zhiqi and Zhou, Hangning and Qiu, Cong and Qin, Hailong and Yang, Mu and Cui, Zhaopeng and Liu, Peidong},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}
```