| --- |
| library_name: pytorch |
| pipeline_tag: image-to-3d |
| license: other |
| base_model: |
| - facebook/dinov2-with-registers-base |
| - google/siglip2-base-patch16-256 |
| datasets: |
| - tracywei/scannet_3drae |
| --- |
| |
| # 3DRAE and 3DDiT |
|
|
| Pretrained weights for **Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale** (NeurIPS 2026). |
|
|
| 3DRAE encodes an arbitrary number of calibrated multi-view images into a fixed-length 3D-grounded representation. 3DDiT performs diffusion modeling directly in this latent space for efficient and spatially consistent 3D scene generation. |
|
|
| - [Paper](https://arxiv.org/abs/2604.11331) |
| - [Project page](https://wswdx.github.io/3DRAE) |
| - [Code](https://github.com/WU-CVGL/3DRAE) |
| - [Processed ScanNet data](https://huggingface.co/datasets/tracywei/scannet_3drae) |
|
|
| https://github.com/user-attachments/assets/9c584612-5fa0-4cfb-b711-046b8c0a2446 |
|
|
| ## Model Variants |
|
|
| | Variant | Latent shape | Reconstruction / NVS | Point maps | 3D generation | |
| | --- | ---: | :---: | :---: | :---: | |
| | `dino2` | `1024 x 768` | Yes | Yes | Yes | |
| | `dino2-256` | `256 x 768` | Yes | No | No | |
| | `siglip2` | `256 x 768` | Yes | No | No | |
|
|
| The released 3DDiT checkpoint is paired with the 1024-token DINOv2 representation. |
|
|
| ## Quick Start |
|
|
| Clone the official implementation and create the recommended Python 3.10 environment: |
|
|
| ```bash |
| git clone https://github.com/WU-CVGL/3DRAE.git |
| cd 3DRAE |
| uv sync --python 3.10 |
| source .venv/bin/activate |
| ``` |
|
|
| Download the complete checkpoint bundle into `checkpoints/`: |
|
|
| ```bash |
| hf download tracywei/3drae \ |
| --exclude "README.md" ".gitattributes" \ |
| --local-dir checkpoints |
| ``` |
|
|
| Verify the downloaded files: |
|
|
| ```bash |
| python tools/verify_checkpoints.py |
| ``` |
|
|
| Run novel-view synthesis on a bundled calibrated example: |
|
|
| ```bash |
| python scripts/reconstruct.py \ |
| --model dino2 \ |
| --scene examples/scene_000 \ |
| --condition-frame-ids 0 5 \ |
| --target-frame-ids 6 7 8 9 \ |
| --output-dir outputs/example_nvs_dino2 |
| ``` |
|
|
| Run single-view conditioned 3D scene generation with a synthetic camera trajectory: |
|
|
| ```bash |
| python scripts/generate.py \ |
| --scene examples/scene_000 \ |
| --appearance-condition-frame-ids 0 \ |
| --trajectory-mode synthetic-orbit \ |
| --output-dir outputs/example_generation_1view_synthetic |
| ``` |
|
|
| See the [official repository](https://github.com/WU-CVGL/3DRAE) for Python APIs, additional conditioning settings, training, feature extraction, dense-view reconstruction, and the interactive Viser viewer. |
|
|
| ## Checkpoint Contents |
|
|
| ```text |
| backbones/ |
| ├── dinov2-with-registers-base/ |
| └── siglip2-base-patch16-256/ |
| dino2/ |
| ├── neck.safetensors |
| ├── decoder.safetensors |
| ├── point_decoder.safetensors |
| ├── stats.safetensors |
| └── 3ddit.safetensors |
| dino2_256/ |
| ├── neck.safetensors |
| ├── decoder.safetensors |
| └── stats.safetensors |
| siglip2/ |
| ├── neck.safetensors |
| ├── decoder.safetensors |
| └── stats.safetensors |
| discriminator/ |
| perceptual/ |
| manifest.json |
| ``` |
|
|
| Files under `discriminator/` and `perceptual/` are required only for Stage 1 training. File provenance, sizes, and SHA256 checksums are recorded in `manifest.json`. |
|
|
| ## Input and Output |
|
|
| Inference expects calibrated multi-view RGB images with camera intrinsics and OpenCV world-to-camera extrinsics. The repository includes ready-to-run examples and documents the exact scene format. |
|
|
| Depending on the selected checkpoint, 3DRAE can output fixed-length 3D latent tokens, novel-view RGB images, world-space point maps, confidence maps, and fused RGB point clouds. 3DDiT generates a 3D latent that can be decoded along arbitrary camera trajectories. |
|
|
| ## Limitations |
|
|
| - The released models require calibrated camera intrinsics and poses; they are not pose-free reconstruction models. |
| - The released 3DDiT checkpoint supports DINOv2-based 1024-token generation. The compact DINOv2 and SigLIP2 variants are reconstruction/NVS models only. |
| - The compact DINOv2 and SigLIP2 variants predict RGB images but do not include point-map decoders. |
| - Generated content may be inaccurate or inconsistent for sparse observations, large viewpoint extrapolation, or domains far from the training distribution. |
| - The model produces implicit latent representations, rendered images, and optional point maps rather than explicit watertight meshes. |
|
|
| ## Sources and Terms |
|
|
| The bundle includes the [DINOv2 with registers base](https://huggingface.co/facebook/dinov2-with-registers-base) and [SigLIP2 base patch16 256](https://huggingface.co/google/siglip2-base-patch16-256) foundation backbones. It also includes the auxiliary DINO discriminator and VGG/LPIPS weights required by the released Stage 1 training configuration. Third-party files remain subject to their original licenses and terms. |
|
|
| The released code is available under the [MIT License](https://github.com/WU-CVGL/3DRAE/blob/main/LICENSE). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{wei20263drae, |
| title = {Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale}, |
| author = {Wei, Dongxu and Xu, Qi and Li, Zhiqi and Zhou, Hangning and Qiu, Cong and Qin, Hailong and Yang, Mu and Cui, Zhaopeng and Liu, Peidong}, |
| booktitle = {Advances in Neural Information Processing Systems}, |
| year = {2026} |
| } |
| ``` |
|
|
