File size: 7,031 Bytes
2e9191e 84d4246 2e9191e da94067 5feb16b da94067 5feb16b da94067 2e9191e 2075869 2e9191e 2075869 ad94054 2e9191e da94067 2e9191e 2075869 ad94054 2e9191e f980e56 711aab9 2e9191e 84d4246 2e9191e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 | ---
license: other
pipeline_tag: image-to-3d
tags:
- 3d-scene-reconstruction
- rgb-d
- single-image
- flow-matching
- textured-mesh
---
# FIRE3D: Feed-forward Interactive 3D Scene Reconstruction Within A Minute
[Paper](https://arxiv.org/pdf/2609.08848) |
[Code](https://github.com/xiahongchi/Fire3D) |
[Inference data](https://huggingface.co/datasets/hongchi/Fire3D)
Fire3D is a unified feed-forward framework that transforms a single RGB image
or casual RGB video into simulation-ready 3D scene assets. It predicts a
compositional scene representation with object-level 6-DoF pose, bounding box,
mesh geometry, and texture, without test-time optimization.
This release provides the inference code, model checkpoints, processed example
inputs, and frozen protocols needed to reproduce Fire3D results on iTHOR,
Imaginarium, ScanNet++, and single-image scenes.
The inference code and installation instructions are maintained at
[xiahongchi/Fire3D](https://github.com/xiahongchi/Fire3D). Processed release
inputs are hosted at
[datasets/hongchi/Fire3D](https://huggingface.co/datasets/hongchi/Fire3D).
## Overview
**Teaser.** Fire3D reconstructs a compositional 3D scene from a single RGB
image or casual RGB video. The output contains object-level pose, geometry,
and material assets that can be rendered, edited, and used in simulation.
[](assets/method_overview.pdf)
**Method overview.** Fire3D lifts RGB-D observations into a shared 3D scene
representation, predicts object instances and 6-DoF poses, and reconstructs
the sparse structure, shape, and PBR fields with three cascaded flow-matching
models. Batched decoders and mesh post-processing produce the final textured
scene assets.
[](assets/hcvae.pdf)
**HC-VAE.** The Hierarchical Compression VAE reduces each sparse SC-VAE shape
or material field from a `32^3 x 32` representation to an `8^3 x 64` latent.
This 32x reduction makes scene-level batched flow sampling practical while
retaining the frozen TRELLIS.2 decoding path.
## Model Bundle
The repository contains the scene perception checkpoint, the three
flow-matching checkpoints, sparse VAE encoders and decoders, latent normalization
statistics, and the TRELLIS.2 shape/PBR decoders used by the frozen release
protocols. `manifest.json` records the byte size and SHA-256 digest of every
inference file; `checksums.sha256` provides the same values in standard form.
Fire3D's Hierarchical Compression VAE (HC-VAE) compresses each sparse SC-VAE
shape and material field from a `32^3` representation with 32 channels into an
`8^3` representation with 64 channels. This 32x reduction enables batched flow
sampling and VAE decoding across many scene instances while retaining the
frozen TRELLIS.2 SC-VAE decoding path.
| Component | Bundle path |
|---|---|
| Scene perception | `perception/model.pt` |
| Sparse-structure flow | `reconstruction/flows/ss/model.pt` |
| Shape flow | `reconstruction/flows/shape/model.pt` |
| PBR flow | `reconstruction/flows/pbr/model.pt` |
| Sparse-structure VAE | `reconstruction/vae/ss/ckpts/encoder.pt`, `decoder.pt` |
| Shape HC-VAE | `reconstruction/vae/shape/ckpts/encoder.pt`, `decoder.pt` |
| PBR HC-VAE | `reconstruction/vae/pbr/ckpts/encoder.pt`, `decoder.pt` |
| DINOv3 encoder | `external/dinov3_vitl16_pretrain_lvd1689m-8aa4cbdd.pth` |
| TRELLIS.2 shape decoder | `external/trellis2/shape_dec_next_dc_f16c32_fp16.safetensors` |
| TRELLIS.2 PBR decoder | `external/trellis2/tex_dec_next_dc_f16c32_fp16.safetensors` |
Public filenames intentionally do not encode private training iteration
numbers.
The sparse-structure VAE encoder and decoder operate on binary `8^3` object
supports through an `8 x 2 x 2 x 2` latent. The normal joint-training recipe
uses BCE-with-logits plus KL and is available in the
[code release](https://github.com/xiahongchi/Fire3D/blob/main/training/README.md#sparse-structure-vae).
When training a new SS flow, regenerate its latent cache and statistics with a
matched VAE encoder. The released pretrained flow remains pinned to its own
latent artifact contract.
The root `config.json` is the model repository's Hugging Face query file. A
normal `snapshot_download` includes it in the model snapshot, allowing the Hub
to maintain the standard repository download count without client-side
telemetry. See the [Hugging Face download-statistics
documentation](https://huggingface.co/docs/hub/models-download-stats).
The default protocols use classifier-free guidance 3, flow and VAE batches of
16, and the batch CuMesh postprocessor. The iTHOR, Imaginarium, and ScanNet++
protocols enable the validated background room-box fit before background
reconstruction.
The released HC-VAE recipe uses one process on one GPU; its configured
per-GPU batch size is therefore the effective global batch size.
## Training Data
Fire3D does not redistribute its training corpora. All source datasets are
publicly available and retain their original licenses and access terms. For
object training data, use
[TRELLIS-500K](https://huggingface.co/datasets/JeffreyXiang/TRELLIS-500K),
which provides the metadata and preparation tools for the ObjaverseXL, ABO,
3D-FUTURE, and HSSD sources used by the object pipeline. Official scene-data
links and Fire3D preprocessing entry points are listed in the
[code release](https://github.com/xiahongchi/Fire3D/tree/main/data_processing).
## Use
```bash
git clone https://github.com/xiahongchi/Fire3D.git
cd Fire3D
bash scripts/install.sh
conda activate fire3d
bash scripts/install_blender.sh
fire3d download --models
fire3d download --data --dataset ithor --scene-id iTHOR_FloorPlan312_physics
bash scripts/run_ithor_example.sh
```
The downloader places this repository under `checkpoints/Fire3D/` and verifies
the release manifest before inference.
## Scope
The released checkpoints are intended for inference through the frozen Fire3D
protocols. Performance can degrade for scenes with inaccurate camera poses or
depth, severe occlusion, unusual scale, non-room backgrounds, or objects far
outside the training distribution. The generated meshes should be reviewed
before use in safety-critical or physically deployed applications.
## Licenses
Original Fire3D code is MIT licensed. This model bundle includes DINOv3 and
selected TRELLIS.2 files, which retain their upstream licenses. Review the
included `licenses/` directory and the repository's
`THIRD_PARTY_NOTICES.md` before use or redistribution.
## Citation
```bibtex
@article{xia2026fire3d,
title={{FIRE3D}: Feed-forward Interactive 3D Scene Reconstruction Within A Minute},
author={Xia, Hongchi and Cheng, Tianhang and Ma, Wei-Chiu and Wang, Shenlong},
journal={arXiv preprint arXiv:2609.08848},
year={2026},
url={https://arxiv.org/pdf/2609.08848}
}
```
|