Fire3D / README.md

Add Fire3D arXiv paper link and citation

84d4246 verified about 2 months ago

7.03 kB

---
license: other
pipeline_tag: image-to-3d
tags:
  - 3d-scene-reconstruction
  - rgb-d
  - single-image
  - flow-matching
  - textured-mesh
---
# FIRE3D: Feed-forward Interactive 3D Scene Reconstruction Within A Minute

[Paper](https://arxiv.org/pdf/2609.08848) |
[Code](https://github.com/xiahongchi/Fire3D) |
[Inference data](https://huggingface.co/datasets/hongchi/Fire3D)

Fire3D is a unified feed-forward framework that transforms a single RGB image
or casual RGB video into simulation-ready 3D scene assets. It predicts a
compositional scene representation with object-level 6-DoF pose, bounding box,
mesh geometry, and texture, without test-time optimization.

This release provides the inference code, model checkpoints, processed example
inputs, and frozen protocols needed to reproduce Fire3D results on iTHOR,
Imaginarium, ScanNet++, and single-image scenes.

The inference code and installation instructions are maintained at
[xiahongchi/Fire3D](https://github.com/xiahongchi/Fire3D). Processed release
inputs are hosted at
[datasets/hongchi/Fire3D](https://huggingface.co/datasets/hongchi/Fire3D).

## Overview

![Fire3D reconstructs compositional, simulation-ready 3D scenes from RGB-D observations.](assets/teaser.png)

**Teaser.** Fire3D reconstructs a compositional 3D scene from a single RGB
image or casual RGB video. The output contains object-level pose, geometry,
and material assets that can be rendered, edited, and used in simulation.

[![Fire3D end-to-end perception and reconstruction method.](assets/method_overview.png)](assets/method_overview.pdf)

**Method overview.** Fire3D lifts RGB-D observations into a shared 3D scene
representation, predicts object instances and 6-DoF poses, and reconstructs
the sparse structure, shape, and PBR fields with three cascaded flow-matching
models. Batched decoders and mesh post-processing produce the final textured
scene assets.

[![Fire3D Hierarchical Compression VAE architecture.](assets/hcvae.png)](assets/hcvae.pdf)

**HC-VAE.** The Hierarchical Compression VAE reduces each sparse SC-VAE shape
or material field from a `32^3 x 32` representation to an `8^3 x 64` latent.
This 32x reduction makes scene-level batched flow sampling practical while
retaining the frozen TRELLIS.2 decoding path.

## Model Bundle

The repository contains the scene perception checkpoint, the three
flow-matching checkpoints, sparse VAE encoders and decoders, latent normalization
statistics, and the TRELLIS.2 shape/PBR decoders used by the frozen release
protocols. `manifest.json` records the byte size and SHA-256 digest of every
inference file; `checksums.sha256` provides the same values in standard form.

Fire3D's Hierarchical Compression VAE (HC-VAE) compresses each sparse SC-VAE
shape and material field from a `32^3` representation with 32 channels into an
`8^3` representation with 64 channels. This 32x reduction enables batched flow
sampling and VAE decoding across many scene instances while retaining the
frozen TRELLIS.2 SC-VAE decoding path.

| Component | Bundle path |
|---|---|
| Scene perception | `perception/model.pt` |
| Sparse-structure flow | `reconstruction/flows/ss/model.pt` |
| Shape flow | `reconstruction/flows/shape/model.pt` |
| PBR flow | `reconstruction/flows/pbr/model.pt` |
| Sparse-structure VAE | `reconstruction/vae/ss/ckpts/encoder.pt`, `decoder.pt` |
| Shape HC-VAE | `reconstruction/vae/shape/ckpts/encoder.pt`, `decoder.pt` |
| PBR HC-VAE | `reconstruction/vae/pbr/ckpts/encoder.pt`, `decoder.pt` |
| DINOv3 encoder | `external/dinov3_vitl16_pretrain_lvd1689m-8aa4cbdd.pth` |
| TRELLIS.2 shape decoder | `external/trellis2/shape_dec_next_dc_f16c32_fp16.safetensors` |
| TRELLIS.2 PBR decoder | `external/trellis2/tex_dec_next_dc_f16c32_fp16.safetensors` |

Public filenames intentionally do not encode private training iteration
numbers.

The sparse-structure VAE encoder and decoder operate on binary `8^3` object
supports through an `8 x 2 x 2 x 2` latent. The normal joint-training recipe
uses BCE-with-logits plus KL and is available in the
[code release](https://github.com/xiahongchi/Fire3D/blob/main/training/README.md#sparse-structure-vae).
When training a new SS flow, regenerate its latent cache and statistics with a
matched VAE encoder. The released pretrained flow remains pinned to its own
latent artifact contract.

The root `config.json` is the model repository's Hugging Face query file. A
normal `snapshot_download` includes it in the model snapshot, allowing the Hub
to maintain the standard repository download count without client-side
telemetry. See the [Hugging Face download-statistics
documentation](https://huggingface.co/docs/hub/models-download-stats).

The default protocols use classifier-free guidance 3, flow and VAE batches of
16, and the batch CuMesh postprocessor. The iTHOR, Imaginarium, and ScanNet++
protocols enable the validated background room-box fit before background
reconstruction.

The released HC-VAE recipe uses one process on one GPU; its configured
per-GPU batch size is therefore the effective global batch size.

## Training Data

Fire3D does not redistribute its training corpora. All source datasets are
publicly available and retain their original licenses and access terms. For
object training data, use
[TRELLIS-500K](https://huggingface.co/datasets/JeffreyXiang/TRELLIS-500K),
which provides the metadata and preparation tools for the ObjaverseXL, ABO,
3D-FUTURE, and HSSD sources used by the object pipeline. Official scene-data
links and Fire3D preprocessing entry points are listed in the
[code release](https://github.com/xiahongchi/Fire3D/tree/main/data_processing).

## Use

```bash
git clone https://github.com/xiahongchi/Fire3D.git
cd Fire3D
bash scripts/install.sh
conda activate fire3d
bash scripts/install_blender.sh
fire3d download --models
fire3d download --data --dataset ithor --scene-id iTHOR_FloorPlan312_physics
bash scripts/run_ithor_example.sh
```

The downloader places this repository under `checkpoints/Fire3D/` and verifies
the release manifest before inference.

## Scope

The released checkpoints are intended for inference through the frozen Fire3D
protocols. Performance can degrade for scenes with inaccurate camera poses or
depth, severe occlusion, unusual scale, non-room backgrounds, or objects far
outside the training distribution. The generated meshes should be reviewed
before use in safety-critical or physically deployed applications.

## Licenses

Original Fire3D code is MIT licensed. This model bundle includes DINOv3 and
selected TRELLIS.2 files, which retain their upstream licenses. Review the
included `licenses/` directory and the repository's
`THIRD_PARTY_NOTICES.md` before use or redistribution.

## Citation

```bibtex
@article{xia2026fire3d,
  title={{FIRE3D}: Feed-forward Interactive 3D Scene Reconstruction Within A Minute},
  author={Xia, Hongchi and Cheng, Tianhang and Ma, Wei-Chiu and Wang, Shenlong},
  journal={arXiv preprint arXiv:2609.08848},
  year={2026},
  url={https://arxiv.org/pdf/2609.08848}
}
```

	---
	license: other
	pipeline_tag: image-to-3d
	tags:
	- 3d-scene-reconstruction
	- rgb-d
	- single-image
	- flow-matching
	- textured-mesh
	---

	# FIRE3D: Feed-forward Interactive 3D Scene Reconstruction Within A Minute

	[Paper](https://arxiv.org/pdf/2609.08848) \|
	[Code](https://github.com/xiahongchi/Fire3D) \|
	[Inference data](https://huggingface.co/datasets/hongchi/Fire3D)

	Fire3D is a unified feed-forward framework that transforms a single RGB image
	or casual RGB video into simulation-ready 3D scene assets. It predicts a
	compositional scene representation with object-level 6-DoF pose, bounding box,
	mesh geometry, and texture, without test-time optimization.

	This release provides the inference code, model checkpoints, processed example
	inputs, and frozen protocols needed to reproduce Fire3D results on iTHOR,
	Imaginarium, ScanNet++, and single-image scenes.

	The inference code and installation instructions are maintained at
	[xiahongchi/Fire3D](https://github.com/xiahongchi/Fire3D). Processed release
	inputs are hosted at
	[datasets/hongchi/Fire3D](https://huggingface.co/datasets/hongchi/Fire3D).

	## Overview

	![Fire3D reconstructs compositional, simulation-ready 3D scenes from RGB-D observations.](assets/teaser.png)

	Teaser. Fire3D reconstructs a compositional 3D scene from a single RGB
	image or casual RGB video. The output contains object-level pose, geometry,
	and material assets that can be rendered, edited, and used in simulation.

	[![Fire3D end-to-end perception and reconstruction method.](assets/method_overview.png)](assets/method_overview.pdf)

	Method overview. Fire3D lifts RGB-D observations into a shared 3D scene
	representation, predicts object instances and 6-DoF poses, and reconstructs
	the sparse structure, shape, and PBR fields with three cascaded flow-matching
	models. Batched decoders and mesh post-processing produce the final textured
	scene assets.

	[![Fire3D Hierarchical Compression VAE architecture.](assets/hcvae.png)](assets/hcvae.pdf)

	HC-VAE. The Hierarchical Compression VAE reduces each sparse SC-VAE shape
	or material field from a `32^3 x 32` representation to an `8^3 x 64` latent.
	This 32x reduction makes scene-level batched flow sampling practical while
	retaining the frozen TRELLIS.2 decoding path.

	## Model Bundle

	The repository contains the scene perception checkpoint, the three
	flow-matching checkpoints, sparse VAE encoders and decoders, latent normalization
	statistics, and the TRELLIS.2 shape/PBR decoders used by the frozen release
	protocols. `manifest.json` records the byte size and SHA-256 digest of every
	inference file; `checksums.sha256` provides the same values in standard form.

	Fire3D's Hierarchical Compression VAE (HC-VAE) compresses each sparse SC-VAE
	shape and material field from a `32^3` representation with 32 channels into an
	`8^3` representation with 64 channels. This 32x reduction enables batched flow
	sampling and VAE decoding across many scene instances while retaining the
	frozen TRELLIS.2 SC-VAE decoding path.

	\| Component \| Bundle path \|
	\|---\|---\|
	\| Scene perception \| `perception/model.pt` \|
	\| Sparse-structure flow \| `reconstruction/flows/ss/model.pt` \|
	\| Shape flow \| `reconstruction/flows/shape/model.pt` \|
	\| PBR flow \| `reconstruction/flows/pbr/model.pt` \|
	\| Sparse-structure VAE \| `reconstruction/vae/ss/ckpts/encoder.pt`, `decoder.pt` \|
	\| Shape HC-VAE \| `reconstruction/vae/shape/ckpts/encoder.pt`, `decoder.pt` \|
	\| PBR HC-VAE \| `reconstruction/vae/pbr/ckpts/encoder.pt`, `decoder.pt` \|
	\| DINOv3 encoder \| `external/dinov3_vitl16_pretrain_lvd1689m-8aa4cbdd.pth` \|
	\| TRELLIS.2 shape decoder \| `external/trellis2/shape_dec_next_dc_f16c32_fp16.safetensors` \|
	\| TRELLIS.2 PBR decoder \| `external/trellis2/tex_dec_next_dc_f16c32_fp16.safetensors` \|

	Public filenames intentionally do not encode private training iteration
	numbers.

	The sparse-structure VAE encoder and decoder operate on binary `8^3` object
	supports through an `8 x 2 x 2 x 2` latent. The normal joint-training recipe
	uses BCE-with-logits plus KL and is available in the
	[code release](https://github.com/xiahongchi/Fire3D/blob/main/training/README.md#sparse-structure-vae).
	When training a new SS flow, regenerate its latent cache and statistics with a
	matched VAE encoder. The released pretrained flow remains pinned to its own
	latent artifact contract.

	The root `config.json` is the model repository's Hugging Face query file. A
	normal `snapshot_download` includes it in the model snapshot, allowing the Hub
	to maintain the standard repository download count without client-side
	telemetry. See the [Hugging Face download-statistics
	documentation](https://huggingface.co/docs/hub/models-download-stats).

	The default protocols use classifier-free guidance 3, flow and VAE batches of
	16, and the batch CuMesh postprocessor. The iTHOR, Imaginarium, and ScanNet++
	protocols enable the validated background room-box fit before background
	reconstruction.

	The released HC-VAE recipe uses one process on one GPU; its configured
	per-GPU batch size is therefore the effective global batch size.

	## Training Data

	Fire3D does not redistribute its training corpora. All source datasets are
	publicly available and retain their original licenses and access terms. For
	object training data, use
	[TRELLIS-500K](https://huggingface.co/datasets/JeffreyXiang/TRELLIS-500K),
	which provides the metadata and preparation tools for the ObjaverseXL, ABO,
	3D-FUTURE, and HSSD sources used by the object pipeline. Official scene-data
	links and Fire3D preprocessing entry points are listed in the
	[code release](https://github.com/xiahongchi/Fire3D/tree/main/data_processing).

	## Use

	```bash
	git clone https://github.com/xiahongchi/Fire3D.git
	cd Fire3D
	bash scripts/install.sh
	conda activate fire3d
	bash scripts/install_blender.sh

	fire3d download --models
	fire3d download --data --dataset ithor --scene-id iTHOR_FloorPlan312_physics
	bash scripts/run_ithor_example.sh
	```

	The downloader places this repository under `checkpoints/Fire3D/` and verifies
	the release manifest before inference.

	## Scope

	The released checkpoints are intended for inference through the frozen Fire3D
	protocols. Performance can degrade for scenes with inaccurate camera poses or
	depth, severe occlusion, unusual scale, non-room backgrounds, or objects far
	outside the training distribution. The generated meshes should be reviewed
	before use in safety-critical or physically deployed applications.

	## Licenses

	Original Fire3D code is MIT licensed. This model bundle includes DINOv3 and
	selected TRELLIS.2 files, which retain their upstream licenses. Review the
	included `licenses/` directory and the repository's
	`THIRD_PARTY_NOTICES.md` before use or redistribution.

	## Citation

	```bibtex
	@article{xia2026fire3d,
	title={{FIRE3D}: Feed-forward Interactive 3D Scene Reconstruction Within A Minute},
	author={Xia, Hongchi and Cheng, Tianhang and Ma, Wei-Chiu and Wang, Shenlong},
	journal={arXiv preprint arXiv:2609.08848},
	year={2026},
	url={https://arxiv.org/pdf/2609.08848}
	}
	```