scene-model-6layer / README.md
haoz19's picture
Model card: drop internal dataset and run names
a913f94 verified
|
Raw History Blame Contribute Delete
5.55 kB
---
license: cc-by-nc-nd-4.0
tags:
- 3d
- depth-estimation
- multilayer-depth
- point-cloud
- diffusion
- scene-understanding
library_name: torch
pipeline_tag: image-to-3d
extra_gated_heading: "Request access to the World Tracing scene model"
extra_gated_description: >
These checkpoints are released for non-commercial research use under the
**CC BY-NC-ND 4.0 license** (Attribution-NonCommercial-NoDerivatives). Please share a few
details below so we can keep a light audit trail of how the
weights are used in the wild.
extra_gated_button_content: "Submit access request"
extra_gated_fields:
Full name: text
Affiliation (university / company): text
Country: country
Primary intended use:
type: select
options:
- Academic research
- Personal / hobbyist project
- Industrial research
- Commercial product
- Other
Brief description of your intended use: text
I agree to cite the World Tracing paper in any publication or release that uses these weights: checkbox
---
# World Tracing — Scene Model (6-layer, 504 × 504, r69e)
> **Weights last updated: 2026-10-30** — `r69e`, iter 428 000.
> This is the **WT-S** scene model of Table 2 in the paper.
## Access
The checkpoints in this repo are released under the **CC BY-NC-ND 4.0 license**.
Downloads are gated behind a short form (basic contact info + intended
use) so we can keep a light audit trail; access is granted right after
you submit it. Then log in with `huggingface-cli login` (or set
`HF_TOKEN`) and run any of the examples from the
[GitHub repo](https://github.com/haoz19/world-tracing) — the `wt`
package picks the token up automatically and `--ckpt r69e` triggers a
normal `hf_hub_download`.
EMA-only release weights for the **r69e** scene model from
[*World Tracing: Generative Pixel-Aligned Geometry Beyond the Visible*](https://haoz19.github.io/world-tracing-page/).
* **Repo**: <https://github.com/haoz19/world-tracing>
* **Project page**: <https://haoz19.github.io/world-tracing-page/>
* **Config name**: `r69e`
* **Architecture**: `MultilayerXYZModel`, 1.5 B params
* **Input**: 504 × 504 full-frame RGB (no alpha)
* **Output**: per-layer XYZ in camera space, 6 stacked depth maps
* **Training data**: synthetic indoor scene renders only (including
3D-FRONT), with 10 3D-FRONT rooms held out for evaluation. No
real-world or sky data — pre-mask the sky externally for outdoor inputs.
For sharper geometry on in-the-wild images, see the 840 × 840 variant
[`scene-model-6layer-840`](https://huggingface.co/haoz19/scene-model-6layer-840)
(`r69l`).
## 3D-FRONT benchmark (Table 2)
`eval/3dfront_test50/` holds the ground truth of the 50 held-out
3D-FRONT frames of Table 2 (input RGB, the first 6 depth-peeled layers as
camera-space XYZ normalised by the median depth, per-layer validity masks,
intrinsics). It is downloaded automatically by
[`benchmarks/3dfront/eval_3dfront.py`](https://github.com/haoz19/world-tracing/tree/main/benchmarks/3dfront),
which evaluates this checkpoint by default and can also score your own
predictions with the same protocol:
```bash
python benchmarks/3dfront/eval_3dfront.py # this model, mean of 8 seeds
python benchmarks/3dfront/eval_3dfront.py --pred-dir preds/ # your method
```
| | MAE | RMSE | AbsRel | L0 CD-L1 | L0 CD-L2 | L0 F@0.05 | All CD-L1 | All CD-L2 | All F@0.05 |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| Paper (WT-S, 1 seed) | 0.0102 | 0.0215 | 0.0114 | 0.0093 | 0.000204 | 0.9867 | 0.0216 | 0.001255 | 0.8951 |
| This checkpoint (mean of 8 seeds) | 0.0085 | 0.0194 | 0.0094 | 0.0081 | 0.000146 | 0.9925 | 0.0208 | 0.001240 | 0.8998 |
The paper numbers come from an earlier checkpoint of the same run
(~100k iterations before this one). The ground-truth frames are renders
of [3D-FRONT](https://tianchi.aliyun.com/specials/promotion/alibaba-3d-scene-dataset)
scenes and are provided for evaluation only, subject to the 3D-FRONT
terms of use.
## Files
| File | Size | Format |
|---|---|---|
| `model.pt` | 5.59 GB | bare `state_dict`, float32 |
| `eval/3dfront_test50/` | 272 MB | 50 × `.npz` + `samples_3dfront_test50.json` |
EMA weights only — ~26 % of the original training checkpoint.
## Usage
```bash
git clone https://github.com/haoz19/world-tracing
cd world-tracing
pip install -e ".[viz]"
python examples/infer_scene.py \
--image examples/test_images/scene/scene_indoor_01_modern_living_room__seed42.png \
--ckpt r69e \
--config r69e \
--out /tmp/wt_scene.rrd
```
Bare `--ckpt r69e` triggers `huggingface_hub.hf_hub_download` against
this repo.
## Citation
```bibtex
@inproceedings{zhang2026worldtracinggenerativepixelaligned,
title = {World Tracing: Generative Pixel-Aligned Geometry Beyond the Visible},
author = {Hao Zhang and Mohamed El Banani and Jen-Hao Cheng and Paul Zhang
and Yi Hua and Ben Mildenhall and Christoph Lassner
and Narendra Ahuja and Gengshan Yang},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2026},
note = {Spotlight},
eprint = {2606.13652},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2606.13652}
}
```
## License
[CC BY-NC-ND 4.0](https://creativecommons.org/licenses/by-nc-nd/4.0/) (Attribution-NonCommercial-NoDerivatives) — see the [GitHub repo](https://github.com/haoz19/world-tracing/blob/main/LICENSE). Non-commercial research use only.