Model card: drop internal dataset and run names

a913f94 verified 15 days ago

5.55 kB

license: cc-by-nc-nd-4.0
tags:
  - 3d
  - depth-estimation
  - multilayer-depth
  - point-cloud
  - diffusion
  - scene-understanding
library_name: torch
pipeline_tag: image-to-3d
extra_gated_heading: Request access to the World Tracing scene model
extra_gated_description: >
  These checkpoints are released for non-commercial research use under the **CC
  BY-NC-ND 4.0 license** (Attribution-NonCommercial-NoDerivatives).  Please
  share a few details below so we can keep a light audit trail of how the
  weights are used in the wild.
extra_gated_button_content: Submit access request
extra_gated_fields:
  Full name: text
  Affiliation (university / company): text
  Country: country
  Primary intended use:
    type: select
    options:
      - Academic research
      - Personal / hobbyist project
      - Industrial research
      - Commercial product
      - Other
  Brief description of your intended use: text
  I agree to cite the World Tracing paper in any publication or release that uses these weights: checkbox

World Tracing — Scene Model (6-layer, 504 × 504, r69e)

Weights last updated: 2026-10-30 — r69e, iter 428 000. This is the WT-S scene model of Table 2 in the paper.

Access

The checkpoints in this repo are released under the CC BY-NC-ND 4.0 license. Downloads are gated behind a short form (basic contact info + intended use) so we can keep a light audit trail; access is granted right after you submit it. Then log in with huggingface-cli login (or set HF_TOKEN) and run any of the examples from the GitHub repo — the wt package picks the token up automatically and --ckpt r69e triggers a normal hf_hub_download.

EMA-only release weights for the r69e scene model from World Tracing: Generative Pixel-Aligned Geometry Beyond the Visible.

Repo: https://github.com/haoz19/world-tracing
Project page: https://haoz19.github.io/world-tracing-page/
Config name: r69e
Architecture: MultilayerXYZModel, 1.5 B params
Input: 504 × 504 full-frame RGB (no alpha)
Output: per-layer XYZ in camera space, 6 stacked depth maps
Training data: synthetic indoor scene renders only (including 3D-FRONT), with 10 3D-FRONT rooms held out for evaluation. No real-world or sky data — pre-mask the sky externally for outdoor inputs.

For sharper geometry on in-the-wild images, see the 840 × 840 variant scene-model-6layer-840 (r69l).

3D-FRONT benchmark (Table 2)

eval/3dfront_test50/ holds the ground truth of the 50 held-out 3D-FRONT frames of Table 2 (input RGB, the first 6 depth-peeled layers as camera-space XYZ normalised by the median depth, per-layer validity masks, intrinsics). It is downloaded automatically by benchmarks/3dfront/eval_3dfront.py, which evaluates this checkpoint by default and can also score your own predictions with the same protocol:

python benchmarks/3dfront/eval_3dfront.py                    # this model, mean of 8 seeds
python benchmarks/3dfront/eval_3dfront.py --pred-dir preds/  # your method

	MAE	RMSE	AbsRel	L0 CD-L1	L0 CD-L2	L0 F@0.05	All CD-L1	All CD-L2	All F@0.05
Paper (WT-S, 1 seed)	0.0102	0.0215	0.0114	0.0093	0.000204	0.9867	0.0216	0.001255	0.8951
This checkpoint (mean of 8 seeds)	0.0085	0.0194	0.0094	0.0081	0.000146	0.9925	0.0208	0.001240	0.8998

The paper numbers come from an earlier checkpoint of the same run (~100k iterations before this one). The ground-truth frames are renders of 3D-FRONT scenes and are provided for evaluation only, subject to the 3D-FRONT terms of use.

Files

File	Size	Format
`model.pt`	5.59 GB	bare `state_dict`, float32
`eval/3dfront_test50/`	272 MB	50 × `.npz` + `samples_3dfront_test50.json`

EMA weights only — ~26 % of the original training checkpoint.

Usage

git clone https://github.com/haoz19/world-tracing
cd world-tracing
pip install -e ".[viz]"

python examples/infer_scene.py \
    --image  examples/test_images/scene/scene_indoor_01_modern_living_room__seed42.png \
    --ckpt   r69e \
    --config r69e \
    --out    /tmp/wt_scene.rrd

Bare --ckpt r69e triggers huggingface_hub.hf_hub_download against this repo.

Citation

@inproceedings{zhang2026worldtracinggenerativepixelaligned,
  title         = {World Tracing: Generative Pixel-Aligned Geometry Beyond the Visible},
  author        = {Hao Zhang and Mohamed El Banani and Jen-Hao Cheng and Paul Zhang
                   and Yi Hua and Ben Mildenhall and Christoph Lassner
                   and Narendra Ahuja and Gengshan Yang},
  booktitle     = {Advances in Neural Information Processing Systems (NeurIPS)},
  year          = {2026},
  note          = {Spotlight},
  eprint        = {2606.13652},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2606.13652}
}

License

CC BY-NC-ND 4.0 (Attribution-NonCommercial-NoDerivatives) — see the GitHub repo. Non-commercial research use only.