Footprint-conditioned building massing — vecset diffusion

Status: active research. Not a production model. Every number below is measured on a single 48-building held-out set with one seed. Nothing here has been reproduced independently, and the project's own hard acceptance criterion is not yet met.

This repository holds the weights, the evaluation harness results, and the reasoning trail for a 49M-parameter latent-set ("vecset") diffusion model that turns a building footprint plus a height into a solid 3D mass. It is the geometry stage of a larger hybrid town-generation pipeline.

The interesting content here is not the checkpoint. It is the chain of measurements that killed four plausible approaches, and a set of measurement traps that cost real GPU-hours to learn.

TL;DR

A decoded-surface loss — supervising the diffusion through a frozen mesh decoder instead of on latent MSE — is the largest single lever found. It bought +0.029 3D IoU for 10 GPU-hours, where tripling training length bought +0.008 for 11.
Correcting where in the noise schedule that loss is applied (t/T 0.40 → 0.55) produced the first selective carve any model in this project has achieved: on 29 of 48 buildings it removes surplus volume without eroding the shape.
It is bimodal. On the other 19 buildings it produces hollow shells. This is a reliability problem, not a capability one.
🔑 Latent distance does not predict decoded quality. Spearman ρ = +0.12 pooled across error families — wrong-signed. A latent-MSE objective cannot rank its own candidates. This one result invalidated a large amount of earlier evidence.

Results

Harness: eval_massing_arms.py, 48 pinned held-out ids, all arms scored in one pass, sampling strength s=0.5. Artifact: massing_arms_eval_band240000.json.

3D IoU is split into missing (shape the model failed to produce) and extra (surplus volume it added). That split is the whole reason this project made progress — the aggregate hides which way a model is failing.

arm	fp-IoU	missing ↓	extra ↓	3D IoU
ground truth	1.000	0.000	0.000	1.000
codec ceiling (encode→decode GT)	0.997	0.000	0.001	0.999
blockout (extruded footprint — the input)	1.000	0.000	0.183	0.845
surface-loss model (`v4`)	0.962	0.002	0.191	0.838
band-fix model (`v5`, this release)	0.954	0.051	0.158	0.737

The band-fix row is a median over a bimodal population, which is misleading on its own. Split:

subset	n	extra ↓	3D IoU
solid (`missing` < 0.15)	29/48	0.149	0.833
hollow	19/48	0.162	0.353

The two modes, same checkpoint. This is what a median of 0.051 missing against a mean of 0.244 actually looks like.

Final checkpoint @240k. Left to right: ground truth · blockout input · surface-loss model · band fix. Row 2 is solid; the other two rows are hollow. Same buildings and camera throughout.

⚠️ A selection effect in the headline number

The project's internal write-up reports the solid subset as "extra 0.149 vs the blockout's 0.183 — a 19% surplus reduction." That comparison is not like-for-like. 0.183 is the blockout's median over all 48 buildings; 0.149 is the model's median over the 29 it happened to succeed on. Scoring the blockout on those same 29 ids gives 0.169.

The honest figure is 0.149 vs 0.169 — an 11.8% surplus reduction, not 19%.

The effect is real and still the first selective carve this project has produced. It is smaller than previously stated, and the subset is selected by the model's own behaviour, so it is a conditional result: "when it works, it removes ~12% of the surplus."

The training trajectory is non-monotonic — do not extrapolate it

checkpoint	fp-IoU	missing	extra	3D IoU
@190k	0.912	0.777	0.135	0.195
@220k	0.903	0.773	0.145	0.200 ← renders showed shredded cages
@230k	0.934	0.027	0.189	0.825 ← recovered
@240k	0.954	0.051	0.158	0.737

A stop was recommended at 220k. The very next checkpoint recovered to 0.825. This is the second time a run in this project was nearly killed during a transient collapse — the earlier one went 0.719 → 0.657 → 0.532 → 0.840 across epochs.

🔑 A 30,000-step window of catastrophic output is not evidence of a dead run in this model.

Method

Task. Footprint polygon (rasterised to a 64×64 mask) + height → solid 3D building mass. ⚠️ Height is a user input, not inferred. The contribution as it stands is footprint + height → mass. "Footprint alone" is future work.

Representation. A frozen third-party Dora-VAE (CVPR 2025, arXiv 2412.17808, Apache-2.0) encodes each building surface into a latent token set, decoded by cross-attention at arbitrary query points. Dora was chosen over a dense-grid codec because its sharp-edge sampling is a direct, published fix for the exact failure mode this project was fighting — losing crisp edges at the autoencoder level.

Generator. VecsetDenoiser — a DiT-style transformer over the token set:


parameters	49.4 M (vs ~947 M for the superseded dense-grid model)
width / depth / heads	768 / 12 / 12
latent channels	64
conditioning	64×64 footprint mask → 16 tokens, cross-attention; AdaLN for timestep
schedule	cosine ᾱ
training set	34,909 LoD2 building meshes

Generation is projection. The model does not synthesise from noise. It receives the extruded footprint ("blockout") as its starting point and learns to carve it. The blockout is therefore a stage inside the method, not a rival baseline — though it remains a valid ablation on whether the learned step earns its compute.

The decoded-surface loss (the actual contribution)

Standard latent-diffusion training regresses ε or x₀ in latent space. Measurement showed that objective cannot rank its own candidates (see below), so the loss was changed to supervise through the frozen decoder:

Predict x̂₀ from the noisy latent.
Decode x̂₀ at ~8,192 sampled query points via DoraCodec(differentiable=True).freeze().
Regress SDF against ground truth at those points.

Design notes, both of which were error-corrected mid-project:

No eikonal term. StEik (NeurIPS 2023) shows the plain eikonal loss becomes an unstable PDE as representation power grows. An earlier attempt in this project confirmed it directly: a gradient-TV term at w=0.1 diverged into rubble, because ε-error is amplified by 1/√ᾱ at high t.
🔑 The noise band matters more than the loss weight. The first run graded at t/T ≈ 0.40 while inference runs at 0.5–0.6. Grading near-clean latents taught the model to copy its input — vs input similarity 0.993. Moving to --surf_t_center 0.55, changing nothing else, produced the selective carve.

Cost is dominated by decode, so it scales with query count, not batch size: 8,192 points = 0.205 s and 5.04 GB per step, against a 305 ms denoiser step. Freezing the decoder yields 0 parameter gradients while still passing gradient to the latent.

The decoded-surface loss, before and after. Every metric column improved — the only lever in this project that did.

Why a frozen third-party autoencoder

Reconstruction quality: the superseded ~947M dense-grid codec against frozen Dora-VAE on the same buildings. Measured 0.00328 vs 0.00552 surface roughness.

🔑 What was ruled out, and how

This is the most transferable part of the work. Each of these closed a direction that looked reasonable.

1. Latent distance is decoupled from decoded quality — the objective cannot rank candidates. Spearman ρ of latent distance vs decoded 3D IoU, 144 candidates from two error families:

pool	n	L2	cosine
pooled	144	+0.120	−0.113
on-manifold only	24	+0.050	−0.188
off-manifold only	120	−0.503	+0.516

A distance metric wants negative ρ. Within one error family it works; pooled — which is the situation any real training run is in — it is worthless and slightly wrong-signed.

The model-free version is starker: a latent at cosine 0.083 decodes to IoU 0.999, while one at cosine 0.995 decodes to IoU 0.053. The 0.083 case is the same mesh re-encoded — furthest-point sampling simply reorders the tokens.

⚠️ Consequence: any "the denoiser is working, cosine improved 0.707 → 0.935" evidence is void. This retroactively invalidated a substantial body of earlier project evidence.

2. Post-hoc correction is exhausted — in both spaces. An SDF-space refiner and a latent-space corrector (zero-init residual 3D U-Net) both plateau at ~0.0047 roughness against a GT floor of 0.0041 and a codec ceiling of 0.0044. Driving latent L1 down 4× barely moved decoded roughness.

3. The codec was never the bottleneck. decode(encode(GT)) scores 0.999 3D IoU. A crisp building is fully representable. The diffusion samples latents that decode badly.

4. Data is not the constraint. All 35,623 meshes audited. ⚠️ Two findings that will bite anyone touching the data: the corpus on disk is inward-wound (35,602 of 35,623) and only comes out correct via load_surfaces — never read the h5 directly. And the meshes are coarse, median 20 faces, which bounds what any sharpness supervision can teach.

5. Training length is exhausted. 180k steps / 41 epochs — 3× the scored run — bought +0.008 IoU.

6. Building size does not explain the bimodality. (new, this pass) Testing the obvious first hypothesis for what separates the 29 solid from the 19 hollow: median GT volume 50,515 vs 43,554 voxels, Mann-Whitney p=0.246, point-biserial r=0.186 (p=0.204). Not significant at n=48. Size is ruled out; footprint complexity and source corpus remain untested.

⚠️ Measurement traps

These cost GPU-hours and near-miss false negatives. They are the highest-value content here.

trap	what happened
Never extrapolate the training curve	0.719 → 0.657 → 0.532 → 0.840. Three monotonic points did not predict the fourth. A stop at the dip would have recorded a false negative.
Always report `vs input`	The generator scores near the blockout by declining to act. At s=0.45 it returned its input at 99.9% and inherited its score. A model can look excellent while making no edit at all.
Medians lie on bimodal outcomes	Median `missing` 0.051, mean 0.244. As a median alone this reads clean. The harness should report a collapse rate.
The aggregate can be flat while geometry degrades	190k → 220k moved IoU 0.195 → 0.200 — "better" — while a building went from a box to a shredded cage.
n=10 probes are not quotable	Adjacent-checkpoint swing (0.59–0.78) is as large as any apparent trend.
Quote the outcome, not the peak	The pre-registered bar was met transiently at 9.5k (`extra` 0.178) and lost by 60k (0.191). Recorded not-met.
Compare on the same ids	See the selection effect above — a subset median against an all-population median inflated a result by 7 points.

The no-op. A model returning 99.9% of its input, scoring near the blockout because it inherited the blockout's score.

Further renders

image	what it shows
`harness-baseline-montage.png`	the 48-id harness baseline — all arms, one pass
`convergence-run-montage.png`	the first convergence run that beat the deployed model
`final-41epoch-montage.png`	41 epochs / 180k steps — where training length was exhausted
`band-fix-230000-comparison.jpg`	@230k — the cages filling back in after the collapse

Acceptance criteria — honest status

criterion	weight	status
1 — visual, human-judged: "would you take the model's output over the extruded footprint?"	primary	✅ passed (human answered yes)
2 — footprint match	hard, non-negotiable	⚠️ 0.962, needs 1.000
3 — 3D IoU	diagnostic only	0.737 median / 0.833 on the solid subset — not a gate

⚠️ Criterion 2 is the live gap and was under-weighted for an entire work cycle in favour of criterion 3, which the project's own specification marks as diagnostic. Anyone continuing this work should weight footprint fidelity first.

Literature positioning

Family this belongs to — vecset / latent-set 3D diffusion. The dominant recipe for open 3D generative models since 2023.

work	venue	arXiv	relevance
3DShape2VecSet	SIGGRAPH/TOG 2023	2301.11445	Introduced the representation. Base recipe for everything below.
Michelangelo	NeurIPS 2023	2306.17115	Vecset diffusion conditions cleanly on external modalities.
CLAY	SIGGRAPH/TOG 2024	2406.13897	Explicit 3D-aware control from primitives (voxels, boxes, point clouds). Direct precedent that footprint conditioning is a solved pattern, not a research risk.
Direct3D	NeurIPS 2024	2405.14832	Continuous latent triplane — the taxonomy is not clean-cut.
TRELLIS	CVPR 2025 Spotlight	2412.01506	Sparse structured latents; pays for resolution only where geometry exists.
Hunyuan3D 2.0	Tencent 2025	2501.12202	Vecset ShapeVAE + flow DiT. Uses importance sampling on edges and corners to preserve sharp detail.
Dora	CVPR 2025	2412.17808	The closest match to this problem. Diagnoses that uniform point sampling loses sharp geometry; fixes it with sharp-edge sampling + dual cross-attention. Matches XCube-VAE with an 8× smaller latent. Used frozen here.

Sharp iso-surface extraction (orthogonal, applies at export): DMTet (NeurIPS 2021), FlexiCubes (SIGGRAPH 2023), GET3D (NeurIPS 2022, notably demonstrated on a buildings category), IGR (ICML 2020, the eikonal term), StEik (NeurIPS 2023 — the stability caution that shaped this loss design).

Where this sits. The vecset recipe, footprint conditioning, and frozen-VAE latent diffusion are all established. What is not standard practice, and is the claim worth testing, is supervising a latent diffusion through its frozen decoder at a deliberately chosen noise band, adopted here because measurement showed the conventional latent objective could not rank its candidates. The band-placement finding — that grading near-clean latents teaches input-copying — is the most transferable result.

⚠️ Scale caveat. Vecset models in the literature train on hundreds of thousands to millions of shapes. This is 34,909. The sharp-edge-sampling training signal transfers at this scale; the capacity/scale advantages CLAY and Hunyuan3D-2 report likely do not.

Files

Start here — the current line of work. These five are scored on the 48-id harness and are what every number above refers to.

file	size	what
`massing-vecset/vecset_v5_surfband_step240000.pth`	189 MB	the band-fix model — final, scored (29/48 solid)
`massing-vecset/vecset_v5_surfband_step230000.pth`	189 MB	best 3D IoU (0.825); post-recovery
`massing-vecset/vecset_v5_surfband_step220000.pth`	189 MB	the collapse checkpoint — kept as evidence
`massing-vecset/vecset_v4_surf.pth`	189 MB	surface-loss model, pre-band-fix (+0.029 IoU)
`massing-vecset/vecset_v3_pair_long_step180000.pth`	189 MB	41-epoch control, no surface loss

Historical — the latest checkpoint of every earlier run. Included so no run is lost, not because each is good. None of these are recommended starting points.

file	size	what
`massing-vecset/stage3a_lod2_deployed.pth`	7.2 GB	superseded ~947M dense-grid baseline — comparison arm
`massing-vecset/vecset_v2_pair_step60000.pth`	189 MB	⚠️ pre-frame-fix — trained on transposed latents
`massing-vecset/vecset_v2_plain.pth`	189 MB	⚠️ pre-frame-fix — same defect
`massing-vecset/vecset_v1.pth`	189 MB	first vecset run
`massing-vecset/vecset_pair_v1.pth`	189 MB	first aligned-pair run
`massing-vecset/vqvae_release_res64.pth`	101 MB	released 64³ VQVAE codec (dense-grid era)
`massing-vecset/vqvae_clean_ft.pth`	101 MB	cleaned VQVAE fine-tune
`massing-vecset/monolith_v1/v2/v3.pth`	47 MB ea	monolith arms from the composition thesis

⚠️ The v1/v2 vecset runs are void, not merely weak. Their training cache had x and z transposed, so they learned a compensating axis swap. Results from them cannot be compared to anything after the frame fix. They are here for provenance only.

Optimizer state is stripped (checkpoints are inference/fine-tune ready, not resume-ready). latent_mu / latent_sd are retained and load-bearing — the denoiser trains on globally normalised latents and decodes to noise without them. Verify with cd massing-vecset && sha256sum -c SHA256SUMS.

⚠️ Keep global latent normalisation. Per-channel normalisation was measured and is harmful — the 16 low-variance channels are collapsed dimensions the decoder ignores.

Also here: the demo serving stack (`demo-serving/`)

Separate from the massing research above. The repo's FastAPI + three.js demo — town generation from a footprint image, SDF sculpting, texture bake — needs its own weights, which are published here so the demo actually runs from a clone.

path	size	what
`demo-serving/outputs/recipe_param_diffusion_b6/`	80 MB	recipe-parameter diffusion — proportions/roof/wings. `recipe_inference.py:44`
`demo-serving/outputs/recipe_param_diffusion_b6_ema/`	2.9 MB	EMA variant
`demo-serving/outputs/part_layout_planner_v2/`	17 MB	window/door/balcony layouts
`demo-serving/outputs/part_set_refiner/`	53 MB	integrates a sculpted mass into the part set
`demo-serving/outputs/part_composer/`	1.3 MB	statistical facade detail
`demo-serving/outputs/refiner_v1/`	22 MB	surface refiner
`demo-serving/logs_building/…-ft-final/ckpt/stage3a_steps-latest.pth`	3.6 GB	snap prior, main (`refine.py:469`)
`demo-serving/logs_building/…-ft/ckpt/stage3a_steps-1000.pth`	3.6 GB	snap prior, autoguidance guide (`refine.py:471`)

Paths mirror the repo layout, so demo-serving/ unpacks straight over a clone:

hf download danvisimhadri/SDFUSION --include 'demo-serving/*' --local-dir /tmp/demo
cp -r /tmp/demo/demo-serving/* .          # into the repo root
./scripts/server/run_web_demo.sh 8099

⚠️ The two snap-prior checkpoints are the same finetune run at different steps — the guide is deliberately the weaker, earlier checkpoint (autoguidance). Don't "upgrade" the guide to the latest; that defeats the mechanism. ⚠️ Optimizer state is stripped from both, so they are inference-only. ⚠️ SDXL / ControlNet / Depth-Anything (~46 GB) are not here — they auto-download to the HF cache on the first texture or render call.

Reproduction

Code: https://github.com/danvisai/SDFusion (branch massing-solid-gate-retrain). REPRODUCING.md covers clone → environment → data regeneration → verification. The corpus is regenerated from 25 MB of committed identity + mesh data rather than shipped: that 25 MB rebuilds a 35 GB SDF field and 17.4 GB of latent caches. Regeneration is equivalent, not bit-identical, and yields 35,623 rows rather than 35,776.

Attribution

The training corpus derives from three open government datasets, each carrying its own attribution terms, which any downstream use must honour:

3DBAG (Netherlands) — 3D BAG, TU Delft
NRW Open Data (Germany) — Geobasis NRW
PLATEAU (Japan) — MLIT Japan

The frozen autoencoder is Dora-VAE (Apache-2.0). Model weights here are released Apache-2.0; the data terms above are separate and are not superseded by it.

Limitations

Bimodal: fails on ~40% of held-out buildings, cause unknown.
Requires height as an input; does not infer it from footprint.
Criterion 2 (footprint fidelity 1.000) not met — measured 0.962.
Single seed, single 48-building held-out set, no independent reproduction.
Trained on European and Japanese LoD2 building stock; no evidence it generalises beyond that.
Meshes are coarse (median 20 faces), bounding achievable sharpness.