Polymorphic / experiments /fourier-splatting /FOURIER_SPLATTING_REPORT.md
UncleanCode's picture
Upload via Autoresearch export_data
8e02a28 verified
|
Raw History Blame Contribute Delete
5.48 kB

Fourier Splatting × AnySplat — Experiment Report

Date: 2026-10-13
Hardware: existing SF Compute H100 80 GB node; no new node created
Status: two-scene prototype, 6,000 scene-optimization iterations per scene

Question

Can the existing AnySplat video-to-3D pipeline be improved by converting its context-only geometry into Fourier-encoded planar primitives and optimizing the scene, instead of retaining a dense Gaussian cloud?

This experiment adapts the upstream Fourier Splatting implementation (https://github.com/mbjurca/FourierSplatting) to AnySplat geometry. It is a downstream, per-scene refinement experiment—not a new feed-forward model.

Evaluation protocol

  • Apartment: 24 frames total; 12 context views for geometry initialization/training and 12 held-out target views.
  • Kitchen: 32 frames total; 16 context views for geometry initialization/training and 16 held-out target views.
  • All images are center-cropped and resized to 448×448.
  • AnySplat context-only inference initializes the Fourier scene. All-frame AnySplat inference is used to obtain target camera poses and align them into the context reconstruction coordinate frame, matching the evaluation protocol. Held-out target pixels are not used for geometry initialization or Fourier training.
  • Initial seed selection uses AnySplat opacity × maximum scale. Fourier Splatting uses K=6 frequencies with staged activation at iterations 300, 600, 900, 1,200, and 1,500.
  • Each scene is optimized for 6,000 iterations. The final LOD sweep changes active Fourier frequency count while keeping primitive count fixed.
  • Metrics: PSNR (higher is better), SSIM (higher is better), and VGG LPIPS (lower is better). Baseline and Fourier metrics use the same VGG LPIPS implementation.

Main held-out results

Scene Method PSNR ↑ SSIM ↑ VGG LPIPS ↓ Primitives
Apartment AnySplat 19.1235 0.6612 0.3154 1,858,896
Apartment Fourier, K=6 20.3469 0.7041 0.3173 99,483
Kitchen AnySplat 21.6155 0.6660 0.2678 1,983,229
Kitchen Fourier, K=6 25.3078 0.8212 0.2586 120,000

Changes vs AnySplat

  • Apartment: +1.2234 dB PSNR, +0.0429 SSIM, LPIPS worsens by 0.0019; 18.68× fewer primitives.
  • Kitchen: +3.6923 dB PSNR, +0.1552 SSIM, LPIPS improves by 0.0092; 16.53× fewer primitives.

The result is encouraging but not uniform across all metrics: apartment LPIPS is marginally worse, while kitchen improves on all three metrics. These are two indoor scenes, not a benchmark sufficient to claim state of the art.

Fourier level-of-detail sweep

Primitive count is held constant within each scene. Truncating Fourier frequencies reduces PLY size and trades off perceptual quality.

Apartment (99,483 primitives)

K PLY size (MB) PSNR ↑ SSIM ↑ VGG LPIPS ↓
1 25.07 19.9081 0.7048 0.3831
2 25.87 20.0612 0.7056 0.3802
3 26.66 20.3420 0.7078 0.3551
4 27.46 20.3825 0.7071 0.3374
5 28.25 20.3771 0.7056 0.3236
6 29.05 20.3469 0.7041 0.3173

Kitchen (120,000 primitives)

K PLY size (MB) PSNR ↑ SSIM ↑ VGG LPIPS ↓
1 30.24 21.9698 0.7638 0.3696
2 31.20 23.3089 0.7772 0.3599
3 32.16 24.3752 0.8006 0.3186
4 33.12 24.9584 0.8137 0.2889
5 34.08 25.1819 0.8190 0.2696
6 35.04 25.3078 0.8212 0.2586

Takeaway from LOD: K=1–3 is substantially worse on LPIPS. K=5 is a useful compromise; K=6 gives the best perceptual metric on both scenes. The K=1 to K=6 PLY-size increase is about 15–16%, while the scene representation remains 16–19× smaller in primitive count than the AnySplat output. This report does not claim an equivalent 16–19× reduction in serialized bytes; those AnySplat PLY bytes were not measured in this run.

Interpretation and limitations

  1. Promising direction: Fourier Splatting looks useful as an offline scene-refinement / compact-representation stage after AnySplat.
  2. Not feed-forward: every scene requires 6,000 optimization iterations. This has not yet been integrated into a low-latency video-to-3D inference path.
  3. No SOTA claim: only two indoor scenes have been tested. More scenes, motion, occlusion, camera-path diversity, runtime and byte-level comparisons are required.
  4. Research-code license: the upstream Fourier Splatting repository states non-commercial research/evaluation terms. Review licensing before using its implementation in a commercial release.

Reproduction commands

From the vendored Fourier Splatting repository, with the project environment and compiled rasterizer installed:

python train.py \
  --source_path /path/to/fourier_scene \
  --model_path /path/to/output \
  --eval --iterations 6000 \
  --test_iterations 3000 4500 6000 \
  --save_iterations 3000 4500 6000 \
  --checkpoint_iterations 3000 4500 6000 \
  --densify_until_iter 0 --densification_interval 100000 \
  --fourier_K 6 \
  --fourier_freq_schedule 300,600,900,1200,1500 \
  --no_dome --resolution 448

The bundle contains the scene transforms and images, seed point clouds, final checkpoints, Fourier PLYs, LOD variants, evaluation outputs, and the preparation/evaluation scripts. Training uses context-only geometry; target images are reserved for held-out evaluation.