README.md

metadata

title: SplatWorld
emoji: 🌊
colorFrom: indigo
colorTo: pink
sdk: gradio
sdk_version: 5.49.1
app_file: app.py
pinned: false
license: cc-by-nc-4.0
tags:
  - gabor
  - wave-optics
  - vae
  - generative
  - faces
  - celeba
  - onnx

SplatWorld

202,599 CelebA faces compressed into one 7.2 MB decoder — and it doesn't store a single pixel.

SplatWorld is a small VAE whose decoder doesn't paint images. It maps a 128‑dimensional point z to 256 Gabor wave packets and lets their interference become the picture. A face is a phase‑locked standing wave. The "fire" you see drifting between faces is those same waves coming unlocked. This repo gives you two ways to walk around inside that world.

---

Two apps

	run it	what it is
app.py (Gradio)	python app.py → opens in a browser	Runs anywhere, including Hugging Face Spaces. Surf with sliders, scrub the zoom, render a zoom video, sample a gallery. This is the Space entry point.
explore_desktop.py (OpenCV)	python explore_desktop.py	The richer local tool: a live window with real mouse‑drag surfing and a hands‑free automated zoom. Needs a display, so it can't run on Spaces.
splat_atlas.py	python splat_atlas.py --dump then --browse	Bakes thousands of thumbnails to disk to survey the whole latent space; click any tile back to a full‑res face.
splat_trainer2.py	python splat_trainer2.py --data_dir …	Rebuild the model from your own faces (needs PyTorch + a CUDA GPU). See Train your own model below.

Both need splat_decoder.onnx (7.2 MB) sitting next to them. The Gradio app still boots without it (showing mock noise), so a Space never hard‑fails on startup — but add the model to see faces.

---

Quick start (browser app)

pip install -r requirements.txt
# put splat_decoder.onnx next to app.py, then:
python app.py                 # http://127.0.0.1:7860

On Hugging Face: create a Gradio Space, upload these files plus splat_decoder.onnx, and it runs. The header block at the top of this README is the Space config.

Quick start (desktop app)

pip install -r requirements-desktop.txt
python explore_desktop.py     # live OpenCV window
python explore_desktop.py --selftest
python explore_desktop.py --gallery 64   # headless: dump faces to ./gallery/

Desktop keys: 1 zoom · 2 surf · 3 atlas · H theory · B gallery · R record · S save · Q quit. In surf, drag to morph, wheel to dive/rise, SPACE for a new face, N to re‑roll the drag plane.

---

The atlas (splat_atlas.py)

The atlas is a heavier, separate tool. It systematically bakes thousands of thumbnails to disk (with every z saved) so you can survey the whole space at a glance and click any tile back to a full‑res face.

python splat_atlas.py --dump --gb 2   # build it ONCE (writes ./atlas/, a few GB)
python splat_atlas.py --browse        # click a tile -> full res + z;  n/p flip pages
python splat_atlas.py --analyze       # fine-band energy vs |z|: the "departure curve"

--browse will say "no atlas found — run --dump first" until you've dumped. Once ./atlas/ exists, app.py's zoom automatically uses the real core faces in it as waypoints.

---

How it works

The decoder is a tiny MLP: z (128-D) → 256 packets × 11 numbers. Each packet is a Gabor atom — position, size (σ), orientation (θ), frequency, and a complex (a, b) amplitude per color channel, i.e. the cosine weight and the sine weight of a little oriented wave. The renderer drops all 256 vibrating drumheads on a 96×96 canvas and sums their ripples with explicit env·cos and env·sin math. No convolutions, no pixels — just interference.

• Core (|z| < 15): the atoms phase‑lock. Peaks and troughs align to cancel everywhere except along a sharp eyebrow or a soft cheekbone. A standing wave — a face.
• Fire (|z| > 35): no training data lives out here, so the decoder stops orchestrating. Envelopes wander, phases drift, packets beat against each other. That decorrelating soup is what you're flying through between faces.

Training is coarse‑to‑fine and you can watch it happen: the envelope code (where the mass sits) commits first, the carrier code (orientation and phase) commits second. Early in training the eyebrows are flat horizontal blobs (envelope only); once the coarse loss plateaus the model finally commits an orientation and the brows snap into their characteristic slant. Amplitude first, phase later.

---

The space between faces (the theory)

Why is there anything between two faces? Because moving a feature from A to B is transport, and transport is where linear models break.

In a fixed additive basis, the only way to move an eyebrow is to fade one atom out while fading another in. Mid‑way both exist, their phases fight, and you get interference. The fire is that fight made visible. This is not new — it's 1990s technology. Eigenfaces built face‑space as linear combinations of basis faces in 1991, and interpolating between two of them produced exactly these ghostly double‑exposures. A purely additive "image model by linear transfer" is ancient, and the fire is its failure mode frozen in place.

There's a loophole worth naming, because it's the direction this whole line of work keeps pointing at. A complex (a, b) atom can translate a feature by rotating its phase — the Fourier shift theorem — instead of crossfading amplitudes. Phase‑transport moves the wave without ever destroying it, so it leaves no ghost. The atoms here already carry that complex structure; a "good" version of this world would move features by rotating phases, not sliding coefficients.

And the size of the gap is a tug‑of‑war you can feel in the trainer. Squeeze faces together (raise the VAE's β) and the morphs get buttery but identities collapse toward a bland mean. Push them apart (low β) and every face is crisp but the fire between them is vast. Crisp‑and‑dense at once — density without collapse — is the actual open problem.

If you want the quantitative version: splat_atlas.py --analyze measures fine‑band energy as a function of |z|. That curve is "the splats appear and grow," plotted. The local geometry of the gaps (how fast the image changes per step of z, direction by direction) is a Riemannian pullback metric — mapping it would tell you exactly where the fire is thick and where faces slide smoothly into each other.

---

Troubleshooting

cv2.error ... buf.u == m.u on OpenCV 5. OpenCV 5's new dnn engine crashes on this graph when the browser app runs the model in a worker thread. Two fixes, either works:

• Install onnxruntime (recommended, and what the Space uses): pip install onnxruntime. The app prefers it automatically and it sidesteps OpenCV entirely.
• If you only have OpenCV, the app now builds a per-thread net and prefers the classic engine, which avoids the assertion. Just update to this version of app.py.

The status line under the title tells you which backend loaded (onnxruntime, opencv-dnn, or mock).

The Space shows mock noise. splat_decoder.onnx isn't next to app.py. Add it (commit it to the Space repo, or use Git LFS).

---

Train your own model (splat_trainer2.py)

The trainer is included so you can rebuild splat_decoder.onnx from your own folder of faces (or retrain on CelebA). It needs PyTorch and a CUDA GPU for real speed; the model here was trained on a 12 GB card.

pip install -r requirements-train.txt      # torch build must match your CUDA

# 1) train  (first run caches the dataset once, then trains from the GPU-resident tensor)
python splat_trainer2.py --data_dir /path/to/faces --beta 0.0005

# 2) export the trained checkpoint to the ONNX the explorers use
python splat_trainer2.py --export          # -> splat_decoder.onnx

# 3) sanity-check the whole pipeline on CPU, no data or GPU needed
python splat_trainer2.py --smoke

How it's fast: it caches the folder to a uint8 array once (decoding 200k JPEGs every epoch was the real bottleneck), keeps the whole dataset resident on the GPU (≈5.6 GB at 96px, no DataLoader, no per-step host→device copy), and trains in gradient steps with a vectorized renderer. --smoke was verified end-to-end (loop-vs-vectorized renderer parity, a short train, export, and cv.dnn↔torch agreement to 1e-4 across a batched dynamic axis).

Useful flags (defaults in brackets):

flag	meaning
--data_dir [./faces]	folder of images (jpg/png/bmp/webp), center-cropped and resized
--image_size [96]	render resolution — bigger costs VRAM fast
--num_packets [256]	Gabor atoms per image
--steps [30000] · --batch [96] · --lr [3e-4]	training length / batch / learning rate
--beta [1.0] · --beta_warmup_steps [3000]	KL weight and its ramp. Low β (e.g. 0.0005) is what gives varied faces; high β collapses to a mean face
--gamma_floater [0.02] · --sigma_ref [0.03]	anti-"floater" penalty: taxes envelopes thinner than sigma_ref (0 disables)
--checkpointing	halve VRAM, double renderer compute — only if you OOM
--out [./runs/splat2]	where model2.pt and the recon/sample grids are written

The export always writes the input/output names (z_latent / rendered_image, opset 17, dynamic batch) that every tool in this repo expects, so a freshly trained model is a drop-in replacement.

Note on ONNX + OpenCV 5. The export uses a dynamic batch axis. OpenCV 5's dnn importer can't parse it, so the explorers load the model with onnxruntime (see Troubleshooting). The trainer's own --smoke uses cv.dnn on a tiny graph and passes; the full 256-packet model needs onnxruntime at inference time.

---

Model card / provenance

Data	CelebA, 202,599 aligned face images
Resolution	96×96 (chosen to fit in VRAM, not for quality)
Latent	128‑D
Packets	256 Gabor atoms, 11 params each
Params	~5.74M
Training	30,000 steps, batch 96, cosine LR, β warm‑up
Export	ONNX, 7.2 MB, dynamic batch axis (many z per forward)
Inference	CPU‑friendly via OpenCV cv2.dnn; no GPU or PyTorch required

The trainer itself is not in this repo — this repo is the explorer. (Training code lives with the model line, e.g. TheSplat5.)

---

Honest ledger

Do not hype. Do not lie. Just show.

What's real and shown here

• The decoder genuinely reconstructs and samples CelebA‑like faces from a 7.2 MB file, live.
• The core→fire shell structure, and the coarse‑to‑fine (envelope‑then‑carrier) commitment, are directly observable in the tools.
• Zoom's z‑path is C0 across identity hand‑offs and across the scale wrap; the surf tangent axes are provably orthonormal (see --selftest).

What's honest to admit

• 96×96 is a memory limit, not an aesthetic. Hair and fine detail struggle; sampled faces skew toward the population mean.
• The "phase‑locked standing wave" framing is a faithful description of a Gabor renderer, not a claim of new physics. The physics/AI resonance here is a representation echo (complex amplitudes = phasors), not a discovery about nature.
• SURF explores a 2‑plane of the tangent space at a time. It does not give you free access to all 128 dimensions at once; press N for a new plane.
• The phase‑transport loophole above is a direction, not something this particular model already does well.

License / data terms. CelebA is released for non‑commercial research use — check the dataset's own terms before you do anything with the outputs. This repo is offered under CC BY‑NC 4.0 to stay consistent with that.

---

Credits

Built by Antti Luode with several AI systems as thinking partners over a couple of long sessions. The interference/eigenface/phase‑transport framing came out of those conversations; the code and the honest ledger are meant to let you check all of it yourself.