Spaces:
Running on Zero
Running on Zero
|
Download README.md from ritianyu/SplatAnything: direct link, hf CLI and curl.
- Browser
- Download file 16.1 kB
-
https://huggingface.co/spaces/ritianyu/SplatAnything/resolve/main/README.md
- Command line
-
hf download hf://spaces/ritianyu/SplatAnything/README.md
-
curl -L -o README.md https://huggingface.co/spaces/ritianyu/SplatAnything/resolve/main/README.md
16.1 kB
| title: SplatAnything (InfiniSplat v3) | |
| emoji: 🌐 | |
| colorFrom: blue | |
| colorTo: gray | |
| sdk: gradio | |
| sdk_version: 6.5.1 | |
| python_version: "3.10" | |
| app_file: space_app.py | |
| pinned: false | |
| # SplatAnything | |
| **Gaussian Spacing Normalization for Feed-Forward 3DGS across Arbitrary Scenes** | |
| [Project page](https://zju3dv.github.io/SplatAnything/) · [Paper](https://arxiv.org/html/2611.08264) | |
| Upload one image or several overlapping images of one static scene, extract images from a video, or upload one full 360° panorama. | |
| Select **RGB-only** to predict cameras, or **Pose-conditioned** to supply camera poses | |
| and intrinsics in `prior_camera.json`. Street, Rig-Recon and Aerial default to 504 × 504; Object, Indoor and Pan-Recon default to 756 × 756. Width and height presets are 504, 756, 896 and 1022; you can also type each dimension manually (positive multiples of 14). | |
| Single-image RGB-only reconstruction is supported; additional overlapping views provide more coverage. Uploading your own images resets the resolution to at most 896 × 896. Each dimension | |
| is capped by the smallest corresponding dimension in the uploaded set, then rounded | |
| down to a multiple of 14 (for example, 800 × 600 becomes 798 × 588). Adding or deleting | |
| images recomputes these defaults; clearing the upload restores 896 × 896. Captured | |
| camera photos and extracted video frames follow the same rule. Original files are | |
| preserved, and you can still edit width and height manually. | |
| The app renders native 3D Gaussian splats in a PlayCanvas viewer. It provides | |
| Gaussian PLY, standalone HTML, and a complete ZIP containing point clouds, cameras, | |
| processed RGB, predicted depth, and the effective configuration. Result links expire | |
| after 24 hours. Anyone given a result link can access that result until it expires; | |
| download it to keep a copy. | |
| ## Run locally | |
| Install a CUDA-compatible PyTorch build first, then: | |
| ```bash | |
| python -m pip install -r requirements.txt | |
| npm ci | |
| python app.py --host 0.0.0.0 --port 8081 | |
| # Open http://127.0.0.1:8081 | |
| ``` | |
| You can also run `python space_app.py`; it installs the pinned viewer package | |
| automatically and listens on `0.0.0.0:7860`. This repository contains the app, inference code, examples, and viewer | |
| dependencies. It runs independently of the GitHub checkout. | |
| ## Checkpoints | |
| Configure the server with complete student and WorldMirror-2 geometry weights. | |
| Local defaults are: | |
| ```text | |
| checkpoints/splatanything.ckpt | |
| checkpoints/worldmirror2.safetensors | |
| checkpoints/skyseg.onnx # optional, for sky removal | |
| ``` | |
| Use `SPLATANYTHING_CKPT`, `WORLDMIRROR2_CKPT`, and `SPLATANYTHING_SKY_CKPT` | |
| to select other local files. The local preparation can link to existing weights; | |
| the external links and checkpoints are ignored by Git and are not portable uploads. | |
| For a hosted Space, upload the two inference checkpoints to a Hugging Face **model** | |
| repository, then configure these Space Variables: | |
| | Variable | Value | | |
| |---|---| | |
| | `SPLATANYTHING_MODEL_REPO` | Your model repository ID (`owner/model`) | | |
| | `SPLATANYTHING_MODEL_REVISION` | Model commit or revision (default `main`) | | |
| | `SPLATANYTHING_MODEL_FILE` | Student filename (default `splatanything.ckpt`) | | |
| | `SPLATANYTHING_GEOMETRY_FILE` | Geometry filename (default `worldmirror2.safetensors`) | | |
| For a private model repository, add `HF_TOKEN` as a Space **Secret** with read | |
| access. Tokens stay on the server. Sky removal uses `skyseg.onnx`, downloaded and | |
| cached on first use from the model repository (default `ritianyu/SplatAnything`). | |
| `SPLATANYTHING_SKY_REPO`, `SPLATANYTHING_SKY_REVISION`, and `SPLATANYTHING_SKY_FILE` | |
| can select a separate sky model; `SPLATANYTHING_SKY_CKPT` overrides this with a | |
| local ONNX file. The CPU session is reused. Sky segmentation runs before the | |
| ZeroGPU reservation, with its time recorded as `sky_and_input_preprocessing`. | |
| Street and Rig-Recon examples enable **Remove sky** by default; other categories | |
| start with it disabled. You can change the checkbox before reconstruction. | |
| The shared `configs/inference.yaml` uses a default voxel size of 0.0018 and a 99.5 scale percentile with the supplied fine-tuned checkpoint. Both camera-mode flags are selected | |
| together by the UI. CLI examples use `python inference.py --example object/1 --mode pose`; | |
| use `--list_examples` to inspect the compact examples included in this bundle. | |
| ## Hugging Face runtime | |
| This folder is self-contained: it includes the inference source, Gradio UI, | |
| portable configuration, small examples, and the npm lockfile. It does not depend on | |
| a sibling GitHub checkout, private filesystem paths, or startup Git clones. | |
| 1. Push this folder to a new **Gradio Space** after reviewing it. | |
| 2. Select GPU hardware and configure the model variables above. | |
| 3. For **ZeroGPU**, set `SPLATANYTHING_ZEROGPU=1` and select ZeroGPU hardware. | |
| Its `spaces.GPU` adapter reserves the GPU only for prediction; model placement | |
| happens at startup and PLY/HTML/ZIP exports run afterward on the CPU. | |
| 4. For a dedicated GPU, leave `SPLATANYTHING_ZEROGPU=0` (the default). | |
| Max images selects the first N input frames; 0 or an empty value uses all frames. User uploads, video frames and camera captures reset it to 0. Video extraction samples 2–32 frames. | |
| `SPLATANYTHING_DEVICE=cpu` is available for small local smoke tests. | |
| `SPLATANYTHING_CONFIG` selects another complete inference YAML. | |
| GPU usage is subject to the hardware and quota settings of the hosted Space. | |
| The original example images live in the dataset | |
| [`ritianyu/SplatAnything_Demo`](https://huggingface.co/datasets/ritianyu/SplatAnything_Demo/tree/main/examples), | |
| under `examples/CATEGORY/SCENE/`. Set `SPLATANYTHING_EXAMPLES_REPO=ritianyu/SplatAnything_Demo` | |
| and `SPLATANYTHING_EXAMPLES_REVISION` to a dataset commit. `space_app.py` downloads | |
| the original images and camera files to a local cache before starting the studio. | |
| This keeps the Space source below its repository storage limit while retaining | |
| all original views and image quality. Local runs use the existing `examples/` | |
| folder; `SPLATANYTHING_EXAMPLES_DIR` or `--examples_dir` can select another folder. | |
| ## Input cameras | |
| Camera IDs must match the image filename stems. Supply OpenCV 4 × 4 | |
| camera-to-world poses and 3 × 3 intrinsics in original-image pixels. | |
| The app updates intrinsics during cropping and resizing. See | |
| [prepare_data.md](prepare_data.md) for camera conversion and validation. | |
| The scene library has six categories: **Object, Indoor, Street, Aerial, Rig-Recon, and Pan-Recon**. Examples are loaded directly from `examples/CATEGORY/SCENE/images/` and the matching `prior_camera.json`, when available. Indoor scenes use RGB-only mode. Street, Rig-Recon and Aerial scenes use 504 × 504; Object, Indoor and Pan-Recon use 756 × 756. The studio displays small previews for faster loading; Max images controls how many original views are reconstructed. Selecting a card loads the scene; reconstruction starts when you click the button. | |
| The studio opens with no selected data and a white viewer with an animated SplatAnything infinity mark. **Clear** resets the current example, uploads, camera captures, camera JSON and result preview for your session, restoring the idle animation. The animation uses local SVG/CSS, has no visible text, and respects reduced-motion preferences. Selecting an example loads its images and settings; the right-hand viewer shows a still captured from the matching HTML’s initial Gaussian camera. Click **Explore 3D** to download and open the precomputed scene from [SplatAnything_Demo](https://huggingface.co/datasets/ritianyu/SplatAnything_Demo). HTML files are fetched only on this click, verified against the pinned dataset revision, and cached for reuse. Click **Reconstruct scene** to run new inference with the current inputs and settings. Example covers use square crops for Object, Indoor, Street, Aerial, and Rig-Recon; Pan-Recon keeps the original panorama aspect ratio. These covers affect previews only: reconstruction uses the complete original images. Captions sit below the image. | |
| Video frame extraction is a CPU operation and resets the mode to RGB-only. | |
| ## Browser camera | |
| In **Images**, click **Open camera** and allow camera access. Capture one image or several overlapping views of one static scene; each snapshot is added to the input preview. The first snapshot starts a new RGB-only scene, replacing any previously selected example or uploads. Later snapshots append to that camera scene. Use **New set** to start over and **Close camera** to release the camera; closing it keeps the captured photos available for reconstruction. Switching input tabs or selecting an example also closes the camera. | |
| The camera belongs to the person visiting the page. It opens only after clicking the button, and the live video stays in the browser; only captured photos are uploaded. Browser camera access requires **HTTPS** (including hosted Hugging Face Spaces) or **localhost**, plus camera permission. A plain HTTP server IP address cannot access the camera. For the local demo, open `http://localhost:8081` on the client machine, with port forwarding if the server is remote. | |
| ## 360° panoramas | |
| Open the **360° Panorama** tab and upload a full 360° × 180° equirectangular JPG, PNG, or WebP image with a 2:1 aspect ratio. Click **Reconstruct panorama**: the demo first generates 24 perspective views (eight headings at three elevations, 60° FOV) and matching camera intrinsics and poses, then runs **Pose-conditioned** reconstruction automatically. No camera JSON upload is required. The resolution and export settings apply to panorama reconstruction too. Conversion runs on the CPU before GPU inference. | |
| The converter is included in this repository at [scripts/convert_panorama.py](scripts/convert_panorama.py), with its implementation in [src/utils/panorama.py](src/utils/panorama.py). It can also prepare an input scene from the command line: | |
| ```bash | |
| python scripts/convert_panorama.py \ | |
| --input /path/to/panorama.jpg \ | |
| --output /path/to/prepared_scene \ | |
| --size 896 | |
| ``` | |
| The resulting scene contains `images/`, `prior_camera.json`, and `panorama.json`. Apply EXIF orientation to the source image before uploading or converting it. | |
| ## Updating example previews | |
| `assets/scene-previews/catalog.json` matches each local `CATEGORY/SCENE` directory to the same numbered directory in the dataset. Its 24 WebP files are actual initial-camera Gaussian renders, packaged with the app so selecting examples does not download large HTML files. The live app does not require Chromium. | |
| To rebuild previews after updating the examples or dataset, install the optional build tooling and run: | |
| ```bash | |
| npm install --no-save playwright | |
| npx playwright install chromium | |
| python scripts/prepare_scene_previews.py | |
| ``` | |
| The script verifies the downloaded HTML hashes and uses the viewer’s `captureFrame` API at time zero, then publishes the new posters and catalog. Restart the demo afterward. | |
| ## Development | |
| Maintain the Gradio UI, ZeroGPU adapter, viewer, and demo examples in this repository. | |
| The GitHub repository contains the command-line inference release; it does not | |
| include the Gradio app or Space packaging scripts. This Space is self-contained | |
| and does not load code from a sibling checkout. | |
| Run the demo regression checks with: | |
| ```bash | |
| python -m unittest discover -s tests -v | |
| ``` | |
| ## Attribution | |
| The demo structure follows [InfiniSplat2](https://huggingface.co/spaces/Edisoneh/InfiniSplat2), | |
| with SplatAnything's own inference pipeline and input camera format. Native Gaussian | |
| viewing uses [PlayCanvas splat-transform](https://github.com/playcanvas/splat-transform) | |
| (installed separately, pinned to 2.6.0). | |
| Original component notices are preserved in [License.txt](License.txt) and [Notice.txt](Notice.txt). | |
| ## Inference speed and progress | |
| Inference Setting → **Max images** uses the first N frames: Object/Indoor/Pan-Recon default to 24, Street/Aerial to 100, Rig-Recon to 150. Uploaded images default to 0 (all frames); uploading a panorama selects 756 × 756 and its 24 generated views; a limit larger than the scene uses all available frames. Previewing precomputed examples remains independent of this setting. | |
| GPU servers preload the two models in the background after startup and reuse them across requests. Set `SPLATANYTHING_PRELOAD=0` to retain loading on the first request. ZeroGPU keeps its existing startup initialization and GPU scheduling. The inline progress indicator reports actual pipeline stages without obscuring the scene. | |
| The interactive Gaussian viewer takes priority: prediction is followed by native Gaussian PLY export and a fast compressed PLY preview, displayed in a shared, cached SuperSplat viewer. Standalone HTML, raw `points.ply`, depth maps and processed RGB files are exported by a bounded CPU background worker. The download panel reports when all files are ready. Gaussian PLY is available immediately. Standalone HTML downloads wait until the background HTML export completes. | |
| **Full result** waits for the background export to finish, then builds and caches the complete ZIP on its first download. It never packages a partially written result. If additional exports fail, the viewer and Gaussian PLY remain available and the app reports the failure. Archives use fast compression and keep already-compressed image formats as-is. Depth, RGB and raw points remain included; model precision and decoder chunk sizes are unchanged. | |
| A local comparison used the same 4.64 million Gaussians, a 960 × 600 Chromium | |
| software WebGL viewport, a shared 100 Mbps server-side bandwidth limit, and three | |
| runs per format. From an already exported native PLY to the first rendered frame, | |
| conversion plus loading took 30.2 s for native PLY, **13.1 s for compressed PLY**, | |
| 21.9 s for SOG, and 25.9 s for standalone HTML (sums of conversion and loading | |
| medians). These measurements exclude model inference and native PLY export; | |
| client hardware and network conditions affect results. On an unrestricted local | |
| connection, native PLY avoided conversion and had the shortest combined wait. | |
| Compressed PLY was selected for the hosted demo. No Gaussians are removed by | |
| this conversion; the native PLY remains available for full-precision download. | |
| `timings.json` records preparation, model loading/lock wait, prediction (including preprocessing and CPU transfer), `gaussian_export`, `preview_compression`, and elapsed `preview_ready`. After the additional files finish, it also records `html_export`, `auxiliary_export` and elapsed `all_files_ready`. Background exports run on CPU after the ZeroGPU prediction reservation has ended. | |
| Inference Setting → **Voxel size** ranges from **0.0015 to 0.0020**, defaults to **0.0018**, with step **0.0001**. This updates the renderer's actual voxel merge size for each request, and is recorded in `run.json` and `config.yaml`. Coarser merging usually reduces Gaussian counts and HTML/PLY size, with a tradeoff in fine detail. It does not reduce the encoder's input tokens or the raw point/depth/RGB output sizes. Precomputed example HTMLs retain their published settings; click Reconstruct scene to apply a new voxel size. | |
| Inference Setting → **max_scale_pct (%)** accepts a percentage greater than 0 and | |
| up to 100, default **99.5**. Gaussians above this scale percentile are removed before | |
| voxel merging: 99.5 trims approximately the largest 0.5%, while 100 keeps all scales | |
| under the percentile filter. Equal scale values at the cutoff can affect the exact | |
| fraction removed. The existing `filter_large_scale_infer` config flag must remain | |
| true for scale filtering (the supplied config enables it). | |
| **filter_flying_points_infer** is off by default. Enable it to filter pixels near | |
| relative depth discontinuities before point-cloud export and Gaussian voxel merging. | |
| Both settings apply to RGB-only, pose-conditioned, video-frame, and panorama | |
| reconstruction, and are recorded in `run.json` and the exported `config.yaml`. | |
| Depth and RGB exports remain unchanged. Changing these settings takes effect on | |
| the next reconstruction; precomputed example viewers retain their published filters. | |