Pi4 / README.md
yyfz233's picture
Reuse input video for RGB previews and remove redundant viewer chrome
76d2cc7
|
Raw History Blame Contribute Delete
10.9 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade
metadata
title: 'π⁴: A Foundation Model for 4D Scene Geometry and Motion'
emoji: 🌌
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 6.31.0
python_version: '3.12'
app_file: app.py
pinned: false
license: cc-by-nc-4.0
short_description: Geometry and motion from RGB images and videos
models:
  - yyfz233/Pi4

π⁴: A Foundation Model for 4D Scene Geometry and Motion

Physical Intelligence Team · Shanghai AI Laboratory

Upload a video or a collection of images to reconstruct geometry and dense motion without supplied camera calibration. The demo reuses Pi4's native interactive viewer for point clouds, camera poses, and dynamic tracks.

Using the Demo

The outer interface reuses the project website's navigation and footer, with the same transparent Shanghai AI Laboratory logo, Physical Intelligence Team lockup, pill buttons, SVG icons, and local Caveat handwritten callouts. The website's Hugging Face button becomes Project Page, annotated see more demos!; no Model weights button is shown. GitHub and GameRealm retain their website links. The technical report stays disabled until its URL is available. The extra Explore demos button is omitted. The footer retains the website branding without an extra button. The full-width white scene viewer sits directly below the navigation. The single-line project title appears inside the empty viewer, above "Your scene, in another dimension." The viewer, input, and examples omit the numbered heading bars to leave more room for content. The input card and example cards sit below the viewer, not beside it. Video and image uploads share one input area. Example clips have real video covers and one-click loading; selecting one does not start inference automatically. Sampling settings are expanded by default; result downloads remain collapsed. The page stays light even when the surrounding Hugging Face UI uses dark mode.

The responsive example gallery includes Figure Skating, Kart, Cycling, Modern Home, Waterfall Surfing, and Rooftop Pursuit. The three additions use the project page's compressed RGB inputs, not rendered reconstructions. Each card restores its sampling preset, enables tracking, and caps inference at 150 sampled frames. The gallery uses three columns on desktop, two on tablets, and one on small phones.

  1. Choose Video or Images, then upload media or select a bundled example.
  2. The defaults match the repository: stride 1, up to 150 sampled frames, and a fixed 512-pixel maximum image side. Figure Skating uses frames 0–149, covering about 12 seconds. Change the stride to cover more of a video, or lower the frame limit to reduce GPU memory use. Images are ordered by filename. Figure Skating uses figure_skating.mp4 at stride 1. Kart · driving and Cycling · seaside select stride 2, up to 150 frames; Kart covers its full ~20-second clip, while Cycling covers its first ~10 seconds. The original three videos match the repository's compressed inputs: 512×288 for Figure Skating/Kart and 288×512 for Cycling, with all source frames and timing retained. The additional website clips use stride 2 for Rooftop Pursuit and 5 for Modern Home and Waterfall Surfing. See examples/README.md for each clip's sampled frame count and coverage; longer clips are not necessarily processed in full.
  3. Enable tracking for geometry and motion, or disable it for geometry only. A single image requires geometry-only mode. The tracking query index refers to the sampled frames, not the original video.
  4. Click Reconstruct & Track, then explore the embedded viewer or open it in a new tab. Download the preview PLY or the complete scene ZIP.

The viewer follows the project website's light, purple-accented design, with a synchronized RGB inset, filmstrip timeline, playback speed, camera frustums, fullscreen mode, and collapsible geometry/trajectory filters. Click or drag the filmstrip, or click the RGB inset, to smoothly return to the selected predicted camera. Scrubbing pauses playback while dragging and resumes it only if it was already playing. The render-budget slider uses random sampling without replacement, defaults to up to 500,000 exported points, and actually reduces the GPU draw count when lowered. The existing 40 MB geometry export budget limits the available points; small scenes contain fewer than two million. Visibility/confidence filters can reduce the number shown further. Display changes never alter inference or downloaded results. Raise the slider toward two million for more detail, or lower it for a lighter view. Only the default display budget changes; the export keeps its full budget.

New point-cloud exports and the preview PLY also sample randomly without replacement after their respective validity/confidence filters. A fixed seed keeps exports reproducible and the slider's random selection stable when the budget changes. Newly loaded geometry joins the random pool incrementally, without reshuffling the entire scene on each frame. Image-frame sampling and tracking queries are unchanged.

Trajectory sampling is independent of the point-cloud budget. The export keeps up to 1,024 finite query tracks, using evenly spaced candidates from the query frame's eligible pixels. In viewer exports with motion masks, eligibility uses the separate 5×5-eroded motion mask at the query and target frames. The browser then uses a fixed random ranking to show up to 256 whole trajectories by default, adjustable with Track count. The same query IDs are used across all frames, and raising the count retains the previous subset. The default trail duration is 1.5 seconds; missing or low-confidence observations can reduce the visible heads and segments without resampling the trajectory IDs.

Dynamic-region hiding uses the native motion probability threshold 0.5, dilated outward by 5 pixels with an 11×11 square kernel at the resized image resolution. Current motion + background applies this expanded mask only to noncurrent frames; Static background only applies it to all frames. Current motion only still uses the current frame's native, undilated mask. This display-only margin does not change model predictions or the separate 5×5 erosion used to filter tracks. Legacy exports retain their original masks; run reconstruction again to include the new margin in the scene export.

The viewer prefetches 32 future frames with up to three concurrent bulk downloads (including tracks), instead of waiting at each eight-frame boundary. Tracks load in the background and their controls become available when ready; they do not block the initial scene or playback. RGB previews are decoded ahead of playback when image fallback is needed, and point chunks are applied in small cooperative batches to keep the page responsive. These changes do not lower the point budget or change inference. A connection slower than the scene's required data rate can still need buffering.

The RGB inset reuses the matching input video's URL and sampled source timestamps when embedded in the Space, without seeking or changing the input player itself. It draws onto a persistent canvas rather than swapping image elements. Image uploads, standalone viewers, changed inputs, and video errors use the exported RGB images instead. These images share requests with the filmstrip, retain downloaded bytes for revisits, and cap the image cache at 32 decoded frames. Temporary failures retry up to three times; failures never replace a valid preview with a broken image. The redundant viewer branding/session bar is removed so the scene uses the iframe height.

Trajectory downloads use a separate two-minute timeout and retry temporary network failures up to three attempts. The layer panel shows loading/error status and a Retry tracks button if needed; it reloads the saved trajectory file without rerunning inference or changing the point budget. Missing, unauthorized, or malformed files are reported rather than repeatedly retried.

All sampled frames are processed together in one forward pass. The demo does not silently reduce resolution, split the input into temporal windows, or retry after an out-of-memory error. Reduce the frame limit if necessary. The displayed forward time excludes GPU allocation, model transfer, and export.

Extract the scene ZIP and use the source repository's viewer to open it locally:

python viewer/serve.py /path/to/extracted-scene --port 8080

Runtime

Select ZeroGPU in this Space's hardware settings. The Space uses the Gradio SDK; only the CUDA forward pass, model transfer, and copying predictions back to CPU run inside @spaces.GPU(duration=120). Frame decoding, CPU weight loading, and scene export do not reserve a GPU. GPU calls are serialized and queued. Access remains subject to Hugging Face's current quota and organization rules.

The model is downloaded at runtime and cached on CPU between requests. The first run may take longer while weights are downloaded and loaded. The dependencies follow the source repository's PyTorch 2.12.1 / Triton 3.7.1 pins and pinned FlexGEMM revision. The Space uses the default PyTorch wheel, not the repository's optional CUDA 12.6 requirements; the selected ZeroGPU hardware must support its CUDA build and sparse kernels. An actual ZeroGPU forward pass is required to validate the hosted runtime.

For private model weights, add HF_TOKEN as a Secret in the Space settings. Use a fine-grained token with read access to yyfz233/Pi4 (or the repository selected by PI4_MODEL_ID). The runtime downloader reads this secret automatically; do not commit the token or put it in a public Variable. preload_from_hub is intentionally omitted because it does not support private repositories. Model downloads happen before GPU allocation.

Optional Space variables:

Variable Default Meaning
PI4_MODEL_ID yyfz233/Pi4 Model repository or local checkpoint
PI4_MODEL_REVISION unset Optional checkpoint revision
PI4_OUTPUT_DIR outputs/space Temporary inference results

Outputs older than six hours are removed when the next request starts; Gradio also periodically expires its upload/download cache. Files disappear when the Space restarts. Media and results are not private storage: anyone with a result URL may access it until it expires. Do not upload sensitive material.

License

Original Pi4 code and model weights are licensed under CC BY-NC 4.0. Third-party materials retain their own terms, including applicable CoWTracker-derived noncommercial research restrictions. See LICENSE, LICENSE_MODEL, NOTICE, and licenses/ for the full terms.

Source: https://github.com/yyfz/Pi4

Model: https://huggingface.co/yyfz233/Pi4