README.md

metadata

title: Simam3D
emoji: 🧭
colorFrom: indigo
colorTo: blue
sdk: docker
app_port: 7860
suggested_hardware: a10g-large
pinned: true
tags:
  - image-to-3d
  - depth-estimation
  - gaussian-splatting
  - multi-view-reconstruction
models:
  - depth-anything/Depth-Anything-V2-Small-hf
  - depth-anything/DA3NESTED-GIANT-LARGE-1.1

Simam3D

Live demo: the verified public Gradio deployment is Simam3D-GPU. This Docker Space is a reproducible metered-GPU build and may remain paused when hardware is not allocated.

Simam3D 0.4.8 is a modular image-to-3D workbench with a tested depth baseline, camera-aware DA3 inference, evidence-aware multi-view fusion, direct Gaussian parameter export when available, uncertainty/reveal diagnostics, and machine-validated output manifests. It does not claim metric-scale geometry, trained Gaussian optimization, or verified hidden-side completion.

This first version intentionally keeps the inference path small and observable so every stage can be tested independently on Hugging Face Spaces. Each run produces inspectable outputs: colorized depth and confidence images, a textured GLB scaffold, a confidence-preserving per-view PLY point cloud, a confidence-weighted fused PLY, a viewer-compatible Gaussian-splat PLY, and an NPZ package containing depth, confidence, extrinsics, and intrinsics arrays. The per-view and fused PLY files retain confidence; fused points also retain the dominant contributing source_view identifier for downstream inspection and evaluation. Fused and Gaussian PLY files also retain a source_mask bitmask showing which input views contributed to each voxel, alongside the dominant pointer and the derived Gaussian opacity/scale initialization. It also emits a shareable evidence-card PNG that places the source, depth, and confidence views beside the measured fusion statistics. The What's Behind the Image? panel lets a user record a scene hypothesis, extracts semantic cue labels, estimates image-space illumination direction, and exports a reveal card pairing the original with the depth result. The reveal is explicitly a hypothesis: a single image cannot verify unseen geometry. Additional views and the manifest provide the evidence trail for future completion models. The analysis also reports a depth-normal photometric light vector; this is a heuristic lighting cue and is not a claim of physically correct illumination. Each run also exports a reveal-uncertainty map and a deterministic next-view plan; these prioritize acquisition around low-confidence or discontinuous regions. It also exports a lightweight 12-frame turntable GIF rendered from the fused points, making the reconstruction immediately inspectable and shareable. The run report projects fused points back into each input camera and records in-frame coverage as a diagnostic for pose or fusion collapse; coverage is a consistency signal, not a geometric accuracy score. It also reports relative depth reprojection error, comparing the fused raster with each input depth map. For troubleshooting, the UI includes an explicitly labeled deterministic export smoke test that bypasses model inference and exercises the complete artifact pipeline on CPU. Its public handler also bypasses the spaces.GPU quota boundary, so it remains useful when ZeroGPU inference credits are exhausted. The reproducibility manifest includes the app/schema version, an explicit provenance claims contract, software versions, SHA-256 fingerprints for every input and exported artifact, and normalized dimensions for every input view. Downloaded runs can be checked with python validate_manifest.py path/to/*_run.json; the validator reports missing or tampered artifacts and also verifies structured failure reports.

Reproducible evaluation

For paired depth arrays, run python evaluate_depth.py prediction.npy ground_truth.npy; for a JSONL dataset, use python evaluate_depth_dataset.py pairs.jsonl. For exported fused or Gaussian geometry, run python evaluate_pointcloud.py exported.ply reference.ply --threshold 0.05, or use evaluate_pointcloud_dataset.py for a JSONL set. Before a calibrated multi-view experiment, validate capture.json with python validate_capture_manifest.py capture.json --check-files. The schema and a copyable example are in CAPTURE_MANIFEST_SCHEMA.md. Run python assess_capture.py capture.json --output capture_quality.json to flag dark, clipped, low-contrast, or likely-blurred source views before inference; this is a media-readiness diagnostic, not a quality benchmark. It also reports exact and near-duplicate view pairs so repeated frames are not mistaken for useful multi-view coverage. For a run-level cross-view consistency report, use python evaluate_reprojection.py fused.ply metadata.npz --output reprojection.json; it reads exported depth and camera arrays and reports coverage plus relative-depth reprojection error. Inspect a generated Gaussian file with python inspect_gaussian_ply.py gaussians.ply to verify 3DGS parameters and provenance fields. Convert a validated Gaussian PLY for common binary splat viewers with python gaussian_ply_to_splat.py gaussians.ply scene.splat; the converter also writes a JSON sidecar documenting the format and source evidence fields. Validate the resulting binary with python validate_splat.py scene.splat before handing it to a renderer. For COLMAP text outputs, use python import_colmap.py cameras.txt images.txt image_root capture.json --held-out IMAGE_ID ... to create the manifest first. See REAL_CAPTURE_GUIDE.md for the complete COLMAP workflow from image folder to held-out evaluation manifest. Use python prepare_inference_manifest.py capture.json inference.json to derive a train-only input manifest while recording excluded held-out views. For a no-network dry run, python make_synthetic_capture.py synthetic_capture creates a labeled calibrated plane fixture with a held-out view and reference PLY. Run python run_synthetic_evaluation.py synthetic_capture to validate the entire fixture, calibration, depth, and geometry-evaluation chain. These tools retain raw versus median-scaled depth results, declared geometric thresholds, held-out splits, and failed samples; they do not turn synthetic fixtures or synthetic camera poses into quality claims.

Multiple manifests can be compared with python summarize_runs.py output, which emits backend, view-count, fusion, coverage, reprojection, fallback, and provenance fields as JSON. Failed manifests are included with their status, failing stage, and structured error details. Use python summarize_runs.py output --format markdown for a compact comparison table suitable for an experiment note or LinkedIn technical article. The dependency-free regression harness can be run with python benchmark_pipeline.py --output benchmark_output. It repeats the CPU fusion fixture at several resolutions and records signatures plus fusion metrics. This proves deterministic geometry/export behavior only; it is not a learned reconstruction-quality benchmark. On Windows, run powershell -ExecutionPolicy Bypass -File run_checks.ps1 before publishing a change; this is the canonical compile and regression gate. The DA3 package-layout resolver is tested independently in da3_compat.py, so optional upstream import changes can be diagnosed without loading model weights. The model-free geometry path can also be exercised with python demo_pipeline.py --output demo_output; it writes deterministic fused and Gaussian artifacts for CPU-only integration testing. The demo also emits a schema-3 manifest, so it can be checked with the same validate_manifest.py tool as a model-backed run. See TROUBLESHOOTING.md for the scheduler, smoke-test, and browser-versus-server failure decision tree. The research direction and required ablations are outlined in PAPER_OUTLINE.md. The verified milestone history is recorded in CHANGELOG.md. The current claim boundary and evidence matrix are in MILESTONE_REPORT.md. Fusion metrics also report the number of distinct dominant source views in the fused result, making view collapse visible without overstating provenance. They report the fraction of voxels supported by multiple views as a separate cross-view consistency signal, plus the number of source views with any support in the fused result. It also emits a dependency-free Gaussian-splat PLY initialized from the fused points. Splat opacity and initial scale are confidence-weighted, allowing the representation to expose uncertainty before any learned Gaussian optimization. Each run includes a JSON manifest recording the backend, model, runtime, number of views, elapsed time, and generated artifact paths. The UI also accepts optional additional images so DA3 can process a small multi-view batch in one request; fusion and novel-view synthesis remain separate, explicitly tracked stages. When the V2 fallback is selected, additional images use a deterministic synthetic orbit pose prior and the run manifest records that limitation; only DA3 camera metadata is treated as camera-aware geometry. Hosted requests are bounded to eight views and a 1536-pixel maximum input side by default, with SIMAM3D_MAX_VIEWS and SIMAM3D_MAX_INPUT_SIDE available for controlled deployments. Inference runs under torch.inference_mode() to avoid retaining training graphs. The heavy inference event is limited to one concurrent request and guarded by a process lock, reducing avoidable V2/DA3 memory contention. Fusion resolution is configurable with SIMAM3D_FUSION_VOXEL_SIZE (default 0.02) and SIMAM3D_FUSION_MAX_POINTS (default 150000); both values are recorded in the run manifest. The provenance bitmask supports up to 30 views because it is exported as a signed 32-bit integer.

The geometry layer is isolated in simam3d_core.py, a dependency-free NumPy module with deterministic voxel fusion and confidence metrics. Its smoke tests can run without CUDA or model weights:

python -c "import runpy; ns=runpy.run_path('tests/test_core.py'); [fn() for name,fn in ns.items() if name.startswith('test_')]"

Relative depth is normalized using a consistent inverse-depth convention for the current scaffold, so exported geometry is useful for visualization but is not yet metrically scaled. DA3 camera calibration and multi-view pose quality should be evaluated before treating measurements as physical dimensions.

Planned backends

• Depth Anything 3: any-view depth, camera pose, confidence, and direct Gaussian estimation.
• TRELLIS.2: primary high-quality mesh and structured-latent backend.
• Hunyuan3D 2.1: alternative shape and PBR-texture backend.
• gsplat: Gaussian initialization, confidence-weighted fusion, and rendering.

Extend3D remains a research reference only. Simam3D will use adapters around these maintained projects so each backend can be tested independently and replaced without rewriting the UI or export layer.

Companion prototypes

• SimamTwin3D: consent-first multi-view human-reference capture audit. It records hashes and capture coverage; it does not claim a perfect digital twin.
• SimamAnything2GS: arbitrary image/video ingest with bounded FFmpeg sampling and a CPU 2.5D preview. It is an ingest smoke test, not learned 3D reconstruction.

These prototypes are intentionally separate so experimental identity/video workflows and general Gaussian reconstruction do not weaken Simam3D's evidence and provenance contract. The falsifiable experiment queue is documented in RESEARCH_BACKLOG.md. The acceptance criteria for depth, fusion, Gaussian, and hidden-side claims are documented in EVALUATION_PROTOCOL.md.

SimamDigital publishing

The live demo currently belongs to the junaid-simamdigital user namespace. For prominent company branding, keep this Space linked from the SimamDigital organization, and later move it to SimamDigital/Simam3D once the organization profile and collection are ready. The pinned flag keeps the Space near the top of its owning profile.

Hosting notes

This repository is currently packaged as a Docker Space, so its T4/L4/A10G hardware choices are metered Space hardware. Hugging Face ZeroGPU is a separate Gradio-SDK-only hosting mode; selecting zero-a10g on this Docker Space is not a drop-in replacement. A future ZeroGPU variant would need a Gradio-SDK packaging pass, the spaces.GPU function boundary, and the ZeroGPU-supported Torch/Python matrix. The Docker version remains the reproducible target for calibrated DA3 experiments. The app already includes an optional no-op-safe spaces.GPU boundary; it does not convert this Docker Space into a ZeroGPU Space by itself. See ZERO_GPU_MIGRATION.md for the separate-variant steps and acceptance checks.

Roadmap

1. Depth baseline, GLB, PLY, camera metadata, confidence fusion, and Gaussian initialization (implemented).
2. Validate DA3NESTED-GIANT-LARGE-1.1 camera-aware and direct-Gaussian outputs against the V2 fallback (verified on the public GPU Space for one-, two-, and three-view requests).
3. Add four-view completion with source-view preservation and confidence-weighted fusion.
4. Add TRELLIS.2 and Hunyuan3D backend selection.
5. Add gsplat optimization and interactive splat preview.

The UI defaults to the V2 stable baseline for a first successful smoke test; DA3 remains available as the explicit camera-aware research backend. The public GPU Space installs the pinned DA3 dependency and keeps the explicit V2 fallback. The Docker image still treats DA3 as optional: requirements-da3.txt can be layered into a research image when the upstream API is known to be compatible. This keeps the baseline Docker build independent of the experimental Git dependency. The Dockerfile exposes the same opt-in as --build-arg SIMAM3D_INSTALL_DA3=1 for reproducible GPU research builds. The optional DA3 source is pinned to a specific upstream commit rather than tracking a moving branch. The container also includes a health check for the Gradio HTTP process, so a reachable UI can be separated from later model-inference failures. Gradio is launched with detailed error display enabled, while the manifest and container logs remain the authoritative diagnostic record. Failed requests also leave a redacted failure manifest with the failing stage, backend, runtime, software versions, input hashes, and exception details for reproducible diagnosis. The Diagnostics panel can list those reports for download after a failed run. The Diagnostics panel checks the optional DA3 package and the expected DepthAnything3 symbol across both known upstream import locations, so an installed but incompatible DA3 package is visible before model inference. When DA3 is selected, its depth, confidence, extrinsics, and intrinsics shapes are validated at the adapter boundary; malformed results trigger the labeled V2 fallback before any geometry export begins, and the exact fallback exception is stored in the run manifest for diagnosis. Backend switching also releases the previously loaded heavyweight model before loading the next one, reducing avoidable GPU memory pressure in long-lived Space processes. Additional views accept both filepath strings and Gradio FileData uploads; unreadable files now produce a direct input error before inference starts. The baseline requirements also pin the CUDA-compatible Torch pair and bound dependency major versions so rebuilds do not silently change the inference or export contract.