Simam3D / README.md
junaid-simamdigital's picture
Document duplicate-view diagnostics
0d78d8d verified
|
Raw History Blame Contribute Delete
16 kB
---
title: Simam3D
emoji: 🧭
colorFrom: indigo
colorTo: blue
sdk: docker
app_port: 7860
suggested_hardware: a10g-large
pinned: true
tags:
- image-to-3d
- depth-estimation
- gaussian-splatting
- multi-view-reconstruction
models:
- depth-anything/Depth-Anything-V2-Small-hf
- depth-anything/DA3NESTED-GIANT-LARGE-1.1
---
# Simam3D
> **Live demo:** the verified public Gradio deployment is
> [Simam3D-GPU](https://huggingface.co/spaces/junaid-simamdigital/Simam3D-GPU).
> This Docker Space is a reproducible metered-GPU build and may remain paused
> when hardware is not allocated.
Simam3D 0.4.8 is a modular image-to-3D workbench with a tested depth baseline,
camera-aware DA3 inference, evidence-aware multi-view fusion, direct Gaussian
parameter export when available, uncertainty/reveal diagnostics, and
machine-validated output manifests. It does not claim metric-scale geometry,
trained Gaussian optimization, or verified hidden-side completion.
This first version intentionally keeps the inference path small and observable
so every stage can be tested independently on Hugging Face Spaces. Each run
produces inspectable outputs: colorized depth and confidence images, a textured
GLB scaffold, a confidence-preserving per-view PLY point cloud, a confidence-weighted fused
PLY, a viewer-compatible Gaussian-splat PLY, and an NPZ package containing
depth, confidence, extrinsics, and intrinsics arrays.
The per-view and fused PLY files retain `confidence`; fused points also retain
the dominant contributing `source_view` identifier for downstream inspection
and evaluation.
Fused and Gaussian PLY files also retain a `source_mask` bitmask showing which
input views contributed to each voxel, alongside the dominant pointer and the
derived Gaussian opacity/scale initialization.
It also emits a shareable evidence-card PNG that places the source, depth, and
confidence views beside the measured fusion statistics.
The **What's Behind the Image?** panel lets a user record a scene hypothesis,
extracts semantic cue labels, estimates image-space illumination direction, and
exports a reveal card pairing the original with the depth result. The reveal is
explicitly a hypothesis: a single image cannot verify unseen geometry. Additional
views and the manifest provide the evidence trail for future completion models.
The analysis also reports a depth-normal photometric light vector; this is a
heuristic lighting cue and is not a claim of physically correct illumination.
Each run also exports a reveal-uncertainty map and a deterministic next-view plan;
these prioritize acquisition around low-confidence or discontinuous regions.
It also exports a lightweight 12-frame turntable GIF rendered from the fused
points, making the reconstruction immediately inspectable and shareable.
The run report projects fused points back into each input camera and records
in-frame coverage as a diagnostic for pose or fusion collapse; coverage is a
consistency signal, not a geometric accuracy score. It also reports relative
depth reprojection error, comparing the fused raster with each input depth map.
For troubleshooting, the UI includes an explicitly labeled deterministic export
smoke test that bypasses model inference and exercises the complete artifact
pipeline on CPU. Its public handler also bypasses the `spaces.GPU` quota
boundary, so it remains useful when ZeroGPU inference credits are exhausted.
The reproducibility manifest includes the app/schema version, an explicit
provenance claims contract, software
versions, SHA-256 fingerprints for every input and exported artifact, and
normalized dimensions for every input view.
Downloaded runs can be checked with `python validate_manifest.py
path/to/*_run.json`; the validator reports missing or tampered artifacts and
also verifies structured failure reports.
## Reproducible evaluation
For paired depth arrays, run `python evaluate_depth.py prediction.npy
ground_truth.npy`; for a JSONL dataset, use `python evaluate_depth_dataset.py
pairs.jsonl`. For exported fused or Gaussian geometry, run `python
evaluate_pointcloud.py exported.ply reference.ply --threshold 0.05`, or use
`evaluate_pointcloud_dataset.py` for a JSONL set. Before a calibrated
multi-view experiment, validate `capture.json` with `python
validate_capture_manifest.py capture.json --check-files`. The schema and a
copyable example are in [CAPTURE_MANIFEST_SCHEMA.md](CAPTURE_MANIFEST_SCHEMA.md).
Run `python assess_capture.py capture.json --output capture_quality.json` to
flag dark, clipped, low-contrast, or likely-blurred source views before
inference; this is a media-readiness diagnostic, not a quality benchmark.
It also reports exact and near-duplicate view pairs so repeated frames are not
mistaken for useful multi-view coverage.
For a run-level cross-view consistency report, use
`python evaluate_reprojection.py fused.ply metadata.npz --output
reprojection.json`; it reads exported depth and camera arrays and reports
coverage plus relative-depth reprojection error.
Inspect a generated Gaussian file with `python inspect_gaussian_ply.py
gaussians.ply` to verify 3DGS parameters and provenance fields.
Convert a validated Gaussian PLY for common binary splat viewers with
`python gaussian_ply_to_splat.py gaussians.ply scene.splat`; the converter also
writes a JSON sidecar documenting the format and source evidence fields.
Validate the resulting binary with `python validate_splat.py scene.splat`
before handing it to a renderer.
For COLMAP text outputs, use `python import_colmap.py cameras.txt images.txt
image_root capture.json --held-out IMAGE_ID ...` to create the manifest first.
See [REAL_CAPTURE_GUIDE.md](REAL_CAPTURE_GUIDE.md) for the complete COLMAP
workflow from image folder to held-out evaluation manifest.
Use `python prepare_inference_manifest.py capture.json inference.json` to
derive a train-only input manifest while recording excluded held-out views.
For a no-network dry run, `python make_synthetic_capture.py synthetic_capture`
creates a labeled calibrated plane fixture with a held-out view and reference
PLY.
Run `python run_synthetic_evaluation.py synthetic_capture` to validate the
entire fixture, calibration, depth, and geometry-evaluation chain.
These tools retain raw versus median-scaled depth results, declared geometric
thresholds, held-out splits, and failed samples; they do not turn synthetic
fixtures or synthetic camera poses into quality claims.
Multiple manifests can be compared with `python summarize_runs.py output`,
which emits backend, view-count, fusion, coverage, reprojection, fallback, and
provenance fields as JSON. Failed manifests are included with their status,
failing stage, and structured error details.
Use `python summarize_runs.py output --format markdown` for a compact comparison
table suitable for an experiment note or LinkedIn technical article.
The dependency-free regression harness can be run with
`python benchmark_pipeline.py --output benchmark_output`. It repeats the CPU
fusion fixture at several resolutions and records signatures plus fusion
metrics. This proves deterministic geometry/export behavior only; it is not a
learned reconstruction-quality benchmark.
On Windows, run `powershell -ExecutionPolicy Bypass -File run_checks.ps1` before
publishing a change; this is the canonical compile and regression gate.
The DA3 package-layout resolver is tested independently in `da3_compat.py`, so
optional upstream import changes can be diagnosed without loading model
weights.
The model-free geometry path can also be exercised with
`python demo_pipeline.py --output demo_output`; it writes deterministic fused
and Gaussian artifacts for CPU-only integration testing.
The demo also emits a schema-3 manifest, so it can be checked with the same
`validate_manifest.py` tool as a model-backed run.
See [TROUBLESHOOTING.md](TROUBLESHOOTING.md) for the scheduler, smoke-test, and
browser-versus-server failure decision tree.
The research direction and required ablations are outlined in
[PAPER_OUTLINE.md](PAPER_OUTLINE.md).
The verified milestone history is recorded in [CHANGELOG.md](CHANGELOG.md).
The current claim boundary and evidence matrix are in
[MILESTONE_REPORT.md](MILESTONE_REPORT.md).
Fusion metrics also report the number of distinct dominant source views in the
fused result, making view collapse visible without overstating provenance.
They report the fraction of voxels supported by multiple views as a separate
cross-view consistency signal, plus the number of source views with any
support in the fused result.
It also emits a dependency-free Gaussian-splat PLY initialized from the fused
points. Splat opacity and initial scale are confidence-weighted, allowing the
representation to expose uncertainty before any learned Gaussian optimization.
Each run includes a JSON manifest recording the backend, model, runtime,
number of views, elapsed time, and generated artifact paths.
The UI also accepts optional additional images so DA3 can process a small
multi-view batch in one request; fusion and novel-view synthesis remain
separate, explicitly tracked stages.
When the V2 fallback is selected, additional images use a deterministic
synthetic orbit pose prior and the run manifest records that limitation; only
DA3 camera metadata is treated as camera-aware geometry.
Hosted requests are bounded to eight views and a 1536-pixel maximum input side
by default, with `SIMAM3D_MAX_VIEWS` and `SIMAM3D_MAX_INPUT_SIDE` available for
controlled deployments. Inference runs under `torch.inference_mode()` to
avoid retaining training graphs. The heavy inference event is limited to one
concurrent request and guarded by a process lock, reducing avoidable V2/DA3
memory contention.
Fusion resolution is configurable with `SIMAM3D_FUSION_VOXEL_SIZE` (default
`0.02`) and `SIMAM3D_FUSION_MAX_POINTS` (default `150000`); both values are
recorded in the run manifest.
The provenance bitmask supports up to 30 views because it is exported as a
signed 32-bit integer.
The geometry layer is isolated in `simam3d_core.py`, a dependency-free NumPy
module with deterministic voxel fusion and confidence metrics. Its smoke tests
can run without CUDA or model weights:
```bash
python -c "import runpy; ns=runpy.run_path('tests/test_core.py'); [fn() for name,fn in ns.items() if name.startswith('test_')]"
```
Relative depth is normalized using a consistent inverse-depth convention for
the current scaffold, so exported geometry is useful for visualization but is
not yet metrically scaled. DA3 camera calibration and
multi-view pose quality should be evaluated before treating measurements as
physical dimensions.
## Planned backends
- **Depth Anything 3**: any-view depth, camera pose, confidence, and direct Gaussian estimation.
- **TRELLIS.2**: primary high-quality mesh and structured-latent backend.
- **Hunyuan3D 2.1**: alternative shape and PBR-texture backend.
- **gsplat**: Gaussian initialization, confidence-weighted fusion, and rendering.
Extend3D remains a research reference only. Simam3D will use adapters around
these maintained projects so each backend can be tested independently and
replaced without rewriting the UI or export layer.
## Companion prototypes
- [SimamTwin3D](https://huggingface.co/spaces/junaid-simamdigital/SimamTwin3D):
consent-first multi-view human-reference capture audit. It records hashes and
capture coverage; it does not claim a perfect digital twin.
- [SimamAnything2GS](https://huggingface.co/spaces/junaid-simamdigital/SimamAnything2GS):
arbitrary image/video ingest with bounded FFmpeg sampling and a CPU 2.5D
preview. It is an ingest smoke test, not learned 3D reconstruction.
These prototypes are intentionally separate so experimental identity/video
workflows and general Gaussian reconstruction do not weaken Simam3D's evidence
and provenance contract.
The falsifiable experiment queue is documented in
[RESEARCH_BACKLOG.md](RESEARCH_BACKLOG.md).
The acceptance criteria for depth, fusion, Gaussian, and hidden-side claims
are documented in [EVALUATION_PROTOCOL.md](EVALUATION_PROTOCOL.md).
## SimamDigital publishing
The live demo currently belongs to the `junaid-simamdigital` user namespace.
For prominent company branding, keep this Space linked from the
[SimamDigital organization](https://huggingface.co/SimamDigital), and later
move it to `SimamDigital/Simam3D` once the organization profile and collection
are ready. The `pinned` flag keeps the Space near the top of its owning profile.
## Hosting notes
This repository is currently packaged as a Docker Space, so its T4/L4/A10G
hardware choices are metered Space hardware. Hugging Face ZeroGPU is a
separate Gradio-SDK-only hosting mode; selecting `zero-a10g` on this Docker
Space is not a drop-in replacement. A future ZeroGPU variant would need a
Gradio-SDK packaging pass, the `spaces.GPU` function boundary, and the
ZeroGPU-supported Torch/Python matrix. The Docker version remains the
reproducible target for calibrated DA3 experiments.
The app already includes an optional no-op-safe `spaces.GPU` boundary; it does
not convert this Docker Space into a ZeroGPU Space by itself.
See [ZERO_GPU_MIGRATION.md](ZERO_GPU_MIGRATION.md) for the separate-variant
steps and acceptance checks.
## Roadmap
1. Depth baseline, GLB, PLY, camera metadata, confidence fusion, and Gaussian initialization (implemented).
2. Validate DA3NESTED-GIANT-LARGE-1.1 camera-aware and direct-Gaussian outputs against the V2 fallback (verified on the public GPU Space for one-, two-, and three-view requests).
3. Add four-view completion with source-view preservation and confidence-weighted fusion.
4. Add TRELLIS.2 and Hunyuan3D backend selection.
5. Add gsplat optimization and interactive splat preview.
The UI defaults to the V2 stable baseline for a first successful smoke test;
DA3 remains available as the explicit camera-aware research backend. The
public GPU Space installs the pinned DA3 dependency and keeps the explicit V2
fallback. The Docker image still treats DA3 as optional: `requirements-da3.txt`
can be layered into a research image when the upstream API is known to be
compatible. This keeps the baseline Docker build independent of the
experimental Git dependency.
The Dockerfile exposes the same opt-in as
`--build-arg SIMAM3D_INSTALL_DA3=1` for reproducible GPU research builds.
The optional DA3 source is pinned to a specific upstream commit rather than
tracking a moving branch.
The container also includes a health check for the Gradio HTTP process, so a
reachable UI can be separated from later model-inference failures.
Gradio is launched with detailed error display enabled, while the manifest and
container logs remain the authoritative diagnostic record.
Failed requests also leave a redacted failure manifest with the failing stage,
backend, runtime, software versions, input hashes, and exception details for
reproducible diagnosis.
The Diagnostics panel can list those reports for download after a failed run.
The Diagnostics panel checks the optional DA3 package and the expected
`DepthAnything3` symbol across both known upstream import locations, so an
installed but incompatible DA3 package is visible before model inference.
When DA3 is selected, its depth, confidence, extrinsics, and intrinsics shapes
are validated at the adapter boundary; malformed results trigger the labeled V2
fallback before any geometry export begins, and the exact fallback exception is
stored in the run manifest for diagnosis.
Backend switching also releases the previously loaded heavyweight model before
loading the next one, reducing avoidable GPU memory pressure in long-lived
Space processes.
Additional views accept both filepath strings and Gradio `FileData` uploads;
unreadable files now produce a direct input error before inference starts.
The baseline requirements also pin the CUDA-compatible Torch pair and bound
dependency major versions so rebuilds do not silently change the inference or
export contract.