Spaces:
Paused
Paused
|
Download README.md from junaid-simamdigital/Simam3D: direct link, hf CLI and curl.
- Browser
- Download file 16 kB
-
https://huggingface.co/spaces/junaid-simamdigital/Simam3D/resolve/main/README.md
- Command line
-
hf download hf://spaces/junaid-simamdigital/Simam3D/README.md
-
curl -L -o README.md https://huggingface.co/spaces/junaid-simamdigital/Simam3D/resolve/main/README.md
16 kB
| title: Simam3D | |
| emoji: 🧭 | |
| colorFrom: indigo | |
| colorTo: blue | |
| sdk: docker | |
| app_port: 7860 | |
| suggested_hardware: a10g-large | |
| pinned: true | |
| tags: | |
| - image-to-3d | |
| - depth-estimation | |
| - gaussian-splatting | |
| - multi-view-reconstruction | |
| models: | |
| - depth-anything/Depth-Anything-V2-Small-hf | |
| - depth-anything/DA3NESTED-GIANT-LARGE-1.1 | |
| # Simam3D | |
| > **Live demo:** the verified public Gradio deployment is | |
| > [Simam3D-GPU](https://huggingface.co/spaces/junaid-simamdigital/Simam3D-GPU). | |
| > This Docker Space is a reproducible metered-GPU build and may remain paused | |
| > when hardware is not allocated. | |
| Simam3D 0.4.8 is a modular image-to-3D workbench with a tested depth baseline, | |
| camera-aware DA3 inference, evidence-aware multi-view fusion, direct Gaussian | |
| parameter export when available, uncertainty/reveal diagnostics, and | |
| machine-validated output manifests. It does not claim metric-scale geometry, | |
| trained Gaussian optimization, or verified hidden-side completion. | |
| This first version intentionally keeps the inference path small and observable | |
| so every stage can be tested independently on Hugging Face Spaces. Each run | |
| produces inspectable outputs: colorized depth and confidence images, a textured | |
| GLB scaffold, a confidence-preserving per-view PLY point cloud, a confidence-weighted fused | |
| PLY, a viewer-compatible Gaussian-splat PLY, and an NPZ package containing | |
| depth, confidence, extrinsics, and intrinsics arrays. | |
| The per-view and fused PLY files retain `confidence`; fused points also retain | |
| the dominant contributing `source_view` identifier for downstream inspection | |
| and evaluation. | |
| Fused and Gaussian PLY files also retain a `source_mask` bitmask showing which | |
| input views contributed to each voxel, alongside the dominant pointer and the | |
| derived Gaussian opacity/scale initialization. | |
| It also emits a shareable evidence-card PNG that places the source, depth, and | |
| confidence views beside the measured fusion statistics. | |
| The **What's Behind the Image?** panel lets a user record a scene hypothesis, | |
| extracts semantic cue labels, estimates image-space illumination direction, and | |
| exports a reveal card pairing the original with the depth result. The reveal is | |
| explicitly a hypothesis: a single image cannot verify unseen geometry. Additional | |
| views and the manifest provide the evidence trail for future completion models. | |
| The analysis also reports a depth-normal photometric light vector; this is a | |
| heuristic lighting cue and is not a claim of physically correct illumination. | |
| Each run also exports a reveal-uncertainty map and a deterministic next-view plan; | |
| these prioritize acquisition around low-confidence or discontinuous regions. | |
| It also exports a lightweight 12-frame turntable GIF rendered from the fused | |
| points, making the reconstruction immediately inspectable and shareable. | |
| The run report projects fused points back into each input camera and records | |
| in-frame coverage as a diagnostic for pose or fusion collapse; coverage is a | |
| consistency signal, not a geometric accuracy score. It also reports relative | |
| depth reprojection error, comparing the fused raster with each input depth map. | |
| For troubleshooting, the UI includes an explicitly labeled deterministic export | |
| smoke test that bypasses model inference and exercises the complete artifact | |
| pipeline on CPU. Its public handler also bypasses the `spaces.GPU` quota | |
| boundary, so it remains useful when ZeroGPU inference credits are exhausted. | |
| The reproducibility manifest includes the app/schema version, an explicit | |
| provenance claims contract, software | |
| versions, SHA-256 fingerprints for every input and exported artifact, and | |
| normalized dimensions for every input view. | |
| Downloaded runs can be checked with `python validate_manifest.py | |
| path/to/*_run.json`; the validator reports missing or tampered artifacts and | |
| also verifies structured failure reports. | |
| ## Reproducible evaluation | |
| For paired depth arrays, run `python evaluate_depth.py prediction.npy | |
| ground_truth.npy`; for a JSONL dataset, use `python evaluate_depth_dataset.py | |
| pairs.jsonl`. For exported fused or Gaussian geometry, run `python | |
| evaluate_pointcloud.py exported.ply reference.ply --threshold 0.05`, or use | |
| `evaluate_pointcloud_dataset.py` for a JSONL set. Before a calibrated | |
| multi-view experiment, validate `capture.json` with `python | |
| validate_capture_manifest.py capture.json --check-files`. The schema and a | |
| copyable example are in [CAPTURE_MANIFEST_SCHEMA.md](CAPTURE_MANIFEST_SCHEMA.md). | |
| Run `python assess_capture.py capture.json --output capture_quality.json` to | |
| flag dark, clipped, low-contrast, or likely-blurred source views before | |
| inference; this is a media-readiness diagnostic, not a quality benchmark. | |
| It also reports exact and near-duplicate view pairs so repeated frames are not | |
| mistaken for useful multi-view coverage. | |
| For a run-level cross-view consistency report, use | |
| `python evaluate_reprojection.py fused.ply metadata.npz --output | |
| reprojection.json`; it reads exported depth and camera arrays and reports | |
| coverage plus relative-depth reprojection error. | |
| Inspect a generated Gaussian file with `python inspect_gaussian_ply.py | |
| gaussians.ply` to verify 3DGS parameters and provenance fields. | |
| Convert a validated Gaussian PLY for common binary splat viewers with | |
| `python gaussian_ply_to_splat.py gaussians.ply scene.splat`; the converter also | |
| writes a JSON sidecar documenting the format and source evidence fields. | |
| Validate the resulting binary with `python validate_splat.py scene.splat` | |
| before handing it to a renderer. | |
| For COLMAP text outputs, use `python import_colmap.py cameras.txt images.txt | |
| image_root capture.json --held-out IMAGE_ID ...` to create the manifest first. | |
| See [REAL_CAPTURE_GUIDE.md](REAL_CAPTURE_GUIDE.md) for the complete COLMAP | |
| workflow from image folder to held-out evaluation manifest. | |
| Use `python prepare_inference_manifest.py capture.json inference.json` to | |
| derive a train-only input manifest while recording excluded held-out views. | |
| For a no-network dry run, `python make_synthetic_capture.py synthetic_capture` | |
| creates a labeled calibrated plane fixture with a held-out view and reference | |
| PLY. | |
| Run `python run_synthetic_evaluation.py synthetic_capture` to validate the | |
| entire fixture, calibration, depth, and geometry-evaluation chain. | |
| These tools retain raw versus median-scaled depth results, declared geometric | |
| thresholds, held-out splits, and failed samples; they do not turn synthetic | |
| fixtures or synthetic camera poses into quality claims. | |
| Multiple manifests can be compared with `python summarize_runs.py output`, | |
| which emits backend, view-count, fusion, coverage, reprojection, fallback, and | |
| provenance fields as JSON. Failed manifests are included with their status, | |
| failing stage, and structured error details. | |
| Use `python summarize_runs.py output --format markdown` for a compact comparison | |
| table suitable for an experiment note or LinkedIn technical article. | |
| The dependency-free regression harness can be run with | |
| `python benchmark_pipeline.py --output benchmark_output`. It repeats the CPU | |
| fusion fixture at several resolutions and records signatures plus fusion | |
| metrics. This proves deterministic geometry/export behavior only; it is not a | |
| learned reconstruction-quality benchmark. | |
| On Windows, run `powershell -ExecutionPolicy Bypass -File run_checks.ps1` before | |
| publishing a change; this is the canonical compile and regression gate. | |
| The DA3 package-layout resolver is tested independently in `da3_compat.py`, so | |
| optional upstream import changes can be diagnosed without loading model | |
| weights. | |
| The model-free geometry path can also be exercised with | |
| `python demo_pipeline.py --output demo_output`; it writes deterministic fused | |
| and Gaussian artifacts for CPU-only integration testing. | |
| The demo also emits a schema-3 manifest, so it can be checked with the same | |
| `validate_manifest.py` tool as a model-backed run. | |
| See [TROUBLESHOOTING.md](TROUBLESHOOTING.md) for the scheduler, smoke-test, and | |
| browser-versus-server failure decision tree. | |
| The research direction and required ablations are outlined in | |
| [PAPER_OUTLINE.md](PAPER_OUTLINE.md). | |
| The verified milestone history is recorded in [CHANGELOG.md](CHANGELOG.md). | |
| The current claim boundary and evidence matrix are in | |
| [MILESTONE_REPORT.md](MILESTONE_REPORT.md). | |
| Fusion metrics also report the number of distinct dominant source views in the | |
| fused result, making view collapse visible without overstating provenance. | |
| They report the fraction of voxels supported by multiple views as a separate | |
| cross-view consistency signal, plus the number of source views with any | |
| support in the fused result. | |
| It also emits a dependency-free Gaussian-splat PLY initialized from the fused | |
| points. Splat opacity and initial scale are confidence-weighted, allowing the | |
| representation to expose uncertainty before any learned Gaussian optimization. | |
| Each run includes a JSON manifest recording the backend, model, runtime, | |
| number of views, elapsed time, and generated artifact paths. | |
| The UI also accepts optional additional images so DA3 can process a small | |
| multi-view batch in one request; fusion and novel-view synthesis remain | |
| separate, explicitly tracked stages. | |
| When the V2 fallback is selected, additional images use a deterministic | |
| synthetic orbit pose prior and the run manifest records that limitation; only | |
| DA3 camera metadata is treated as camera-aware geometry. | |
| Hosted requests are bounded to eight views and a 1536-pixel maximum input side | |
| by default, with `SIMAM3D_MAX_VIEWS` and `SIMAM3D_MAX_INPUT_SIDE` available for | |
| controlled deployments. Inference runs under `torch.inference_mode()` to | |
| avoid retaining training graphs. The heavy inference event is limited to one | |
| concurrent request and guarded by a process lock, reducing avoidable V2/DA3 | |
| memory contention. | |
| Fusion resolution is configurable with `SIMAM3D_FUSION_VOXEL_SIZE` (default | |
| `0.02`) and `SIMAM3D_FUSION_MAX_POINTS` (default `150000`); both values are | |
| recorded in the run manifest. | |
| The provenance bitmask supports up to 30 views because it is exported as a | |
| signed 32-bit integer. | |
| The geometry layer is isolated in `simam3d_core.py`, a dependency-free NumPy | |
| module with deterministic voxel fusion and confidence metrics. Its smoke tests | |
| can run without CUDA or model weights: | |
| ```bash | |
| python -c "import runpy; ns=runpy.run_path('tests/test_core.py'); [fn() for name,fn in ns.items() if name.startswith('test_')]" | |
| ``` | |
| Relative depth is normalized using a consistent inverse-depth convention for | |
| the current scaffold, so exported geometry is useful for visualization but is | |
| not yet metrically scaled. DA3 camera calibration and | |
| multi-view pose quality should be evaluated before treating measurements as | |
| physical dimensions. | |
| ## Planned backends | |
| - **Depth Anything 3**: any-view depth, camera pose, confidence, and direct Gaussian estimation. | |
| - **TRELLIS.2**: primary high-quality mesh and structured-latent backend. | |
| - **Hunyuan3D 2.1**: alternative shape and PBR-texture backend. | |
| - **gsplat**: Gaussian initialization, confidence-weighted fusion, and rendering. | |
| Extend3D remains a research reference only. Simam3D will use adapters around | |
| these maintained projects so each backend can be tested independently and | |
| replaced without rewriting the UI or export layer. | |
| ## Companion prototypes | |
| - [SimamTwin3D](https://huggingface.co/spaces/junaid-simamdigital/SimamTwin3D): | |
| consent-first multi-view human-reference capture audit. It records hashes and | |
| capture coverage; it does not claim a perfect digital twin. | |
| - [SimamAnything2GS](https://huggingface.co/spaces/junaid-simamdigital/SimamAnything2GS): | |
| arbitrary image/video ingest with bounded FFmpeg sampling and a CPU 2.5D | |
| preview. It is an ingest smoke test, not learned 3D reconstruction. | |
| These prototypes are intentionally separate so experimental identity/video | |
| workflows and general Gaussian reconstruction do not weaken Simam3D's evidence | |
| and provenance contract. | |
| The falsifiable experiment queue is documented in | |
| [RESEARCH_BACKLOG.md](RESEARCH_BACKLOG.md). | |
| The acceptance criteria for depth, fusion, Gaussian, and hidden-side claims | |
| are documented in [EVALUATION_PROTOCOL.md](EVALUATION_PROTOCOL.md). | |
| ## SimamDigital publishing | |
| The live demo currently belongs to the `junaid-simamdigital` user namespace. | |
| For prominent company branding, keep this Space linked from the | |
| [SimamDigital organization](https://huggingface.co/SimamDigital), and later | |
| move it to `SimamDigital/Simam3D` once the organization profile and collection | |
| are ready. The `pinned` flag keeps the Space near the top of its owning profile. | |
| ## Hosting notes | |
| This repository is currently packaged as a Docker Space, so its T4/L4/A10G | |
| hardware choices are metered Space hardware. Hugging Face ZeroGPU is a | |
| separate Gradio-SDK-only hosting mode; selecting `zero-a10g` on this Docker | |
| Space is not a drop-in replacement. A future ZeroGPU variant would need a | |
| Gradio-SDK packaging pass, the `spaces.GPU` function boundary, and the | |
| ZeroGPU-supported Torch/Python matrix. The Docker version remains the | |
| reproducible target for calibrated DA3 experiments. | |
| The app already includes an optional no-op-safe `spaces.GPU` boundary; it does | |
| not convert this Docker Space into a ZeroGPU Space by itself. | |
| See [ZERO_GPU_MIGRATION.md](ZERO_GPU_MIGRATION.md) for the separate-variant | |
| steps and acceptance checks. | |
| ## Roadmap | |
| 1. Depth baseline, GLB, PLY, camera metadata, confidence fusion, and Gaussian initialization (implemented). | |
| 2. Validate DA3NESTED-GIANT-LARGE-1.1 camera-aware and direct-Gaussian outputs against the V2 fallback (verified on the public GPU Space for one-, two-, and three-view requests). | |
| 3. Add four-view completion with source-view preservation and confidence-weighted fusion. | |
| 4. Add TRELLIS.2 and Hunyuan3D backend selection. | |
| 5. Add gsplat optimization and interactive splat preview. | |
| The UI defaults to the V2 stable baseline for a first successful smoke test; | |
| DA3 remains available as the explicit camera-aware research backend. The | |
| public GPU Space installs the pinned DA3 dependency and keeps the explicit V2 | |
| fallback. The Docker image still treats DA3 as optional: `requirements-da3.txt` | |
| can be layered into a research image when the upstream API is known to be | |
| compatible. This keeps the baseline Docker build independent of the | |
| experimental Git dependency. | |
| The Dockerfile exposes the same opt-in as | |
| `--build-arg SIMAM3D_INSTALL_DA3=1` for reproducible GPU research builds. | |
| The optional DA3 source is pinned to a specific upstream commit rather than | |
| tracking a moving branch. | |
| The container also includes a health check for the Gradio HTTP process, so a | |
| reachable UI can be separated from later model-inference failures. | |
| Gradio is launched with detailed error display enabled, while the manifest and | |
| container logs remain the authoritative diagnostic record. | |
| Failed requests also leave a redacted failure manifest with the failing stage, | |
| backend, runtime, software versions, input hashes, and exception details for | |
| reproducible diagnosis. | |
| The Diagnostics panel can list those reports for download after a failed run. | |
| The Diagnostics panel checks the optional DA3 package and the expected | |
| `DepthAnything3` symbol across both known upstream import locations, so an | |
| installed but incompatible DA3 package is visible before model inference. | |
| When DA3 is selected, its depth, confidence, extrinsics, and intrinsics shapes | |
| are validated at the adapter boundary; malformed results trigger the labeled V2 | |
| fallback before any geometry export begins, and the exact fallback exception is | |
| stored in the run manifest for diagnosis. | |
| Backend switching also releases the previously loaded heavyweight model before | |
| loading the next one, reducing avoidable GPU memory pressure in long-lived | |
| Space processes. | |
| Additional views accept both filepath strings and Gradio `FileData` uploads; | |
| unreadable files now produce a direct input error before inference starts. | |
| The baseline requirements also pin the CUDA-compatible Torch pair and bound | |
| dependency major versions so rebuilds do not silently change the inference or | |
| export contract. | |