junaid-simamdigital commited on
Commit
cc2febb
·
verified ·
1 Parent(s): 662c670

docs: link Gaussian inspector

Browse files
Files changed (1) hide show
  1. README.md +60 -237
README.md CHANGED
@@ -1,246 +1,69 @@
1
  ---
2
- title: Simam3D
3
  emoji: 🧭
4
  colorFrom: indigo
5
  colorTo: blue
6
- sdk: docker
7
- app_port: 7860
8
- suggested_hardware: a10g-large
9
- pinned: true
10
- tags:
11
- - image-to-3d
12
- - depth-estimation
13
- - gaussian-splatting
14
- - multi-view-reconstruction
15
  models:
16
  - depth-anything/Depth-Anything-V2-Small-hf
17
  - depth-anything/DA3NESTED-GIANT-LARGE-1.1
18
  ---
19
 
20
- # Simam3D
21
-
22
- > **Live demo:** the verified public Gradio deployment is
23
- > [Simam3D-GPU](https://huggingface.co/spaces/junaid-simamdigital/Simam3D-GPU).
24
- > This Docker Space is a reproducible metered-GPU build and may remain paused
25
- > when hardware is not allocated.
26
-
27
- Simam3D 0.4.8 is a modular image-to-3D workbench with a tested depth baseline,
28
- camera-aware DA3 inference, evidence-aware multi-view fusion, direct Gaussian
29
- parameter export when available, uncertainty/reveal diagnostics, and
30
- machine-validated output manifests. It does not claim metric-scale geometry,
31
- trained Gaussian optimization, or verified hidden-side completion.
32
-
33
- This first version intentionally keeps the inference path small and observable
34
- so every stage can be tested independently on Hugging Face Spaces. Each run
35
- produces inspectable outputs: colorized depth and confidence images, a textured
36
- GLB scaffold, a confidence-preserving per-view PLY point cloud, a confidence-weighted fused
37
- PLY, a viewer-compatible Gaussian-splat PLY, and an NPZ package containing
38
- depth, confidence, extrinsics, and intrinsics arrays.
39
- The per-view and fused PLY files retain `confidence`; fused points also retain
40
- the dominant contributing `source_view` identifier for downstream inspection
41
- and evaluation.
42
- Fused and Gaussian PLY files also retain a `source_mask` bitmask showing which
43
- input views contributed to each voxel, alongside the dominant pointer and the
44
- derived Gaussian opacity/scale initialization.
45
- It also emits a shareable evidence-card PNG that places the source, depth, and
46
- confidence views beside the measured fusion statistics.
47
- The **What's Behind the Image?** panel lets a user record a scene hypothesis,
48
- extracts semantic cue labels, estimates image-space illumination direction, and
49
- exports a reveal card pairing the original with the depth result. The reveal is
50
- explicitly a hypothesis: a single image cannot verify unseen geometry. Additional
51
- views and the manifest provide the evidence trail for future completion models.
52
- The analysis also reports a depth-normal photometric light vector; this is a
53
- heuristic lighting cue and is not a claim of physically correct illumination.
54
- Each run also exports a reveal-uncertainty map and a deterministic next-view plan;
55
- these prioritize acquisition around low-confidence or discontinuous regions.
56
- It also exports a lightweight 12-frame turntable GIF rendered from the fused
57
- points, making the reconstruction immediately inspectable and shareable.
58
- The run report projects fused points back into each input camera and records
59
- in-frame coverage as a diagnostic for pose or fusion collapse; coverage is a
60
- consistency signal, not a geometric accuracy score. It also reports relative
61
- depth reprojection error, comparing the fused raster with each input depth map.
62
- For troubleshooting, the UI includes an explicitly labeled deterministic export
63
- smoke test that bypasses model inference and exercises the complete artifact
64
- pipeline on CPU.
65
- The reproducibility manifest includes the app/schema version, an explicit
66
- provenance claims contract, software
67
- versions, SHA-256 fingerprints for every input and exported artifact, and
68
- normalized dimensions for every input view.
69
- Downloaded runs can be checked with `python validate_manifest.py
70
- path/to/*_run.json`; the validator reports missing or tampered artifacts and
71
- also verifies structured failure reports.
72
-
73
- ## Reproducible evaluation
74
-
75
- For paired depth arrays, run `python evaluate_depth.py prediction.npy
76
- ground_truth.npy`; for a JSONL dataset, use `python evaluate_depth_dataset.py
77
- pairs.jsonl`. For exported fused or Gaussian geometry, run `python
78
- evaluate_pointcloud.py exported.ply reference.ply --threshold 0.05`, or use
79
- `evaluate_pointcloud_dataset.py` for a JSONL set. Before a calibrated
80
- multi-view experiment, validate `capture.json` with `python
81
- validate_capture_manifest.py capture.json --check-files`. The schema and a
82
- copyable example are in [CAPTURE_MANIFEST_SCHEMA.md](CAPTURE_MANIFEST_SCHEMA.md).
83
- For COLMAP text outputs, use `python import_colmap.py cameras.txt images.txt
84
- image_root capture.json --held-out IMAGE_ID ...` to create the manifest first.
85
- These tools retain raw versus median-scaled depth results, declared geometric
86
- thresholds, held-out splits, and failed samples; they do not turn synthetic
87
- fixtures or synthetic camera poses into quality claims.
88
-
89
- Multiple manifests can be compared with `python summarize_runs.py output`,
90
- which emits backend, view-count, fusion, coverage, reprojection, fallback, and
91
- provenance fields as JSON. Failed manifests are included with their status,
92
- failing stage, and structured error details.
93
- The dependency-free regression harness can be run with
94
- `python benchmark_pipeline.py --output benchmark_output`. It repeats the CPU
95
- fusion fixture at several resolutions and records signatures plus fusion
96
- metrics. This proves deterministic geometry/export behavior only; it is not a
97
- learned reconstruction-quality benchmark.
98
- On Windows, run `powershell -ExecutionPolicy Bypass -File run_checks.ps1` before
99
- publishing a change; this is the canonical compile and regression gate.
100
- The DA3 package-layout resolver is tested independently in `da3_compat.py`, so
101
- optional upstream import changes can be diagnosed without loading model
102
- weights.
103
- The model-free geometry path can also be exercised with
104
- `python demo_pipeline.py --output demo_output`; it writes deterministic fused
105
- and Gaussian artifacts for CPU-only integration testing.
106
- The demo also emits a schema-3 manifest, so it can be checked with the same
107
- `validate_manifest.py` tool as a model-backed run.
108
- See [TROUBLESHOOTING.md](TROUBLESHOOTING.md) for the scheduler, smoke-test, and
109
- browser-versus-server failure decision tree.
110
- The research direction and required ablations are outlined in
111
- [PAPER_OUTLINE.md](PAPER_OUTLINE.md).
112
- The verified milestone history is recorded in [CHANGELOG.md](CHANGELOG.md).
113
- Fusion metrics also report the number of distinct dominant source views in the
114
- fused result, making view collapse visible without overstating provenance.
115
- They report the fraction of voxels supported by multiple views as a separate
116
- cross-view consistency signal, plus the number of source views with any
117
- support in the fused result.
118
- It also emits a dependency-free Gaussian-splat PLY initialized from the fused
119
- points. Splat opacity and initial scale are confidence-weighted, allowing the
120
- representation to expose uncertainty before any learned Gaussian optimization.
121
- Each run includes a JSON manifest recording the backend, model, runtime,
122
- number of views, elapsed time, and generated artifact paths.
123
- The UI also accepts optional additional images so DA3 can process a small
124
- multi-view batch in one request; fusion and novel-view synthesis remain
125
- separate, explicitly tracked stages.
126
- When the V2 fallback is selected, additional images use a deterministic
127
- synthetic orbit pose prior and the run manifest records that limitation; only
128
- DA3 camera metadata is treated as camera-aware geometry.
129
- Hosted requests are bounded to eight views and a 1536-pixel maximum input side
130
- by default, with `SIMAM3D_MAX_VIEWS` and `SIMAM3D_MAX_INPUT_SIDE` available for
131
- controlled deployments. Inference runs under `torch.inference_mode()` to
132
- avoid retaining training graphs. The heavy inference event is limited to one
133
- concurrent request and guarded by a process lock, reducing avoidable V2/DA3
134
- memory contention.
135
- Fusion resolution is configurable with `SIMAM3D_FUSION_VOXEL_SIZE` (default
136
- `0.02`) and `SIMAM3D_FUSION_MAX_POINTS` (default `150000`); both values are
137
- recorded in the run manifest.
138
- The provenance bitmask supports up to 30 views because it is exported as a
139
- signed 32-bit integer.
140
-
141
- The geometry layer is isolated in `simam3d_core.py`, a dependency-free NumPy
142
- module with deterministic voxel fusion and confidence metrics. Its smoke tests
143
- can run without CUDA or model weights:
144
-
145
- ```bash
146
- python -c "import runpy; ns=runpy.run_path('tests/test_core.py'); [fn() for name,fn in ns.items() if name.startswith('test_')]"
147
- ```
148
-
149
- Relative depth is normalized using a consistent inverse-depth convention for
150
- the current scaffold, so exported geometry is useful for visualization but is
151
- not yet metrically scaled. DA3 camera calibration and
152
- multi-view pose quality should be evaluated before treating measurements as
153
- physical dimensions.
154
-
155
- ## Planned backends
156
-
157
- - **Depth Anything 3**: any-view depth, camera pose, confidence, and direct Gaussian estimation.
158
- - **TRELLIS.2**: primary high-quality mesh and structured-latent backend.
159
- - **Hunyuan3D 2.1**: alternative shape and PBR-texture backend.
160
- - **gsplat**: Gaussian initialization, confidence-weighted fusion, and rendering.
161
-
162
- Extend3D remains a research reference only. Simam3D will use adapters around
163
- these maintained projects so each backend can be tested independently and
164
- replaced without rewriting the UI or export layer.
165
-
166
- ## Companion prototypes
167
-
168
- - [SimamTwin3D](https://huggingface.co/spaces/junaid-simamdigital/SimamTwin3D):
169
- consent-first multi-view human-reference capture audit. It records hashes and
170
- capture coverage; it does not claim a perfect digital twin.
171
- - [SimamAnything2GS](https://huggingface.co/spaces/junaid-simamdigital/SimamAnything2GS):
172
- arbitrary image/video ingest with bounded FFmpeg sampling and a CPU 2.5D
173
- preview. It is an ingest smoke test, not learned 3D reconstruction.
174
-
175
- These prototypes are intentionally separate so experimental identity/video
176
- workflows and general Gaussian reconstruction do not weaken Simam3D's evidence
177
- and provenance contract.
178
- The falsifiable experiment queue is documented in
179
- [RESEARCH_BACKLOG.md](RESEARCH_BACKLOG.md).
180
- The acceptance criteria for depth, fusion, Gaussian, and hidden-side claims
181
- are documented in [EVALUATION_PROTOCOL.md](EVALUATION_PROTOCOL.md).
182
-
183
- ## SimamDigital publishing
184
-
185
- The live demo currently belongs to the `junaid-simamdigital` user namespace.
186
- For prominent company branding, keep this Space linked from the
187
- [SimamDigital organization](https://huggingface.co/SimamDigital), and later
188
- move it to `SimamDigital/Simam3D` once the organization profile and collection
189
- are ready. The `pinned` flag keeps the Space near the top of its owning profile.
190
-
191
- ## Hosting notes
192
-
193
- This repository is currently packaged as a Docker Space, so its T4/L4/A10G
194
- hardware choices are metered Space hardware. Hugging Face ZeroGPU is a
195
- separate Gradio-SDK-only hosting mode; selecting `zero-a10g` on this Docker
196
- Space is not a drop-in replacement. A future ZeroGPU variant would need a
197
- Gradio-SDK packaging pass, the `spaces.GPU` function boundary, and the
198
- ZeroGPU-supported Torch/Python matrix. The Docker version remains the
199
- reproducible target for calibrated DA3 experiments.
200
- The app already includes an optional no-op-safe `spaces.GPU` boundary; it does
201
- not convert this Docker Space into a ZeroGPU Space by itself.
202
- See [ZERO_GPU_MIGRATION.md](ZERO_GPU_MIGRATION.md) for the separate-variant
203
- steps and acceptance checks.
204
-
205
- ## Roadmap
206
-
207
- 1. Depth baseline, GLB, PLY, camera metadata, confidence fusion, and Gaussian initialization (implemented).
208
- 2. Validate DA3NESTED-GIANT-LARGE-1.1 camera-aware and direct-Gaussian outputs against the V2 fallback (verified on the public GPU Space for one-, two-, and three-view requests).
209
- 3. Add four-view completion with source-view preservation and confidence-weighted fusion.
210
- 4. Add TRELLIS.2 and Hunyuan3D backend selection.
211
- 5. Add gsplat optimization and interactive splat preview.
212
-
213
- The UI defaults to the V2 stable baseline for a first successful smoke test;
214
- DA3 remains available as the explicit camera-aware research backend. The
215
- public GPU Space installs the pinned DA3 dependency and keeps the explicit V2
216
- fallback. The Docker image still treats DA3 as optional: `requirements-da3.txt`
217
- can be layered into a research image when the upstream API is known to be
218
- compatible. This keeps the baseline Docker build independent of the
219
- experimental Git dependency.
220
- The Dockerfile exposes the same opt-in as
221
- `--build-arg SIMAM3D_INSTALL_DA3=1` for reproducible GPU research builds.
222
- The optional DA3 source is pinned to a specific upstream commit rather than
223
- tracking a moving branch.
224
- The container also includes a health check for the Gradio HTTP process, so a
225
- reachable UI can be separated from later model-inference failures.
226
- Gradio is launched with detailed error display enabled, while the manifest and
227
- container logs remain the authoritative diagnostic record.
228
- Failed requests also leave a redacted failure manifest with the failing stage,
229
- backend, runtime, software versions, input hashes, and exception details for
230
- reproducible diagnosis.
231
- The Diagnostics panel can list those reports for download after a failed run.
232
- The Diagnostics panel checks the optional DA3 package and the expected
233
- `DepthAnything3` symbol across both known upstream import locations, so an
234
- installed but incompatible DA3 package is visible before model inference.
235
- When DA3 is selected, its depth, confidence, extrinsics, and intrinsics shapes
236
- are validated at the adapter boundary; malformed results trigger the labeled V2
237
- fallback before any geometry export begins, and the exact fallback exception is
238
- stored in the run manifest for diagnosis.
239
- Backend switching also releases the previously loaded heavyweight model before
240
- loading the next one, reducing avoidable GPU memory pressure in long-lived
241
- Space processes.
242
- Additional views accept both filepath strings and Gradio `FileData` uploads;
243
- unreadable files now produce a direct input error before inference starts.
244
- The baseline requirements also pin the CUDA-compatible Torch pair and bound
245
- dependency major versions so rebuilds do not silently change the inference or
246
- export contract.
 
1
  ---
2
+ title: Simam3D ZeroGPU
3
  emoji: 🧭
4
  colorFrom: indigo
5
  colorTo: blue
6
+ sdk: gradio
7
+ app_file: app.py
8
+ suggested_hardware: zero-a10g
9
+ pinned: false
 
 
 
 
 
10
  models:
11
  - depth-anything/Depth-Anything-V2-Small-hf
12
  - depth-anything/DA3NESTED-GIANT-LARGE-1.1
13
  ---
14
 
15
+ # Simam3D ZeroGPU
16
+
17
+ Gradio-SDK deployment variant of Simam3D for Hugging Face ZeroGPU. It reuses
18
+ the same tested depth, fusion, Gaussian export, provenance, and failure-report
19
+ code as the Docker Space. ZeroGPU availability and queue time are platform
20
+ controlled; this variant does not claim a successful model run until a real
21
+ request returns a validated manifest.
22
+
23
+ ## Verified capabilities
24
+
25
+ - Depth Anything V2 stable-baseline inference on GPU.
26
+ - Depth Anything 3 camera-aware inference using the Nested checkpoint.
27
+ - Direct DA3 Gaussian export with deterministic opacity-based point capping.
28
+ - GLB, point-cloud, fused-point-cloud, Gaussian PLY, uncertainty, turntable,
29
+ evidence-card, and reproducibility-manifest outputs.
30
+ - Input-diversity and fusion-support diagnostics that distinguish duplicate,
31
+ near-duplicate, synthetic-prior, and native-camera evidence.
32
+
33
+ See [HOSTED_VERIFICATION.md](HOSTED_VERIFICATION.md) for live run IDs and
34
+ metrics, [PAPER_OUTLINE.md](PAPER_OUTLINE.md) for the research protocol, and
35
+ [RESEARCH_BACKLOG.md](RESEARCH_BACKLOG.md) for falsifiable follow-up ideas.
36
+ The claim boundaries and acceptance criteria are defined in
37
+ [EVALUATION_PROTOCOL.md](EVALUATION_PROTOCOL.md).
38
+
39
+ For public project stories, see the [Simam AI Lab case-study series](CASE_STUDIES/README.md)
40
+ and the [LinkedIn launch pack](CASE_STUDIES/LINKEDIN-POSTS.md).
41
+
42
+ Maintainers can reproduce the no-model hosted check with
43
+ [`hosted_smoke_test.py`](hosted_smoke_test.py), and use
44
+ [`TROUBLESHOOTING.md`](TROUBLESHOOTING.md) for failure-stage guidance.
45
+
46
+ For quantitative work, use [`evaluate_depth.py`](evaluate_depth.py) and
47
+ [`evaluate_depth_dataset.py`](evaluate_depth_dataset.py) for depth arrays,
48
+ [`evaluate_pointcloud.py`](evaluate_pointcloud.py) and
49
+ [`evaluate_pointcloud_dataset.py`](evaluate_pointcloud_dataset.py) for exported
50
+ 3D/Gaussian positions, and validate calibrated inputs with
51
+ [`validate_capture_manifest.py`](validate_capture_manifest.py). The copyable
52
+ manifest format is documented in
53
+ [`CAPTURE_MANIFEST_SCHEMA.md`](CAPTURE_MANIFEST_SCHEMA.md).
54
+ Use [`inspect_gaussian_ply.py`](inspect_gaussian_ply.py) to inspect exported
55
+ Gaussian parameters and evidence fields.
56
+ COLMAP text models can be converted with
57
+ [`import_colmap.py`](import_colmap.py) before validation.
58
+
59
+ ## Important limitations
60
+
61
+ This Space does not claim metric-scale reconstruction, learned Gaussian
62
+ optimization, verified unseen-side geometry, or generated novel-view images.
63
+ The displayed turntable is a point/Gaussian export preview, not proof that the
64
+ hidden side of an object was reconstructed correctly.
65
+
66
+ The public GPU deployment installs the pinned DA3 package so both the
67
+ camera-aware path and the V2 baseline can be tested. Start with the explicit
68
+ export smoke test to separate UI/GPU scheduling from model inference; a DA3
69
+ failure is reported and falls back to V2 rather than being hidden.