Spaces:
Running on Zero
Running on Zero
| # Simam3D evaluation protocol | |
| This document defines what evidence is required before Simam3D makes a quality | |
| claim. Current hosted results are engineering and contract evidence; they are | |
| not results on a named reconstruction benchmark. | |
| ## Claim levels | |
| | Claim | Minimum evidence | Current status | | |
| | --- | --- | --- | | |
| | Depth prediction is executable | Named model, fixed input, successful manifest, runtime record | Implemented and hosted | | |
| | Multi-view geometry is reproducible | Repeated signatures, input hashes, pose provenance, fusion metrics | Implemented on controlled fixtures | | |
| | Depth-to-camera geometry is internally correct | Independent analytic surface, RMSE/max-error threshold, multiple resolutions | Implemented in `benchmark_pipeline.py` | | |
| | Depth is quantitatively accurate | Named dataset split, metric definitions, baseline, confidence interval | Not yet measured | | |
| | Depth evaluation is reproducible | Paired `.npy` prediction/ground-truth arrays, mask, orientation flag, standard metrics | Implemented in `evaluate_depth.py`; dataset run still pending | | |
| | Gaussian export is direct from a model | Native source, candidate/export counts, portable file inspection | DA3 direct path verified | | |
| | Exported geometry is quantitatively close | Reference point cloud, declared threshold, symmetric Chamfer and F-score | Implemented in `evaluate_pointcloud.py`; reference run pending | | |
| | Cross-view consistency is measurable | Exported point cloud plus depth/intrinsics/extrinsics NPZ | Implemented in `evaluate_reprojection.py`; calibrated run pending | | |
| | Gaussian reconstruction is higher quality | Held-out views, renderer, PSNR/SSIM/LPIPS or equivalent, baseline | Not yet claimed | | |
| | Hidden-side geometry is correct | Held-out unseen-side views and a predeclared threshold | Not yet claimed | | |
| ## Required run record | |
| Every evaluation row should retain: | |
| - input and artifact SHA-256 hashes; | |
| - model identifier and software versions; | |
| - hardware and elapsed time; | |
| - view count, unique/duplicate status, and camera-pose source; | |
| - fused point count, source-view masks, projection coverage, and reprojection | |
| error where applicable; | |
| - Gaussian source (`da3_direct` or `fused_point_initialization`); | |
| - whether any backend fallback occurred; | |
| - the exact manifest validator result. | |
| ## Experiment matrix | |
| ### A. Single-view baseline | |
| Run the V2 baseline and DA3 path on the same fixed image set. Report successful | |
| completion rate, depth metrics where ground truth exists, runtime, peak memory, | |
| and artifact validity. Keep metric-scale and unseen-side claims false. | |
| ### B. Camera-aware multi-view | |
| Use calibrated captures with known intrinsics and extrinsics. Compare one view, | |
| two views with a wide baseline, three or more views with controlled baselines, | |
| and duplicate/near-duplicate controls. Report held-out-view reprojection error, | |
| coverage, source-view support, and failure cases. Synthetic orbit poses remain | |
| a control condition, not a substitute for calibration. | |
| The CPU benchmark also records a support-pruning ablation at minimum support | |
| of two views. This is an implementation diagnostic: retention alone is not a | |
| quality improvement unless paired with held-out-view measurements. | |
| ### C. Gaussian representation | |
| Separate three questions: can the system export a valid portable Gaussian file; | |
| are its parameters native to the model or initialized from depth; and does | |
| optimization improve held-out-view quality? Only the first two are currently | |
| implemented for the DA3 path. The third needs a renderer, fixed optimization | |
| budget, and comparison against the unoptimized export. | |
| ### D. “What’s Behind the Image?” | |
| Treat hidden content as a hypothesis. Evaluate it only where held-out backside | |
| data exists, and score the generated hypothesis against that evidence. A | |
| plausible reveal without held-out evaluation is not proof of correct hidden | |
| geometry. | |
| ## Reproducibility rules | |
| - Pin model revisions and dependency versions. | |
| - Store deterministic fixtures and their hashes. | |
| - Run each condition at least twice for repeatability checks. | |
| - Predeclare thresholds and exclusions before inspecting results. | |
| - Publish failures, fallbacks, and quota limitations alongside successes. | |
| - Never use a synthetic-prior status as evidence of calibrated camera agreement. | |
| For a real depth evaluation, prepare matching positive-depth `.npy` arrays and | |
| run `python evaluate_depth.py prediction.npy ground_truth.npy`. Use | |
| `--inverse-depth` for inverse-depth predictions. Record both `raw` and | |
| `median_scaled` results; only the former is appropriate when metric scale is | |
| known. | |
| For a multi-sample run, create one JSONL row per pair with `prediction` and | |
| `ground_truth` paths (plus optional `id`, `mask`, and `inverse_depth`) and run | |
| `python evaluate_depth_dataset.py pairs.jsonl`. The report retains per-sample | |
| results and failed rows instead of silently dropping unusable inputs. | |
| For point-cloud or Gaussian position evaluation, use | |
| `python evaluate_pointcloud.py exported.ply reference.ply --threshold 0.05`. | |
| The evaluator reports both directions of nearest-neighbour error, precision, | |
| recall, and F-score. A reference capture and predeclared coordinate alignment | |
| are required before interpreting these numbers as reconstruction quality. | |
| For multiple 3D samples, use JSONL rows with `prediction`, `reference`, and an | |
| optional `threshold` and run `python evaluate_pointcloud_dataset.py pairs.jsonl`. | |
| The aggregate retains per-sample scores and failures. | |
| For a run-level consistency diagnostic, use `python evaluate_reprojection.py | |
| fused.ply metadata.npz --output reprojection.json`. It reports per-view and | |
| aggregate projection coverage and relative-depth reprojection error. These are | |
| consistency signals, not substitutes for held-out geometry ground truth. | |
| Use [EXPERIMENT_TEMPLATE.md](EXPERIMENT_TEMPLATE.md) to record the dataset, | |
| calibration, conditions, measurements, failure analysis, and claim boundary | |
| for each run. | |
| Before a calibrated multi-view run, validate a manifest containing each image, | |
| 3x3 intrinsics, 4x4 world-to-camera extrinsics, and explicit `train` or | |
| `held_out` split with `python validate_capture_manifest.py capture.json | |
| --check-files`. A manifest without a held-out view cannot support a quality | |
| claim. | |
| ## Current honest conclusion | |
| Simam3D demonstrates a reproducible depth/fusion/export workbench, native DA3 | |
| Gaussian parameter export, and explicit evidence bookkeeping. It does not yet | |
| demonstrate metric-scale accuracy, trained Gaussian optimization, or verified | |
| unseen-side completion. Those are the next experiments, not implied features. | |