# Simam3D evaluation protocol

This document defines what evidence is required before Simam3D makes a quality
claim. Current hosted results are engineering and contract evidence; they are
not results on a named reconstruction benchmark.

## Claim levels

| Claim | Minimum evidence | Current status |
| --- | --- | --- |
| Depth prediction is executable | Named model, fixed input, successful manifest, runtime record | Implemented and hosted |
| Multi-view geometry is reproducible | Repeated signatures, input hashes, pose provenance, fusion metrics | Implemented on controlled fixtures |
| Depth-to-camera geometry is internally correct | Independent analytic surface, RMSE/max-error threshold, multiple resolutions | Implemented in `benchmark_pipeline.py` |
| Depth is quantitatively accurate | Named dataset split, metric definitions, baseline, confidence interval | Not yet measured |
| Depth evaluation is reproducible | Paired `.npy` prediction/ground-truth arrays, mask, orientation flag, standard metrics | Implemented in `evaluate_depth.py`; dataset run still pending |
| Gaussian export is direct from a model | Native source, candidate/export counts, portable file inspection | DA3 direct path verified |
| Exported geometry is quantitatively close | Reference point cloud, declared threshold, symmetric Chamfer and F-score | Implemented in `evaluate_pointcloud.py`; reference run pending |
| Cross-view consistency is measurable | Exported point cloud plus depth/intrinsics/extrinsics NPZ | Implemented in `evaluate_reprojection.py`; calibrated run pending |
| Gaussian reconstruction is higher quality | Held-out views, renderer, PSNR/SSIM/LPIPS or equivalent, baseline | Not yet claimed |
| Hidden-side geometry is correct | Held-out unseen-side views and a predeclared threshold | Not yet claimed |

## Required run record

Every evaluation row should retain:

- input and artifact SHA-256 hashes;
- model identifier and software versions;
- hardware and elapsed time;
- view count, unique/duplicate status, and camera-pose source;
- fused point count, source-view masks, projection coverage, and reprojection
  error where applicable;
- Gaussian source (`da3_direct` or `fused_point_initialization`);
- whether any backend fallback occurred;
- the exact manifest validator result.

## Experiment matrix

### A. Single-view baseline

Run the V2 baseline and DA3 path on the same fixed image set. Report successful
completion rate, depth metrics where ground truth exists, runtime, peak memory,
and artifact validity. Keep metric-scale and unseen-side claims false.

### B. Camera-aware multi-view

Use calibrated captures with known intrinsics and extrinsics. Compare one view,
two views with a wide baseline, three or more views with controlled baselines,
and duplicate/near-duplicate controls. Report held-out-view reprojection error,
coverage, source-view support, and failure cases. Synthetic orbit poses remain
a control condition, not a substitute for calibration.

The CPU benchmark also records a support-pruning ablation at minimum support
of two views. This is an implementation diagnostic: retention alone is not a
quality improvement unless paired with held-out-view measurements.

### C. Gaussian representation

Separate three questions: can the system export a valid portable Gaussian file;
are its parameters native to the model or initialized from depth; and does
optimization improve held-out-view quality? Only the first two are currently
implemented for the DA3 path. The third needs a renderer, fixed optimization
budget, and comparison against the unoptimized export.

### D. “What’s Behind the Image?”

Treat hidden content as a hypothesis. Evaluate it only where held-out backside
data exists, and score the generated hypothesis against that evidence. A
plausible reveal without held-out evaluation is not proof of correct hidden
geometry.

## Reproducibility rules

- Pin model revisions and dependency versions.
- Store deterministic fixtures and their hashes.
- Run each condition at least twice for repeatability checks.
- Predeclare thresholds and exclusions before inspecting results.
- Publish failures, fallbacks, and quota limitations alongside successes.
- Never use a synthetic-prior status as evidence of calibrated camera agreement.

For a real depth evaluation, prepare matching positive-depth `.npy` arrays and
run `python evaluate_depth.py prediction.npy ground_truth.npy`. Use
`--inverse-depth` for inverse-depth predictions. Record both `raw` and
`median_scaled` results; only the former is appropriate when metric scale is
known.

For a multi-sample run, create one JSONL row per pair with `prediction` and
`ground_truth` paths (plus optional `id`, `mask`, and `inverse_depth`) and run
`python evaluate_depth_dataset.py pairs.jsonl`. The report retains per-sample
results and failed rows instead of silently dropping unusable inputs.

For point-cloud or Gaussian position evaluation, use
`python evaluate_pointcloud.py exported.ply reference.ply --threshold 0.05`.
The evaluator reports both directions of nearest-neighbour error, precision,
recall, and F-score. A reference capture and predeclared coordinate alignment
are required before interpreting these numbers as reconstruction quality.

For multiple 3D samples, use JSONL rows with `prediction`, `reference`, and an
optional `threshold` and run `python evaluate_pointcloud_dataset.py pairs.jsonl`.
The aggregate retains per-sample scores and failures.

For a run-level consistency diagnostic, use `python evaluate_reprojection.py
fused.ply metadata.npz --output reprojection.json`. It reports per-view and
aggregate projection coverage and relative-depth reprojection error. These are
consistency signals, not substitutes for held-out geometry ground truth.

Use [EXPERIMENT_TEMPLATE.md](EXPERIMENT_TEMPLATE.md) to record the dataset,
calibration, conditions, measurements, failure analysis, and claim boundary
for each run.

Before a calibrated multi-view run, validate a manifest containing each image,
3x3 intrinsics, 4x4 world-to-camera extrinsics, and explicit `train` or
`held_out` split with `python validate_capture_manifest.py capture.json
--check-files`. A manifest without a held-out view cannot support a quality
claim.

## Current honest conclusion

Simam3D demonstrates a reproducible depth/fusion/export workbench, native DA3
Gaussian parameter export, and explicit evidence bookkeeping. It does not yet
demonstrate metric-scale accuracy, trained Gaussian optimization, or verified
unseen-side completion. Those are the next experiments, not implied features.