EVALUATION_PROTOCOL.md

Simam3D evaluation protocol

This document defines what evidence is required before Simam3D makes a quality claim. Current hosted results are engineering and contract evidence; they are not results on a named reconstruction benchmark.

Claim levels

Claim	Minimum evidence	Current status
Depth prediction is executable	Named model, fixed input, successful manifest, runtime record	Implemented and hosted
Multi-view geometry is reproducible	Repeated signatures, input hashes, pose provenance, fusion metrics	Implemented on controlled fixtures
Depth-to-camera geometry is internally correct	Independent analytic surface, RMSE/max-error threshold, multiple resolutions	Implemented in benchmark_pipeline.py
Depth is quantitatively accurate	Named dataset split, metric definitions, baseline, confidence interval	Not yet measured
Depth evaluation is reproducible	Paired .npy prediction/ground-truth arrays, mask, orientation flag, standard metrics	Implemented in evaluate_depth.py; dataset run still pending
Gaussian export is direct from a model	Native source, candidate/export counts, portable file inspection	DA3 direct path verified
Exported geometry is quantitatively close	Reference point cloud, declared threshold, symmetric Chamfer and F-score	Implemented in evaluate_pointcloud.py; reference run pending
Cross-view consistency is measurable	Exported point cloud plus depth/intrinsics/extrinsics NPZ	Implemented in evaluate_reprojection.py; calibrated run pending
Gaussian reconstruction is higher quality	Held-out views, renderer, PSNR/SSIM/LPIPS or equivalent, baseline	Not yet claimed
Hidden-side geometry is correct	Held-out unseen-side views and a predeclared threshold	Not yet claimed

Required run record

Every evaluation row should retain:

• input and artifact SHA-256 hashes;
• model identifier and software versions;
• hardware and elapsed time;
• view count, unique/duplicate status, and camera-pose source;
• fused point count, source-view masks, projection coverage, and reprojection error where applicable;
• Gaussian source (da3_direct or fused_point_initialization);
• whether any backend fallback occurred;
• the exact manifest validator result.

Experiment matrix

A. Single-view baseline

Run the V2 baseline and DA3 path on the same fixed image set. Report successful completion rate, depth metrics where ground truth exists, runtime, peak memory, and artifact validity. Keep metric-scale and unseen-side claims false.

B. Camera-aware multi-view

Use calibrated captures with known intrinsics and extrinsics. Compare one view, two views with a wide baseline, three or more views with controlled baselines, and duplicate/near-duplicate controls. Report held-out-view reprojection error, coverage, source-view support, and failure cases. Synthetic orbit poses remain a control condition, not a substitute for calibration.

The CPU benchmark also records a support-pruning ablation at minimum support of two views. This is an implementation diagnostic: retention alone is not a quality improvement unless paired with held-out-view measurements.

C. Gaussian representation

Separate three questions: can the system export a valid portable Gaussian file; are its parameters native to the model or initialized from depth; and does optimization improve held-out-view quality? Only the first two are currently implemented for the DA3 path. The third needs a renderer, fixed optimization budget, and comparison against the unoptimized export.

D. “What’s Behind the Image?”

Treat hidden content as a hypothesis. Evaluate it only where held-out backside data exists, and score the generated hypothesis against that evidence. A plausible reveal without held-out evaluation is not proof of correct hidden geometry.

Reproducibility rules

• Pin model revisions and dependency versions.
• Store deterministic fixtures and their hashes.
• Run each condition at least twice for repeatability checks.
• Predeclare thresholds and exclusions before inspecting results.
• Publish failures, fallbacks, and quota limitations alongside successes.
• Never use a synthetic-prior status as evidence of calibrated camera agreement.

For a real depth evaluation, prepare matching positive-depth .npy arrays and run python evaluate_depth.py prediction.npy ground_truth.npy. Use --inverse-depth for inverse-depth predictions. Record both raw and median_scaled results; only the former is appropriate when metric scale is known.

For a multi-sample run, create one JSONL row per pair with prediction and ground_truth paths (plus optional id, mask, and inverse_depth) and run python evaluate_depth_dataset.py pairs.jsonl. The report retains per-sample results and failed rows instead of silently dropping unusable inputs.

For point-cloud or Gaussian position evaluation, use python evaluate_pointcloud.py exported.ply reference.ply --threshold 0.05. The evaluator reports both directions of nearest-neighbour error, precision, recall, and F-score. A reference capture and predeclared coordinate alignment are required before interpreting these numbers as reconstruction quality.

For multiple 3D samples, use JSONL rows with prediction, reference, and an optional threshold and run python evaluate_pointcloud_dataset.py pairs.jsonl. The aggregate retains per-sample scores and failures.

For a run-level consistency diagnostic, use python evaluate_reprojection.py fused.ply metadata.npz --output reprojection.json. It reports per-view and aggregate projection coverage and relative-depth reprojection error. These are consistency signals, not substitutes for held-out geometry ground truth.

Use EXPERIMENT_TEMPLATE.md to record the dataset, calibration, conditions, measurements, failure analysis, and claim boundary for each run.

Before a calibrated multi-view run, validate a manifest containing each image, 3x3 intrinsics, 4x4 world-to-camera extrinsics, and explicit train or held_out split with python validate_capture_manifest.py capture.json --check-files. A manifest without a held-out view cannot support a quality claim.

Current honest conclusion

Simam3D demonstrates a reproducible depth/fusion/export workbench, native DA3 Gaussian parameter export, and explicit evidence bookkeeping. It does not yet demonstrate metric-scale accuracy, trained Gaussian optimization, or verified unseen-side completion. Those are the next experiments, not implied features.