fishxinyu/flexcombine-bench/evaluation/eval_camera_new - Storage Bucket

Camera evaluation — MegaSaM poses + RotErr / TransErr / CamMC

The camera-accuracy protocol from CamCloneMaster (arXiv 2506.03140, Sec. 5.1): estimate per-frame camera poses from both the generated clip and its camera reference with MegaSaM, then score the two trajectories with the three errors defined by CamI2V.

This replaces the earlier 2D similarity-transform metric (zoom / trans), which could only represent scale and pan and therefore mis-measured any orbit, arc or dolly — for the reference and the generation alike.

The metric

Given per-frame camera-to-world poses c2w (N,4,4) for each video, expressed relative to their own first frame (rel[i] = c2w[0]⁻¹ @ c2w[i], so the two reconstructions' arbitrary world frames drop out):

metric	definition
RotErr	Σᵢ arccos((tr(R_refᵢᵀ R_genᵢ) − 1) / 2) — geodesic rotation error, radians
TransErr	Σᵢ ‖t_genᵢ − t_refᵢ‖₂
CamMC	Σᵢ ‖[R|t]_genᵢ − [R|t]_refᵢ‖₂ — L2 over the flattened 3×4

All three are sums over frames and lower is better, matching the papers (CamCloneMaster's 1.49 / 2.37 / 3.50 are sums over 77 frames). *_per_frame means are emitted alongside, and are the quantity to use whenever clips differ in length.

Translations are scale-normalized first, each trajectory by its own max ‖t‖. This is not optional: both trajectories come from monocular reconstruction, so their scales are arbitrary and unrelated. It is CamI2V's _rel variant; the _abs variant aligns to a known ground-truth trajectory via a mono-depth scale factor, which does not apply here because both sides are estimated. The consequence to keep in mind: TransErr measures the shape of the translation path, not its magnitude.

The formulas are a direct port of CamI2V's reference implementation (ZGCTroy/CamI2V, evaluation/glomap_evaluation.py and evaluation/utils/common.py).

Read the errors against the control

Every metric is also reported as <metric>_chance: the static-camera control, an identity pose at every frame scored against the same reference. That is the chance level for this condition, and it is computed analytically so it is exact and free (which is why --shuffle-refs is a no-op here).

References differ enormously in motion magnitude, so a raw RotErr partly reports which camera the sample drew. <metric>_rel = <metric> / <metric>_chance puts them on one scale: below 1 means the generated shot beat a locked-off camera, at or above 1 means it did not.

Files

file	what it is
metric.py	the metric math. Pure torch/numpy on (N,4,4) pose stacks — no MegaSaM, no CUDA, no GPU.
megasam.py	drives the vendored MegaSaM pipeline for one video; caches poses by content hash.
eval.py	the evaluator run_eval.sh calls: pairs generated↔reference by stem, writes results.json / results.csv.
compare_metrics.py	the discrimination test — matched vs mismatched reference retrieval, new metric vs old, side by side.
test_metric.py	synthetic-trajectory checks for metric.py. Runs in any env with torch.
setup_megasam.sh	creates the flexbench-megasam env, builds the CUDA extensions, fetches checkpoints.

Setup

bash evaluation/eval_camera_new/setup_megasam.sh
conda run -n flexbench-megasam python -c \
  "import torch, droid_backends, lietorch, torch_scatter, xformers; print(torch.__version__)"

The pinned stack is python 3.10 / torch 2.4.0+cu121 / xformers 0.0.27.post2 / torch_scatter 2.1.2+pt24cu121, built against /usr/local/cuda-12.8. Do not bump casually: torch 2.4.0 is the newest version with a prebuilt torch_scatter wheel, a matching xformers (UniDepth imports it unconditionally via nystrom_attention), and a working DROID-SLAM build. The nvcc 12.8 / cu121-torch gap is a minor-version mismatch — cpp_extension warns and proceeds.

Usage

# via the pipeline
evaluation/run_eval.sh --generated-dir <run> --camera

# standalone
conda run -n flexbench-megasam python evaluation/eval_camera_new/eval.py \
    --generated-dir  <run>/videos \
    --dataset-dir    <dataset>/camera \
    --output-dir     <out>/camera \
    --pose-cache-dir <out>/camera/poses

Pose estimation costs roughly 2–4 minutes of GPU per clip and is cached by video content hash, not by filename — the FlexBench layout copies the same camera reference once per subject, so identical files are only reconstructed once. Re-scoring against cached poses is instant.

Pipeline

megasam.py runs three stages of the official MegaSaM repo (vendored unmodified at tools/mega-sam) as subprocesses:

1. Depth-Anything/run_videos.py — per-frame relative inverse depth
2. UniDepth/scripts/demo_mega-sam.py — per-frame metric depth + FOV
3. camera_tracking_scripts/test_demo.py — DROID-SLAM tracking + global bundle adjustment → poses

cvd_opt/ (consistent video depth) is deliberately skipped: it consumes the poses rather than producing them, so it cannot change the camera metric. That is also why no RAFT checkpoint is needed. MegaSaM's intermediates are large — the _droid.npz embeds full RGB and depth stacks — so everything except the (N,4,4) pose array and the 3×3 intrinsic is deleted per clip.

Frames are extracted as PNG only, never mixed with JPEG: test_demo.py globs *.jpg then *.png and concatenates the two sorted lists, so a mixed directory would reorder frames relative to the depth priors.

Limitations

• MegaSaM is a monocular reconstruction. On a low-parallax or heavily dynamic clip it can return a near-degenerate trajectory. Those samples are flagged (degenerate, ref_max_trans, gen_max_trans) rather than reported as if they were sound.
• Absolute scale is unrecoverable from monocular video — see the normalization note above.
• Camera motion is all this sees: not scene coherence, temporal flicker, or visual quality.