docs/error_decomposition_design.md

Error decomposition and calibrated conservatism: experiment design

Status: design, 2026-07-26. Supersedes the ad-hoc conservatism sweep as the paper's safety story. Nothing here is measured yet; every number below is a placeholder for a run.

Why redesign

The paper currently answers "how close is the ViT to the oracle?" by comparing both on the same terrain. That measures amortization error and nothing else. Two questions it cannot answer:

1. How much of the deployed error comes from the terrain being wrong rather than the network being wrong?
2. How much conservatism is actually needed, and can it be justified rather than swept?

The second is the paper's weakest section: safety_buffer_m is a knob tuned until the false-safe rate looked acceptable. A reviewer is entitled to call that fitting the metric.

The setup already supports this, and one arm is missing

make_dataset.py draws inputs from the noisy observation and labels from the clean surface (noisy_observation, "privileged information"). So the ViT is not merely an amortizer — it is trained as a denoising estimator: it maps a corrupted DEM to the value function of the true one.

evaluate.py computes value_gt on the clean grid and value_base analytically on the observed grid. There is no arm that runs the HJ oracle on the observed terrain. That is the missing cell, and it is the one that matters: it is the honest version of "why not just run the solver?"

The estimand

At cell p, truth is V*(p) — the joint HJ reach-avoid value on the true surface with true dynamics. Every estimator is scored against it, and no claim is made below the oracle's own discretization floor (grid 0.0022, temporal 0.0030 — results/{grid,temporal}_convergence.csv).

The 2x2, plus one control

	on true terrain	on observed terrain
exact solve	A = V* (reference)	B = HJ(z_obs) ← missing today
learned	E = ViT(z_true) (control)	D = ViT(z_obs) (deployed)

plus C = analytic margin on observed = today's baseline.

Clean decomposition, each term separately measurable:

• pure amortization E − A — network error with perfect terrain
• pure perception B − A — terrain error propagated through an exact solve
• deployed D − A — what the aircraft actually suffers
• denoising credit |B − A| − |D − A| — how much better (or worse) the learned map is than the exact solver given the same corrupted input

E1 — The decomposition (core)

Compute A, B, C, D, E over held-out synthetic terrains x states x platforms. Report MAE, signed-error quantiles, and false-safe rate for each arm.

The hypothesis worth testing: |D − A| < |B − A|. If the ViT beats the exact solver on the DEM you actually have — because it was trained against ground truth and learned the error structure — then the amortization argument stops being "25x faster" and becomes "running the exact solve on your real data is worse than our map, and also slower." That directly answers the strongest reviewer objection to the paper, which is that a 25x speedup over a 0.78 s solve does not motivate learning anything.

If instead |D − A| > |B − A|, that is a real negative result and the amortization claim stays a speed claim. Either outcome is publishable; the experiment must be run without a preferred answer.

E2 — The control that decides whether E1 means anything

An in-distribution win is nearly vacuous: the ViT saw this exact noise model (spatially-correlated Gaussian, corr_px=3, sigma <= 1 m) during training, so beating a noise-blind solver is unsurprising.

The claim only survives if the advantage holds under error the network was never trained on:

• structured bias — error correlated with slope/aspect (a depth pipeline is worst on steep faces), not zero-mean
• different correlation length — corr_px well outside training
• systematic offset — a vertical datum error, constant over the tile
• real residuals — the actual error field from E5

Report the denoising credit separately for in-distribution and out-of-distribution error. Only the latter supports the headline.

E3 — Observation-error sweep and the DEM-quality spec

Sweep sigma from 0 to ~10 m, far past the 1 m training cap (--z-noise-max defaults to 1.0, which is small next to real DEM error and tiny next to depth-pipeline error). Locate:

• the crossover where perception error overtakes amortization error — beyond it, improving the network is wasted effort and the DEM is the bottleneck;
• where the ViT leaves its training distribution and degrades;
• the sigma at which false-safe crosses a deployment threshold.

Deliverable: a DEM-quality requirement derived from measured propagation ("this system needs elevation data better than X m RMSE"), which is an actionable engineering spec and not currently anywhere in the paper.

E4 — Conservatism as a certificate, not a knob

Replace the swept safety_buffer_m with split conformal calibration. On a held-out calibration split, take the optimism residual r = max(0, V_pred − V*) and set the buffer to its (1−alpha) quantile. Applied at the safety threshold this bounds the false-safe rate at alpha with a finite-sample, distribution-free guarantee — a proof rather than a sweep, and the same knob the paper already ships.

Variants to compare:

1. marginal — one global buffer; valid but wasteful
2. Mondrian / conditional — binned by (relief, AGL, platform), giving per-regime validity. This is precisely the "bounded validity envelope" the limitations section asks for, and it addresses both the heavy-airframe deficit and the low-AGL/high-relief blow-up with a principled mechanism instead of a hand-off rule.
3. margin-adaptive — error concentrates at the footprint boundary, so scale the buffer with local |margin|.

Headline figure: the price of the guarantee. Fraction of the true safe footprint retained at alpha in {1%, 5%, 10%}, per variant. Conservatism stops being free and starts having a measured cost — which is the honest way to present it.

Stated limitation, not to be buried: conformal validity needs exchangeability between calibration and test. Held-out synthetic terrains qualify. Real terrain does not — that is distribution shift, and it is exactly why the real-DEM false-safe numbers diverge. Framing must be "calibrated on synthetic, measured on real", never "guaranteed on real".

E5 — Depth-from-imagery: can we fly without a prior DEM?

The onboard-realistic case, and the one that actually exercises the Depth Anything lineage the paper leans on: run DA-V2 in its native role on NAIP RGB, fit the affine (scale/shift) alignment monocular relative-inverse-depth requires, and compare the resulting surface against 3DEP as truth. Then propagate through the oracle and score with E1's arms.

Honest prior: nadir aerial imagery has weak monocular depth cues and severe scale ambiguity, so absolute elevation will likely be poor even if relative structure (ridges, valleys) survives. A bounded negative result here is a genuine contribution — it justifies the DEM-based design against the imagery-based landing-zone literature the related-work section cites, rather than leaving the choice unargued. Its residual field also feeds E2 as the most realistic available error model.

E6 — Floors and controls

• oracle discretization floor — already measured; every claim reported relative to it, nothing asserted below it
• E (ViT on clean input) — the pure-amortization control the paper lacks
• label-noise ceiling — buffered (b=40) vs raw-oracle labels, since val MAE 0.018 is agreement with the buffered label and is not comparable to raw-oracle numbers

What this changes in the paper

Sections 5-6 currently report "ViT vs oracle, same terrain" plus a swept conservatism knob. The replacement is: a complete error decomposition with the missing exact-solver-on-observed arm, an out-of-distribution control that decides whether the denoising claim is real, a DEM-quality spec, and a conservatism mechanism with a finite-sample guarantee and a measured price.

The risk to manage: E1 may show the ViT does not beat the exact solver on observed terrain. Design the writing so that outcome is still a contribution (a clean statement of where learned maps help and where they do not) rather than a result to be buried.