# Error decomposition and calibrated conservatism: experiment design

Status: design, 2026-07-26. Supersedes the ad-hoc conservatism sweep as the
paper's safety story. Nothing here is measured yet; every number below is a
placeholder for a run.

## Why redesign

The paper currently answers "how close is the ViT to the oracle?" by
comparing both **on the same terrain**. That measures amortization error and
nothing else. Two questions it cannot answer:

1. How much of the deployed error comes from the *terrain being wrong*
   rather than the *network being wrong*?
2. How much conservatism is actually needed, and can it be justified rather
   than swept?

The second is the paper's weakest section: `safety_buffer_m` is a knob tuned
until the false-safe rate looked acceptable. A reviewer is entitled to call
that fitting the metric.

## The setup already supports this, and one arm is missing

`make_dataset.py` draws **inputs from the noisy observation and labels from
the clean surface** (`noisy_observation`, "privileged information"). So the
ViT is not merely an amortizer — it is trained as a *denoising estimator*:
it maps a corrupted DEM to the value function of the true one.

`evaluate.py` computes `value_gt` on the clean grid and `value_base`
analytically on the observed grid. **There is no arm that runs the HJ oracle
on the observed terrain.** That is the missing cell, and it is the one that
matters: it is the honest version of "why not just run the solver?"

## The estimand

At cell `p`, truth is `V*(p)` — the joint HJ reach-avoid value on the true
surface with true dynamics. Every estimator is scored against it, and no
claim is made below the oracle's own discretization floor (grid 0.0022,
temporal 0.0030 — `results/{grid,temporal}_convergence.csv`).

## The 2x2, plus one control

| | on true terrain | on observed terrain |
|---|---|---|
| **exact solve** | **A** = `V*` (reference) | **B** = `HJ(z_obs)` ← *missing today* |
| **learned** | **E** = `ViT(z_true)` (control) | **D** = `ViT(z_obs)` (deployed) |

plus **C** = analytic margin on observed = today's baseline.

Clean decomposition, each term separately measurable:

- **pure amortization** `E − A` — network error with perfect terrain
- **pure perception** `B − A` — terrain error propagated through an exact solve
- **deployed** `D − A` — what the aircraft actually suffers
- **denoising credit** `|B − A| − |D − A|` — how much better (or worse) the
  learned map is than the exact solver *given the same corrupted input*

## E1 — The decomposition (core)

Compute A, B, C, D, E over held-out synthetic terrains x states x platforms.
Report MAE, signed-error quantiles, and false-safe rate for each arm.

**The hypothesis worth testing:** `|D − A| < |B − A|`. If the ViT beats the
exact solver on the DEM you actually have — because it was trained against
ground truth and learned the error structure — then the amortization argument
stops being "25x faster" and becomes *"running the exact solve on your real
data is worse than our map, and also slower."* That directly answers the
strongest reviewer objection to the paper, which is that a 25x speedup over a
0.78 s solve does not motivate learning anything.

If instead `|D − A| > |B − A|`, that is a real negative result and the
amortization claim stays a speed claim. Either outcome is publishable; the
experiment must be run without a preferred answer.

## E2 — The control that decides whether E1 means anything

An in-distribution win is nearly vacuous: the ViT saw this exact noise model
(spatially-correlated Gaussian, `corr_px=3`, sigma <= 1 m) during training, so
beating a noise-blind solver is unsurprising.

The claim only survives if the advantage holds under **error the network was
never trained on**:

- **structured bias** — error correlated with slope/aspect (a depth pipeline
  is worst on steep faces), not zero-mean
- **different correlation length** — `corr_px` well outside training
- **systematic offset** — a vertical datum error, constant over the tile
- **real residuals** — the actual error field from E5

Report the denoising credit separately for in-distribution and
out-of-distribution error. Only the latter supports the headline.

## E3 — Observation-error sweep and the DEM-quality spec

Sweep sigma from 0 to ~10 m, far past the 1 m training cap (`--z-noise-max`
defaults to 1.0, which is small next to real DEM error and tiny next to
depth-pipeline error). Locate:

- the crossover where perception error overtakes amortization error — beyond
  it, improving the network is wasted effort and the DEM is the bottleneck;
- where the ViT leaves its training distribution and degrades;
- the sigma at which false-safe crosses a deployment threshold.

Deliverable: **a DEM-quality requirement derived from measured propagation**
("this system needs elevation data better than X m RMSE"), which is an
actionable engineering spec and not currently anywhere in the paper.

## E4 — Conservatism as a certificate, not a knob

Replace the swept `safety_buffer_m` with **split conformal calibration**.
On a held-out calibration split, take the optimism residual
`r = max(0, V_pred − V*)` and set the buffer to its `(1−alpha)` quantile.
Applied at the safety threshold this bounds the false-safe rate at `alpha`
with a finite-sample, distribution-free guarantee — a proof rather than a
sweep, and the same knob the paper already ships.

Variants to compare:

1. **marginal** — one global buffer; valid but wasteful
2. **Mondrian / conditional** — binned by (relief, AGL, platform), giving
   *per-regime* validity. This is precisely the "bounded validity envelope"
   the limitations section asks for, and it addresses both the heavy-airframe
   deficit and the low-AGL/high-relief blow-up with a principled mechanism
   instead of a hand-off rule.
3. **margin-adaptive** — error concentrates at the footprint boundary, so
   scale the buffer with local `|margin|`.

**Headline figure: the price of the guarantee.** Fraction of the true safe
footprint retained at alpha in {1%, 5%, 10%}, per variant. Conservatism stops
being free and starts having a measured cost — which is the honest way to
present it.

**Stated limitation, not to be buried:** conformal validity needs
exchangeability between calibration and test. Held-out synthetic terrains
qualify. Real terrain does not — that is distribution shift, and it is
exactly why the real-DEM false-safe numbers diverge. Framing must be
"calibrated on synthetic, *measured* on real", never "guaranteed on real".

## E5 — Depth-from-imagery: can we fly without a prior DEM?

The onboard-realistic case, and the one that actually exercises the Depth
Anything lineage the paper leans on: run DA-V2 in its native role on NAIP RGB,
fit the affine (scale/shift) alignment monocular relative-inverse-depth
requires, and compare the resulting surface against 3DEP as truth. Then
propagate through the oracle and score with E1's arms.

Honest prior: nadir aerial imagery has weak monocular depth cues and severe
scale ambiguity, so absolute elevation will likely be poor even if relative
structure (ridges, valleys) survives. A bounded negative result here is a
genuine contribution — it justifies the DEM-based design against the
imagery-based landing-zone literature the related-work section cites, rather
than leaving the choice unargued. Its residual field also feeds E2 as the
most realistic available error model.

## E6 — Floors and controls

- **oracle discretization floor** — already measured; every claim reported
  relative to it, nothing asserted below it
- **E (ViT on clean input)** — the pure-amortization control the paper lacks
- **label-noise ceiling** — buffered (`b=40`) vs raw-oracle labels, since
  `val MAE 0.018` is agreement with the *buffered* label and is not
  comparable to raw-oracle numbers

## What this changes in the paper

Sections 5-6 currently report "ViT vs oracle, same terrain" plus a swept
conservatism knob. The replacement is: a complete error decomposition with
the missing exact-solver-on-observed arm, an out-of-distribution control that
decides whether the denoising claim is real, a DEM-quality spec, and a
conservatism mechanism with a finite-sample guarantee and a measured price.

The risk to manage: E1 may show the ViT does *not* beat the exact solver on
observed terrain. Design the writing so that outcome is still a contribution
(a clean statement of where learned maps help and where they do not) rather
than a result to be buried.