reach-down-vit-release / docs /error_decomposition_design.md

Publish complete fixed-wing ViT runs, dataset provenance, and evaluation artifacts

728cafc verified 17 days ago

8.59 kB

# Error decomposition and calibrated conservatism: experiment design

Status: design, 2026-07-26. Supersedes the ad-hoc conservatism sweep as the
paper's safety story. Nothing here is measured yet; every number below is a
placeholder for a run.

## Why redesign

The paper currently answers "how close is the ViT to the oracle?" by
comparing both **on the same terrain**. That measures amortization error and
nothing else. Two questions it cannot answer:

1. How much of the deployed error comes from the *terrain being wrong*
   rather than the *network being wrong*?
2. How much conservatism is actually needed, and can it be justified rather
   than swept?

The second is the paper's weakest section: `safety_buffer_m` is a knob tuned
until the false-safe rate looked acceptable. A reviewer is entitled to call
that fitting the metric.

## The setup already supports this, and one arm is missing

`make_dataset.py` draws **inputs from the noisy observation and labels from
the clean surface** (`noisy_observation`, "privileged information"). So the
ViT is not merely an amortizer — it is trained as a *denoising estimator*:
it maps a corrupted DEM to the value function of the true one.

`evaluate.py` computes `value_gt` on the clean grid and `value_base`
analytically on the observed grid. **There is no arm that runs the HJ oracle
on the observed terrain.** That is the missing cell, and it is the one that
matters: it is the honest version of "why not just run the solver?"

## The estimand

At cell `p`, truth is `V*(p)` — the joint HJ reach-avoid value on the true
surface with true dynamics. Every estimator is scored against it, and no
claim is made below the oracle's own discretization floor (grid 0.0022,
temporal 0.0030 — `results/{grid,temporal}_convergence.csv`).

## The 2x2, plus one control

| | on true terrain | on observed terrain |
|---|---|---|
| **exact solve** | **A** = `V*` (reference) | **B** = `HJ(z_obs)` ← *missing today* |
| **learned** | **E** = `ViT(z_true)` (control) | **D** = `ViT(z_obs)` (deployed) |

plus **C** = analytic margin on observed = today's baseline.

Clean decomposition, each term separately measurable:

- **pure amortization** `E − A` — network error with perfect terrain
- **pure perception** `B − A` — terrain error propagated through an exact solve
- **deployed** `D − A` — what the aircraft actually suffers
- **denoising credit** `|B − A| − |D − A|` — how much better (or worse) the
  learned map is than the exact solver *given the same corrupted input*

## E1 — The decomposition (core)

Compute A, B, C, D, E over held-out synthetic terrains x states x platforms.
Report MAE, signed-error quantiles, and false-safe rate for each arm.

**The hypothesis worth testing:** `|D − A| < |B − A|`. If the ViT beats the
exact solver on the DEM you actually have — because it was trained against
ground truth and learned the error structure — then the amortization argument
stops being "25x faster" and becomes *"running the exact solve on your real
data is worse than our map, and also slower."* That directly answers the
strongest reviewer objection to the paper, which is that a 25x speedup over a
0.78 s solve does not motivate learning anything.

If instead `|D − A| > |B − A|`, that is a real negative result and the
amortization claim stays a speed claim. Either outcome is publishable; the
experiment must be run without a preferred answer.

## E2 — The control that decides whether E1 means anything

An in-distribution win is nearly vacuous: the ViT saw this exact noise model
(spatially-correlated Gaussian, `corr_px=3`, sigma <= 1 m) during training, so
beating a noise-blind solver is unsurprising.

The claim only survives if the advantage holds under **error the network was
never trained on**:

- **structured bias** — error correlated with slope/aspect (a depth pipeline
  is worst on steep faces), not zero-mean
- **different correlation length** — `corr_px` well outside training
- **systematic offset** — a vertical datum error, constant over the tile
- **real residuals** — the actual error field from E5

Report the denoising credit separately for in-distribution and
out-of-distribution error. Only the latter supports the headline.

## E3 — Observation-error sweep and the DEM-quality spec

Sweep sigma from 0 to ~10 m, far past the 1 m training cap (`--z-noise-max`
defaults to 1.0, which is small next to real DEM error and tiny next to
depth-pipeline error). Locate:

- the crossover where perception error overtakes amortization error — beyond
  it, improving the network is wasted effort and the DEM is the bottleneck;
- where the ViT leaves its training distribution and degrades;
- the sigma at which false-safe crosses a deployment threshold.

Deliverable: **a DEM-quality requirement derived from measured propagation**
("this system needs elevation data better than X m RMSE"), which is an
actionable engineering spec and not currently anywhere in the paper.

## E4 — Conservatism as a certificate, not a knob

Replace the swept `safety_buffer_m` with **split conformal calibration**.
On a held-out calibration split, take the optimism residual
`r = max(0, V_pred − V*)` and set the buffer to its `(1−alpha)` quantile.
Applied at the safety threshold this bounds the false-safe rate at `alpha`
with a finite-sample, distribution-free guarantee — a proof rather than a
sweep, and the same knob the paper already ships.

Variants to compare:

1. **marginal** — one global buffer; valid but wasteful
2. **Mondrian / conditional** — binned by (relief, AGL, platform), giving
   *per-regime* validity. This is precisely the "bounded validity envelope"
   the limitations section asks for, and it addresses both the heavy-airframe
   deficit and the low-AGL/high-relief blow-up with a principled mechanism
   instead of a hand-off rule.
3. **margin-adaptive** — error concentrates at the footprint boundary, so
   scale the buffer with local `|margin|`.

**Headline figure: the price of the guarantee.** Fraction of the true safe
footprint retained at alpha in {1%, 5%, 10%}, per variant. Conservatism stops
being free and starts having a measured cost — which is the honest way to
present it.

**Stated limitation, not to be buried:** conformal validity needs
exchangeability between calibration and test. Held-out synthetic terrains
qualify. Real terrain does not — that is distribution shift, and it is
exactly why the real-DEM false-safe numbers diverge. Framing must be
"calibrated on synthetic, *measured* on real", never "guaranteed on real".

## E5 — Depth-from-imagery: can we fly without a prior DEM?

The onboard-realistic case, and the one that actually exercises the Depth
Anything lineage the paper leans on: run DA-V2 in its native role on NAIP RGB,
fit the affine (scale/shift) alignment monocular relative-inverse-depth
requires, and compare the resulting surface against 3DEP as truth. Then
propagate through the oracle and score with E1's arms.

Honest prior: nadir aerial imagery has weak monocular depth cues and severe
scale ambiguity, so absolute elevation will likely be poor even if relative
structure (ridges, valleys) survives. A bounded negative result here is a
genuine contribution — it justifies the DEM-based design against the
imagery-based landing-zone literature the related-work section cites, rather
than leaving the choice unargued. Its residual field also feeds E2 as the
most realistic available error model.

## E6 — Floors and controls

- **oracle discretization floor** — already measured; every claim reported
  relative to it, nothing asserted below it
- **E (ViT on clean input)** — the pure-amortization control the paper lacks
- **label-noise ceiling** — buffered (`b=40`) vs raw-oracle labels, since
  `val MAE 0.018` is agreement with the *buffered* label and is not
  comparable to raw-oracle numbers

## What this changes in the paper

Sections 5-6 currently report "ViT vs oracle, same terrain" plus a swept
conservatism knob. The replacement is: a complete error decomposition with
the missing exact-solver-on-observed arm, an out-of-distribution control that
decides whether the denoising claim is real, a DEM-quality spec, and a
conservatism mechanism with a finite-sample guarantee and a measured price.

The risk to manage: E1 may show the ViT does *not* beat the exact solver on
observed terrain. Design the writing so that outcome is still a contribution
(a clean statement of where learned maps help and where they do not) rather
than a result to be buried.

	# Error decomposition and calibrated conservatism: experiment design

	Status: design, 2026-07-26. Supersedes the ad-hoc conservatism sweep as the
	paper's safety story. Nothing here is measured yet; every number below is a
	placeholder for a run.

	## Why redesign

	The paper currently answers "how close is the ViT to the oracle?" by
	comparing both on the same terrain. That measures amortization error and
	nothing else. Two questions it cannot answer:

	1. How much of the deployed error comes from the terrain being wrong
	rather than the network being wrong?
	2. How much conservatism is actually needed, and can it be justified rather
	than swept?

	The second is the paper's weakest section: `safety_buffer_m` is a knob tuned
	until the false-safe rate looked acceptable. A reviewer is entitled to call
	that fitting the metric.

	## The setup already supports this, and one arm is missing

	`make_dataset.py` draws **inputs from the noisy observation and labels from
	the clean surface** (`noisy_observation`, "privileged information"). So the
	ViT is not merely an amortizer — it is trained as a denoising estimator:
	it maps a corrupted DEM to the value function of the true one.

	`evaluate.py` computes `value_gt` on the clean grid and `value_base`
	analytically on the observed grid. **There is no arm that runs the HJ oracle
	on the observed terrain.** That is the missing cell, and it is the one that
	matters: it is the honest version of "why not just run the solver?"

	## The estimand

	At cell `p`, truth is `V*(p)` — the joint HJ reach-avoid value on the true
	surface with true dynamics. Every estimator is scored against it, and no
	claim is made below the oracle's own discretization floor (grid 0.0022,
	temporal 0.0030 — `results/{grid,temporal}_convergence.csv`).

	## The 2x2, plus one control

	\| \| on true terrain \| on observed terrain \|
	\|---\|---\|---\|
	\| exact solve \| A = `V` (reference) \| B* = `HJ(z_obs)` ← missing today \|
	\| learned \| E = `ViT(z_true)` (control) \| D = `ViT(z_obs)` (deployed) \|

	plus C = analytic margin on observed = today's baseline.

	Clean decomposition, each term separately measurable:

	- pure amortization `E − A` — network error with perfect terrain
	- pure perception `B − A` — terrain error propagated through an exact solve
	- deployed `D − A` — what the aircraft actually suffers
	- denoising credit `\|B − A\| − \|D − A\|` — how much better (or worse) the
	learned map is than the exact solver given the same corrupted input

	## E1 — The decomposition (core)

	Compute A, B, C, D, E over held-out synthetic terrains x states x platforms.
	Report MAE, signed-error quantiles, and false-safe rate for each arm.

	The hypothesis worth testing: `\|D − A\| < \|B − A\|`. If the ViT beats the
	exact solver on the DEM you actually have — because it was trained against
	ground truth and learned the error structure — then the amortization argument
	stops being "25x faster" and becomes *"running the exact solve on your real
	data is worse than our map, and also slower."* That directly answers the
	strongest reviewer objection to the paper, which is that a 25x speedup over a
	0.78 s solve does not motivate learning anything.

	If instead `\|D − A\| > \|B − A\|`, that is a real negative result and the
	amortization claim stays a speed claim. Either outcome is publishable; the
	experiment must be run without a preferred answer.

	## E2 — The control that decides whether E1 means anything

	An in-distribution win is nearly vacuous: the ViT saw this exact noise model
	(spatially-correlated Gaussian, `corr_px=3`, sigma <= 1 m) during training, so
	beating a noise-blind solver is unsurprising.

	The claim only survives if the advantage holds under **error the network was
	never trained on**:

	- structured bias — error correlated with slope/aspect (a depth pipeline
	is worst on steep faces), not zero-mean
	- different correlation length — `corr_px` well outside training
	- systematic offset — a vertical datum error, constant over the tile
	- real residuals — the actual error field from E5

	Report the denoising credit separately for in-distribution and
	out-of-distribution error. Only the latter supports the headline.

	## E3 — Observation-error sweep and the DEM-quality spec

	Sweep sigma from 0 to ~10 m, far past the 1 m training cap (`--z-noise-max`
	defaults to 1.0, which is small next to real DEM error and tiny next to
	depth-pipeline error). Locate:

	- the crossover where perception error overtakes amortization error — beyond
	it, improving the network is wasted effort and the DEM is the bottleneck;
	- where the ViT leaves its training distribution and degrades;
	- the sigma at which false-safe crosses a deployment threshold.

	Deliverable: a DEM-quality requirement derived from measured propagation
	("this system needs elevation data better than X m RMSE"), which is an
	actionable engineering spec and not currently anywhere in the paper.

	## E4 — Conservatism as a certificate, not a knob

	Replace the swept `safety_buffer_m` with split conformal calibration.
	On a held-out calibration split, take the optimism residual
	`r = max(0, V_pred − V*)` and set the buffer to its `(1−alpha)` quantile.
	Applied at the safety threshold this bounds the false-safe rate at `alpha`
	with a finite-sample, distribution-free guarantee — a proof rather than a
	sweep, and the same knob the paper already ships.

	Variants to compare:

	1. marginal — one global buffer; valid but wasteful
	2. Mondrian / conditional — binned by (relief, AGL, platform), giving
	per-regime validity. This is precisely the "bounded validity envelope"
	the limitations section asks for, and it addresses both the heavy-airframe
	deficit and the low-AGL/high-relief blow-up with a principled mechanism
	instead of a hand-off rule.
	3. margin-adaptive — error concentrates at the footprint boundary, so
	scale the buffer with local `\|margin\|`.

	Headline figure: the price of the guarantee. Fraction of the true safe
	footprint retained at alpha in {1%, 5%, 10%}, per variant. Conservatism stops
	being free and starts having a measured cost — which is the honest way to
	present it.

	Stated limitation, not to be buried: conformal validity needs
	exchangeability between calibration and test. Held-out synthetic terrains
	qualify. Real terrain does not — that is distribution shift, and it is
	exactly why the real-DEM false-safe numbers diverge. Framing must be
	"calibrated on synthetic, measured on real", never "guaranteed on real".

	## E5 — Depth-from-imagery: can we fly without a prior DEM?

	The onboard-realistic case, and the one that actually exercises the Depth
	Anything lineage the paper leans on: run DA-V2 in its native role on NAIP RGB,
	fit the affine (scale/shift) alignment monocular relative-inverse-depth
	requires, and compare the resulting surface against 3DEP as truth. Then
	propagate through the oracle and score with E1's arms.

	Honest prior: nadir aerial imagery has weak monocular depth cues and severe
	scale ambiguity, so absolute elevation will likely be poor even if relative
	structure (ridges, valleys) survives. A bounded negative result here is a
	genuine contribution — it justifies the DEM-based design against the
	imagery-based landing-zone literature the related-work section cites, rather
	than leaving the choice unargued. Its residual field also feeds E2 as the
	most realistic available error model.

	## E6 — Floors and controls

	- oracle discretization floor — already measured; every claim reported
	relative to it, nothing asserted below it
	- E (ViT on clean input) — the pure-amortization control the paper lacks
	- label-noise ceiling — buffered (`b=40`) vs raw-oracle labels, since
	`val MAE 0.018` is agreement with the buffered label and is not
	comparable to raw-oracle numbers

	## What this changes in the paper

	Sections 5-6 currently report "ViT vs oracle, same terrain" plus a swept
	conservatism knob. The replacement is: a complete error decomposition with
	the missing exact-solver-on-observed arm, an out-of-distribution control that
	decides whether the denoising claim is real, a DEM-quality spec, and a
	conservatism mechanism with a finite-sample guarantee and a measured price.

	The risk to manage: E1 may show the ViT does not beat the exact solver on
	observed terrain. Design the writing so that outcome is still a contribution
	(a clean statement of where learned maps help and where they do not) rather
	than a result to be buried.