sensorflow/seqeval/DESIGN.md

# seqeval — Efficient Sequential Regression Detection: Technical Design

**Goal.** Detect small model regressions (~2 percentage points on specific
classes/conditions) on every model update WITHOUT evaluating the full corpus,
with statistical validity under (a) sequential looks, (b) clustered
(non-independent) frames, (c) stratified non-proportional sampling, and
(d) many simultaneous hypotheses.

This document records the design *and the judgment calls*: which candidate
techniques were adopted, which were rejected, and why. The reference
implementation lives in this package and is exercised by `tests/test_seqeval/`.

---

## 1. Problem formulation

For each metric m (recall is the driver metric in v1) and each population
stratum h, let Δ_h = m_candidate(h) − m_baseline(h) measured on the same data.

With a **practical-significance margin** δ (default 0.005 = 0.5pp):

```
H0 (no material regression):  Δ_h ≥ −δ
H1 (material regression):     Δ_h < −δ
```

plus the separate equivalence-style claim used for PASS: reject `Δ_h ≤ −δ`
in favor of `Δ_h > −δ`.

**Absolute vs relative deltas.** The default driver is **absolute percentage
points**, for three reasons:

1. Safety cost is absolute: a 2pp recall drop means the same number of extra
   missed pedestrians whether baseline recall was 0.98 or 0.80. Relative
   deltas overstate changes on strong baselines (0.98→0.96 is "only −2%
   relative") and understate them on weak ones.
2. Absolute deltas keep Bernoulli variance algebra and margin semantics simple
   and uniform across strata; relative margins make the null hypothesis depend
   on the (estimated) baseline value, contaminating the test with baseline
   estimation error.
3. Absolute margins are what promotion policies elsewhere in this platform
   already use (`megaeval.analysis.DEFAULT_PROMOTION_POLICY` is expressed in
   absolute drops), so the gate composes with existing policy language.

Relative deltas are still *reported* in every evidence record (`rel_delta`)
because they are useful for human interpretation, especially for rare classes
with weak baselines. They are never the decision variable.

**Practical vs statistical significance.** With ~320k objects, a 0.05pp drop
can be "statistically significant" yet operationally meaningless. Hence the
margin δ: nothing is declared REGRESSION unless the evidence shows the drop
exceeds δ, and nothing is declared PASS unless the evidence shows any drop is
smaller than δ. Statistical significance against a point null of exactly zero
is deliberately not used anywhere.

## 2. The statistical unit: clusters, not frames

Frames/objects are not independent: objects in one container (scene/track)
share weather, lighting, scenario, scene-level hardness and — critically —
correlated model failures (one bad scene produces many misses). Treating
objects as iid understates variance by the design effect

```
DEFF = 1 + (m̄ − 1) · ICC        (m̄ = size-weighted mean cluster size)
```

and every "significant" result computed that way is anti-conservative.

Adopted design:

* **Inference unit = cluster**: all sampled objects of a stratum in one
  container. Per-cluster mean paired differences (bounded in [−1, 1]) are the
  observations fed to the confidence sequences and e-processes. This is valid
  under *arbitrary* within-cluster correlation — no correlation model is
  assumed — and whole clusters are sampled so the unit is never split
  (`planner.py` selects container groups, not objects).
* **Diagnostics, not corrections**: ICC (one-way ANOVA estimator), DEFF and
  n_eff = n/DEFF are computed per node (`units.py`) and attached to every
  evidence record. They are reporting artifacts; validity does not depend on
  them because clustering is handled structurally.
* An interesting empirical note the machinery surfaces: pairing itself removes
  most cluster correlation (the shared scene effect hits both models and
  cancels in d_i), so ICC on paired differences is small (~0.07 in the worked
  example, DEFF ≈ 1.2) even when ICC on raw outcomes is large. We keep the
  cluster unit anyway — the residual correlation is real and free to handle.
* **Clustered bootstrap** (resampling clusters, not objects) is the prescribed
  fallback for metrics that don't decompose into per-object outcomes (§3). It
  is not needed for the v1 recall driver.

Rejected: variance-inflation-only corrections (divide n by an assumed DEFF≈2
and pretend units are iid). Rejected because ICC varies by stratum and model
pair; a fixed inflation is either wasteful or invalid, and it composes poorly
with anytime-valid methods whose guarantees are per-observation.

## 3. Paired evaluation as the core design

Both models are evaluated on the **same sampled units** (`paired.py`), and all
inference is on the per-unit difference d_i = c_i − b_i ∈ {−1, 0, +1}.

Why this is the single biggest efficiency lever: for two independent samples,
Var(m̂_c − m̂_b) ≈ 2·p(1−p)/n (~0.18/n at p=0.9). For paired evaluation,
Var(d) = ν − Δ², where ν = P(models disagree) — the discordance rate. Adjacent
model versions agree on almost everything (measured ν ≈ 0.03–0.06 in the
harness), so pairing cuts required samples by roughly 3–10×. The paired
design also removes the between-unit difficulty variance *and* the shared
scene effects (§2).

**McNemar correspondence.** For binary per-object outcomes the information in
the paired design is exactly the discordant counts: n01 (baseline right,
candidate wrong) vs n10 (the reverse) — McNemar's setup. We do not use the
classical McNemar *test* (it is fixed-n); instead the same discordant-pair
information drives (a) the sequential e-process on d_i and (b) the
Beta-Binomial posterior of §14, which is literally a Bayesian McNemar. The
counts n01/n10 are reported in attribution because they are the most
interpretable summary of "what changed".

**Which metrics get which treatment:**

| Metric | Per-object outcome? | Sequential treatment |
|---|---|---|
| recall (per-GT-object TP/FN) | yes, Bernoulli | confidence sequence / e-process on paired d_i ✔ (v1 driver) |
| precision (per-prediction) | yes, but the unit set differs per model (each model's own predictions); pair at the cluster level | CS on cluster-level paired precision differences |
| safety_recall, per-stratum recall | yes | same as recall ✔ |
| mean IoU (matched pairs) | bounded continuous per object | CS on paired IoU differences (empirical-Bernstein handles [0,1] naturally) |
| AP / mAP / AUC | **no** — rank statistics over the whole set; not a mean of per-object terms | **paired clustered bootstrap** at checkpoints (resample clusters, recompute both models' AP on the same resample, CI on the difference). No anytime-valid guarantee: schedule at *pre-registered* checkpoints (end of each budget stage) with alpha-spending across the (few, fixed) looks |

This split is a deliberate honesty boundary: Bernoulli-style anytime-valid
treatment is used exactly where a metric is a mean of bounded per-unit terms;
composite rank metrics get bootstrap at a small number of pre-committed looks
rather than pretending they support continuous monitoring.

## 4. Sampling design: hybrid Neyman + risk floors + safety minimums

Candidates considered:

* **Proportional allocation** — matches the population, but starves exactly
  the strata we care about (pedestrian|night is ~5% of objects; a proportional
  10k sample gives ~500 units, hopeless for 2pp detection). Rejected as the
  primary rule.
* **Equal allocation** — maximal per-stratum power for fixed budget, but
  wastes budget on huge easy strata and makes population-level estimates
  needlessly noisy. Rejected.
* **Risk-weighted only** (as in `megaeval.sampling`) — right instinct, no
  optimality story; can be arbitrarily far from variance-optimal. Rejected as
  the *sole* rule, kept as a multiplier.
* **Neyman allocation** n_h ∝ N_h·σ_h — variance-optimal for the *population*
  estimate, but per-stratum guarantees are incidental, and σ_h must not come
  from candidate outcomes (bias firewall, §13). Adopted as the baseline using
  a candidate-independent PRIOR variance profile (documented rates by
  class/condition; a pilot baseline-only estimate would also qualify).

**Adopted hybrid** (`planner.py`), in order: Neyman shares from prior σ_h →
multiplicative risk weights (VRU ×1.5, night ×1.3) → mandatory minimum per
stratum (200) → pre-registered safety-critical strata get a floor sized for
the target MDE (2500 by default, §10) → truncate at N_h, select whole
clusters in a seeded shuffled order.

**Unbiasedness under oversampling.** Every sampled object carries the
Horvitz–Thompson weight w_h = N_h/n_h. Population and class-level estimators
are weight-corrected (weighted cluster means feed the overall/class/difficulty
nodes), so the classic stratified identity `E[Σ_h W_h p̂_h] = p` holds and
oversampling safety strata does not bias headline estimates. Per-stratum
(leaf) inference is self-weighting and needs no correction.

## 5. Multi-stage pipeline

```
sanity (≈500 units)  →  stratified screening (~15% of plan)
    →  sequential confirmation (batched to 100% of plan, anytime-valid)
    →  targeted escalation (reserve clusters, suspect/safety strata only)
```

* **Sanity**: fingerprints resolved, frozen plan hashed, smoke batch checks
  for degenerate outputs (rate ∉ (0.2, 1], discordance ≥ 0.5 ⇒ abort — a
  broken export should fail in seconds, not consume the budget).
* **Screening** flags "suspect" strata early (e-value ≥ 1.5 — soft evidence,
  *never* a decision) to prioritize escalation later. Screening consumes the
  same frozen plan; it does not select units based on outcomes (§13).
* **Sequential confirmation** processes the remaining plan in batches; all
  decisions (REGRESSION / PASS) can fire at any batch boundary because the
  underlying processes are anytime-valid.
* **Escalation** pulls from each stratum's pre-frozen reserve (shuffled at
  plan time) for strata still undecided that are suspect or safety-primary,
  up to a hard cap. Escalation to *full* evaluation of a stratum is the
  terminal step of the same mechanism (the reserve is the rest of the
  stratum).

## 6. Sequential methodology: anytime-valid confidence sequences, not CUSUM

**Why repeated fixed-sample CIs are wrong.** A 95% CI computed at n has 5%
miscoverage *at that single n*. Recompute after every batch and stop on the
first exclusion of the null and the type-I error inflates monotonically —
by the law of the iterated logarithm the statistic crosses any fixed-n
boundary infinitely often under the null, so "peek until significant" has
error probability → 1. Evaluation gates that check after every batch are
exactly this setting.

**Adopted: confidence sequences / e-processes** (`sequential.py`):

* Empirical-Bernstein confidence sequence (Waudby-Smith & Ramdas 2023,
  predictable plug-in): a CS valid *simultaneously over all n*, so the gate
  may look after every batch and stop whenever it likes. Empirical-Bernstein
  rather than Hoeffding because paired differences have tiny variance
  (ν ≪ 1/4) and EB width scales with the realized variance; a Hoeffding CS
  would be ~3–4× wider here (Hoeffding rejected for power, kept as a mental
  baseline).
* One-sided **betting e-processes** for the two directed nulls. The wealth
  process K_t = Π(1 − λ_t(X_t − m)) is a nonnegative supermartingale under
  H0: μ ≥ m for any predictable λ_t in the admissible range; Ville's
  inequality gives P(∃t: K_t ≥ 1/α) ≤ α. Bet sizing = max(empirical-Bernstein
  recipe, aGRAPA/approximate-Kelly λ ≈ drift/(σ̂² + drift²)), clipped —
  sizing affects power only, never validity. E-values (not just CIs) are kept
  because they compose: they feed e-BH in the hierarchy (§8) and multiply
  across independent looks/runs if evidence is ever pooled.
* Batch-incremental updates, sticky decisions, per-batch trajectory recording
  (`log e vs n` with the 1/α boundary) for the dashboard.

**Where CUSUM legitimately fits — and why not here.** CUSUM/Page tests are
designed for *change-point detection in an ongoing stream*: an unknown time
at which a monitored process shifts, with guarantees phrased as average run
length (ARL) to false alarm and detection delay. That is the right tool for
**production drift monitoring** (a deployed model's live metric degrading at
an unknown time — a reasonable future addition next to
`sensorflow/evaluation/regression.py`). It is the wrong tool for **gated A/B
evaluation**, where there is no change-point: the candidate is a fixed,
already-different model, the hypothesis is about a fixed mean difference, ARL
semantics don't map to per-release type-I error, and CUSUM provides neither
confidence intervals for effect size nor e-values for hierarchical multiple
testing. CUSUM is therefore explicitly rejected for the gate.
(Group-sequential designs — O'Brien-Fleming etc. — were also considered:
valid, but they require pre-committing to a fixed number of looks and a
maximum n, which fights the budget-staged, stop-any-time controller; the
anytime-valid framework strictly dominates for this use.)

## 7. Mandatory three-outcome decisions

Non-detection is NEVER equivalence. Each node ends in exactly one of:

* **REGRESSION** — the regression e-process crossed 1/α_allocated: confirmed
  drop beyond the margin. Gate: block.
* **PASS** — the equivalence e-process crossed 1/α_pass: confirmed that any
  drop is smaller than the margin (this is a one-sided equivalence /
  non-inferiority claim, the sequential analogue of a TOST bound). Gate:
  allow, but only when the overall node AND every pre-registered safety
  primary individually PASS.
* **INSUFFICIENT_EVIDENCE** — budget exhausted first. Gate: expand within
  budget if possible, else report with explicit language: *"not proven
  equivalent — do not treat as a pass."* (`controller.INSUFFICIENT_LANGUAGE`
  is exactly that sentence; the tiny-budget test asserts it.)

## 8. Hierarchical multiple testing

Testing ~28 nodes (overall, 6 classes, 18 class×condition strata, 3 difficulty
bands) at α each would give a family error of ~1−0.95²⁸ ≈ 76%. Naive
Bonferroni over everything destroys power for the one node that matters.
Adopted structure (`hierarchy.py`):

```
Level 1  overall                        α·0.30   single test
Safety   pre-registered primaries       α·0.30   split equally, tested individually
Level 2  class                          α·0.15   e-BH within level
Level 3  class × condition strata       α·0.15   e-BH within level (non-safety)
Level 4  difficulty bands               α·0.10   e-BH within level
```

Budgets sum to α, so the union bound gives family-wise control across levels;
within a level, **e-BH** (BH on e-values) controls FDR *under arbitrary
dependence* — mandatory here because levels re-aggregate the same units and
strata share scenes. e-BH composes with optional stopping because an
e-process value at any stopping time is a valid e-value (this is the reason
the implementation carries e-values and not just CSs).

**Gatekeeping and the masking trap.** Classic top-down gatekeeping ("test
children only where the parent rejected") was **rejected** for the regression
direction: a 2pp pedestrian-night drop inside a globally *improved* model is
invisible at level 1, and a gate that only descends on level-1 rejection would
never see it. Instead every level has its own reserved budget, and
**safety-critical strata are pre-registered primary hypotheses** with
dedicated alpha, tested individually and exempt from e-BH masking by
better-behaved siblings. The power test in the suite is exactly this case:
overall metric up ~0.3pp, pedestrian-night down 2pp — and the stratum node
must fire. Alpha propagation is top-down only in the *reporting* sense
(attribution descends into difficulty bands of regressed strata); decisions
never require a parent rejection.

## 9. Rare-class policy

* **Minimum counts**: every non-empty stratum gets ≥ 200 sampled objects
  (or all of N_h if smaller) — below that, even 10pp effects are undetectable
  and the honest answer is INSUFFICIENT_EVIDENCE, which the three-outcome
  logic produces naturally.
* **Oversampling with weight correction**: rare + risky strata get risk
  multipliers and floors (§4); HT weights keep population estimates unbiased.
* **Per-class thresholds**: the margin δ and alpha allocation are policy
  fields; safety primaries effectively run at tighter per-node alpha with far
  larger minimum samples. (A per-stratum δ_h map is a straightforward policy
  extension; v1 keeps a single δ for legibility.)
* **Escalation**: rare strata that end INSUFFICIENT but suspect consume their
  reserve up to the whole stratum. If a stratum is exhausted and still
  undecided, the ledger says so explicitly (stopping_reason
  `escalation_exhausted`) — the decision is a visible budget/detectability
  statement, not silence.

## 10. Sample sizes and power (with the design effect)

Fixed-n planning numbers for a Bernoulli recall metric at baseline p₀ = 0.9,
one-sided α = 0.05, power 0.9 ((z₀.₉₅+z₀.₉)² ≈ 8.57). Paired variance uses
Var(d) ≈ ν − Δ² with measured-typical discordance ν ≈ 0.04 + Δ:

| detectable Δ | unpaired, per model arm | paired, iid units | paired, DEFF = 2 | paired, DEFF = 2, anytime (~×1.35) |
|---:|---:|---:|---:|---:|
| 1pp  | 16,100 | 4,280 | 8,550 | ~11,500 |
| 2pp  |  4,150 | 1,280 | 2,550 |  ~3,450 |
| 5pp  |    745 |   300 |   600 |    ~810 |
| 10pp |    214 |   111 |   223 |    ~300 |

Notes: (i) pairing beats doubling the corpus — the unpaired column is per
*arm*; (ii) the anytime-valid overhead (~30–40% at these scales, shrinking
with the e-process's adaptive bets) buys the right to stop at any time, which
in expectation *saves* samples whenever the effect is larger than the MDE
(the 10pp test stops ~15× before budget); (iii) measured DEFF on paired
differences was ≈1.2, not 2 — the table's DEFF=2 column is the conservative
planning figure requested, the ledger reports the realized one. The default
safety floor (2,500) is the 2pp row with the realized design effect plus
head-room.

## 11. Compute-budget staging

Default policy on a 320k corpus: plan ≈ 25k objects, staged
`[15%, 40%, 100%]` of plan (≈ 3.8k → 10k → 25k) + escalation reserve, i.e. a
10K→50K→250K→full ladder scaled to plan size. Advancement rules: advance
while any node is undecided and budget remains; stop immediately on any
confirmed REGRESSION (sticky, anytime-valid) or when overall + all safety
primaries PASS with nothing undecided; escalate only suspect/safety strata;
terminal state otherwise INSUFFICIENT_EVIDENCE with per-stratum sample
accounting in the ledger.

## 12. Prediction caching

Baseline predictions are computed once per (dataset fingerprint, model
fingerprint) and reused across every candidate update
(`paired.PredictionCache`, disk + memory). Fingerprints cover everything
predictions depend on — dataset identity/seed/size, model version + effects
config — so **invalidation is by construction**: any change produces a new
key; nothing stale can ever be served. A compute counter proves in tests that
the second candidate evaluation performs zero baseline recomputation. In a
real deployment the fingerprint inputs become dataset content hash + model
checkpoint digest + inference config (thresholds, NMS, quantization), and the
cache stores per-unit predictions rather than a success vector.

## 13. Sampling-bias prevention

* The plan is built by a function that **cannot see candidate outcomes** (no
  such parameter exists), from population structure + a prior variance
  profile only, then persisted with seed, config and a content hash over the
  selected and reserve unit ids. The hash is recorded in every run's lineage;
  two candidates with opposite outcomes provably receive byte-identical plans
  (asserted in tests).
* Escalation draws from a reserve whose order was frozen at plan time, so
  even adaptive *budget* decisions never become adaptive *selection*.
* **If outcome-adaptive sampling is ever added** (e.g. active sampling of
  high-disagreement regions), estimates must be corrected with
  inverse-propensity weights w_i = 1/π_i under recorded, everywhere-positive
  selection probabilities π_i, and CS validity must be re-established for the
  weighted observations (bounded w_i·d_i, e.g. via truncated propensities).
  Without that correction, adaptive selection biases Δ̂ toward whatever the
  selector chases. v1 deliberately does not do adaptive selection.

## 14. Bayesian complement

Alongside every frequentist decision, `sequential.py` computes a Beta-Binomial
posterior P(Δ < −δ | data) from the discordant pairs (θ = P(regression pair |
discordant) with Beta(1,1) prior; Δ = ν(1−2θ)). Operational comparison:

* *For* the posterior: a single monotone number ("99.99% probable this is a
  real regression") that program managers read correctly without training;
  natural for a decision-theoretic cost tradeoff.
* *Against* (as the gate): posteriors have no frequentist stopping guarantee
  under continuous monitoring without committing to a prior + loss and
  accepting their audit burden; e-values give the same "accumulating
  evidence" narrative *with* an error guarantee any auditor can check
  (e ≥ 1/α), and they compose across strata (e-BH) and across looks by
  multiplication.

Verdict: **e-values gate, posterior explains.** Both are in the ledger.

## 15. Failure modes and mitigations

| Failure mode | Mitigation in this design |
|---|---|
| Peeking inflates type-I | everything anytime-valid; fixed-n intervals never gate (§6) |
| Correlated frames overstate n | cluster inference units; DEFF/n_eff in every record (§2) |
| Masking: overall improves, stratum regresses | reserved per-level alpha + pre-registered safety primaries (§8) |
| Multiplicity across strata | hierarchical budgets + e-BH within level (§8) |
| "No signal" read as "safe" | three-outcome logic; INSUFFICIENT carries "not proven equivalent" (§7) |
| Outcome-dependent sampling bias | frozen hashed plans, frozen reserves, no adaptive selection (§13) |
| Rare strata silently undecidable | floors, oversampling+weights, escalation, explicit exhaustion reason (§9) |
| Stale baseline predictions | fingerprint-keyed cache, invalidation by construction (§12) |
| Broken candidate export burns budget | sanity smoke stage aborts in ~500 units (§5) |
| Trivial-magnitude "significance" | practical margin δ in both null hypotheses (§1) |
| Irreproducible decisions | seeds + fingerprints + plan hash + config in lineage; end-to-end determinism asserted in tests |
| Metric doesn't decompose per object (mAP) | pre-registered checkpoint paired clustered bootstrap, not fake sequential (§3) |

## 16. Final recommendation (defaults shipped in `controller.DEFAULT_POLICY`)

Paired, cluster-unit, stratified evaluation against a frozen hybrid
Neyman/risk-floor plan; empirical-Bernstein confidence sequences + betting
e-processes with a 0.5pp practical margin; three-outcome decisions; hierarchical
alpha budget (30% overall / 30% safety primaries / 15% class / 15% stratum /
10% difficulty) with e-BH within levels; staged budget 15%→40%→100% of a ~25k
plan with targeted escalation; fingerprint-keyed baseline cache; Beta-Binomial
posterior as explanatory complement. CUSUM reserved for future production
drift monitoring, not release gating.

## 17. Worked example: planted 2pp pedestrian-night regression (real run)

Setup: population `320k` objects / 26,666 containers (megaeval synthetic,
seed 42). Baseline `model-v41`. Candidate `model-v42` with planted effects
{pedestrian|night: −2pp, global: +0.4pp} — the masking case. Default policy
(δ = 0.5pp, α = 0.05, plan target 24k). Ground truth measured on the full
corpus afterwards: overall Δ = **+0.28pp** (candidate looks *better*
headline-wise), pedestrian|night Δ = **−1.63pp** realized (N = 17,561,
baseline recall 0.801).

Run (deterministic, reproducible from lineage):

* Plan: 25,699 objects frozen across 18 strata; pedestrian|night allocated
  2,501 (floor-driven; weight w = 7.02) + 15,060-object reserve.
* Decision: **REGRESSION** at 26,705 objects total = **8.4% of the corpus**
  (~12× cheaper than a full paired evaluation), stopping reason
  `regression_confirmed`, during escalation of the suspect safety primary.
* Pedestrian|night node at stop: n = 3,507 objects in 1,195 clusters
  (ICC 0.065, DEFF 1.20, n_eff 2,923); Δ̂ = −1.94pp,
  95% CS [−3.97pp, −0.48pp]; discordant pairs n01 = 110 vs n10 = 42;
  log e-value **7.50** vs boundary log(1/α_safety) = **4.89**
  (e ≈ 1,808 ≥ 133); Beta-Binomial P(Δ < −δ | data) = 0.99999.
* Evidence trajectory (n, log e): (129, 1.5) → (1002, 1.6) → (2501, 2.8) →
  (3006, 4.9) → (3507, 7.5) — the dashboard plots exactly this against the
  4.89 boundary.
* Overall node: Δ̂ = +0.03pp, CS [−0.51pp, +0.38pp] — correctly *not*
  regressed and *not yet* provably equivalent; the hierarchy is what surfaced
  the stratum despite the healthy headline.
* Ledger: 28 node records, each with the full §15-proof field set; lineage
  carries plan hash, fingerprints, seed and the entire statistical config.

The same engine with a null candidate (no planted effect) across 24 Monte
Carlo repetitions produced **zero** false REGRESSION verdicts (α = 0.05), and
with a planted 10pp effect stopped at ~2.8k samples — 11% of plan, <1% of the
corpus (both asserted in `tests/test_seqeval/`).