docs/07_protocol_shift.md

# 07 — Process-recipe and campaign shift (RQ4)

[← Back to 06: Explainability](06_explainability.md) | [README](../README.md)

Covers **Phase 10**. Every number is generated into
[`outputs/reports/`](../outputs/reports/).

**RQ4: does the screening system remain safe when a new process recipe or a new
production campaign appears — and if not, how quickly does the guarantee break?**

---

## 0. The confound that qualifies everything below

Measured in Phase 5 and repeated here rather than left in a footnote:

> **12 of 18 recipe families occur in exactly ONE batch, covering 59 cells —
> 48% of the cohort.** Recipe holdout is therefore *partly* a campaign holdout
> and **is not an independent axis** from batch holdout. Any statement of the
> form "the model generalises to new recipes" is entangled with "the model
> generalises to new campaigns" and cannot be cleanly separated on this dataset.

**However, the two schemes behave very differently, which partially
disentangles them** — see §2.

## 1. The headline, and it is a safety finding

Under **campaign shift** (train on batches 1–2, test on batch 3):

| Axis | In-distribution | Batch-3 OOD | Change |
|---|---:|---:|---:|
| RMSE (log₁₀) | 0.0640 | 0.1233 | **+92.5%** |
| MAPE (raw) | 10.45% | 18.77% | +79.7% |
| **Conformal coverage** | **0.9073** | **0.4250** | **−53.2%** |
| **Interval width** | **0.1975** | **0.1352** | **−31.5%** |
| Cost per cell | 6.879 | 7.200 | +4.7% |
| SHAP ranking stability | 1.000 | 0.673 | — |

### The dangerous signature: the intervals get *narrower* as the model gets worse

Read the two bold rows together. **RMSE nearly doubles while the prediction
interval shrinks by 31.5%.** The model does not merely degrade — it becomes
*confidently wrong*, and coverage collapses to 42.5% against a nominal 90%.

This is the single most consequential result in Phase 10, because it is the
worst possible failure mode for a system whose value proposition is a bounded
escape rate, and because **an accuracy-only monitor cannot see it**. A plant
watching prediction error would notice something; a plant watching interval
width would conclude the model had become *more* confident.

The mechanism is the conformal calibration set. Calibration residuals are
computed on batches 1–2, where the model fits well, so the half-width inherited
by batch 3 reflects an easier distribution than the one it is applied to.
Exchangeability — the assumption every conformal guarantee rests on — is
violated, and Phase 3 measured that violation directly
(**KS D = 0.651, p = 1.7 × 10⁻¹¹**).

### The Phase 3 measurement predicted this

The degradation is not a surprise, and showing the coherence is stronger than
reporting it cold. Phase 3 recorded the Gate-1 ΔQ(V) relationship weakening
across campaigns — **R² = 0.901 → 0.665 → 0.647** for batches 1 → 2 → 3 — and a
KS statistic of 0.651 between the in-distribution cohort and batch 3. A model
qualified on the first two campaigns was always going to find the third harder;
Phase 10 measures how much, and on which axes.

### The escape-rate number here is uninformative, and that is stated

Escape rate falls to 0.0000 on batch 3. **That is a property of the test set,
not evidence of safety**: batch 3 contains **1 cell below the warranty target
out of 40**, while batch 2 contains 41 of 43. The escape guarantee is only
meaningfully testable where unfit cells exist. Coverage, by contrast, is
measurable everywhere and is what collapsed.

## 2. Recipe shift is mild; campaign shift is severe

| Scheme | Coverage change | RMSE change | SHAP ranking stability |
|---|---:|---:|---:|
| **batch_holdout** (campaign) | **−53.2%** | **+92.5%** | 0.673 |
| **recipe_holdout** (process) | **−2.5%** | +15.5% | 0.937 |

**This partially disentangles the confound.** If recipe holdout were simply a
campaign holdout wearing a different label, the two rows would look alike. They
do not: coverage falls 53% under campaign shift and 2.5% under recipe shift, a
factor of 20.

The reading this supports — offered as an inference, not a clean separation the
data cannot deliver — is that **the damaging axis is the production campaign,
not the charging recipe.** Holding out an unseen recipe family costs little;
holding out a later campaign costs a great deal. Since the recipe-holdout folds
still draw training data from all three campaigns, they retain the campaign
information the batch holdout removes.

## 3. Group F ablation — and Phase 9 predicted part of it

Phase 9's H4 found cross-validated RMSE **unchanged to four decimal places**
(0.0597 → 0.0597) when the recipe descriptors were removed, so the
in-distribution ablation here was expected to be undramatic. **It is:**

| Scheme | Metric | With Group F | Without | Change |
|---|---|---:|---:|---:|
| cell_grouped | RMSE | 0.0640 | 0.0649 | **+1.3%** |
| cell_grouped | coverage | 0.9073 | 0.9192 | +1.3% |
| recipe_holdout | RMSE | 0.0740 | 0.0733 | **−0.9%** |
| recipe_holdout | coverage | 0.8850 | 0.8693 | −1.8% |
| **batch_holdout** | **RMSE** | **0.1233** | **0.0681** | **−44.7%** |
| **batch_holdout** | **coverage** | **0.4250** | **0.7750** | **+82.4%** |

**In-distribution and under recipe shift the recipe descriptors are worth
essentially nothing — exactly as Phase 9 predicted.** That is the confirmation
half.

**Under campaign shift they are actively harmful, and dramatically so.**
Removing them nearly halves RMSE and almost doubles coverage, taking it from a
catastrophic 42.5% to a merely poor 77.5%.

This is precisely the risk the build plan anticipated when it required Group F
to be flagged and ablated: *a model that memorises a recipe-to-lifetime mapping
will not generalise to a new recipe.* The refinement measurement adds is that
the memorisation costs nothing while the recipes are familiar and a great deal
when they are not — and batch 3 uses recipe families largely absent from
batches 1–2, which is the same confound from §0 seen from the other side.

**Note the two results are consistent, not contradictory.** H4 asked whether the
model *needs* the recipe features (it does not: RMSE unchanged when removed).
Phase 10 asks whether *having* them hurts under shift (it does). A feature can
be redundant in-distribution and harmful out-of-distribution at the same time —
redundancy means the information exists elsewhere, not that the feature is inert.

## 4. Recommended revalidation policy

The findings above imply a concrete monitoring trigger rather than a general
exhortation to retrain.

**1. Do not ship the recipe descriptors in the deployed model.** They cost 1.3%
of in-distribution RMSE and buy a 44.7% RMSE penalty and a 33-point coverage
loss the moment a campaign shifts. On this evidence the `no_recipe` feature set
is the operationally correct choice, and the in-line-measurable subset is a
further conservative option.

**2. Monitor interval width against realised error, not either alone.** The
failure signature is *narrowing intervals with rising error*. A monitor watching
only error would react late; one watching only width would read the failure as
increased confidence. The ratio of realised absolute error to nominal half-width
is the quantity that moves unambiguously — it should sit near a stable value and
rising above it is the trigger.

**3. Recalibrate conformal quantiles on every new campaign before relying on the
guarantee.** The guarantee is a statement about exchangeable data; a new
campaign is by construction a new distribution. Recalibration needs only labelled
cells from the new campaign, not a retrained model — and Phase 7 established the
floor: **at least 19 calibration cells for α = 0.05, and 99 for α = 0.01**,
which α = 0.01 makes unreachable at realistic qualification-lot sizes.

**4. Treat a large KS statistic against the qualification cohort as a
revalidation trigger.** Phase 3 measured KS D = 0.651 for batch 3, and the
system degraded accordingly. This dataset provides one shifted campaign, so it
can demonstrate the association but cannot calibrate a threshold — a plant would
need several campaigns to set one, and that is stated as a limitation rather
than a number invented here.

**5. Monitor SHAP ranking stability as a slower secondary signal.** Ranking
correlation fell to 0.673 under campaign shift against 0.937 under recipe shift,
so it tracks the damaging axis. It is a weaker and noisier signal than coverage
and is suggested as corroboration, not as a primary trigger.

## 5. Limitations

- **The confound in §0 is not resolved, only partially disentangled.** §2 shows
  the two schemes behave very differently, which is evidence that the campaign
  axis dominates, but with 12 of 18 recipe families confined to one batch this
  dataset cannot separate them cleanly.
- **One shifted campaign.** Every OOD conclusion rests on a single batch-3
  holdout, so it can show *that* degradation occurs and roughly how much, but
  cannot estimate variability across campaigns. Its metrics are reported as a
  single split with that justification recorded in the artifact.
- **Escape-based OOD metrics are uninformative** because batch 3 holds one unfit
  cell in 40 (§1).
- **The revalidation thresholds are directional, not calibrated.** Numbers like
  "KS above some value" would require several campaigns to establish.

---

[← Explainability](06_explainability.md) · [README](../README.md) · [Operator console →](../app/README.md)