# Validation

**This is research validation, not a clinical trial.** The model is a forecasting aid; results below are on a selected charted cohort and synthetic stress data.

## 1. External real-data validation

**Data:** public Creighton cycle-length corpus, USA/Canada 1990–2014. DOI [10.7278/rgp0q-5p086](https://doi.org/10.7278/rgp0q-5p086), **CC BY-NC 3.0**. 3,324 rows / 581 women; 562 women with usable sequences. **Not redistributed**; obtain from the source.

**Split:** even-ID women only (odd IDs were used for a separate research calibration; the published `dochia-v1` model uses no corpus-fitted weights). 283 women, **858 walk-forward forecasts** with at least 3 prior cycles. Forecasts are genuinely prospective: cycle *t* uses only cycles 1…*t*−1.

**Model:** `params/dochia-v1.json` (published priors only; no fitted calibration).

| Metric | Value |
|---|---|
| MAE (central guess) | **2.67 d** |
| Median absolute error | **1.48 d** |
| RMSE | 5.14 d |
| Within 1 day | 35% |
| Within 2 days | 62% |
| Within 3 days | 75% |
| Within 7 days | 94% |
| 80% interval coverage | **82.0%** |
| 95% interval coverage | **94.4%** |
| Width of 80% / 95% interval | 8.8 / 15.8 d |
| CRPS | 2.048 |
| Log score | 2.54 |
| PIT KS | 0.073 |
| Abstain rate | 8.2% |

`coverage95` is the 2.5th–97.5th percentile interval. "Abstain" means the model withholds a point date because variability/history is insufficient; the interval is still available.

### Baseline comparison (same forecasts)

A re-implemented standard normal hierarchical model — the kind commonly shipped in apps — on the same records:

| Model | MAE | cov80 | cov95 | width80 | log score |
|---|---|---|---|---|---|
| Baseline normal hierarchical | 2.69 | 76.9% | 90.3% | 6.9 | 2.85 |
| Dochia | **2.67** | **82.0%** | **94.4%** | 8.8 | **2.54** |

**Interpretation:** point accuracy is at parity (a 0.02-day difference, noise). The gain is that the intervals mean what they say: the baseline's "80%" range actually held 77% of the time; Dochia's holds 82%. The cost is a slightly wider interval.

### Strata (same test women)

| Stratum | n | MAE | cov80 | cov95 |
|---|---|---|---|---|
| New users (1–2 prior cycles; descriptive) | 455 | 3.23 | 70.8% | 88.4% |
| 3–5 prior cycles | 471 | 3.09 | 80.7% | 93.0% |
| ≥6 prior cycles | 387 | 2.16 | 83.7% | 96.1% |
| Regular (within-SD tertile) | 296 | 1.19 | 88.5% | 99.0% |
| Medium variability | 297 | 1.94 | 79.8% | 93.9% |
| Variable (within-SD tertile) | 265 | 5.15 | 77.4% | 89.8% |

Forecasts sharpen with history; variable users get wider intervals and more abstention.

## 2. Synthetic stress suites

These are **not evidence about real people**. They are generated from published summary statistics (see [TESTDATA.md](TESTDATA.md)) and test robustness across regimes. Minimum 3 prior cycles; synthetic suites exclude any real rows.

| Suite | n | MAE | cov80 | cov95 | width80 | abstain |
|---|---|---|---|---|---|---|
| synthetic_literature | 1,848 | 3.47 | 82.2% | 93.6% | 12.5 | 21.9% |
| synthetic_stress | 1,486 | 6.33 | 74.0% | 91.9% | 17.1 | 45.4% |
| synthetic_all | 3,334 | 4.75 | 78.5% | 92.8% | 14.6 | 32.3% |

Per stress generator (Dochia):

| Generator | n | MAE | cov80 | cov95 | abstain |
|---|---|---|---|---|---|
| adolescent | 259 | 6.13 | 76.4% | 93.8% | 62.2% |
| perimenopausal | 210 | 7.23 | 69.0% | 91.0% | 67.1% |
| pcos_like | 223 | 10.63 | 58.3% | 88.3% | 89.2% |
| postpartum | 58 | 15.54 | 50.0% | 74.1% | 93.1% |
| hormonal_contraception | 366 | 4.02 | 82.8% | 95.6% | 10.1% |
| mislogged_recall | 370 | 4.21 | 79.5% | 92.2% | 22.2% |

**Reading this:** on hard regimes the model widens and abstains; where it still speaks, coverage is below nominal. This is expected and is the reason the model must abstain, not pretend. `hormonal_contraception` has regular withdrawal bleeds but no ovulatory cycle — the model cannot infer this from cycle lengths; apps must collect contraception status and must not make phase/fertility claims for those users.

## 3. What this does and does not establish

**Established in this validation:**
- On a charted, largely regular North American cohort, the model achieves roughly nominal 80/95 interval coverage and retains point accuracy comparable to a normal hierarchical baseline.
- The corrected ovulation-variance identity and heavy-tail allowance produce more honest intervals.
- Abstention activates for high-variability histories.

**Not established:**
- Accuracy on general app users (likely worse; published app-scale history-only MAE is ~3.5 days).
- Performance in PCOS, adolescence, perimenopause, postpartum, or non-Western/low-resource populations.
- Any phase/ovulation or clinical outcome claim.
- Prospective calibration after release (the model is intended for local-only apps with no telemetry).

## 4. Reproduce

The public repo includes the model and test harness; the external real-data validation can be reproduced with the optional scripts:

```bash
cd research/
python3 fetch_creighton.py      # downloads the public CC BY-NC data locally (do not redistribute)
python3 evaluate_creighton.py   # dochia-v1 model, even-ID held-out walk-forward
```

The research calibration fitted on odd-ID women is **not** included: it is derived from CC BY-NC data and is not part of the published model. If you need it for research, obtain permission for the data and fit it yourself.

Synthetic reproduction:

```bash
cd testing/
python3 build_suites.py
python3 evaluate_suites.py
```