Validation
This is research validation, not a clinical trial. The model is a forecasting aid; results below are on a selected charted cohort and synthetic stress data.
1. External real-data validation
Data: public Creighton cycle-length corpus, USA/Canada 1990–2014. DOI 10.7278/rgp0q-5p086, CC BY-NC 3.0. 3,324 rows / 581 women; 562 women with usable sequences. Not redistributed; obtain from the source.
Split: even-ID women only (odd IDs were used for a separate research calibration; the published dochia-v1 model uses no corpus-fitted weights). 283 women, 858 walk-forward forecasts with at least 3 prior cycles. Forecasts are genuinely prospective: cycle t uses only cycles 1…t−1.
Model: params/dochia-v1.json (published priors only; no fitted calibration).
| Metric | Value |
|---|---|
| MAE (central guess) | 2.67 d |
| Median absolute error | 1.48 d |
| RMSE | 5.14 d |
| Within 1 day | 35% |
| Within 2 days | 62% |
| Within 3 days | 75% |
| Within 7 days | 94% |
| 80% interval coverage | 82.0% |
| 95% interval coverage | 94.4% |
| Width of 80% / 95% interval | 8.8 / 15.8 d |
| CRPS | 2.048 |
| Log score | 2.54 |
| PIT KS | 0.073 |
| Abstain rate | 8.2% |
coverage95 is the 2.5th–97.5th percentile interval. "Abstain" means the model withholds a point date because variability/history is insufficient; the interval is still available.
Baseline comparison (same forecasts)
A re-implemented standard normal hierarchical model — the kind commonly shipped in apps — on the same records:
| Model | MAE | cov80 | cov95 | width80 | log score |
|---|---|---|---|---|---|
| Baseline normal hierarchical | 2.69 | 76.9% | 90.3% | 6.9 | 2.85 |
| Dochia | 2.67 | 82.0% | 94.4% | 8.8 | 2.54 |
Interpretation: point accuracy is at parity (a 0.02-day difference, noise). The gain is that the intervals mean what they say: the baseline's "80%" range actually held 77% of the time; Dochia's holds 82%. The cost is a slightly wider interval.
Strata (same test women)
| Stratum | n | MAE | cov80 | cov95 |
|---|---|---|---|---|
| New users (1–2 prior cycles; descriptive) | 455 | 3.23 | 70.8% | 88.4% |
| 3–5 prior cycles | 471 | 3.09 | 80.7% | 93.0% |
| ≥6 prior cycles | 387 | 2.16 | 83.7% | 96.1% |
| Regular (within-SD tertile) | 296 | 1.19 | 88.5% | 99.0% |
| Medium variability | 297 | 1.94 | 79.8% | 93.9% |
| Variable (within-SD tertile) | 265 | 5.15 | 77.4% | 89.8% |
Forecasts sharpen with history; variable users get wider intervals and more abstention.
2. Synthetic stress suites
These are not evidence about real people. They are generated from published summary statistics (see TESTDATA.md) and test robustness across regimes. Minimum 3 prior cycles; synthetic suites exclude any real rows.
| Suite | n | MAE | cov80 | cov95 | width80 | abstain |
|---|---|---|---|---|---|---|
| synthetic_literature | 1,848 | 3.47 | 82.2% | 93.6% | 12.5 | 21.9% |
| synthetic_stress | 1,486 | 6.33 | 74.0% | 91.9% | 17.1 | 45.4% |
| synthetic_all | 3,334 | 4.75 | 78.5% | 92.8% | 14.6 | 32.3% |
Per stress generator (Dochia):
| Generator | n | MAE | cov80 | cov95 | abstain |
|---|---|---|---|---|---|
| adolescent | 259 | 6.13 | 76.4% | 93.8% | 62.2% |
| perimenopausal | 210 | 7.23 | 69.0% | 91.0% | 67.1% |
| pcos_like | 223 | 10.63 | 58.3% | 88.3% | 89.2% |
| postpartum | 58 | 15.54 | 50.0% | 74.1% | 93.1% |
| hormonal_contraception | 366 | 4.02 | 82.8% | 95.6% | 10.1% |
| mislogged_recall | 370 | 4.21 | 79.5% | 92.2% | 22.2% |
Reading this: on hard regimes the model widens and abstains; where it still speaks, coverage is below nominal. This is expected and is the reason the model must abstain, not pretend. hormonal_contraception has regular withdrawal bleeds but no ovulatory cycle — the model cannot infer this from cycle lengths; apps must collect contraception status and must not make phase/fertility claims for those users.
3. What this does and does not establish
Established in this validation:
- On a charted, largely regular North American cohort, the model achieves roughly nominal 80/95 interval coverage and retains point accuracy comparable to a normal hierarchical baseline.
- The corrected ovulation-variance identity and heavy-tail allowance produce more honest intervals.
- Abstention activates for high-variability histories.
Not established:
- Accuracy on general app users (likely worse; published app-scale history-only MAE is ~3.5 days).
- Performance in PCOS, adolescence, perimenopause, postpartum, or non-Western/low-resource populations.
- Any phase/ovulation or clinical outcome claim.
- Prospective calibration after release (the model is intended for local-only apps with no telemetry).
4. Reproduce
The public repo includes the model and test harness; the external real-data validation can be reproduced with the optional scripts:
cd research/
python3 fetch_creighton.py # downloads the public CC BY-NC data locally (do not redistribute)
python3 evaluate_creighton.py # dochia-v1 model, even-ID held-out walk-forward
The research calibration fitted on odd-ID women is not included: it is derived from CC BY-NC data and is not part of the published model. If you need it for research, obtain permission for the data and fit it yourself.
Synthetic reproduction:
cd testing/
python3 build_suites.py
python3 evaluate_suites.py