| # Validation |
|
|
| **This is research validation, not a clinical trial.** The model is a forecasting aid; results below are on a selected charted cohort and synthetic stress data. |
|
|
| ## 1. External real-data validation |
|
|
| **Data:** public Creighton cycle-length corpus, USA/Canada 1990–2014. DOI [10.7278/rgp0q-5p086](https://doi.org/10.7278/rgp0q-5p086), **CC BY-NC 3.0**. 3,324 rows / 581 women; 562 women with usable sequences. **Not redistributed**; obtain from the source. |
|
|
| **Split:** even-ID women only (odd IDs were used for a separate research calibration; the published `dochia-v1` model uses no corpus-fitted weights). 283 women, **858 walk-forward forecasts** with at least 3 prior cycles. Forecasts are genuinely prospective: cycle *t* uses only cycles 1…*t*−1. |
|
|
| **Model:** `params/dochia-v1.json` (published priors only; no fitted calibration). |
|
|
| | Metric | Value | |
| |---|---| |
| | MAE (central guess) | **2.67 d** | |
| | Median absolute error | **1.48 d** | |
| | RMSE | 5.14 d | |
| | Within 1 day | 35% | |
| | Within 2 days | 62% | |
| | Within 3 days | 75% | |
| | Within 7 days | 94% | |
| | 80% interval coverage | **82.0%** | |
| | 95% interval coverage | **94.4%** | |
| | Width of 80% / 95% interval | 8.8 / 15.8 d | |
| | CRPS | 2.048 | |
| | Log score | 2.54 | |
| | PIT KS | 0.073 | |
| | Abstain rate | 8.2% | |
|
|
| `coverage95` is the 2.5th–97.5th percentile interval. "Abstain" means the model withholds a point date because variability/history is insufficient; the interval is still available. |
|
|
| ### Baseline comparison (same forecasts) |
|
|
| A re-implemented standard normal hierarchical model — the kind commonly shipped in apps — on the same records: |
|
|
| | Model | MAE | cov80 | cov95 | width80 | log score | |
| |---|---|---|---|---|---| |
| | Baseline normal hierarchical | 2.69 | 76.9% | 90.3% | 6.9 | 2.85 | |
| | Dochia | **2.67** | **82.0%** | **94.4%** | 8.8 | **2.54** | |
|
|
| **Interpretation:** point accuracy is at parity (a 0.02-day difference, noise). The gain is that the intervals mean what they say: the baseline's "80%" range actually held 77% of the time; Dochia's holds 82%. The cost is a slightly wider interval. |
|
|
| ### Strata (same test women) |
|
|
| | Stratum | n | MAE | cov80 | cov95 | |
| |---|---|---|---|---| |
| | New users (1–2 prior cycles; descriptive) | 455 | 3.23 | 70.8% | 88.4% | |
| | 3–5 prior cycles | 471 | 3.09 | 80.7% | 93.0% | |
| | ≥6 prior cycles | 387 | 2.16 | 83.7% | 96.1% | |
| | Regular (within-SD tertile) | 296 | 1.19 | 88.5% | 99.0% | |
| | Medium variability | 297 | 1.94 | 79.8% | 93.9% | |
| | Variable (within-SD tertile) | 265 | 5.15 | 77.4% | 89.8% | |
|
|
| Forecasts sharpen with history; variable users get wider intervals and more abstention. |
|
|
| ## 2. Synthetic stress suites |
|
|
| These are **not evidence about real people**. They are generated from published summary statistics (see [TESTDATA.md](TESTDATA.md)) and test robustness across regimes. Minimum 3 prior cycles; synthetic suites exclude any real rows. |
|
|
| | Suite | n | MAE | cov80 | cov95 | width80 | abstain | |
| |---|---|---|---|---|---|---| |
| | synthetic_literature | 1,848 | 3.47 | 82.2% | 93.6% | 12.5 | 21.9% | |
| | synthetic_stress | 1,486 | 6.33 | 74.0% | 91.9% | 17.1 | 45.4% | |
| | synthetic_all | 3,334 | 4.75 | 78.5% | 92.8% | 14.6 | 32.3% | |
| |
| Per stress generator (Dochia): |
| |
| | Generator | n | MAE | cov80 | cov95 | abstain | |
| |---|---|---|---|---|---| |
| | adolescent | 259 | 6.13 | 76.4% | 93.8% | 62.2% | |
| | perimenopausal | 210 | 7.23 | 69.0% | 91.0% | 67.1% | |
| | pcos_like | 223 | 10.63 | 58.3% | 88.3% | 89.2% | |
| | postpartum | 58 | 15.54 | 50.0% | 74.1% | 93.1% | |
| | hormonal_contraception | 366 | 4.02 | 82.8% | 95.6% | 10.1% | |
| | mislogged_recall | 370 | 4.21 | 79.5% | 92.2% | 22.2% | |
|
|
| **Reading this:** on hard regimes the model widens and abstains; where it still speaks, coverage is below nominal. This is expected and is the reason the model must abstain, not pretend. `hormonal_contraception` has regular withdrawal bleeds but no ovulatory cycle — the model cannot infer this from cycle lengths; apps must collect contraception status and must not make phase/fertility claims for those users. |
|
|
| ## 3. What this does and does not establish |
|
|
| **Established in this validation:** |
| - On a charted, largely regular North American cohort, the model achieves roughly nominal 80/95 interval coverage and retains point accuracy comparable to a normal hierarchical baseline. |
| - The corrected ovulation-variance identity and heavy-tail allowance produce more honest intervals. |
| - Abstention activates for high-variability histories. |
|
|
| **Not established:** |
| - Accuracy on general app users (likely worse; published app-scale history-only MAE is ~3.5 days). |
| - Performance in PCOS, adolescence, perimenopause, postpartum, or non-Western/low-resource populations. |
| - Any phase/ovulation or clinical outcome claim. |
| - Prospective calibration after release (the model is intended for local-only apps with no telemetry). |
|
|
| ## 4. Reproduce |
|
|
| The public repo includes the model and test harness; the external real-data validation can be reproduced with the optional scripts: |
|
|
| ```bash |
| cd research/ |
| python3 fetch_creighton.py # downloads the public CC BY-NC data locally (do not redistribute) |
| python3 evaluate_creighton.py # dochia-v1 model, even-ID held-out walk-forward |
| ``` |
|
|
| The research calibration fitted on odd-ID women is **not** included: it is derived from CC BY-NC data and is not part of the published model. If you need it for research, obtain permission for the data and fit it yourself. |
|
|
| Synthetic reproduction: |
|
|
| ```bash |
| cd testing/ |
| python3 build_suites.py |
| python3 evaluate_suites.py |
| ``` |
|
|
