| --- |
| license: apache-2.0 |
| tags: |
| - sepsis |
| - clinical |
| - early-warning |
| - time-series |
| - tabular-classification |
| - conformal-prediction |
| library_name: prodrome |
| pipeline_tag: tabular-classification |
| --- |
| |
| # prodrome-clock |
|
|
| Hourly early warning for sepsis in intensive care, from the |
| [Prodrome](https://github.com/rhs2/prodrome) pipeline. Trained on hospital system A of the |
| PhysioNet/CinC 2019 challenge corpus and evaluated on system B, |
| which it never saw. |
|
|
| **This is a research artefact. It is not a medical device, it has not been cleared or approved by any regulator, and it must not be used to inform clinical decisions.** |
|
|
| ## Read this before the numbers |
|
|
| - **It predicts a recorded clinical suspicion, not a biological event.** The label |
| turns on six hours before a clinician documented suspicion of infection alongside a |
| rise in SOFA score. Every label in this field is a proxy. |
| - **A three-feature clock reaches 100% of this model's utility score.** A baseline that alerts on nothing but how long the patient has been in the unit scores 0.2413 against this model's 0.2413. Septic stays in this corpus run substantially longer than the rest, and the challenge utility metric rewards that. What this model adds is visible in the ranking, the in-window detection rate and the calibration, not in the headline score. |
| - **The model leans heavily on the workup and the clock.** An ablation fitting the |
| same model on measured values alone reaches roughly half the utility, while the |
| ordering and staleness columns alone reach most of it. Nobody orders a lactate for a |
| patient they are relaxed about, so the test-ordering pattern encodes clinician |
| suspicion, and the label is a record of clinician suspicion. |
| - **Two hospital systems in one country is not the world.** The only supported claim |
| is the transfer measured below. |
| - **The reported lead times are generous.** A long median largely reflects patients |
| flagged early in a long stay, which is why the stricter in-window detection rate is |
| reported beside it. |
|
|
| ## Results |
|
|
| Measured by `prodrome eval` on corpus `ef46d3e6ec24`, split seed |
| 7. External results first, because that is the number that means |
| something. |
|
|
| | tested on | utility | AUPRC | AUROC | detected (ever) | detected (in window) | median lead | alerts per patient-day | precision among alerted | calibration error | |
| |---|---|---|---|---|---|---|---|---|---| |
| | site B (**external**, never trained on) | 0.2413 | 0.047 | 0.628 | 0.347 | 0.065 | 45.5 h | 0.04 | 0.346 | 0.097 | |
| | site A (internal) | 0.2938 | 0.080 | 0.642 | 0.369 | 0.101 | 30.5 h | 0.04 | 0.506 | 0.090 | |
|
|
| `utility` is the published challenge scoring function, where alerting through the whole |
| beneficial window scores 1 and never alerting scores 0. `precision among alerted` is |
| the share of alerted patients who go on to become septic, against a base rate of about |
| 7 percent. |
|
|
| ## What it promises |
|
|
| > With probability at least 95% over the calibration draw, at most 4.0 alerts are raised per patient-day and at least 30% of alerted patients go on to develop sepsis (observed on the calibration slice: 0.04 alerts per patient-day, 45% precision). |
|
|
| Checked on the external test patients, that promise held: precision among alerted patients was 0.346 against a floor of 0.30, and the alert rate was 0.04 against a budget of 4.0 per patient-day. |
|
|
| The threshold is not a default. It is chosen on held-out calibration patients as the |
| lowest cut whose precision, at its exact Clopper-Pearson lower bound, clears the |
| operator's floor and whose alert rate stays inside the operator's budget. When no |
| threshold qualifies, the policy alerts on nothing rather than implying a guarantee it |
| cannot support. |
|
|
| ## What the model is reading |
|
|
| | ablation | features | utility | share of full | auroc | auprc | in window | alerts/pt-day | |
| |---|---|---|---|---|---|---|---| |
| | full | 205 | 0.2468 | 100% | 0.8016 | 0.0724 | 0.153 | 0.11 | |
| | physiology_only | 125 | 0.1159 | 47% | 0.7697 | 0.0604 | 0.117 | 0.08 | |
| | process_only | 75 | 0.2159 | 87% | 0.7263 | 0.0523 | 0.109 | 0.09 | |
| | clock_only | 3 | 0.2428 | 98% | 0.6656 | 0.0488 | 0.074 | 0.06 | |
| | workup_only | 72 | 0.2119 | 86% | 0.7539 | 0.0520 | 0.115 | 0.10 | |
| | no_time_index | 202 | 0.2316 | 94% | 0.7937 | 0.0707 | 0.156 | 0.14 | |
|
|
| The process and clock columns alone reach 87% of the full model's utility, more than physiology alone at 47%. This model is substantially reading the workup rather than the patient, which is worth knowing before anyone calls it an early warning. |
|
|
| Measured on site B (external). |
|
|
|
|
| ## Subgroups on the external site |
|
|
| A model that works on average and fails on the over-eighties is a model that fails, so |
| the breakdown is here rather than on request. |
|
|
| | dimension | group | patients | septic | utility | AUROC | detection rate | |
| |---|---|---|---|---|---|---| |
| | unit | micu | 6,923 | 390 | 0.1658 | 0.542 | 0.251 | |
| | age_band | 80_plus | 2,322 | 129 | 0.1949 | 0.598 | 0.279 | |
| | age_band | 50_to_64 | 6,278 | 351 | 0.2268 | 0.612 | 0.316 | |
| | age_band | under_50 | 4,818 | 276 | 0.2351 | 0.610 | 0.341 | |
| | sex | male | 10,732 | 647 | 0.2357 | 0.628 | 0.342 | |
| | sex | female | 9,268 | 495 | 0.2486 | 0.627 | 0.354 | |
| | unit | sicu | 6,982 | 428 | 0.2720 | 0.689 | 0.393 | |
| | age_band | 65_to_79 | 6,582 | 386 | 0.2736 | 0.665 | 0.402 | |
| | unit | unknown | 6,095 | 324 | 0.2888 | 0.649 | 0.401 | |
|
|
| ## Intended use |
|
|
| Research and education: reproducing the reported numbers, comparing a new method |
| against a measured baseline on an openly licensed corpus, and studying calibrated |
| abstention and cross-hospital transfer. |
|
|
| **Out of scope:** any clinical use, any deployment that informs patient care, any |
| triage or resource allocation, and any claim about a population unlike the two |
| hospital systems in this corpus. |
|
|
| ## How to use it |
|
|
| ```bash |
| pip install prodrome |
| prodrome init |
| prodrome data fetch # the corpus, under its own licence, 42 MB |
| prodrome data prepare |
| prodrome train && prodrome calibrate && prodrome eval |
| ``` |
|
|
| The model file here is loadable by `prodrome.core.registry` through the configuration |
| recorded alongside it. A clinical model without its feature pipeline is not a model, so |
| the preprocessing travels with the weights. |
|
|
| ## Training data |
|
|
| | | | |
| |---|---| |
| | corpus | PhysioNet/CinC Challenge 2019, version 1.0.0 | |
| | patients | 40,336 across two hospital systems | |
| | rows | 1,552,210, one per ICU hour | |
| | septic | 2,932 patients, 7.3 percent | |
| | trained on | system A, 4067 held-out patients at test | |
| | corpus digest | `ef46d3e6ec2497f7279ef352acdc32521cf54d42bb785ba2f3df4f2e179d854f` | |
|
|
| ## Licence and attribution |
|
|
| The model weights and the Prodrome source are Apache-2.0. |
|
|
| This model was produced from the PhysioNet/Computing in Cardiology Challenge 2019 database, which is available under the [Open Database License (ODbL) v1.0](https://opendatacommons.org/licenses/odbl/1-0/). Under section 4.3 of that licence this notice accompanies the model as a Produced Work. The database itself is neither redistributed with this model nor included in the source repository. Cite Reyna et al., *Early Prediction of Sepsis from Clinical Data: The PhysioNet/Computing in Cardiology Challenge 2019*, Critical Care Medicine 48(2), 2020. |
|
|
| ## Reproducing this card |
|
|
| Every number above is generated from the committed evaluation reports by |
| `prodrome publish card`, so the card cannot drift from what the harness measured. |
|
|
