In this episode, I’ll discuss the importance of external validation of clinical prediction models.
When a prediction model is developed, it is only known to be valid in the data set that it was developed from. In order to determine a prediction model’s reproducibility and generalizability to new and different patients it must undergo a process called external validation.
Either the same set of researchers or, ideally, a new set of researchers apply the prediction model to another data set and attempt to replicate the results. If successful, the model is considered externally valid and its broad application becomes more legitimate.
Implementing the prediction model widely prior to external validation can lead to undesirable consequences if the model turns out to not have external validity.
Such an example was recently published in JAMA of the Epic Sepsis Model (ESM), a proprietary sepsis prediction model that was widely used at hundreds of US hospitals to predict the development of sepsis in inpatients.
In order to validate the model a retrospective cohort study was conducted among almost 28,000 inpatients in the academic health system of the University of Michigan, Ann Arbor.
The authors found that:
The ESM had a hospitalization-level area under the receiver operating characteristic curve of 0.63 (95% CI, 0.62-0.64). The ESM identified 183 of 2552 patients with sepsis (7%) who did not receive timely administration of antibiotics, highlighting the low sensitivity of the ESM in comparison with contemporary clinical practice. The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue.
The implications of widespread adoption before external validation of this model are that it did a poor job of predicting the onset of sepsis, and contributed a large number of unnecessary clinical alerts, wasting time, resources, and contributing to alert fatigue. This could happen with any model that is widely implemented before external validation and serves as a warning against implementing a prediction model too soon in the research stage. Whenever you see a new clinical prediction model, be sure to identify whether or when it will be externally validated.
To access my free download area with 20 different resources to help you in your practice, go to pharmacyjoe.com/free.
If you like this post, check out my book – A Pharmacist’s Guide to Inpatient Medical Emergencies: How to respond to code blue, rapid response calls, and other medical emergencies.
Leave a Reply