A machine learning model demonstrated good discrimination of in-hospital mortality for adult patients at the time of admission in a recent validation study at multiple hospitals.
Researchers at Duke University Health System in North Carolina trained the model using electronic health record (EHR) data from 43,180 hospitalizations of 31,003 unique adult patients admitted to one system hospital (hospital A) from Oct. 1, 2014, to Dec. 31, 2015. They built a total of 195 model features using 57 EHR data elements, including patient demographic characteristics (five data elements), laboratory test results (33 data elements), vital signs (nine data elements), and medication administrations (10 data elements) that were available between hospital presentation and time of admission. A dashboard displayed patient risk scores to clinicians. The study evaluated the model in retrospective and prospective validation cohorts in three hospitals at the health system. All hospitalizations of adult patients with an order placed for inpatient admission were included (including those who died in the ED after such an order was placed).
The model was retrospectively and temporally validated in a separate cohort of 16,122 hospitalizations of 13,094 patients admitted to hospital A from March 1 to Aug. 31, 2018. It was also retrospectively and externally validated using two separate cohorts from two different community-based hospitals. Hospital B had a cohort of 6,586 hospitalizations of 5,613 patients, and hospital C had a cohort of 4,086 hospitalizations of 3,428 patients, all occurring from March 1 to Aug. 31, 2018. Then the model was integrated into the EHR and prospectively validated in a cohort of 5,273 hospitalizations of 4,525 patients admitted to hospital A from Feb. 14 to April 15, 2019. The main outcome was in-hospital mortality (defined as a discharge disposition of “expired”), and model performance was quantified using the area under the receiver operating characteristic curve (AUROC) and area under the precision recall curve (AUPRC). Results were published online Feb. 7 by JAMA Network Open.
Overall, the study included 75,247 hospitalizations (median patient age, 59.5 years; 45.9% involving male patients), 2,021 (2.7%) of which resulted in in-hospital mortality. The in-hospital mortality rates for the training validation; retrospective validations at hospitals A, B, and C; and prospective validation cohorts were 3.0%, 2.7%, 1.8%, 2.1%, and 1.6%, respectively. The AUROCs were 0.87 (95% CI, 0.83 to 0.89), 0.85 (95% CI, 0.83 to 0.87), 0.89 (95% CI, 0.86 to 0.92), 0.84 (95% CI, 0.80 to 0.89), and 0.86 (95% CI, 0.83 to 0.90), respectively. The AUPRCs were 0.29 (95% CI, 0.25 to 0.37), 0.17 (95% CI, 0.13 to 0.22), 0.22 (95% CI, 0.14 to 0.31), 0.13 (95% CI, 0.08 to 0.21), and 0.14 (95% CI, 0.09 to 0.21), respectively.
The 2018 temporal and external validation cohorts showed that if the model were implemented in each setting at a threshold that corresponds to a sensitivity (recall) of 60%, the positive predictive value would be 9.8% in hospital A, 8.0% in hospital B, and 9.1% in hospital C. In the prospective validation cohort, using a threshold that corresponds to a sensitivity of 60% would lead to a positive predictive value of 8.5%.
The results may not be generalizable to hospitals in other regions and were due in large part to the institution's technology infrastructure, among other limitations, the study authors noted. They added that the model should not be used to prioritize patients for admission to the ICU or to limit care delivery.
“While we demonstrate that the model prediction accuracy is maintained prospectively, it is unknown whether these predictions will actually improve clinical or operational outcomes. Future work will evaluate the implementation of a clinical workflow,” they wrote, adding that the first use they plan to evaluate is identification of patients for palliative care consultation.