Predicting Readmission or Death After Discharge From the ICU: External Validation and Retraining of a Machine Learning Model.

Objectives: Many machine learning (ML) models have been developed for application in the ICU, but few models have been subjected to external validation. The performance of these models in new settings therefore remains unknown. The objective of this study was to assess the performance of an existing decision support tool based on a ML model predicting readmission or death within 7 days after ICU discharge before, during, and after retraining and recalibration.
Design: A gradient boosted ML model was developed and validated on electronic health record data from 2004 to 2021. We performed an independent validation of this model on electronic health record data from 2011 to 2019 from a different tertiary care center.
Setting: Two ICUs in tertiary care centers in The Netherlands.
Patients: Adult patients who were admitted to the ICU and stayed for longer than 12 hours.
Interventions: None.
Measurements and Main Results: We assessed discrimination by area under the receiver operating characteristic curve (AUC) and calibration (slope and intercept). We retrained and recalibrated the original model and assessed performance via a temporal validation design. The final retrained model was cross-validated on all data from the new site. Readmission or death within 7 days after ICU discharge occurred in 577 of 10,052 ICU admissions (5.7%) at the new site. External validation revealed moderate discrimination with an AUC of 0.72 (95% CI 0.67-0.76). Retrained models showed improved discrimination with AUC 0.79 (95% CI 0.75-0.82) for the final validation model. Calibration was poor initially and good after recalibration via isotonic regression.
Conclusions: In this era of expanding availability of ML models, external validation and retraining are key steps to consider before applying ML models to new settings. Clinicians and decision-makers should take this into account when considering applying new ML models to their local settings.
Competing Interests: Drs. Fornasa and Cinà received funding from Pacmed; they disclosed work for hire; they disclosed the off-label product use of Pacmed Critical. Dr. Elbers disclosed that Amsterdam University Medical Center is entitled to royalties from jointly developed models. The remaining authors have disclosed that they do not have any potential conflicts of interest.
(Copyright © 2022 The Author(s). Published by Wolters Kluwer Health, Inc. on behalf of the Society of Critical Care Medicine and Wolters Kluwer Health, Inc.)