Universal Risk Scores and Local Relevance: Feasible in the Digital Health Age?

OBJECTIVE: Develop and test the performance of electronic versions of the Children’s Hospital of Pittsburgh pediatric risk of mortality (CHP e-PRISM-IV) and pediatric logistic organ dysfunction (CHP e-PELOD-2) scores. DESIGN: Retrospective, single-center cohort derived from structured electronic health record data. SETTING: Large, quaternary pediatric intensive care unit (PICU) at a freestanding, university-affiliated children’s hospital. PATIENTS: All encounters with a PICU admission between January 1, 2009 and December 31, 2017 identified using electronic definitions of inpatient encounter. MEASUREMENTS: The main outcome was predictive validity of each score for hospital mortality, assessed as model discrimination and calibration. Discrimination was examined with the area under the receiver operating characteristics curve (AUROC) and the area under the precision-recall curve (AUPRC). Calibration was assessed with the Hosmer-Lemeshow goodness of fit test and calculation of a standardized mortality ratio (SMR). Models were recalibrated with new regression coefficients in a training subset of 75% of encounters selected randomly from all years of the cohort and the calibrated models were tested in the remaining 25% of the cohort. Content validity was assessed by examining correlation between electronic versions of the scores and prospectively calculated data (CHP e-PRISM-IV) and an alternative informatics approach (CHP e-PELOD-2). RESULTS: The cohort included 21,335 encounters. Correlation coefficients indicated strong agreement between different methods of score calculation. Uncalibrated AUROCs were 0.96 (95% confidence interval 0.95-0.97) for CHP e-PELOD-2 and 0.87 (0.85-0.89) for e-CHP PRISM-IV for inpatient mortality. The uncalibrated CHP e-PRISM-IV SMR was 0.63 (0.59-0.66), demonstrating strong agreement with previous, prospective evaluation at the study center. The uncalibrated CHP e-PELOD-2 SMR was 0.20 (0.18-0.21). All models required recalibrating (all Hosmer-Lemeshow goodness of fit P