Search

Background: New methods such as machine learning techniques have been increasingly used to enhance the performance of risk predictions for clinical decision-making. However, commonly reported performance metrics may not be sufficient to capture the advantages of these newly proposed models for their adoption by health care professionals to improve care. Machine learning models often improve risk estimation for certain subpopulations that may be missed by these metrics.Methods and Results: This article addresses the limitations of commonly reported metrics for performance comparison and proposes additional metrics. Our discussions cover metrics related to overall performance, discrimination, calibration, resolution, reclassification, and model implementation. Models for predicting acute kidney injury after percutaneous coronary intervention are used to illustrate the use of these metrics.Conclusions: We demonstrate that commonly reported metrics may not have sufficient sensitivity to identify improvement of machine learning models and propose the use of a comprehensive list of performance metrics for reporting and comparing clinical risk prediction models. [ABSTRACT FROM AUTHOR]

Background: Intraoperative data may improve models predicting postoperative events. We evaluated the effect of incorporating intraoperative variables to the existing preoperative model on the predictive performance of the model for coronary artery bypass graft.Methods: We analyzed 378 572 isolated coronary artery bypass graft cases performed across 1083 centers, using the national Society of Thoracic Surgeons Adult Cardiac Surgery Database between 2014 and 2016. Outcomes were operative mortality, 5 postoperative complications, and composite representation of all events. We fitted models by logistic regression or extreme gradient boosting (XGBoost). For each modeling approach, we used preoperative only, intraoperative only, or pre+intraoperative variables. We developed 84 models with unique combinations of the 3 variable sets, 2 variable selection methods, 2 modeling approaches, and 7 outcomes. Each model was tested in 20 iterations of 70:30 stratified random splitting into development/testing samples. Model performances were evaluated on the testing dataset using the C statistic, area under the precision-recall curve, and calibration metrics, including the Brier score.Results: The mean patient age was 65.3 years, and 24.7% were women. Operative mortality, excluding intraoperative death, occurred in 1.9%. In all outcomes, models that considered pre+intraoperative variables demonstrated significantly improved Brier score and area under the precision-recall curve compared with models considering pre or intraoperative variables alone. XGBoost without external variable selection had the best C statistics, Brier score, and area under the precision-recall curve values in 4 of the 7 outcomes (mortality, renal failure, prolonged ventilation, and composite) compared with logistic regression models with or without variable selection. Based on the calibration plots, risk restratification for mortality showed that the logistic regression model underestimated the risk in 11 114 patients (9.8%) and overestimated in 12 005 patients (10.6%). In contrast, the XGBoost model underestimated the risk in 7218 patients (6.4%) and overestimated in 0 patients (0%).Conclusions: In isolated coronary artery bypass graft, adding intraoperative variables to preoperative variables resulted in improved predictions of all 7 outcomes. Risk models based on XGBoost may provide a better prediction of adverse events to guide clinical care. [ABSTRACT FROM AUTHOR]

Background: Visit-to-visit variability (VVV) in blood pressure values has been reported in clinical studies. However, little is known about VVV in clinical practice and whether it is associated with patient characteristics in real-world setting. Methods: We conducted a retrospective cohort study to quantify VVV in systolic blood pressure (SBP) values in a real-world setting. We included adults (age ≥18 years) with at least 2 outpatient visits between January 1, 2014 and October 31, 2018 from Yale New Haven Health System. Patient-level measures of VVV included SD and coefficient of variation of a given patient's SBP across visits. We calculated patient-level VVV overall and by patient subgroups. We further developed a multilevel regression model to assess the extent to which VVV in SBP was explained by patient characteristics. Results: The study population included 537 218 adults, with a total of 7 721 864 SBP measurements. The mean age was 53.4 (SD 19.0) years, 60.4% were women, 69.4% were non-Hispanic White, and 18.1% were on antihypertensive medications. Patients had a mean body mass index of 28.4 (5.9) kg/m2 and 22.6%, 8.0%, 9.7%, and 5.6% had a history of hypertension, diabetes, hyperlipidemia, and coronary artery disease, respectively. The mean number of visits per patient was 13.3, over an average period of 2.4 years. The mean (SD) intraindividual SD and coefficient of variation of SBP across visits were 10.6 (5.1) mm Hg and 0.08 (0.04). These measures of blood pressure variation were consistent across patient subgroups defined by demographic characteristics and medical history. In the multivariable linear regression model, only 4% of the variance in absolute standardized difference was attributable to patient characteristics. Conclusions: The VVV in real-world practice poses challenges for management of patients with hypertension based on blood pressure readings in outpatient settings and suggest the need to go beyond episodic clinic evaluation. [ABSTRACT FROM AUTHOR]

Background: The current ability to predict readmissions in patients with heart failure is modest at best. It is unclear whether machine learning techniques that address higher dimensional, nonlinear relationships among variables would enhance prediction. We sought to compare the effectiveness of several machine learning algorithms for predicting readmissions.Methods and Results: Using data from the Telemonitoring to Improve Heart Failure Outcomes trial, we compared the effectiveness of random forests, boosting, random forests combined hierarchically with support vector machines or logistic regression (LR), and Poisson regression against traditional LR to predict 30- and 180-day all-cause readmissions and readmissions because of heart failure. We randomly selected 50% of patients for a derivation set, and a validation set comprised the remaining patients, validated using 100 bootstrapped iterations. We compared C statistics for discrimination and distributions of observed outcomes in risk deciles for predictive range. In 30-day all-cause readmission prediction, the best performing machine learning model, random forests, provided a 17.8% improvement over LR (mean C statistics, 0.628 and 0.533, respectively). For readmissions because of heart failure, boosting improved the C statistic by 24.9% over LR (mean C statistic 0.678 and 0.543, respectively). For 30-day all-cause readmission, the observed readmission rates in the lowest and highest deciles of predicted risk with random forests (7.8% and 26.2%, respectively) showed a much wider separation than LR (14.2% and 16.4%, respectively).Conclusions: Machine learning methods improved the prediction of readmission after hospitalization for heart failure compared with LR and provided the greatest predictive range in observed readmission rates. [ABSTRACT FROM AUTHOR]

Use of machine learning (ML) in clinical research is growing steadily given the increasing availability of complex clinical data sets. ML presents important advantages in terms of predictive performance and identifying undiscovered subpopulations of patients with specific physiology and prognoses. Despite this popularity, many clinicians and researchers are not yet familiar with evaluating and interpreting ML analyses. Consequently, readers and peer-reviewers alike may either overestimate or underestimate the validity and credibility of an ML-based model. Conversely, ML experts without clinical experience may present details of the analysis that are too granular for a clinical readership to assess. Overwhelming evidence has shown poor reproducibility and reporting of ML models in clinical research suggesting the need for ML analyses to be presented in a clear, concise, and comprehensible manner to facilitate understanding and critical evaluation. We present a recommendation for transparent and structured reporting of ML analysis results specifically directed at clinical researchers. Furthermore, we provide a list of key reporting elements with examples that can be used as a template when preparing and submitting ML-based manuscripts for the same audience. [ABSTRACT FROM AUTHOR]

Background: New methods such as machine learning techniques have been increasingly used to enhance the performance of risk predictions for clinical decision-making. However, commonly reported performance metrics may not be sufficient to capture the advantages of these newly proposed models for their adoption by health care professionals to improve care. Machine learning models often improve risk estimation for certain subpopulations that may be missed by these metrics., Methods and Results: This article addresses the limitations of commonly reported metrics for performance comparison and proposes additional metrics. Our discussions cover metrics related to overall performance, discrimination, calibration, resolution, reclassification, and model implementation. Models for predicting acute kidney injury after percutaneous coronary intervention are used to illustrate the use of these metrics., Conclusions: We demonstrate that commonly reported metrics may not have sufficient sensitivity to identify improvement of machine learning models and propose the use of a comprehensive list of performance metrics for reporting and comparing clinical risk prediction models.