Search

Background: New methods such as machine learning techniques have been increasingly used to enhance the performance of risk predictions for clinical decision-making. However, commonly reported performance metrics may not be sufficient to capture the advantages of these newly proposed models for their adoption by health care professionals to improve care. Machine learning models often improve risk estimation for certain subpopulations that may be missed by these metrics.Methods and Results: This article addresses the limitations of commonly reported metrics for performance comparison and proposes additional metrics. Our discussions cover metrics related to overall performance, discrimination, calibration, resolution, reclassification, and model implementation. Models for predicting acute kidney injury after percutaneous coronary intervention are used to illustrate the use of these metrics.Conclusions: We demonstrate that commonly reported metrics may not have sufficient sensitivity to identify improvement of machine learning models and propose the use of a comprehensive list of performance metrics for reporting and comparing clinical risk prediction models. [ABSTRACT FROM AUTHOR]

Randomized trials of hypertension have seldom examined heterogeneity in response to treatments over time and the implications for cardiovascular outcomes. Understanding this heterogeneity, however, is a necessary step toward personalizing antihypertensive therapy. We applied trajectory-based modeling to data on 39 763 study participants of the ALLHAT (Antihypertensive and Lipid-Lowering Treatment to Prevent Heart Attack Trial) to identify distinct patterns of systolic blood pressure (SBP) response to randomized medications during the first 6 months of the trial. Two trajectory patterns were identified: immediate responders (85.5%), on average, had a decreasing SBP, whereas nonimmediate responders (14.5%), on average, had an initially increasing SBP followed by a decrease. Compared with those randomized to chlorthalidone, participants randomized to amlodipine (odds ratio, 1.20; 95% confidence interval [CI], 1.10-1.31), lisinopril (odds ratio, 1.88; 95% CI, 1.73-2.03), and doxazosin (odds ratio, 1.65; 95% CI, 1.52-1.78) had higher adjusted odds ratios associated with being a nonimmediate responder (versus immediate responder). After multivariable adjustment, nonimmediate responders had a higher hazard ratio of stroke (hazard ratio, 1.49; 95% CI, 1.21-1.84), combined cardiovascular disease (hazard ratio, 1.21; 95% CI, 1.11-1.31), and heart failure (hazard ratio, 1.48; 95% CI, 1.24-1.78) during follow-up between 6 months and 2 years. The SBP response trajectories provided superior discrimination for predicting downstream adverse cardiovascular events than classification based on difference in SBP between the first 2 measurements, SBP at 6 months, and average SBP during the first 6 months. Our findings demonstrate heterogeneity in response to antihypertensive therapies and show that chlorthalidone is associated with more favorable initial response than the other medications. [ABSTRACT FROM AUTHOR]

Background: The current ability to predict readmissions in patients with heart failure is modest at best. It is unclear whether machine learning techniques that address higher dimensional, nonlinear relationships among variables would enhance prediction. We sought to compare the effectiveness of several machine learning algorithms for predicting readmissions.Methods and Results: Using data from the Telemonitoring to Improve Heart Failure Outcomes trial, we compared the effectiveness of random forests, boosting, random forests combined hierarchically with support vector machines or logistic regression (LR), and Poisson regression against traditional LR to predict 30- and 180-day all-cause readmissions and readmissions because of heart failure. We randomly selected 50% of patients for a derivation set, and a validation set comprised the remaining patients, validated using 100 bootstrapped iterations. We compared C statistics for discrimination and distributions of observed outcomes in risk deciles for predictive range. In 30-day all-cause readmission prediction, the best performing machine learning model, random forests, provided a 17.8% improvement over LR (mean C statistics, 0.628 and 0.533, respectively). For readmissions because of heart failure, boosting improved the C statistic by 24.9% over LR (mean C statistic 0.678 and 0.543, respectively). For 30-day all-cause readmission, the observed readmission rates in the lowest and highest deciles of predicted risk with random forests (7.8% and 26.2%, respectively) showed a much wider separation than LR (14.2% and 16.4%, respectively).Conclusions: Machine learning methods improved the prediction of readmission after hospitalization for heart failure compared with LR and provided the greatest predictive range in observed readmission rates. [ABSTRACT FROM AUTHOR]

Prior claims analyses suggest that the use of intravenous inotropic therapy for patients hospitalized with heart failure varies substantially by hospital. Whether differences in the clinical characteristics of the patients explain observed differences in the use of inotropic therapy is not known.We sought to characterize institutional variation in inotrope use among patients hospitalized with heart failure before and after accounting for clinical factors of patients. Hierarchical generalized linear regression models estimated risk-standardized hospital-level rates of inotrope use within 209 hospitals participating in Get With The Guidelines-Heart Failure (GWTG-HF) registry between 2005 and 2011. The association between risk-standardized rates of inotrope use and clinical outcomes was determined. Overall, an inotropic agent was administered in 7691 of 126 564 (6.1%) heart failure hospitalizations: dobutamine 43%, dopamine 24%, milrinone 17%, or a combination 16%. Patterns of inotrope use were stable during the 7-year study period. Use of inotropes varied significantly between hospitals even after accounting for patient and hospital characteristics (median risk-standardized hospital rate, 5.9%; interquartile range, 3.7%-8.6%; range, 1.3%-32.9%). After adjusting for case-mix and hospital structural differences, model intraclass correlation indicated that 21% of the observed variation in inotrope use was potentially attributable to random hospital effects (ie, institutional preferences). Hospitals with higher risk-standardized inotrope use had modestly longer risk-standardized length of stay (P=0.005) but had no difference in risk-standardized inpatient mortality (P=0.12)Use of intravenous inotropic agents during hospitalization for heart failure varies significantly among US hospitals even after accounting for patient and hospital factors. [ABSTRACT FROM AUTHOR]

Background: New methods such as machine learning techniques have been increasingly used to enhance the performance of risk predictions for clinical decision-making. However, commonly reported performance metrics may not be sufficient to capture the advantages of these newly proposed models for their adoption by health care professionals to improve care. Machine learning models often improve risk estimation for certain subpopulations that may be missed by these metrics., Methods and Results: This article addresses the limitations of commonly reported metrics for performance comparison and proposes additional metrics. Our discussions cover metrics related to overall performance, discrimination, calibration, resolution, reclassification, and model implementation. Models for predicting acute kidney injury after percutaneous coronary intervention are used to illustrate the use of these metrics., Conclusions: We demonstrate that commonly reported metrics may not have sufficient sensitivity to identify improvement of machine learning models and propose the use of a comprehensive list of performance metrics for reporting and comparing clinical risk prediction models.

Background: Despite increasing attention on reducing relatively costly hospital practices while maintaining the quality of care, few studies have examined how hospitals use the intensive care unit (ICU), a high-cost setting, for patients admitted with heart failure (HF). We characterized hospital patterns of ICU admission for patients with HF and determined their association with the use of ICU-level therapies and patient outcomes., Methods and Results: We identified 166 224 HF discharges from 341 hospitals in the 2009-2010 Premier Perspective database. We excluded hospitals with <25 HF admissions, patients <18 years old, and transfers. We defined ICU as including medical ICU, coronary ICU, and surgical ICU. We calculated the percent of patients admitted directly to an ICU. We compared hospitals in the top quartile (high ICU admission) with the remaining quartiles. The median percentage of ICU admission was 10% (interquartile range, 6%-16%; range, 0%-88%). In top-quartile hospitals, treatments requiring an ICU were used less often; the percentage of ICU days receiving mechanical ventilation was 6% for the top quartile versus 15% for the others; noninvasive positive pressure ventilation, 8% versus 19%; vasopressors and/or inotropes, 9% versus 16%; vasodilators, 6% versus 12%; and any of these interventions, 26% versus 51%. Overall HF in-hospital risk-standardized mortality was similar (3.4% versus 3.5%; P=0.2)., Conclusions: ICU admission rates for HF varied markedly across hospitals and lacked association with in-hospital risk-standardized mortality. Greater ICU use correlated with fewer patients receiving ICU interventions. Judicious ICU use could reduce resource consumption without diminishing patient outcomes.

Background: The intensive practice style of hospitals with high procedure rates may result in higher costs of care for medically managed patients. We sought to determine how costs for patients with heart failure (HF) not receiving procedures compare between hospital groups defined by their overall use of procedures., Methods and Results: We identified all 2009 to 2010 adult HF hospitalizations in hospitals capable of performing invasive procedures that had at least 25 HF hospitalizations in the Perspective database from Premier, Inc. We divided hospitals into 2 groups by the proportion of patients with HF receiving invasive percutaneous or surgical procedures: low (>0%-10%) and high (≥ 10%). The standard costs of hospitalizations at each hospital were risk adjusted using patient demographics and comorbidities. We used the Wilcoxon rank sum test to assess cost, length of stay, and mortality outcome differences between the 2 groups. Median risk-standardized costs among low-procedural HF hospitalizations were $5259 (interquartile range, $4683-$6814) versus $6965 (interquartile range, $5981-$8235) for hospitals with high procedure use (P<0.001). Median length of stay was 4 days for both groups. Risk-standardized mortality rates were 5.4% (low procedure) and 5.0% (high procedure) (P=0.009). We did not identify any single service area that explained the difference in costs between hospital groups, but these hospitals had higher costs for most service areas., Conclusion: Among patients who do not receive invasive procedures, the cost of HF hospitalization is higher in more procedure-intense hospitals compared with hospitals that perform fewer procedures.