Search

Quality measurement plays an increasing role in U.S. health care. Measures inform quality improvement efforts, public reporting of variations in quality of care across providers and hospitals, and high‐stakes financial decisions. To be meaningful in these contexts, measures should be reliable and not heavily impacted by chance variations in sampling or measurement. Several different methods are used in practice by measure developers and endorsers to evaluate reliability; however, there is uncertainty and debate over differences between these methods and their interpretations. We review methods currently used in practice, pointing out differences that can lead to disparate reliability estimates. We compare estimates from 14 different methods in the case of two sets of mental health quality measures within a large health system. We find that estimates can differ substantially and that these discrepancies widen when sample size is reduced. [ABSTRACT FROM AUTHOR]

Objective: To examine the sensitivity of split‐sample reliability estimates to the random split of the data and propose alternative methods for improving the stability of the split‐sample method. Data Sources and Study Setting: Data were simulated to reflect a variety of real‐world quality measure distributions and scenarios. There is no date range to report as the data are simulated. Study Design: Simulation studies of split‐sample reliability estimation were conducted under varying practical scenarios. Data Collection/Extraction Methods: All data were simulated using functions in R. Principal Findings: Single split‐sample reliability estimates can be very dependent on the random split of the data, especially in low sample size and low variability settings. Averaging split‐sample estimates over many splits of the data can yield a more stable reliability estimate. Conclusions: Measure developers and evaluators using the split‐sample reliability method should average a series of reliability estimates calculated from many resamples of the data without replacement to obtain a more stable reliability estimate. [ABSTRACT FROM AUTHOR]