A novel graphical evaluation of agreement

Abstract Background The Bland-Altman plot with the limits of agreement has been widely used as an absolute index for assessing test-retest reliability or reproducibility between two measurements. We have observed that in the settings where the relative index such as concordance correlation coefficient (CCC) or intraclass correlation coefficient is employed, the limits of agreement approach may be inconsistent with the scaled index. Particularly, the broad width of the limits of agreement may indicate a lack of agreement when the two measurements are highly concordant but an acceptable difference is not known and the common variance of the data is large. This research aims to create a novel, CCC-based guidance for graphical evaluation of reproducibility or reliability. Methods The concordance correlation coefficient is used to create a 100(1-α)% reference band from two measurements. Simulation studies and real examples, including the peak expiratory flow rate data in Bland and Altman’s paper and the test-retest reproducibility data of the Radiomics study, are implemented to assess the use of the reference band. Results In the absence of an acceptable difference between measurements, we found that the limits of agreement may not be consistent with the concordance correlation coefficient. Our simulation study results and real data application show that the proposed method can provide practitioners with a novel graphical evaluation that is consistent with results from the concordance correlation coefficient. Conclusions Our proposed novel scaled index-based guidance can be used for the graphical evaluation of reproducibility or reliability and may have advantages over the limits of agreement in settings where the concordance correlation coefficient is employed.