Search

Agreement measures are useful metrics to both compare different evaluations of the same diagnostic outcomes and validate new rating systems or devices. While many of them have been proposed in the literature so far, Cohen’s n is still the de facto standard in gauging the agreement. Information Agreement (IA) is a novel two-observers information-theoretic-based metric introduced to overcome all the limitations and alleged pitfalls of Cohen’s n. It offers an operative meaning to the agreement since it measures – in both dichotomous and multi-value ordered-categorical cases-the information shared between two raters through the virtual diagnostic channel connecting them: the more information exchanged between the raters, the higher their agreement. Unfortunately, this measure is only able to deal with agreement matrices whose values are all strictly positive numbers. This work extends IA by admitting also 0 as a possible value for the entries of an agreement matrix. Moreover, a Python software library to compute the extended version of IA, together with some of the most used agreement measures, is presented and tested.

Introduction:Acquired cranial asymmetry is prevalent in infants today and largely attributed to the supine sleep position recommended for infant safety. There is a risk of permanent cranial asymmetry, so prevention and early detection are important. A prevention project was initiated in Sweden, and an intervention was planned. The aim of this study was to evaluate reliability of assessors judging infant cranial asymmetry in order to evaluate if they could be considered reliable interchangeable assessors in the planned intervention.Materials and Methodology:Five assessors were taught how to assess infant cranial asymmetry using illustrated severity assessments. They were intra-rater and inter-rater reliability tested by taking a photograph test-retest and an infant test. Agreement matrices were devised to illustrate assessor agreement based on bothtypeanddegreeof cranial asymmetry. Agreement based on degree of asymmetry was analyzed by calculating AC2 using quadratic weights. Results were adjusted to arrive at the perceived genuine agreement and interpreted according to Landis and Koch’s strength of agreement intervals.Results:In the photograph test, mean percentage of perfect intra-rater agreement was 73. Adjusted mean intra-rater AC2 was 0.69 [0.63; 0.76], and adjusted inter-rater AC2s were 0.72 [0.64; 0.81] and 0.71 [0.63; 0.79]. In the infant test, the adjusted inter-rater AC2 was 0.73 [0.60; 0.87]. Results indicate substantial strength of assessor agreement.Conclusion:Assessors were reliable and interchangeable. In a larger clinical context, results indicate that educating child health care nurses to assess infant cranial asymmetry can be used for early detection.