A head-to-head comparison of the inter- and intraobserver agreement of COVID-RADS and CO-RADS grading systems in a population with high estimated prevalence of COVID-19

Purpose To evaluate the inter- and intraobserver agreement of COVID-RADS and CO-RADS reporting systems among differently experienced radiologists in a population with high estimated prevalence of COVID-19. Materials and Methods Chest CT scans of patients with clinically-epidemiologically diagnosed COVID-19 were retrieved from an open-source MosMedData dataset, randomised, and independently assigned COVID-RADS and CO-RADS grades by an abdominal radiology fellow, thoracic imaging fellow and a consultant cardiothoracic radiologist. The inter- and intraobserver agreement of the two systems were assessed using the Fleiss’ and Cohen’s kappa coefficients, respectively. Results A total of 200 studies were included in the analysis. Both systems demonstrated moderate interobserver agreement, with kappa values of 0.51 (95% CI: 0.46-0.56) and 0.55 (95% CI: 0.50-0.59) for COVID-RADS and CO-RADS, respectively. When COVID-RADS and CO-RADS grades were dichotomised at cut-off values of 2B and 4 to evaluate the agreement between grades representing different levels of clinical suspicion for COVID-19, the interobserver agreement became substantial with kappa values of 0.74 (95% CI: 0.66-0.82) for COVID-RADS and 0.73 (95% CI: 0.65-0.81) for CO-RADS. The median intraobserver agreement was considerably higher for CO-RADS reaching 0.81 (95% CI: 0.43-0.76) compared with 0.60 (95% CI: 0.43-0.76) of COVID-RADS. Conclusions COVID-RADS and CO-RADS showed comparable interobserver agreement, which was moderate when grades were compared head-to-head and substantial when grades were dichotomised to better reflect the underlying levels of suspicion for COVID-19. The median intraobserver agreement of CO-RADS was, however, considerably higher compared with COVID-RADS. Advances in knowledge This paper provides a comprehensive review of the newly introduced COVID-19 chest CT reporting systems, which will help radiologists of all sub-specialties and experience levels make an informed decision on which system to use in their own practice.