A head-to-head comparison of the intra- and interobserver agreement of COVID-RADS and CO-RADS grading systems in a population with high estimated prevalence of COVID-19

OBJECTIVE: To evaluate the inter- and intraobserver agreement of COVID-RADS and CO-RADS reporting systems among differently experienced radiologists in a population with high estimated prevalence of COVID-19. METHODS AND MATERIALS: Chest CT scans of patients with clinically-epidemiologically diagnosed COVID-19 were retrieved from an open-source MosMedData data set, randomised, and independently assigned COVID-RADS and CO-RADS grades by an abdominal radiology fellow, thoracic imaging fellow and a consultant cardiothoracic radiologist. The inter- and intraobserver agreement of the two systems were assessed using the Fleiss' and Cohen's κ coefficients, respectively. RESULTS: A total of 200 studies were included in the analysis. Both systems demonstrated moderate interobserver agreement, with κ values of 0.51 95% confidence interval (CI): 0.46-0.56 and 0.55 (95% CI: 0.50-0.59) for COVID-RADS and CO-RADS, respectively. When COVID-RADS and CO-RADS grades were dichotomised at cut-off values of 2B and 4 to evaluate the agreement between grades representing different levels of clinical suspicion for COVID-19, the interobserver agreement became substantial with κ values of 0.74 (95% CI: 0.66-0.82) for COVID-RADS and 0.73 (95% CI: 0.65-0.81) for CO-RADS. The median intraobserver agreement was considerably higher for CO-RADS reaching 0.81 (95% CI: 0.43-0.76) compared with 0.60 (95% CI: 0.43-0.76) of COVID-RADS. CONCLUSIONS: COVID-RADS and CO-RADS showed comparable interobserver agreement, which was moderate when grades were compared head-to-head and substantial when grades were dichotomised to better reflect the underlying levels of suspicion for COVID-19. The median intraobserver agreement of CO-RADS was, however, considerably higher compared with COVID-RADS. ADVANCES IN KNOWLEDGE: This paper provides a comprehensive review of the newly introduced COVID-19 chest CT reporting systems, which will help radiologists of all sub-specialties and experience levels make an informed decision on which system to use in their own practice.