Abstract Background/purpose Development and validation of AI tools for diagnostic imaging requires high-quality annotations. Dedicated research readings are considered superior to the clinical radiologic report (CRR). However, the practices for generating CRR varies in rigor. The purpose of this study was to validate CRRs produced using a post-conference multiple disciplinary reader workflow by comparing them to dedicated research readings of trauma radiographs. Materials and Methods This retrospective study included consecutive patients referred for radiography at the indication of trauma at two university hospitals. For the index test, the CRRs were evaluated based on their description of eight common diagnoses. The reference standard was established as the agreement of two certified reporting technologists and arbitrated by a senior musculoskeletal radiologist. Sensitivity, specificity, positive and negative predictive values were calculated. Results The study sample consisted of 618 consecutive examinations (median age 52 years IQR, 24, 76, 351 female). Fracture incidence was 36%. Incidence of other findings ranged from 1% (bone lesions) to 10% (degenerative disease). The sensitivities of the CRRs were: fracture (97% 94 to 99%), luxation (87% 69 to 96%), degenerative disease (67% 53 to 78%), effusion (67% 46 to 83%), old fracture (64% 48 to 78%), subluxation (44% 14 to 79%), halisteresis (30% 13 to 53%) and bone lesion (25% 1 to 81%). Specificity ranged from halisteresis (94% 92 to 96%) to subluxation (100% 99 to 100%) and luxation (100% 99 to 100%). Conclusion We found that the workflow at the tested hospitals generates clinical radiologic reports of trauma radiographs with a diagnostic accuracy for the assessment of fracture and luxation suitable for research quality labelling.
Bruun et al. (Fri,) studied this question.