Automated Computer-Assisted Categorization of Radiology Reports

OBJECTIVE: The objective of our study was to create and validate an automated computerized method for the categorization of narrative text radiograph reports. MATERIALS AND METHODS: Using commercially available software with embedded Boolean logic, we created a text search algorithm to categorize reports of radiography examinations into "fracture,"normal," and "neither normal nor fracture." The algorithm was refined and optimized through repeated testing on 512 consecutive ankle radiography reports from a single clinical imaging center. The final algorithm was applied on a different set of 750 consecutive radiography reports of the spine and extremities produced at three different clinical imaging sites and interpreted by 44 different radiologists. Expert reviewers assessed the accuracy of the final classification. The chi-square test or Fisher's exact test was performed to determine the reproducibility of results across different clinical imaging sites. RESULTS: The computerized classification was highly accurate for the classification of radiography reports into "normal" (specificity, 91.6%; sensitivity, 91.3%), "neither normal nor fracture"(sensitivity, 87.8%; specificity, 94.9%), and "fracture"(sensitivity, 94.1%; specificity, 98.1%) categories. This performance showed no significant difference across the three sites (p >0.05). CONCLUSION: Computerized categorization of narrative-text radiography reports is highly sensitive and specific and can be used to classify reports from different imaging sites generated by different radiologists. This method can be an extremely powerful tool in future cost-effectiveness studies, health care policy studies, operations assessments, and quality control.