AI-driven quality assurance for emergency department documentation: Pilot comparison with physician peer review of chest pain cases

Abstract Objectives To evaluate the feasibility and reliability of an artificial intelligence (AI)-driven quality assurance system for emergency chest pain documentation compared to traditional physician reviewers. Methods We developed an automated quality assurance solution leveraging commercial Large Language Models that are customized based on College of Physicians and Surgeons of Ontario (CPSO) peer review standards. Seventeen emergency chest pain cases of varying quality were independently reviewed by an AI auditor, six community emergency physicians, and original case writers using a 3-point evaluation scale across nine dimensions. Agreement was measured using Lin's Concordance Correlation Coefficient (CCC) for overall score agreement, Root Mean Square Error (RMSE) for overall score accuracy, and Cohen's Kappa statistics for categorical dimension scores. Results Inter-rater agreement among human reviewers showed substantial variability, with overall score CCCs ranging from 0.28 to 0.86. The AI auditor demonstrated agreement levels within the range of human variability, with the best performing model (CPSO-customized Claude Sonnet 4) with a CCC value of 0.85 and RMSE of 0.25 with respect to the average human scores on the 3-point scale. For pass/fail decisions, AI agreement with human reviewers ranged from 65% to 94% with Kappa scores of 0.27 to 0.85, comparable to inter-human reviewer agreement Kappa scores of -0.12 to 0.68. Conclusions The AI auditor performed comparably to physician reviewers in determining documentation quality and pass/fail outcomes. Given the potential variability in human assessments, AI-driven quality assurance may provide a more consistent evaluation of quality, with the added advantage of the mass-processing of many cases in near real-time.