Documenting Care with AI: A Comparative Analysis of Commercial Scribe Tools

Clinical documentation is a leading cause of clinician burnout. Artificial intelligence (AI) scribes use ambient generative AI to create text summaries of clinical encounters. While several commercial AI scribe tools are available, there are limited comparisons of the performance of these tools. This study evaluated the quality and accuracy of notes generated by four commercially available AI scribes using two simulated internal medicine and surgical patient encounters. AI-generated notes were compared to transcripts, with errors quantified and quality assessed using the Physician Documentation Quality Instrument (PDQI), evaluating succinctness, usefulness, and comprehensiveness. Omission errors constituted 71% of the total errors, addition errors made up 19.4%, and incorrect facts accounted for 6.5% of the errors identified among vendors. Vendor A had the lowest error rate (12.2%) and balanced note length and quality. Vendor C, with the highest omission error rate (24.4%), produced the shortest notes, while Vendor B generated the longest notes with the highest addition error rate (7.3%). PDQI-9 scores ranged from 35.5 to 39.5, with Vendor C & D achieving the highest score. These findings emphasize the need for ongoing AI tool oversight and highlight future research opportunities, including the impact of AI scribes on communication, safety, and documentation quality.