Large language models (LLM) are increasingly used in clinical medicine for tasks such as automated note generation. However, LLM-generated notes remain vulnerable to transcription errors, raising concerns about their reliability in clinical practice. We analyzed the types and rates of LLM mis-transcription errors (deletions, substitutions, and additions) and LLM mis-attribution errors (assigning dialogue to an incorrect speaker) in transcripts generated by a single LLM and tested whether error rates differed by speaker role and speaker sex. We also examined plausible sources of LLM-related error, including overlapping speech and speaking turn taking, and hypothesized that higher-quality audio would be associated with fewer transcription errors. In this retrospective single center study, an LLM (NotebookLM) generated speaker-labeled transcripts from audio recordings of twelve standardized-patient (SP) medical student encounters involving three SPs in a single simulated clinical scenario. Six encounters were re-recorded with higher-fidelity audio (HFA) to evaluate the effect of recording quality on errors. LLM-generated transcripts were compared with gold-standard transcripts. Outcomes included target word errors (substitutions and deletions), insertions, turn-taking errors, mis-attributed speaker word errors, semantic errors (errors that changed the meaning of a word or phrase), medical terminology errors, speaking turns, and overlapping speech. Interactions averaged 2,226 ± 252 words and the mean transcription error frequencies were 73 ± 26 target word errors (3.3% of target words), 22 ± 13 substitutions (1.0%), 51 ± 22 deletions (2.3%), and 9 ± 4 insertions (0.4%). There were 19 ± 5 semantic word/phrase errors, of which 8 ± 4 were due to medical terminology errors. For speaker attribution accuracy, there were 15 ± 12 turn-taking errors (7.3% of all speaking turns), 5 ± 6 semantic turn-taking errors (errors that altered meaning), and 48 ± 39 mis-attributed-speaker word errors. Overlapping speech accounted for 19.1% of total word errors and 16.3% of mis-attributed speaker word errors. Speaking turns were correlated with target word errors and insertions (r = 0.41), turn-taking errors (r = 0.62), and mis-attributed speaker word errors (r = 0.51). HFA recordings reduced but did not eliminate the errors. In this single simulated clinical scenario involving three SPs and 12 SP–student interactions, overlapping speech, turn-taking, medical terminology, and audio fidelity were frequent contributors to transcription errors with NotebookLM. Though LLM transcription errors were modest, even small numbers of errors can have a meaningful impact on documentation. These findings suggest that caution is warranted when relying on LLMs for fully autonomous clinical note generation. LLMs may be most appropriately used in a supportive role, such as assisting clinicians in reviewing and improving physician-authored documentation, rather than replacing clinician involvement in the documentation process.
Nolan et al. (Tue,) studied this question.