A mixed-methods evaluation of a language model-powered breast cancer note augmentation tool for patients

Large language models (LLMs) have the potential to make clinical notes more understandable to patients, but tools must be tested with patients pre-deployment. We developed a patient-facing tool that uses an LLM (GPT-4-031) to generate “augmentations” to notes: simplified note text, defined medical terminology, frequently asked questions, extracted key information, and to-do items. We evaluated the tool with 200 healthy female participants and 8 notes. Compared to control, participants with access to augmentations had greater self-reported confidence (2.83 ± 0.05 augmentations, 2.18 ± 0.07 control, p<0.001) and scored higher on comprehension questions (0.63 ± 0.02 augmentations, 0.54 ± 0.04 control, p<0.001). We conducted interviews with 7 participants with a history of breast cancer and their own notes who described ease of use and perceived improvements in note understanding, but also reported concerns around LLMs. Three clinicians conducted an error analysis and found most errors to be minor and related to assumed context.