Leveraging generative artificial intelligence (AI) to improve patient communication: A qualitative assessment of AI-generated patient-friendly radiology report summaries for patients with cancer.

618 Background: The 21st Century Cures Act mandates the release of full radiology reports to patients (pt). These reports, designed for medical professionals, are challenging for pts to interpret. Oncology (Onc) pts undergo frequent imaging for disease assessment, often receiving scan results days before their Onc visit. Reviewing these complex reports without guidance can lead to confusion, anxiety, and increased inquiries to providers. To address this, we leveraged Generative AI to create pt-friendly imaging report summaries (Summ). Here we report the results of qualitative assessment of these Summ by medical professionals. Methods: Using our established HIPAA-compliant institutional pipeline for clinical LLM use case assessment, we developed and optimized our prompt to generate pt-friendly Summ of imaging reports. GPT-4 was selected based on the initial performance assessment of multiple LLMs. A total of 240 AI-generated Summ were randomly assigned to 7 medical professionals (3 med oncologists, 3 Onc APPs, and 1 radiologist), with each Summ reviewed by 3 independent reviewers. To mitigate leniency bias, 16 challenge Summ (intended to have issues) were included in the dataset. Reviewers evaluated Summ against the original reports (de-identified) using 5 categorical (Yes/No) and 6 Likert scale (LS) questions (ranging from very poor to very good). Given the imbalanced score distribution, inter-rater agreement was assessed using Gwet's AC1 statistic on a 0 to 1 scale. Results: A total of 672 reviews (224 evaluable Summ × 3 reviewers) were analyzed. Reviewers rated the Summ favorably across categorical questions: avoiding medical jargon (95%), not including any information that wasn’t present in the original report (98%), not missing any crucial details (97%), not misrepresenting any crucial details (95%), and adequate specificity of the findings (91%). LS ratings were predominantly "good" or "very good" (Table). Inter-rater agreement was high, with Gwet's AC1 scores of 0.85 for categorical questions and 0.75 for LS ratings. Conclusions: AI-generated pt-friendly Summ of imaging reports demonstrated high accuracy and clarity, highlighting its potential to augment pt understanding and reduce provider workload. A clinical pilot involving 4 alpha users (Onc providers) sharing the summaries via pt portal is ongoing. Criteria Very poor Poor Mediocre Good Very good % (n) Accuracy 0 (0) 0 (1) 3 (20) 34 (228) 63 (442) Avoiding fabricated content 0 (0) 0 (2) 1 (9) 17 (114) 81 (546) Completeness 0 (0) 0 (3) 4 (27) 32 (216) 63 (426) Specificity 0 (0) 1 (8) 9 (59) 31 (207) 59 (396) Avoiding potentially misleading or harmful information 0 (0) 1 (6) 5 (32) 32 (213) 63 (421) Clarity for a non-medical reader 0 (1) 0 (3) 5 (33) 32 (216) 62 (419)