575 Using Artificial Intelligence to Generate Medical Literature for Patients: A Comparison of Three Different Large Language Models

Aim Large language models (LLMs) are artificial intelligence (AI) that use deep learning techniques to understand, summarize and generate content. The potential of LLMs in healthcare is predicted to be vast. This study aims to examine the usefulness of LLMs in generating patient information leaflets (PILs). Method Prompts were created to generate PILs for 4 topics in urology (circumcision, nephrectomy, overactive bladder syndrome, and transurethral resection of the prostate). Prompts were given to 3 LLMs: ChatGPT-4, PaLM 2 (Google) and Llama-2 (Meta). PILs were evaluated using a validated assessment checklist by blinded reviewers and a 5-point Likert scale. Readability assessments for leaflets were conducted using the Average Reading Level Consensus Calculator. Results PaLM 2 had the highest overall average score (3.575), followed by Llama-2 (3.3375) and ChatGPT-4 (3.08). PaLM 2 generated PILs were of the highest quality in all topics except one and was the only AI to include images. Medical accuracy of leaflets was mixed, with all LLMs generating content containing minor errors and some instances of major error. Readability analysis identified PaLM 2 PILs as the easiest to read with an average reading level of age 14-15 (slightly difficult). Llama-2 leaflets were the most difficult with an average reading level of age 16-17 (fairly difficult). Conclusions LLMs can generate PILs that will help reduce the workload on healthcare professionals. However, the content generated requires human input for appropriateness and factual content. The content is also generated at a reading age above the average level for adults.