188 Assessing the Quality and Readability of Online Patient Information: ENT UK Patient E-Leaflets Versus ChatGPT Responses

Introduction The evolution of artificial intelligence has introduced new ways to disseminate health information, including natural language processing models like ChatGPT. However, the quality and readability of such digitally generated information remains understudied. This study is the first to compare the quality and readability of digitally generated health information against leaflets produced by professionals. Method Patient information leaflets for five ENT UK leaflets, and their corresponding ChatGPT responses, were evaluated using the Ensuring Quality Information for Patients (EQIP) tool and readability tools including the Flesch Reading Ease Score (FRES) and Flesch-Kincaid Grade Level (FKGL). Statistical analysis was performed to identify differences between leaflets, assessors, and sources of information. Results ENT UK leaflets were of moderate quality, scoring a median EQIP of 23. Statistically significant differences in overall EQIP score were identified between ENT UK leaflets but ChatGPT responses were of uniform quality. Non-specialist doctors rated the highest EQIP scores while medical students scored the lowest. The mean readability of ENT UK leaflets was higher than ChatGPT responses. The overall EQIP scores negatively correlated with readability metrics of ChatGPT responses. In short, the information quality of ENT UK leaflets was moderate and varied between topics. Equivalent ChatGPT information provided comparable content quality but with reduced readability. Conclusions ChatGPT patient information leaflets had comparable content, but were less readable, and required a higher reading age than ENT UK leaflets. With the increasing use of online health resources, this study underscores the need for a balanced approach that optimises both the quality and readability of patient education material.