Comparing AI-generated and human-written plain language summaries: text-based readability measures and evaluations by lay readers and experts

Plain language summaries (PLSs) are lay-friendly summaries of scientific publications. As large language models (LLM) have become a popular tool to summarize and rewrite texts, researchers investigated how well they perform in producing PLSs. Previous studies found that AI-generated PLSs were more readable than human-written PLSs. However, none of those studies used an evidence-based writing guideline to ensure that the PLSs were generated according to established quality standards. Furthermore, none of those studies investigated PLSs in psychology. This is an important gap given the clear need for lay-friendly, accurate information about psychological scientific evidence, particularly amid the prevalence of inaccurate popular psychology content in the media. To address this gap, in a collaborative project between psychologists and computational linguists, an LLM-based app was developed that generates PLSs of psychological meta-analyses based on an evidence-based guideline. This study aims to evaluate its output across three complementary areas: (1) text-based measure of readability, (2) comprehensibility and credibility, as perceived by lay readers, (3) expert-assessed quality.