Evaluation of automated assessments of systematic review adherence to the PRISMA 2020 statement: study protocol

<ns3:p>Background The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) 2020 statement aims to help users write complete and accurate systematic review reports. Peer-reviewers and meta-researchers often assess adherence of systematic reviews to PRISMA 2020. This process can be time consuming, particularly when evaluating many systematic reviews. Automated approaches using large language models (LLMs) have the potential to accelerate this process and produce more comprehensive assessments. Objective To evaluate the performance of LLMs prompted to undertake comprehensive assessments of adherence to PRISMA 2020. Methods We will conduct a validation study using a diagnostic test accuracy analysis framework. We will assemble a sample of 200 published systematic reviews assessing the effects of interventions on human health, which will be divided into a training set and a test set. To assess adherence, we will reframe 95 reporting elements in PRISMA 2020 (which provide granular reporting recommendations for 41 checklist items) into one or more questions. We will iteratively develop few-shot prompts for use in several LLMs and refine them using systematic reviews included in the training set. Final LLM prompts will be applied to all systematic reviews in the test set and LLM responses compared with consensus responses of two humans (our reference standard). We will estimate the performance of each LLM assessment against the reference standard by calculating percentage agreement, Gwet’s Agreement Coefficient, sensitivity, specificity, positive predictive value, negative predictive value, and the F1 score. Conclusion We will determine which questions in our comprehensive tool for assessing adherence to PRISMA 2020 can be accurately automated by LLMs. This knowledge will help inform which questions need the most human oversight by meta-researchers, peer reviewers and other interest holders seeking to assess adherence.</ns3:p>