Can LLMs Accelerate Systematic Literature Reviews?

Raw data for study: Systematic literature reviews (SLRs) are highly valuable to researchers and practitioners because they provide rigorous, unbiased summaries of existing evidence, identify alternative approaches to problem solving, and reveal knowledge gaps that can guide future research. However, conducting an SLR can be laborious and time consuming, particularly during full-text screening and data extraction, and it is natural to ask whether Large Language Models (LLMs) can accelerate the process. LLMs may reduce reviewer workload, but it is still unclear how closely their judgments align with human reviewers, whether their outputs may influence human decisions, and how much human oversight is needed. This study evaluates the data extraction phase of the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) process. We report on the inter-rater reliability (using Krippendorff's α) between human reviewers with and without LLM output (coding and evidential text), between consensus codes agreed before and after consulting LLM output, and with and without access to LLM output during data extraction. We conclude that there is strong evidence that access to LLM output accelerates human reviewers' convergence without noticeably biasing that consensus. We propose an approach to incorporate LLMs into the coding process, while maintaining the supervision of human researchers. Based on preliminary benchmarks, we expect at least a four-times speedup, mostly due to the use of the LLM coding and evidence by the human reviewer as a starting point for their coding.