Automation of literature screening for meta-analyses in the field of psychology: an evaluation of two text-mining approaches

Background The process of consolidating research through meta-analyses is time-consuming and costly. Using text-mining approaches to (semi-)automate the step of screening and selecting articles eligible for inclusion might potentially save hours of workload and costs. A lack of evaluative research on the use of text-mining for literature screening prevents the widespread adoption of text-mining approaches among the psychology research community. This study explores the possibilities of applying text-mining approaches to facilitate or (semi-)automate screening articles for inclusion in psychology meta-analyses. Methods The performances of two publicly accessible text-mining approaches were evaluated and compared against three recent meta-analyses from different subfields of psychology. Results Averaged over 10 text-mining runs per meta-analysis, across the three meta-analyses the approaches achieved recall performances between 62.5% and 99.3%, specificities between 60.1% and 92.7%, and saved between 54.0% and 92.0% of screening workload. Higher recall was observed when more articles were screened manually in the text-mining process. Conclusions These findings suggest that text-mining approaches can substantially reduce screening workload while maintaining high recall. These efficiency gains come with a trade-off: the more workload is saved, the lower the recall performance. Text-mining can therefore serve as a valuable tool for the initial identification of potentially eligible studies or as a secondary screening aid, but the risk of missing relevant studies should be mitigated by retaining an appropriate amount of manual screening.