Evaluating machine learning tools to assist title and abstract screening in systematic literature reviews: a report based on the EULAR RA Management Recommendations Task Force

ABSTRACTObjectives Systematic literature reviews (SLRs) provide the scientific basis for European Alliance of Associations for Rheumatology (EULAR) task force projects, but they are highly time- and labour-intensive in an ever-growing research landscape, repetitive, and susceptible to human error. In this study, we evaluate the performance of machine learning (ML) models developed for semiautomated title and abstract screening, aiming at supporting and accelerating future review processes. Methods Title and abstract screening of 4 SLRs, conducted to inform the 2025 update of the EULAR management recommendations for rheumatoid arthritis, was replicated using different ML models, with manual screening serving as reference standard. Eligible software was identified via the Systematic Review Toolbox, online and ML-based searches, and reference checking. Tools were included if they were actively maintained, accessible, and offered screening functionality beyond commonly used reference managers. Results Nine tools employing ML-based record prioritisation and relevance prediction were identified, of which 3 met all inclusion criteria and were evaluated in detail. These were applied to search results from 3 SLRs on randomised controlled trials (RCTs) and 1 that also included observational studies. Across reviews, substantial workload reductions were achieved (mean 77.8%, SD 12.8%) while consistently capturing over 95% of all relevant records. Additionally, ML-supported classification and filtering of RCTs prior to screening resulted in a mean reduction of abstracts to screen by 57.3% (SD 7.4%). Conclusions Replication of manual title and abstract screening demonstrates the considerable potential of ML tools to support systematic reviews, while highlighting important limitations and pitfalls. Prospective evaluations will identify optimal strategies for integrating these tools into future review processes while maintaining high methodological standards.