Validation of an LLM-assisted title and abstract screener on two published systematic reviews

Technical report, version 1.0. Background: Large language models (LLMs) are increasingly tried as first-pass screeners for systematic reviews. A screener that removes records from human reading is useful only if it keeps almost every study the review finally includes, so recall is the key measure. Methods: A recall-first screener (qwen3.8-27b served by Groq) labelled each record INCLUDE, UNSURE or EXCLUDE; INCLUDE and UNSURE both go to a human. It was tested on two published reviews from the open SYNERGY collection, after checking that each dataset's labelled inclusions appear in the review's own reference list: Meijboom et al. 2022 (switching back from biosimilar to originator TNF-α inhibitors; 881 records, 37 included studies) and van der Waal et al. 2023 (the role older adults with cancer prefer in treatment decisions; 1,962 records, 33 included studies). Eligibility criteria were written from each paper and frozen with checksums before screening. Each evaluation set held every included study plus 200 randomly sampled excluded records. Results: The screener kept 37 of 37 included studies in the first review (recall 100%; 95% CI 90.6–100) and 30 of 33 in the second (90.9%; 76.4–96.9), below the 95% level commonly used as a reference point. Specificity was 78.0% and 92.0%, and an estimated 74.7% and 90.6% of records would not have needed manual reading. All three missed studies were qualitative studies excluded under a frozen criterion that the review's authors applied more loosely. Conclusions: On these two reviews the screener worked as a first pass but not as a sole screener. Eligibility criteria should be piloted on a sample of records before any screener is relied on.