Large Language Model-Assisted Title and Abstract Screening for A Scoping Review of Circulating miRNA Monitoring: Pilot Validation Against a Human Reviewer and Quantifying the Error of Keyword Pre-Filters

This study evaluated an open-weight LLM and keyword rules for title/abstract screening of circulating microRNA (miRNA) monitoring biomarker reviews. PubMed and Europe PMC searches returned 3,204 records (1,976 after deduplication). Automated keyword rules excluded 1,222 records; 754 were screened by gpt-oss-120b (n = 628) or gpt-oss-20b (n = 126) at temperature 0. One reviewer labelled 225 records; a second independently labelled a stratified 113-record sample, and 21 disagreements were adjudicated to consensus (2 Include, 3 Maybe, 16 Exclude), serving as the reference, though blinding of the first reviewer to LLM outputs could not be verified. The primary outcome was sensitivity for Include or Maybe against a pre-set threshold of 0.95. The LLM returned Include 173, Maybe 102, and Exclude 479. Against the consensus reference, sensitivity was 0.89 (95% CI 0.77–0.95), specificity 0.90 (95% CI 0.83–0.95), and kappa 0.65 (95% CI 0.54–0.76). Of 100 randomly sampled LLM-excluded records, 5 (5%) were reference-Include; in all five, the LLM's stated reason involved prognostic or clinical-event outcomes rather than drug response, resistance, or toxicity. Among keyword-excluded records, 11/75 (15%) were reference-Include, rising to 11/50 (22%) for the rule 'no human circulating miRNA + treatment context'. The sensitivity point estimate fell below 0.95, but its 95% CI (0.77–0.95) includes 0.95, so this pilot cannot rule out that the threshold is met; keyword pre-filters excluded relevant records at a rate that made them unsafe without audit. In this domain, an LLM can prioritise records but cannot replace human screening. The authors propose clearer outcome criteria, single-model runs, stratified human audits of every automated exclusion, and pre-registered performance thresholds.