Answers are grounded in this paper and cite their sources.
Objectives The exponential growth in medical literature has strained traditional evidence synthesis approaches, prompting interest in artificial intelligence (AI) as a potential solution. Thus, we sought to assess the concordance and time savings between humans and AI in the systematic review process for a specialized otolaryngology topic. Methods A mock systematic review on immunotherapy for HPV+ oropharyngeal squamous cell carcinoma was conducted. Title/abstract and full‐text were independently performed by human reviewers using Covidence and by Claude by Anthropic, model Sonnet 3.7. Concordance at each phase was assessed with sensitivity and specificity, and total time expenditure was documented for both approaches. Claude Sonnet 4.5 was subsequently applied to full‐text screening of a published systematic review as proof‐of‐concept validation. Results Of the 1337 abstracts screened, human reviewers identified 43 for full‐text review and ultimately included 17 manuscripts. Claude demonstrated an overall sensitivity of 17.6% (3/17) and specificity of 96.3% (1271/1320) when performing independent screening of all abstracts. When provided human‐curated abstracts for full‐text screening in an “AI‐assisted” methodology, it achieved 100% (17/17) sensitivity and 57.7% (15/26) specificity. Validation against a published review achieved 100% sensitivity and 97.8% specificity. AI‐driven screening required 325 active human minutes versus 1340 m for traditional human review, representing a theoretical active human time savings of over 75%. Conclusion While AI cannot yet replace human judgment in systematic reviews, hybrid methodologies combining AI speed with human clinical expertise may represent the most promising approach for maintaining both efficiency and quality in evidence synthesis for rapidly evolving medical fields. Level of Evidence N/A.
No takes yet. Share an insight, caveat, or question.
Shah et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: