Quality Evaluation of Generative AI–Based Search Strategies in Systematic Reviews and Comparison of Search Performance with Human Expert (Medical Librarian)

Reproducible and sensitive literature search strategies are essential to the methodological rigor of systematic reviews (SRs). Recently, generative artificial intelligence (AI) tools such as ChatGPT and Gemini have been explored as aids for search strategy development; however, concerns remain regarding logical errors, limited understanding of controlled vocabularies, and reproducibility. This study evaluates the quality of search strategies generated by generative AI for a specific clinical topic and compares their retrieval performance with those developed by professional medical librarians. The research topic was defined as the effectiveness of psychotherapy for patients with obesity. Relevant Cochrane Reviews were identified in the Cochrane Library database using a search strategy incorporating both MeSH terms and text word keywords. Based on the final reference lists of the included reviews, a gold standard comprising a total of 139 studies was constructed. The comparative experiment involved generative AI models (ChatGPT-5, Gemini 2.5) and a human expert (medical librarian). The AI group operated in a zero-shot environment using three levels of prompts based on researcher proficiency, while the human expert formulated strategies adhering to the Cochrane Handbook and PRISMA-S guidelines. The derived search strategy was applied to three major academic databases (PubMed, EMBASE, and the Cochrane Library). Its performance was evaluated qualitatively using the PRESS 2015 Evidence-Based Guideline Checklist and quantitatively by calculating sensitivity (recall) and precision. As prompt levels advanced, generative AI showed improved structural systematicity in search strategies; however, limitations persisted in the accurate application of database-specific syntax (truncation, field tags) and controlled vocabularies (MeSH/Emtree). In the performance evaluation, the human expert achieved the highest sensitivity (54.7%), followed by Gemini 2.5 (advanced prompt) at 49.6% and ChatGPT-5 (intermediate prompt) at 46.8%. Precision was low (less than 1%) for all groups. The analysis of missing studies revealed that human expert omitted some recent intervention-related literature, such as digital health studies, due to relying on traditional conceptual boundaries of “psychotherapy.” Generative AI is a useful auxiliary tool for deriving initial search concepts and expanding natural language terms; however, verification by medical librarians remains essential to ensure appropriate controlled vocabulary use and syntactic accuracy. Medical librarians, in contrast, developed comprehensive and reproducible high-sensitivity search strategies for systematic reviews, though further refinement is required to improve precision. Accordingly, this study proposes a Human–AI Collaboration model that integrates generative AI automation with the methodological expertise of medical librarians, highlighting the continued importance of medical librarians in high-quality evidence synthesis in the AI era. Future research should encompass diverse clinical questions beyond a single topic and continuously evaluate the performance of rapidly evolving AI models. J Korean Med Libr Assoc 2025;52(1):28-47