ABSTRACT Background Public health events of international concern highlight the need for up‐to‐date evidence curated using sustainable processes that are accessible. In development of the Global Repository of Epidemiological Parameters (grEPI) we explore the performance of an agentic‐AI assisted pipeline (GREP‐Agent) for screening evidence which capitalizes on recent advancements in large language models (LLMs). Methods In this study, the performance of the GREP‐Agent was evaluated on a data set of 2000 citations from a systematic review on measles using four LLMs (GPT4o, GPT4o‐mini, Llama3.1, and Phi4). The GREP‐Agent framework integrates multiple LLMs and human feedback to fine‐tune its performance, optimize workload reduction and accuracy in screening research articles. The impact on performance of each part of this Agentic‐AI system is presented and measured by accuracy, precision, recall, and F1‐score metrics. Results The results show how each phase of the GREP‐Agent system incrementally improves accuracy regardless of the LLM. We found that GREP‐Agent was able to increase sensitivity across a broad range of open source and proprietary LLMs to 84.2%–88.9% after fine‐tuning and to 86.4%–95.3% by varying workload reduction strategies. Performance was significantly impacted by the clarity of the screening questions and setting thresholds for optimized workload reduction strategies. Conclusions The GREP‐Agent shows promise in improving the efficiency and effectiveness of evidence synthesis in dynamic public health contexts. Further development and refinement of adaptable human‐in‐the‐loop AI systems for screening literature are essential to support future public health response activities, while maintaining a human‐centric approach.
Hu et al. (Tue,) studied this question.