A proof-of-concept performance evaluation of six large language models for semi-automated data extraction in a systematic review on infectious diseases clinical trials

Objective Large language models (LLMs) are increasingly being used in the systematic review process. Our objective was to explore data extraction performance of six LLMs from a published systematic review of infectious diseases randomised controlled trials. The LLMs used were categorised as ‘fast' (ChatGPT-5 Thinking, Gemini 2.5 Flash) or ‘reasoning' (ChatGPT-5 Instant, Gemini 2.5 Pro, ChatGPT o3), and included a research-oriented model (NotebookLM). Methods In this proof-of-concept study, we randomly selected 50 open access articles and compared data extraction performance of each LLM to the gold-standard manual extraction across 49 different extraction fields, stratified by field domains and data types. We used a standardised prompt for each LLM with full-text articles uploaded in batches of five portable document files. Outputs were assigned 1 for concordant or 0 for not concordant using standardised matching rules. Descriptive statistics including means and medians were calculated across the 49 fields and 50 articles. Results Overall mean concordance (95% CI) was 0.75 (0.72-0.79) for Gemini 2.5 Flash, 0.69 (0.62-0.76) for NotebookLM, 0.67 (0.63-0.70) for ChatGPT o3 (legacy reasoning model), 0.63 (0.58-0.68) for GPT-5 Instant, 0.58 (0.51-0.64) for Gemini 2.5 Pro, and 0.57 (0.48-0.66) for GPT-5 Thinking. Mean concordance for data types were 0.66 (0.64-0.69), 0.63 (0.60-0.66) and 0.62 (0.58-0.66) for binary/categorical, string/text and numeric respectively. Conclusion Our study suggests the use of LLMs to assist in systematic review data extraction is feasible, although performance varied across models. Given the rapid pace of LLM development, ongoing evaluations of newer models are warranted.