By quantitatively integrating results of meta-analyses, second-order meta-analyses (SOMAs) from educational psychology address far-reaching questions of interest to adjacent scientific fields, policy, and educational practice. Conducting a SOMA, however, is time-consuming, particularly the extraction of statistical results from meta-analyses. Large Language Models (LLMs) have the potential to accelerate data extraction, as evidenced by evaluations of data extraction from primary studies. Evaluations of their accuracy in extracting results from meta-analyses to conduct SOMA are needed. We compare the accuracy of three LLMs with that of human experts extracting data from 156 educational meta-analyses investigating students’ achievement, randomly chosen from the Visible Learning database. LLMs and humans extracted effect sizes, the number of included studies, and the number of students. To assess accuracy, we established a gold standard benchmark by resolving discrepancies between LLMs and human experts through expert adjudication. Results show that the distributions of data from all coders and the standard were similar and showed no systematic bias. Accuracy reached ICCs of 0.95/0.81 for the two human experts, and 0.96/0.97/0.96 for LLMs (Gemini2.5Pro/GPT4.1/GPTo3), with percentage agreement of 86%/80% (humans) and 81%/78%/77% (LLMs). The results demonstrate that LLMs achieve data extraction accuracy comparable to that of human experts. We discuss the conditions under which data extracted by LLMs, humans, or a hybrid of both can be considered validated for use in educational SOMAs. Show a way to improve Second-Order-Meta-analyses. Assessing accuracy in data extraction by LLMs compared to experts. Experts and LLMs extracted data from 156 meta-analyses on student achievement. LLMs approach expert human accuracy. Gold standard determination from consensus calibration by multiple experts.
Jansen et al. (Fri,) studied this question.