Evaluating Generative AI to Extract Qualitative Data from Peer-Reviewed Documents

Abstract Uptake of AI tools in knowledge production processes is rapidly growing. Here, we explore the ability of generative AI tools to reliably extract qualitative data from peer-reviewed documents. Specifically, we evaluate the capacity of multiple AI tools to analyse literature and extract relevant information for a systematic literature review, comparing the results to those of human reviewers. We address how well AI tools can discern the presence of relevant contextual data, whether the outputs of AI tools are comparable to human extractions, and whether the difficulty of question influences the performance of the extraction. While the AI tools we tested (GPT4-Turbo and Elicit) were not reliable in discerning the presence or absence of contextual data, at least one of the AI tools consistently returned responses that were on par with human reviewers. These results highlight the utility of AI tools in the extraction phase of evidence synthesis for supporting human-led reviews and underscore the ongoing need for human oversight.